Method and system for realizing two-dimensional appearance detection of PCBA (Printed Circuit Board Assembly) by multi-scale image processing method

Abstract

The invention relates to the technical field of image processing, and discloses a method and a system for realizing two-dimensional appearance detection of a PCBA (Printed Circuit Board Assembly) by a multi-scale image processing method.The method comprises the following steps: acquiring a multi-scale appearance image of the PCBA, and performing deep semantic fusion to obtain a reconstructed appearance image; performing wavelet frequency domain feature coding on the reconstructed appearance image to perform image adaptive enhancement on the multi-scale appearance image to obtain an enhanced appearance image; performing background separation on the enhanced appearance image to obtain a background separation image, and performing positive and negative structural element sliding processing on the background separation image to obtain a morphological processing image; and carrying out region segmentation on the morphological processing image to obtain a target attention region, identifying a multi-scale texture change degree of the target attention region, and carrying out appearance analysis on the PCBA to obtain an appearance detection report. According to the invention, the appearance detection accuracy of the PCBA can be improved.

Description

TECHNICAL FIELD

[0001] The application relates to a two-dimensional appearance detection method and system for PCBA based on a multi-scale image processing method, and belongs to the technical field of image processing. BACKGROUND

[0002] PCBA refers to printed circuit board assembly, which is formed by mounting electronic components on a printed circuit board (PCB) through welding, plug-in and other processes to form an electronic assembly with specific functions. The quality of PCBA directly affects the performance and reliability of electronic products. Therefore, in the electronic manufacturing industry, PCBA plays an extremely important role, and thus it is very important to control the quality of PCBA. Generally, the quality of PCBA is detected through appearance detection.

[0003] At present, the appearance detection of PCBA is mostly based on traditional image processing algorithms combined with single-scale detection technology. The appearance image of PCBA is collected, and then edge detection, threshold segmentation and other means are used to process the image. Then, the processed image is judged for defects by using a preset resolution. Although this method has a fast detection speed, due to the increasing integration of PCBA and the decreasing size of components, the type of defects tends to be diversified, which leads to great limitations of the traditional method in this case, and the method is prone to miss detection and false detection, resulting in low accuracy of appearance detection of PCBA. SUMMARY

[0004] The application provides a two-dimensional appearance detection method and system for PCBA based on a multi-scale image processing method, which mainly aims to improve the accuracy of appearance detection of PCBA.

[0005] To achieve the above purpose, the two-dimensional appearance detection method for PCBA based on the multi-scale image processing method provided by the application comprises the following steps:

[0006] Collecting a multi-scale appearance image of PCBA, performing feature progressive extraction on the multi-scale appearance image to obtain multi-scale features, performing deep semantic fusion on the multi-scale appearance image based on the multi-scale features to obtain a reconstructed appearance image;

[0007] Performing wavelet frequency domain feature coding on the reconstructed appearance image to obtain wavelet coding features, querying the multi-scale features of the reconstructed appearance image, and performing multi-scale feature fusion on the multi-scale features and the wavelet coding features to obtain joint coding features, performing image adaptive enhancement on the multi-scale appearance image based on the joint coding features to obtain an enhanced appearance image;

[0008] calculate an inter-class variance of the enhanced appearance image, perform background separation on the enhanced appearance image by using the inter-class variance to obtain a background separation image, and perform positive and negative structure element sliding processing on the background separation image to obtain a morphology processing image;

[0009] calculate an integral image of the morphology processing image, perform region segmentation on the morphology processing image by using the integral image to obtain a target attention region, identify a multi-scale texture variation degree of the target attention region, perform appearance analysis on the PCBA based on the multi-scale texture variation degree to obtain an appearance detection report.

[0010] Optionally, the multi-scale appearance image is subjected to feature progressive extraction to obtain a multi-scale feature, including:

[0011] the multi-scale appearance image is decomposed into a low-frequency structure layer, a medium-frequency texture layer, and a high-frequency detail layer;

[0012] the low-frequency structure layer is subjected to smoothing processing to obtain a low-frequency feature;

[0013] the medium-frequency texture layer is subjected to gradient histogram processing to obtain a medium-frequency feature;

[0014] the high-frequency detail layer is subjected to filter convolution to obtain a high-frequency feature;

[0015] the low-frequency feature, the medium-frequency feature, and the high-frequency feature are subjected to feature splicing to obtain a multi-scale feature.

[0016] Optionally, the multi-scale appearance image is subjected to deep semantic fusion based on the multi-scale feature to obtain a reconstructed appearance image, including:

[0017] the multi-scale feature is subjected to multi-channel pruning processing to obtain a pruned feature;

[0018] the pruned feature is subjected to feature compression to obtain a compressed feature;

[0019] the compressed feature is subjected to multi-receptive field feature fusion to obtain a fused feature;

[0020] a cross-region semantic relationship of the fused feature is constructed to convert the fused feature into graph structure data to obtain a reconstructed appearance image.

[0021] Optionally, the reconstructed appearance image is subjected to wavelet frequency domain feature coding to obtain a wavelet coding feature, including:

[0022] the reconstructed appearance image is subjected to local contrast stretching processing to obtain a contrast image;

[0023] the contrast image is subjected to edge sharpening processing to obtain a sharpened image;

[0024] performing dual-tree complex wavelet decomposition on the sharpened image to obtain multi-level image subbands;

[0025] performing adaptive quantization on the multi-level image subbands to obtain quantized features;

[0026] performing cross-direction joint coding on the quantized features to obtain wavelet coded features.

[0027] Optionally, based on the joint coded features, performing image adaptive enhancement on the multi-scale appearance image to obtain an enhanced appearance image, comprising:

[0028] performing effective subband screening on the joint coded features to obtain preferred coded features;

[0029] using the preferred coded features to construct a contrast gain map of the multi-scale appearance image;

[0030] using the contrast gain map to perform wavelet inverse transform reconstruction on the multi-scale appearance image to obtain an inverse transform reconstructed image;

[0031] performing anti-sharpening mask processing on the inverse transform reconstructed image to obtain an enhanced appearance image.

[0032] Optionally, calculating an inter-class variance of the enhanced appearance image, comprising:

[0033] performing grayscale processing on the enhanced appearance image and calculating an average grayscale value of the image after grayscale processing;

[0034] calculating a background class pixel probability and a foreground class pixel probability of the enhanced appearance image;

[0035] based on the background class pixel probability, calculating a background class average grayscale value of the enhanced appearance image;

[0036] based on the foreground class pixel probability, calculating a foreground class average grayscale value of the enhanced appearance image;

[0037] based on the average grayscale value, the background class pixel probability, the foreground class pixel probability, the background class average grayscale value and the foreground class average grayscale value, calculating an inter-class variance of the enhanced appearance image using the following formula:

[0038] ω=w1×(δ1-ε) 2 +w2×(δ2-ε) 2

[0039] Wherein, omega represents the inter-class variance, w1 represents the background class pixel probability, w2 represents the foreground class pixel probability, delta 1 represents the background class average gray value, delta 2 represents the foreground class average gray value, and epsilon represents the average gray value.

[0040] Optionally, the background separation image is subjected to a positive and negative structure element sliding processing to obtain a morphological processing image, comprising:

[0041] Query the image size of the background separation image;

[0042] Based on the image size, an element sliding structure of the background separation image is constructed;

[0043] The background separation image is subjected to image erosion using the element sliding structure, and the background separation image is subjected to image dilation using the element sliding structure to obtain a morphological processing image.

[0044] Optionally, the morphological processing image is subjected to region segmentation using the integral image to obtain a target attention region, comprising:

[0045] The integral image is used to construct a separation point of the morphological processing image;

[0046] The morphological processing image is subjected to non-maximum suppression with the separation point as the center to obtain a suppression image;

[0047] An image edge of the suppression image is identified;

[0048] Based on the image edge, the suppression image is subjected to region segmentation to obtain a target attention region.

[0049] Optionally, the multi-scale texture variation degree of the target attention region is identified, comprising:

[0050] A multi-scale decomposition image is obtained by performing multi-scale decomposition on a target attention region image corresponding to the target attention region;

[0051] A gray level co-occurrence matrix under each scale of the multi-scale decomposition image is calculated;

[0052] The gray level co-occurrence matrix is used to identify the contrast, texture smoothness and homogeneity of the multi-scale decomposition image under different scales;

[0053] Based on the contrast, the texture smoothness and the homogeneity, texture characteristic analysis is performed on the target attention region to determine the multi-scale texture variation degree of the target attention region.

[0054] In order to solve the above problems, the application further provides a multi-scale image processing method for realizing a two-dimensional appearance detection system of a PCBA, the system comprising:

[0055] an image reconstruction module, configured to collect a multi-scale appearance image of the PCBA, perform feature progressive extraction on the multi-scale appearance image to obtain multi-scale features, perform deep semantic fusion on the multi-scale appearance image based on the multi-scale features, and obtain a reconstructed appearance image;

[0056] an image enhancement module, configured to perform wavelet frequency domain feature coding on the reconstructed appearance image to obtain wavelet coding features, query the multi-scale features of the reconstructed appearance image, perform multi-scale feature fusion on the multi-scale features and the wavelet coding features to obtain joint coding features, perform image self-adaptive enhancement on the multi-scale appearance image based on the joint coding features, and obtain an enhanced appearance image;

[0057] an image morphology processing module, configured to calculate an inter-class variance of the enhanced appearance image, perform background separation on the enhanced appearance image by using the inter-class variance to obtain a background separation image, and perform positive and negative structure element sliding processing on the background separation image to obtain a morphology processing image;

[0058] an appearance detection module, configured to calculate an integral image of the morphology processing image, perform region segmentation on the morphology processing image by using the integral image to obtain a target attention region, identify a multi-scale texture variation degree of the target attention region, perform appearance analysis on the PCBA based on the multi-scale texture variation degree, and obtain an appearance detection report.

[0059] Compared with the problems in the background art, the multi-scale appearance image of the PCBA is collected to obtain image data at different scales, and the multi-scale appearance image is structured in multiple levels to facilitate image analysis at different fine granularities. Then, the image is subjected to multi-channel pruning, feature compression, and multi-receptive field feature fusion to realize semantic fusion of the image, eliminate redundancy and conflicts, and make the image semantics more accurate and complete. Further, the image is subjected to wavelet decomposition to extract different frequency feature information of the reconstructed appearance image, so as to highlight the details and textures of the image. The multi-scale features of the image are fused with the wavelet coding features to make full use of the advantages of the two kinds of features, so that the details of the image are more optimized. The contrast gain map is constructed by performing optimized offspring screening on the image, and the appearance image is subjected to detail enhancement by using the contrast gain map, so that the key information is more obvious. Further, the inter-class variance of the image is calculated to distinguish the background of the image, and background separation is performed to improve the analysis efficiency. Then, morphological processing is performed to further improve the image quality, so that the analysis result is more accurate. Further, the integral image of the enhanced image is calculated to identify the target analysis region in the image, so as to achieve the effect of accurately positioning the target analysis region. Then, the multi-scale texture variation degree analysis is performed on the target analysis region to complete the appearance detection of the PCBA, thereby improving the appearance detection accuracy of the PCBA. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A schematic diagram of a flow chart of a method for implementing a two-dimensional appearance inspection method for PCBA using a multi-scale image processing method provided in one embodiment of the present invention;

[0061] Figure 2 A schematic diagram of a module for implementing a two-dimensional appearance inspection method for PCBAs using the multi-scale image processing method provided in one embodiment of the present invention.

[0062] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0063] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0064] The embodiments of the present application provide a multi-scale image processing method for implementing a two-dimensional appearance inspection method for PCBAs. The execution entity for implementing the multi-scale image processing method for implementing the two-dimensional appearance inspection method for PCBAs includes, but is not limited to, at least one of electronic devices such as a server and a terminal that can be configured to execute the method provided in the embodiments of the present application. In other words, the multi-scale image processing method for implementing the two-dimensional appearance inspection method for PCBAs can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster.

[0065] Example 1:

[0066] Reference Figure 1 FIG. 1 is a flow chart of a method for implementing a two-dimensional appearance inspection method for PCBA using a multi-scale image processing method according to an embodiment of the present invention. In this embodiment, the method for implementing a two-dimensional appearance inspection method for PCBA using the multi-scale image processing method includes:

[0067] S1. Collect multi-scale appearance images of the PCBA, perform progressive feature extraction on the multi-scale appearance images to obtain multi-scale features, and perform deep semantic fusion on the multi-scale appearance images based on the multi-scale features to obtain a reconstructed appearance image.

[0068] The embodiment of the present invention can obtain image data of the PCBA at different scales by collecting multi-scale appearance images of the PCBA, thereby understanding the overall layout and subtle structural information of the PCBA.

[0069] The PCBA refers to printed circuit board assembly, which is a component with specific circuit functions formed by mounting various electronic components (such as resistors, capacitors, chips, etc.) on a printed circuit board (PCB) through welding, patching and other processes.

[0070] Optionally, the multi-scale appearance image can be collected by using a multi-resolution high-definition camera.

[0071] Further, the embodiment of the present application can obtain more comprehensive and representative multi-scale features by progressively extracting features from the multi-scale appearance image, so as to facilitate the user to understand the characteristics of the PCBA under different observation granularities.

[0072] The multi-scale feature refers to image features under different resolutions or frequency domains.

[0073] As an embodiment of the present application, the multi-scale appearance image is progressively extracted to obtain multi-scale features, comprising:

[0074] The multi-scale appearance image is decomposed into a low-frequency structure layer, a medium-frequency texture layer and a high-frequency detail layer;

[0075] The low-frequency structure layer is smoothed to obtain a low-frequency feature;

[0076] The medium-frequency texture layer is processed by a gradient histogram to obtain a medium-frequency feature;

[0077] The high-frequency detail layer is filtered and convolved to obtain a high-frequency feature;

[0078] The low-frequency feature, the medium-frequency feature and the high-frequency feature are spliced to obtain a multi-scale feature.

[0079] The low-frequency structure layer refers to the slowly changing and relatively smooth part of the image, such as the PCBA shape, the main component layout, etc., the medium-frequency texture layer refers to the texture information of the component surface on the PCBA and some not obvious structural changes, such as the circuit board texture, and the high-frequency detail layer refers to the part of the image with intense changes and rich details, such as the PCBA solder joints, pins, etc.

[0080] In the specific implementation process, the multi-scale appearance image can be subjected to Gaussian filtering and downsampling operations multiple times by using a Gaussian pyramid technique to construct an image pyramid with different resolutions, and then the multi-scale appearance image is decomposed into a low-frequency structure layer, a medium-frequency texture layer and a high-frequency detail layer by using the image pyramid, the low-frequency feature can be obtained by processing the low-frequency structure layer image by using a Gaussian smoothing filter, the medium-frequency feature can be obtained by calculating the gradient of the medium-frequency texture layer image using a gradient operator (such as a Sobel operator), and then counting the gradient direction histogram, and the high-frequency feature can be obtained by convolving the high-frequency detail layer using a convolution layer in a convolutional neural network, and the multi-scale feature can be formed by directly splicing the low-frequency feature, the medium-frequency feature and the high-frequency feature.

[0081] Further, the embodiment of the present application obtains a reconstructed appearance image by performing deep semantic fusion on the multi-scale appearance image based on the multi-scale feature, which effectively integrates the information in different scale features, eliminates the redundancy and conflict between features, and makes the reconstructed appearance image more accurate and complete in semantics.

[0082] The reconstructed appearance image refers to an image obtained by integrating and optimizing image features at different scales, eliminating the redundancy and conflict between features, and then performing deep semantic fusion.

[0083] As an embodiment of the present application, the deep semantic fusion on the multi-scale appearance image based on the multi-scale feature to obtain a reconstructed appearance image comprises:

[0084] The multi-scale feature is subjected to multi-channel pruning processing to obtain a pruned feature.

[0085] The pruned feature is subjected to feature compression to obtain a compressed feature.

[0086] The compressed feature is subjected to multi-receptive field feature fusion to obtain a fused feature.

[0087] The cross-region semantic relationship of the fused feature is constructed to convert the fused feature into graph structure data to obtain a reconstructed appearance image.

[0088] The pruned feature refers to a feature obtained by subjecting the multi-scale feature to multi-channel pruning processing, and the compressed feature refers to a low-dimensional feature obtained by subjecting the pruned feature to feature dimension reduction processing.

[0089] Optionally, the pruning feature can use a gradient-based method to calculate the contribution of each channel feature in the multi-scale feature to the PCBA appearance detection task, and then remove the channels with low contribution to obtain, and the specific needs of the actual appearance detection task, such as the need to detect the appearance of the protective shell, the appearance of the pin, and the integrity of the element, etc., the compression feature can be obtained by performing dimensionality reduction processing on the pruning feature using principal component analysis, and the fusion feature can be obtained by processing the compression feature using a multi-scale convolution kernel in a convolutional neural network to obtain feature information under different receptive fields, such as in PCBA appearance detection, small receptive fields can capture element detail features, and large receptive fields can grasp overall structure features, and these different receptive field features are spliced or weighted and fused to obtain a fusion feature containing multi-scale information, and the reconstructed appearance image can be obtained by using a graph neural network technology, regarding different regions in the fusion feature as nodes of a graph, determining edges and weights according to the semantic correlation between regions to construct cross-region semantic relationships, and then modeling the fusion feature based on the cross-region semantic relationships to obtain a graph structure data.

[0090] S2, wavelet frequency domain feature coding is performed on the reconstructed appearance image to obtain wavelet coding features, multi-scale features of the reconstructed appearance image are queried, and multi-scale feature fusion is performed on the multi-scale features and the wavelet coding features to obtain joint coding features, and based on the joint coding features, image adaptive enhancement is performed on the multi-scale appearance image to obtain an enhanced appearance image.

[0091] In the embodiment of the application, the wavelet coding features obtained by performing wavelet frequency domain feature coding on the reconstructed appearance image can extract feature information of the reconstructed appearance image at different frequencies, so as to better highlight the details and textures of the image.

[0092] The wavelet coding features refer to features obtained by performing wavelet frequency domain feature coding on the reconstructed appearance image.

[0093] As an embodiment of the application, the wavelet coding features obtained by performing wavelet frequency domain feature coding on the reconstructed appearance image include:

[0094] The reconstructed appearance image is subjected to local contrast stretching processing to obtain a contrast image.

[0095] The contrast image is subjected to edge sharpening processing to obtain a sharpened image.

[0096] The sharpened image is subjected to dual-tree complex wavelet decomposition to obtain multi-level image subbands.

[0097] The multi-level image subbands are subjected to adaptive quantization to obtain quantized features.

[0098] The quantized features are subjected to cross-direction joint coding to obtain wavelet coding features.

[0099] The contrast image refers to an image obtained by dividing an image into a plurality of local regions, respectively linearly stretching pixel gray values in each region according to minimum and maximum values of the pixel gray values, and improving the contrast of the local regions of the image, the sharpened image refers to an image obtained by performing edge sharpening processing on the image, and the multi-level image subband refers to an image subband obtained by performing wavelet decomposition on the image in different scales and directions, and the subband image can capture feature information of the image in different scales and directions.

[0100] Optionally, the contrast image can divide the reconstructed appearance image into a plurality of non-overlapping local regions, calculate minimum and maximum values of the gray values of each local region, and then obtain the linear stretching of the pixel gray values in the region according to the two values, the sharpened image can be obtained by performing convolution operation on the contrast image by using a Laplace operator and enhancing the gray value change of the edge and detail part of the image, the multi-level image subband can be obtained by decomposing the sharpened image by using a dual-tree complex wavelet transform algorithm, decomposing the image into image subbands in different scales and directions, and obtaining the subbands, the quantized feature can be obtained by determining a suitable quantization step for each subband according to the statistical characteristics (such as mean, variance, etc.) of each image subband, and then quantizing the coefficients in the subband, and the wavelet coding feature can be obtained by using Huffman coding to compress and encode the quantized coefficients.

[0101] Further, the embodiment of the present application can fully utilize the advantages of the two features, make up for the deficiency of a single feature, make the joint coding feature more representative and robust, and better reflect the overall feature and detail feature of the PCBA appearance image by querying the multi-scale features of the reconstructed appearance image, fusing the multi-scale features and the wavelet coding features, and obtaining the joint coding feature.

[0102] Optionally, the joint coding feature can fuse the multi-scale features and the wavelet coding features in different scale dimensions by using feature splicing or weighted summation.

[0103] Further, the embodiment of the present application can make the defect and other key information in the PCBA appearance image more obvious, and improve the visual quality of the image by performing image adaptive enhancement on the multi-scale appearance image based on the joint coding feature, and obtaining the enhanced appearance image.

[0104] As an embodiment of the present application, the image adaptive enhancement is performed on the multi-scale appearance image based on the joint coding feature, and the enhanced appearance image is obtained, including:

[0105] The joint coding feature is subjected to effective subband screening, and an optimal coding feature is obtained.

[0106] constructing a contrast gain map of the multi-scale appearance image by using the preferred encoding features;

[0107] reconstructing an inverse transform reconstructed image by inverse wavelet transform of the multi-scale appearance image by using the contrast gain map;

[0108] obtaining an enhanced appearance image by performing unsharp masking on the inverse transform reconstructed image.

[0109] The preferred encoding features refer to the encoding features obtained by high-frequency energy distribution screening of joint encoding features, the contrast gain map refers to a coefficient matrix dynamically generated by gradient or brightness distribution of a local region, and the inverse transform reconstructed image refers to an image reconstructed after low-frequency energy part in the multi-scale appearance image.

[0110] The preferred encoding features can be obtained by calculating sub-generation energy corresponding to the joint encoding features, then removing 20% of the sub-generation energy with the lowest energy, and retaining the remaining encoding features, the contrast gain map can analyze the mapping relationship between the preferred encoding features and the image contrast, then calculate the contrast gain value of each pixel position according to the mapping relationship, and construct the contrast gain map according to the contrast gain value, the inverse transform reconstructed image can be obtained by applying the contrast gain map to the wavelet coefficients of the multi-scale appearance image, adjusting the wavelet coefficients, and performing inverse wavelet transform, and the enhanced appearance image can be obtained by performing blur processing on the inverse transform reconstructed image using an image blurring method to obtain a blurred image, then subtracting the blurred image from the original image to obtain a sharpening mask, and finally superimposing the sharpening mask on the original image.

[0111] S3, calculating the inter-class variance of the enhanced appearance image, performing background separation on the enhanced appearance image by using the inter-class variance to obtain a background separation image, and performing sliding processing of positive and negative structure elements on the background separation image to obtain a morphological processing image.

[0112] The inter-class variance of the enhanced appearance image can be calculated to obtain a statistical quantity for measuring the difference between different categories (usually referring to foreground and background) in the image, and then the image can be conveniently separated from the background, and the image analysis efficiency is improved.

[0113] The inter-class variance refers to a segmentation threshold value that can best distinguish the foreground and background of the image.

[0114] As an embodiment of the present application, the calculation of the inter-class variance of the enhanced appearance image comprises:

[0115] performing grayscale processing on the enhanced appearance image, and calculating the average gray value of the image after grayscale processing;

[0116] calculating a background class pixel probability and a foreground class pixel probability of the enhanced appearance image;

[0117] calculating a background class average gray value of the enhanced appearance image based on the background class pixel probability;

[0118] calculating a foreground class average gray value of the enhanced appearance image based on the foreground class pixel probability;

[0119] calculating an inter-class variance of the enhanced appearance image based on the average gray value, the background class pixel probability, the foreground class pixel probability, the background class average gray value and the foreground class average gray value by using the following formula:

[0120] ω = w1 × (δ1 - ε) 2 + w2 × (δ2 - ε) 2

[0121] wherein ω represents the inter-class variance, w1 represents the background class pixel probability, w2 represents the foreground class pixel probability, δ1 represents the background class average gray value, δ2 represents the foreground class average gray value, and ε represents the average gray value.

[0122] wherein the background class pixel probability refers to a proportion of a number of pixels classified as a background part in an image to a total number of pixels in the image, and the foreground class pixel probability refers to a ratio of a number of pixels identified as a foreground part in the image to the total number of pixels in the image.

[0123] In the implementation process, a gray threshold value can be preset (a gray value that can make the inter-class variance maximum can be automatically calculated as the threshold value through an iteration method), pixels with a gray value less than the threshold value are classified as the background class, and pixels with a gray value greater than or equal to the threshold value are classified as the foreground class, then the number of background class pixels and the number of foreground class pixels are counted respectively, then the background class pixel probability is obtained by dividing the number of background class pixels by the total number of pixels in the image, and the foreground class pixel probability is obtained by dividing the number of foreground class pixels by the total number of pixels in the image, the background class average gray value can be obtained by adding the gray values of all the background class pixels and then dividing the sum by the number of background class pixels, and the calculation principle of the foreground class average gray value is the same as that of the background class average gray value, and thus is not described herein.

[0124] It needs to be further explained that the inter-class variance calculation formula is determined by calculating the probabilities of the background class and the foreground class pixels and the differences between their respective average gray values and the average gray value of the image, w1 and w2 respectively represent the probabilities of the background class and the foreground class pixels, reflect the proportions of the two types of pixels in the image, δ1 and δ2 are respectively the background class average gray value and the foreground class average gray value, ε is the average gray value of the image, (δ1 - ε) 2 and (δ2 - ε) 2The deviation degrees of the average gray values of the background class and the foreground class from the overall average gray value of the image are measured, the formula quantifies the difference degrees between the foreground and the background by multiplying the deviation degrees by the respective pixel probabilities and adding them, the greater the inter-class variance value, the more obvious the gray difference between the foreground and the background, that is, the higher the discrimination degree of different classes (foreground and background) in the image, in image analysis, such as processing of the enhanced appearance image, the image can be well divided into the foreground and the background through the inter-class variance, which is helpful for image segmentation and target recognition.

[0125] Further, the embodiment of the present application can remove the background information, highlight the foreground target, and make subsequent analysis and processing of the target object easier by separating the background of the enhanced appearance image through the inter-class variance to obtain the background separation image.

[0126] The background separation image refers to an image obtained by separating the background and the foreground of an image.

[0127] Optionally, the background separation image can be separated from the background part of the enhanced appearance image according to the inter-class variance calculation result.

[0128] Further, the embodiment of the present application can further optimize the quality of the image, and make the shape of the foreground object more clear and accurate by performing the positive and negative structural element sliding processing on the background separation image to obtain the morphological processing image.

[0129] The morphological processing image refers to an image obtained by performing morphological processing on the background separation image.

[0130] As an embodiment of the present application, the positive and negative structural element sliding processing is performed on the background separation image to obtain the morphological processing image, including:

[0131] Querying the image size of the background separation image;

[0132] Based on the image size, an element sliding structure of the background separation image is constructed;

[0133] The element sliding structure is used to perform image erosion on the background separation image, and the element sliding structure is used to perform image dilation on the background separation image to obtain the morphological processing image.

[0134] The element sliding structure refers to a structural element in morphological image processing, which is a self-defined shape (such as a common rectangle, circle, cross, etc.) and size template.

[0135] Optionally, the element sliding structure can be created according to the size of the background separation image and the preset structural element shape (such as rectangle, circle, etc.) and size requirements. The image erosion of the background separation image can be achieved by sliding the constructed element sliding structure on the background separation image in a certain order (such as from left to right, from top to bottom) to remove some boundary pixels. The image expansion of the background separation image can be achieved by sliding the element sliding structure on the eroded image (such as from left to right, from top to bottom) to supplement the missing pixels.

[0136] S4. Calculate an integral image of the morphologically processed image, use the integral image to perform region segmentation on the morphologically processed image to obtain a target region of interest, identify a multi-scale texture variation of the target region of interest, and perform an appearance analysis on the PCBA based on the multi-scale texture variation to obtain an appearance inspection report.

[0137] The embodiment of the present invention can quickly calculate the pixel feature values ​​of specific areas (such as components, solder joints, etc.) by calculating the integral image of the morphologically processed image, which helps to quickly locate and identify these areas.

[0138] The integral image refers to a data structure used to quickly calculate the sum of image regions.

[0139] In the specific implementation process, the morphologically processed image can be converted into a grayscale image, and then the integralImage() function in MATLAB can be used to directly calculate the integral image of the grayscale image.

[0140] Furthermore, the embodiments of the present invention utilize the integral image to perform region segmentation on the morphologically processed image to obtain a target region of interest, which can narrow the detection range and focus on the key parts that need to be inspected, thereby improving the accuracy and efficiency of PCBA appearance inspection. For example, the housing, pins, and other parts of the PCBA can be separated and analyzed separately, thereby avoiding mutual interference and improving analysis efficiency.

[0141] As an embodiment of the present invention, the step of performing region segmentation on the morphologically processed image using the integral image to obtain a target region of interest includes:

[0142] constructing separation points of the morphologically processed image using the integral image;

[0143] Taking the separation point as the center, performing non-maximum suppression on the morphologically processed image to obtain a suppressed image;

[0144] identifying image edges of the suppressed image;

[0145] Based on the image edges, the suppressed image is segmented to obtain a target region of interest.

[0146] Wherein, the separation point refers to the boundary of different regions in the image or the place where the feature changes obviously, the pixel and the surrounding area have significant difference, which can be used as the key node to distinguish different regions, and the image edge refers to the boundary between the target object and the background or different target objects.

[0147] Optionally, the separation point can be used to quickly calculate the pixel sum of different regions in the image, and then the point with the pixel sum between 150-200 is selected to determine (specifically, the value obtained by calculation in the actual application process is set), the suppression image can be divided into a local area with the separation point as the center, the pixel values in the region are compared, only the local maximum value point is reserved, the rest of the pixel values are set as non-maximum value and suppressed (reduce the gray value or mark rejection) to obtain, and the image edge can be obtained by using the Canny edge detection algorithm to perform gradient calculation and threshold processing on the suppression image to extract the edge profile of the target object in the image.

[0148] The multi-scale texture variation degree of the target attention region can be more comprehensive and detailed to identify the texture features of the PCBA surface, capture possible defects or abnormal texture information, and provide a more reliable basis for accurately judging the appearance of the PCBA.

[0149] Wherein, the multi-scale texture variation degree refers to the degree of image texture variation of the image under multiple scales or multiple resolutions.

[0150] As an embodiment of the present application, the multi-scale texture variation degree of the target attention region comprises:

[0151] Performing multi-scale decomposition on the attention region image corresponding to the target attention region to obtain a multi-scale decomposition image;

[0152] Calculating the gray level co-occurrence matrix under each scale of the multi-scale decomposition image;

[0153] Using the gray level co-occurrence matrix to identify the contrast, texture smoothness and homogeneity of the multi-scale decomposition image under different scales;

[0154] Based on the contrast, the texture smoothness and the homogeneity, performing texture characteristic analysis on the target attention region to determine the multi-scale texture variation degree of the target attention region.

[0155] The gray level co-occurrence matrix refers to a mathematical tool of a statistical method for describing the spatial distribution of gray levels in an image, the contrast refers to a characteristic quantity of the gray level co-occurrence matrix, and is used for measuring the clear degree of texture and the severe degree of gray level change in the image, the correlation can be used to represent the linear correlation between pixel gray values in a local region in the image, and reflects the regularity and directionality of the texture of the image, and the texture smoothness reflects the smooth degree of the texture in the image, and the higher the energy value is, the smoother the texture in the image is, and the distribution of pixel gray values is more concentrated, and vice versa, the lower the energy value is, the coarser the texture is.

[0156] Optionally, the multi-scale decomposition image can be obtained by using a Gaussian pyramid algorithm to perform multi-scale decomposition on the attention region image corresponding to the target attention region, the gray level co-occurrence matrix can be constructed by statistically analyzing the occurrence frequency of pixel pairs in terms of gray values and spatial positions under preset distance and angle parameters (for example, the distance is 1, and the angles are 0°, 45°, 90° and 135°), the contrast can be obtained by calculating the sum of the products of the element values in the gray level co-occurrence matrix and the corresponding gray level difference values, the texture smoothness can be obtained by summing the square values of the element values in the gray level co-occurrence matrix, the homogeneity can be obtained by calculating the sum of the element values in the gray level co-occurrence matrix divided by the gray level distance plus 1, and the multi-scale texture change degree can be determined by analyzing the change trend and rule of the texture of the target attention region under different scales according to the numerical differences of the contrast, the texture smoothness and the homogeneity under different scales.

[0157] According to the multi-scale texture change degree, the appearance of the PCBA is analyzed to obtain an appearance detection report, the identified multi-scale texture change degree can be used as a basis to accurately evaluate and analyze the appearance of the PCBA, to determine whether there are defects, flaws and other problems, and to generate a detailed appearance detection report.

[0158] In the implementation process, the appearance of the PCBA can be preliminarily judged by contrast, if the contrast value is large (such as contrast greater than 40, which should be set in combination with the actual application scene), it indicates that the appearance of the PCBA is obvious, at this time, the historical detection image can be directly used for image matching to realize appearance detection, such as detecting whether the packaging shell has defects, cracks, whether the pin is broken or bent, etc., if the contrast value is not large enough (less than 35), it indicates that the appearance of the PCBA is not obvious enough, at this time, the texture smoothness and homogeneity need to be combined to analyze (generally, the texture smoothness and homogeneity are positively correlated), if the texture smoothness is in the range of 0.15-0.3, and the homogeneity is in the range of 0.8-0.95, it can be considered that the appearance of the PCBA is good and there is no problem, if the texture smoothness is less than 0.15 and the homogeneity is less than 0.8, it is considered that there is a defect, it needs to be further explained that this is an idealized example, in the actual PCBA detection, the reasonable threshold and judgment standard need to be determined in combination with the specific product characteristics and a large amount of actual detection data.

[0159] Embodiment 2

[0160] As shown in Figure 2 is a function module diagram of a two-dimensional appearance detection system of a PCBA realized by the multi-scale image processing method of the application.

[0161] The two-dimensional appearance detection system 200 of the PCBA realized by the multi-scale image processing method of the application can be installed in an electronic device. According to the realized functions, the two-dimensional appearance detection system of the PCBA realized by the multi-scale image processing method can include an image reconstruction module 201, an image enhancement module 202, an image morphology processing module 203 and an appearance detection module 204. The modules of the application can also be called units, which refer to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0162] In the embodiments of the application, the functions of each module / unit are as follows:

[0163] The image reconstruction module 201 is used to collect multi-scale appearance images of the PCBA, perform feature progressive extraction on the multi-scale appearance images to obtain multi-scale features, perform deep semantic fusion on the multi-scale appearance images based on the multi-scale features, and obtain a reconstructed appearance image;

[0164] The image enhancement module 202 is configured to perform wavelet frequency domain feature coding on the reconstructed appearance image to obtain wavelet coding features, query multi-scale features of the reconstructed appearance image, perform multi-scale feature fusion on the multi-scale features and the wavelet coding features to obtain joint coding features, and perform image self-adaptive enhancement on the multi-scale appearance image based on the joint coding features to obtain an enhanced appearance image.

[0165] The image morphology processing module 203 is configured to calculate an inter-class variance of the enhanced appearance image, perform background separation on the enhanced appearance image by using the inter-class variance to obtain a background separation image, and perform positive and negative structure element sliding processing on the background separation image to obtain a morphology processing image.

[0166] The appearance detection module 204 is configured to calculate an integral image of the morphology processing image, perform region segmentation on the morphology processing image by using the integral image to obtain a target attention region, identify a multi-scale texture variation degree of the target attention region, and perform appearance analysis on the PCBA based on the multi-scale texture variation degree to obtain an appearance detection report.

[0167] In detail, the multi-scale image processing method in the embodiments of the present application realizes the use of the modules in the two-dimensional appearance detection system 200 of the PCBA in the same technical means as the multi-scale image processing method for realizing the two-dimensional appearance detection method of the PCBA in the above-mentioned Figure 1 embodiments, and can produce the same technical effects, which will not be described here.

[0168] It is obvious for those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and the present application can be realized in other specific forms without departing from the spirit or essential characteristics of the present application.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application but not limit the present application, and although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. A multi-scale image processing method for realizing two-dimensional appearance detection of a PCBA, characterized in that, The method comprises: Collecting multi-scale appearance images of the PCBA, performing feature progressive extraction on the multi-scale appearance images to obtain multi-scale features, performing deep semantic fusion on the multi-scale appearance images based on the multi-scale features to obtain a reconstructed appearance image; Performing wavelet frequency domain feature coding on the reconstructed appearance image to obtain wavelet coding features, querying multi-scale features of the reconstructed appearance image, and performing multi-scale feature fusion on the multi-scale features and the wavelet coding features to obtain joint coding features, performing image self-adaptive enhancement on the multi-scale appearance images based on the joint coding features to obtain an enhanced appearance image; Calculating the inter-class variance of the enhanced appearance image, performing background separation on the enhanced appearance image using the inter-class variance to obtain a background separation image, performing sliding processing of positive and negative structure elements on the background separation image to obtain a morphological processing image; Calculating the integral image of the morphological processing image, performing region segmentation on the morphological processing image using the integral image to obtain a target attention region, identifying the multi-scale texture variation degree of the target attention region, and performing appearance analysis on the PCBA based on the multi-scale texture variation degree to obtain an appearance detection report.

2. The multi-scale image processing method for detecting the two-dimensional appearance of a PCBA according to claim 1, wherein, Performing feature progressive extraction on the multi-scale appearance images to obtain multi-scale features comprises: Decomposing the multi-scale appearance images into a low-frequency structure layer, a medium-frequency texture layer, and a high-frequency detail layer; Performing smoothing processing on the low-frequency structure layer to obtain a low-frequency feature; Performing gradient histogram processing on the medium-frequency texture layer to obtain a medium-frequency feature; Performing filter convolution on the high-frequency detail layer to obtain a high-frequency feature; Performing feature splicing on the low-frequency feature, the medium-frequency feature, and the high-frequency feature to obtain multi-scale features.

3. The multi-scale image processing method for detecting the two-dimensional appearance of a PCBA according to claim 1, wherein, Performing deep semantic fusion on the multi-scale appearance images based on the multi-scale features to obtain a reconstructed appearance image comprises: Performing multi-channel pruning processing on the multi-scale features to obtain pruned features; Performing feature compression on the pruned features to obtain compressed features; Performing multi-receptive field feature fusion on the compressed features to obtain fused features; Constructing cross-region semantic relationships of the fused features to convert the fused features into graph structure data to obtain a reconstructed appearance image.

4. The multi-scale image processing method for detecting the two-dimensional appearance of the PCBA according to claim 1, wherein, Performing wavelet frequency domain feature coding on the reconstructed appearance image to obtain wavelet coding features comprises: Performing local contrast stretching processing on the reconstructed appearance image to obtain a contrast image; Performing edge sharpening processing on the contrast image to obtain a sharpened image; Performing dual-tree complex wavelet decomposition on the sharpened image to obtain multi-level image subbands; Performing adaptive quantization on the multi-level image subbands to obtain quantized features; Performing cross-direction joint coding on the quantized features to obtain wavelet coding features.

5. The multi-scale image processing method for detecting the two-dimensional appearance of a PCBA according to claim 1, wherein, Performing image self-adaptive enhancement on the multi-scale appearance images based on the joint coding features to obtain an enhanced appearance image comprises: Performing effective subband screening on the joint coding features to obtain optimal coding features; Using the optimal coding features to construct a contrast gain map of the multi-scale appearance images; inverse wavelet transform reconstruction is performed on the multi-scale appearance image based on the contrast gain map, to obtain an inverse transform reconstructed image; inverse unsharp masking is performed on the inverse transform reconstructed image, to obtain an enhanced appearance image.

6. The multi-scale image processing method for detecting the two-dimensional appearance of a PCBA according to claim 1, wherein, inter-class variance of the enhanced appearance image is calculated, including: gray processing is performed on the enhanced appearance image, and average gray value of the image after the gray processing is calculated; background class pixel probability and foreground class pixel probability of the enhanced appearance image are calculated; background class average gray value of the enhanced appearance image is calculated based on the background class pixel probability; foreground class average gray value of the enhanced appearance image is calculated based on the foreground class pixel probability; inter-class variance of the enhanced appearance image is calculated based on the average gray value, the background class pixel probability, the foreground class pixel probability, the background class average gray value and the foreground class average gray value, by using the following formula: ω = w1 x (δ1 - ε) 2 + w2 x (δ2 - ε) 2 wherein ω represents the inter-class variance, w1 represents the background class pixel probability, w2 represents the foreground class pixel probability, δ1 represents the background class average gray value, δ2 represents the foreground class average gray value, and ε represents the average gray value.

7. The multi-scale image processing method for detecting the two-dimensional appearance of a PCBA according to claim 1, wherein, morphological processing is performed on the background separation image by using a positive and negative structure element sliding process, to obtain a morphologically processed image, including: image size of the background separation image is queried; an element sliding structure of the background separation image is constructed based on the image size; image erosion is performed on the background separation image by using the element sliding structure, and image dilation is performed on the background separation image by using the element sliding structure, to obtain the morphologically processed image.

8. The multi-scale image processing method of claim 1, wherein the method of detecting a two-dimensional appearance of a PCBA is characterized by, the integral image is used to perform region segmentation on the morphologically processed image, to obtain a target attention region, including: a separation point of the morphologically processed image is constructed by using the integral image; non-maximum suppression is performed on the morphologically processed image with the separation point as the center, to obtain a suppressed image; image edges of the suppressed image are identified; region segmentation is performed on the suppressed image based on the image edges, to obtain the target attention region.

9. The multi-scale image processing method of claim 1, wherein the method of detecting a two-dimensional appearance of a PCBA is characterized by, the multi-scale texture variation degree of the target attention region is identified, including: a multi-scale decomposition is performed on an attention region image corresponding to the target attention region, to obtain a multi-scale decomposition image; a gray level co-occurrence matrix under each scale of the multi-scale decomposition image is calculated; the gray level co-occurrence matrix is used to identify contrast, texture smoothness and homogeneity of the multi-scale decomposition image under different scales; texture characteristic analysis is performed on the target attention region based on the contrast, the texture smoothness and the homogeneity, to determine the multi-scale texture variation degree of the target attention region.

10. A multi-scale image processing method for implementing a two-dimensional appearance detection system of a PCBA, characterized in that, the system includes: an image reconstruction module, configured to collect a multi-scale appearance image of a PCBA, perform feature progressive extraction on the multi-scale appearance image, to obtain multi-scale features, perform deep semantic fusion on the multi-scale appearance image based on the multi-scale features, to obtain a reconstructed appearance image; An image enhancement module is configured to perform wavelet frequency domain feature coding on the reconstructed appearance image to obtain wavelet coding features, query multi-scale features of the reconstructed appearance image, perform multi-scale feature fusion on the multi-scale features and the wavelet coding features to obtain joint coding features, perform image self-adaptive enhancement on the multi-scale appearance image based on the joint coding features, and obtain an enhanced appearance image; An image morphology processing module is configured to calculate an inter-class variance of the enhanced appearance image, perform background separation on the enhanced appearance image by using the inter-class variance to obtain a background separation image, and perform positive and negative structure element sliding processing on the background separation image to obtain a morphology processing image; An appearance detection module is configured to calculate an integral image of the morphology processing image, perform region segmentation on the morphology processing image by using the integral image to obtain a target attention region, identify a multi-scale texture variation degree of the target attention region, perform appearance analysis on the PCBA based on the multi-scale texture variation degree, and obtain an appearance detection report.

Images