Circuit board welding fault identification method and system based on machine vision

Abstract

The present invention provides a circuit board welding fault recognition method and system based on machine vision, which relates to the field of visual recognition technology, including using a structured light projection system and a multi-camera imaging system to acquire an image of a circuit board welding area, using three-dimensional morphological information and realizing accurate positioning of a defective area in an area of ​​interest through morphological operations and edge detection algorithms; obtaining high-precision three-dimensional point cloud data of the defective area after accurate positioning, extracting a multi-scale curvature feature descriptor, and fusing the multi-scale curvature feature descriptor with a visual feature to form a multi-feature descriptor; inputting the multi-feature descriptor into a defect classification and recognition module based on ensemble learning to obtain the type and severity of the welding defect; and intuitively displaying the three-dimensional spatial distribution and severity of the welding defect according to the defect recognition result.

Description

Technical Field

[0001] The present invention relates to visual recognition technology, and in particular to a circuit board welding fault recognition method and system based on machine vision. Background Art

[0002] Circuit boards are key components of electronic products, and their welding quality directly affects the reliability and service life of electronic products. Welding failures are common quality problems in circuit board production, mainly including cold solder joints, leaking solder joints, short circuits, and arching. Traditional circuit board welding quality inspection mainly relies on manual visual inspection, which has problems such as low inspection efficiency and strong subjectivity of inspection results, making it difficult to meet the high quality and high efficiency requirements of the modern electronic manufacturing industry.

[0003] There are still some shortcomings in the existing circuit board welding fault recognition methods. On the one hand, most methods use single-view images for analysis, which makes it difficult to fully characterize the three-dimensional morphological features of solder joints, and the recognition accuracy of some complex welding faults is not high. On the other hand, the existing methods mainly rely on manually designed feature descriptors, lack the mining of deep semantic information of solder joint images, and are difficult to adapt to changes in different solder joint types and imaging conditions. In addition, traditional methods are highly dependent on algorithm parameters, and their generalization ability and robustness need to be improved. Summary of the invention

[0004] The embodiments of the present invention provide a method and system for identifying circuit board welding faults based on machine vision, which can solve the problems in the prior art.

[0005] According to a first aspect of the embodiments of the present invention,

[0006] Provides a circuit board welding fault identification method based on machine vision, including:

[0007] The image of the soldering area of ​​the circuit board is acquired by using a structured light projection system and a multi-camera imaging system. The three-dimensional morphology information of the soldering area is acquired by using the parallax principle. The imaging quality of the soldering area is optimized by adjusting the light source parameters in real time in combination with an adaptive light source system based on a dual-optical coaxial light source and a multi-angle annular light source. The original images acquired by the structured light projection system and the multi-camera imaging system are subjected to distortion correction and image registration. The registered images are subjected to adaptive noise reduction and background equalization processing by using wavelet transform and morphological filtering algorithms to obtain the pre-processed soldering area image.

[0008] A method combining a watershed algorithm based on graph theory and an adaptive threshold segmentation algorithm is used to automatically segment the preprocessed welding area image to obtain the region of interest. The 3D morphology information is used and morphological operations and edge detection algorithms are used to accurately locate the defect area in the region of interest. For the defect area after accurate positioning, the triangulation principle is used to reconstruct the defect area based on the multi-view images and structured light information obtained by the multi-camera imaging system to obtain high-precision 3D point cloud data of the defect area. For the high-precision 3D point cloud data of the reconstructed defect area, a multi-scale curvature feature descriptor is extracted and the multi-scale curvature feature descriptor is converted into a pixel value. The descriptor is fused with the visual feature to form a multi-feature descriptor, and the visual feature includes geometric features, grayscale statistical features, texture features and three-dimensional morphology features; the multi-feature descriptor is input into a defect classification and recognition module based on ensemble learning, and the defect classification and recognition module adopts a stacking integration strategy to combine multiple heterogeneous classifiers into a strong classifier, and the heterogeneous classifiers include support vector machines, K nearest neighbors, and random forests. The weights of heterogeneous classifiers are dynamically adjusted through an adaptive boosting algorithm, and the multi-feature descriptors are classified by the strong classifier. At the same time, a confidence threshold is introduced to adaptively correct the classification results to obtain the type and severity of welding defects;

[0009] In the training process of the defect classification and recognition module, a data enhancement method based on physical simulation is used. By performing physical lighting simulation on the three-dimensional CAD model of the circuit board welding area, virtual solder joint images under different defect types and imaging conditions are generated to expand the training data set and improve the generalization ability of the defect classification and recognition module. The virtual solder joint images contain corresponding three-dimensional morphological information, which complements the real collected data to jointly construct a diverse training data set. According to the results of defect recognition, a high-precision motion control system is used, including a linear motor, a grating ruler and a motion control card, to achieve precise positioning and continuous scanning of the circuit board within the camera's field of view, accurately locate the detected defective area in combination with the three-dimensional morphological information, and visualize the defective area in the form of color markings or heat maps to intuitively display the three-dimensional spatial distribution and severity of welding defects.

[0010] In an optional embodiment,

[0011] The steps of using a structured light projection system and a multi-camera imaging system to obtain images of the soldering area of ​​the circuit board, obtaining the three-dimensional morphology information of the soldering area through the parallax principle, and optimizing the imaging quality of the soldering area by adjusting the light source parameters in real time in combination with an adaptive light source system based on a dual-light path coaxial light source and a multi-angle annular light source include:

[0012] A multi-camera imaging system with a coaxial light source and a structured light projection system are used to collect multi-view images of the welding area through the multi-camera imaging system, and an image of the welding area surface projected with a structured light stripe pattern is obtained through the structured light projection system; the multi-camera imaging system includes a plurality of industrial cameras with different imaging viewing angles, and the optical axis of the structured light projection system forms a triangulation relationship with the optical axis of the multi-camera imaging system;

[0013] Build an adaptive light source system, which includes a group of coaxial light sources and a group of annular light sources; the coaxial light source adopts a focus-adjustable lens, and the size and projection direction of the output light spot can be dynamically adjusted by adjusting the focal length and incident angle of the focus-adjustable lens; the annular light source includes two groups of independently controlled LED annular arrays in inner and outer zones, and the light intensity at different positions in the welding area is adjusted by controlling the switch and brightness of LEDs in different zones;

[0014] Calibrate the multi-camera imaging system to obtain the internal and external parameters of each industrial camera; use the internal and external parameters to perform distortion correction on the acquired multi-view images, obtain the correspondence between images of different viewpoints through feature matching, calculate and optimize the homography transformation matrix of each viewpoint, and register the multi-view images;

[0015] Perform three-dimensional reconstruction on the registered image to obtain three-dimensional point cloud data of the solder joint surface, calculate the phase information of the projector pixel points according to the structured light fringe pattern, convert the phase information into three-dimensional coordinates in combination with the internal and external parameters, and calculate and obtain the high-density point cloud of the solder joint surface;

[0016] A three-dimensional model of the surface morphology of the welding area is established based on the three-dimensional point cloud data; the establishment of the three-dimensional model includes: splicing the three-dimensional point clouds obtained from different perspectives through coordinate transformation and rigid body registration algorithm to obtain complete three-dimensional morphology information of the welding area;

[0017] In the process of collecting images of the welding area, the lighting parameters of the coaxial light source and the annular light source are controlled through real-time calculation and feedback of image quality evaluation indicators, so as to adaptively optimize the imaging quality of the welding point; the image quality evaluation indicators include image contrast, brightness uniformity, and signal-to-noise ratio.

[0018] In an optional embodiment,

[0019] The steps of performing distortion correction and image registration on the original images acquired by the structured light projection system and the multi-camera imaging system, and performing adaptive noise reduction and background equalization processing on the registered images using wavelet transform and morphological filtering algorithms to obtain the preprocessed welding area image include:

[0020] The original welding image obtained by the structured light projection system and the multi-camera imaging system is subjected to distortion correction. The intrinsic parameter matrix and distortion coefficient obtained by camera calibration are used to establish a mapping relationship between pixel coordinates and normalized image plane coordinates to eliminate radial distortion and tangential distortion introduced by lens distortion, and obtain a distortion-corrected image.

[0021] Perform feature matching and homography transformation estimation on the distortion-corrected image, extract the scale-invariant feature points of the image and calculate their descriptors, use nearest neighbor matching and Lowe's ratio test to screen reliable matching point pairs, use random sampling consistency algorithm to eliminate false matches, and combine with Levenberg-Marquardt algorithm to optimize the homography matrix for fine registration, perform projection transformation and resample the image, and achieve spatial alignment of images from different perspectives;

[0022] The registered image is decomposed by wavelet, and the Daubechies wavelet basis is used to decompose the image in multiple scales to obtain low-frequency approximate coefficients and high-frequency detail coefficients in the horizontal, vertical and diagonal directions; the Bayesian shrinkage criterion is used to adaptively estimate the optimal threshold of each sub-band coefficient, and the soft threshold processing method is used to adaptively shrink the high-frequency detail coefficient to suppress image noise;

[0023] The low-frequency approximate coefficients obtained by wavelet decomposition are transformed by morphology top-hat transformation, and the structural elements are used to extract the areas in the image that are brighter than the neighborhood, highlighting the uneven background illumination components; the top-hat transformation results are then subjected to morphological closing operation to fill small dark areas and smooth the background; the closing operation result is subtracted from the low-frequency approximate coefficients to obtain the low-frequency component after illumination correction;

[0024] The high-frequency detail coefficients processed by adaptive threshold and the low-frequency approximate coefficients after background correction are used for wavelet reconstruction to obtain the welding area image after noise reduction and background equalization.

[0025] In an optional embodiment,

[0026] The method combining the graph-based watershed algorithm and the adaptive threshold segmentation algorithm is used to automatically segment the preprocessed welding area image. The steps of obtaining the region of interest include:

[0027] The preprocessed welding area image is mapped into an undirected weighted graph, the nodes of which correspond to image pixels, the edges connect adjacent pixels, and the edge weight is the negative exponential function of the grayscale difference of adjacent pixels. The undirected weighted graph is recursively divided using the minimum spanning tree algorithm, and multiple connected branches are obtained by removing the edges with weights less than the increasing threshold. Each connected branch corresponds to a watershed area, and the watershed areas are merged. The average grayscale difference and boundary shape difference of adjacent areas are calculated, and the areas where both are less than the given threshold are merged into new watershed areas.

[0028] The grayscale histogram of the preprocessed welding area image is calculated and smoothed. The global threshold is adaptively determined by maximizing the inter-class variance using the maximum inter-class variance method, and the welding area image is binarized using the global threshold. Pixels with grayscale values ​​greater than the threshold are segmented into foreground areas, and pixels with grayscale values ​​less than or equal to the threshold are segmented into background areas.

[0029] Perform logical AND operations on the watershed area and the foreground area obtained by threshold segmentation to extract the common foreground area, and perform morphological opening and closing operations on the foreground area to fill small holes, remove small spots, and smooth the area boundaries;

[0030] The connected areas are extracted from the foreground area after morphological processing, and the regions of interest corresponding to the welding points are screened out according to the area, perimeter and circularity of the region to achieve automatic segmentation of the welding area image.

[0031] In an optional embodiment,

[0032] The steps of using 3D morphology information and achieving accurate positioning of defect areas within the region of interest through morphological operations and edge detection algorithms include:

[0033] The disparity map corresponding to the region of interest is obtained by stereo matching of multi-view images, and the disparity map is converted into a three-dimensional point cloud according to the camera calibration parameters. The three-dimensional point cloud is denoised and smoothed, and the point cloud surface is reconstructed using the moving least squares method to obtain a smooth three-dimensional morphology of the region of interest.

[0034] The three-dimensional morphology of the region of interest is projected back to the two-dimensional plane to obtain a depth image, and the depth image is subjected to morphological top-hat transformation to extract the region that is more prominent than the neighboring region, and the top-hat transformation result is binarized and morphologically closed to obtain a preliminary defect candidate region;

[0035] Perform geometric feature analysis on the defect candidate area, set feature thresholds based on prior knowledge of area, aspect ratio, and circularity, and filter out areas that do not meet the defect characteristics to obtain refined defect areas;

[0036] Calculate the gradient magnitude map and gradient direction map of the image of the region of interest, use the edge detection operator to perform edge detection to obtain a binary edge map, and perform morphological refinement on the binary edge map to obtain an edge skeleton line with a single pixel width;

[0037] The refined defect area and the edge skeleton line are logically ANDed to extract the precise boundary of the defect area, and the precise boundary of the defect area is subjected to geometric feature analysis and smoothing to remove small protrusions and depressions to achieve precise positioning of the defect area.

[0038] In an optional embodiment,

[0039] For the defect area after precise positioning, the defect area is three-dimensionally reconstructed using the triangulation principle according to the multi-view images and structured light information obtained by the multi-camera imaging system to obtain high-precision three-dimensional point cloud data of the defect area; for the reconstructed high-precision three-dimensional point cloud data of the defect area, a multi-scale curvature feature descriptor is extracted, and the multi-scale curvature feature descriptor is fused with the visual feature to form a multi-feature descriptor, the steps include:

[0040] The structured light sensor is used to obtain the three-dimensional data of the welding spot area. The structured light projector and multiple cameras are arranged around the welding workpiece. The corresponding relationship between the fringe code and the spatial coordinates is established by projecting the coded fringe pattern and capturing its deformed image.

[0041] Perform 2D feature matching on multi-view images, use scale-invariant feature transformation algorithm to extract key points of multi-view images and calculate their descriptors, and find the corresponding pairs of points with the same name between different viewpoints through the nearest neighbor distance ratio matching strategy;

[0042] Combined with the camera calibration parameters, the three-dimensional coordinates of the same-name points in the defect area are calculated using the triangulation principle, and the welding area is reconstructed in three dimensions. The outliers caused by mismatching are eliminated through statistical filtering methods to obtain high-precision three-dimensional point cloud data of the defect area.

[0043] For the high-precision three-dimensional point cloud of the reconstructed defect area, the principal component analysis method is used to estimate the normal vector and curvature information of each point, and the curvature mean and curvature variance in the point neighborhood are extracted and spliced ​​to form a multi-scale curvature feature descriptor;

[0044] The multi-scale curvature feature descriptor is fused with the visual feature descriptor, and the multi-feature descriptor of the defect area is obtained by feature splicing.

[0045] In an optional embodiment,

[0046] The multi-feature descriptor is input into a defect classification and recognition module based on ensemble learning. The defect classification and recognition module adopts a stacking integration strategy to combine multiple heterogeneous classifiers into a strong classifier. The heterogeneous classifiers include support vector machines, K nearest neighbors, and random forests. The weights of the heterogeneous classifiers are dynamically adjusted through an adaptive boosting algorithm. The multi-feature descriptors are classified using a strong classifier, and a confidence threshold is introduced to adaptively correct the classification results. The steps of obtaining the type and severity of welding defects include:

[0047] Construct a set of heterogeneous classifiers including support vector machine, K nearest neighbor and random forest, use multi-feature descriptors to train each heterogeneous classifier separately, and optimize the kernel function parameters and penalty coefficient of support vector machine, the optimal number of neighbors of K nearest neighbor classifier, and the number of decision trees and maximum tree depth parameters of random forest classifier through grid search and cross validation;

[0048] The trained heterogeneous classifiers are combined into a strong classifier using an adaptive boosting algorithm. The sample weights are adjusted iteratively so that the samples misclassified by the previous round of classifiers gain attention in the next round. The weight coefficient of each heterogeneous classifier is calculated based on the classification error rate to construct a weighted decision function for the strong classifier.

[0049] The soft classification confidence vector is calculated for the defect type given by the strong classifier, and the maximum confidence is compared with the set confidence threshold. When the maximum confidence is less than the threshold, the classification result of the sample is considered unreliable. The defect type of the sample is re-predicted by the K-nearest neighbor algorithm based on the weighted Euclidean distance of the multi-feature descriptor of the sample and the known defect sample, and the classification result of the welding defect after adaptive correction is updated; the geometric features of the area and length of the defect area are mapped into severity levels to obtain the quantitative evaluation results of the welding defects.

[0050] According to a second aspect of the embodiments of the present invention,

[0051] Provide the subject name of the circuit board welding fault identification system based on machine vision, including:

[0052] The first unit is used to obtain images of the soldering area of ​​the circuit board using a structured light projection system and a multi-camera imaging system, obtain the three-dimensional morphology information of the soldering area through the parallax principle, and optimize the imaging quality of the soldering area by adjusting the light source parameters in real time in combination with an adaptive light source system based on a dual-optical coaxial light source and a multi-angle annular light source; perform distortion correction and image registration on the original images obtained by the structured light projection system and the multi-camera imaging system, and use wavelet transform and morphological filtering algorithms to perform adaptive noise reduction and background equalization on the registered images to obtain the pre-processed soldering area images;

[0053] The second unit is used to automatically segment the pre-processed welding area image by combining the watershed algorithm based on graph theory and the adaptive threshold segmentation algorithm to obtain the region of interest, and accurately locate the defect area in the region of interest by using the three-dimensional morphology information and morphological operations and edge detection algorithms; for the defect area after accurate positioning, the triangulation principle is used to reconstruct the defect area in three dimensions based on the multi-view images and structured light information obtained by the multi-camera imaging system to obtain high-precision three-dimensional point cloud data of the defect area; for the high-precision three-dimensional point cloud data of the reconstructed defect area, a multi-scale curvature feature descriptor is extracted, and the multi-scale curvature feature descriptor is converted into the multi-scale curvature feature descriptor. The feature descriptor is fused with the visual feature to form a multi-feature descriptor, wherein the visual feature includes geometric features, grayscale statistical features, texture features and three-dimensional morphology features; the multi-feature descriptor is input into a defect classification and recognition module based on ensemble learning, wherein the defect classification and recognition module adopts a stacking integration strategy to combine multiple heterogeneous classifiers into a strong classifier, wherein the heterogeneous classifiers include support vector machines, K nearest neighbors and random forests, and the weights of the heterogeneous classifiers are dynamically adjusted through an adaptive boosting algorithm, and the multi-feature descriptors are classified by the strong classifier, and a confidence threshold is introduced to adaptively correct the classification results, thereby obtaining the type and severity of the welding defects;

[0054] The third unit is used to use a data enhancement method based on physical simulation during the training process of the defect classification and recognition module. By performing physical lighting simulation on the three-dimensional CAD model of the circuit board welding area, virtual solder joint images under different defect types and imaging conditions are generated to expand the training data set and improve the generalization ability of the defect classification and recognition module. The virtual solder joint images contain corresponding three-dimensional morphological information, which complements the real collected data to jointly construct a diverse training data set. According to the results of defect recognition, a high-precision motion control system is used, including a linear motor, a grating ruler and a motion control card, to achieve precise positioning and continuous scanning of the circuit board within the camera's field of view, accurately locate the detected defective area in combination with the three-dimensional morphological information, and visualize the defective area in the form of color markings or heat maps to intuitively display the three-dimensional spatial distribution and severity of welding defects.

[0055] According to a third aspect of the embodiments of the present invention,

[0056] An electronic device is provided, comprising:

[0057] processor;

[0058] a memory for storing processor-executable instructions;

[0059] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0060] According to a fourth aspect of the embodiments of the present invention,

[0061] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0062] This paper proposes a method for image acquisition and three-dimensional reconstruction of the soldering area of ​​a circuit board. A multi-camera imaging system with coaxial light sources and a structured light projection system are combined to obtain high-quality multi-view images of the solder joint area through chessboard calibration, image distortion correction, and registration. The high-density three-dimensional point cloud of the solder joint surface is reconstructed by using the principle of structured light fringe projection decoding and triangulation, and the three-dimensional morphology model of the solder joint is generated by point cloud splicing and gridding. An adaptive light source system consisting of a dual-optical coaxial light source and a multi-angle annular light source is introduced. The solder joint imaging quality is dynamically optimized through image quality evaluation feedback and closed-loop control of light source parameters. The solder joint images acquired and reconstructed by this method have obvious advantages in resolution, viewing angle coverage, and illumination consistency. The obtained three-dimensional model of the solder joint accurately depicts the spatial characteristics of the solder joint morphology, laying a good data foundation for subsequent defect recognition. In addition, the adaptive light source optimization strategy effectively improves the stability of image quality and reduces the quality difference between viewing angles, creating favorable conditions for achieving robust and high-precision defect recognition.

[0063] This paper proposes a solder joint ROI extraction method that combines graph theory watershed and adaptive threshold segmentation. First, the solder joint image is mapped to an undirected weighted graph, and the watershed region is obtained by recursive partitioning of the minimum spanning tree and region merging, which effectively suppresses over-segmentation; then the maximum inter-class variance method is used to adaptively determine the global threshold and perform foreground and background segmentation on the image; finally, the final solder joint region of interest is obtained by logical AND operation of the watershed region and the threshold segmentation result, and after morphological processing and screening of the area, perimeter and circularity of the connected region. This method overcomes the shortcomings of traditional watershed that is easy to over-segment and threshold segmentation is not robust enough, and can automatically adapt to different solder joint images, with stable and reliable segmentation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 A schematic diagram of a flow chart of a method for identifying circuit board welding faults based on machine vision according to an embodiment of the present invention;

[0065] Figure 2 The figure is a schematic diagram of the structure of a circuit board welding fault identification system based on machine vision according to an embodiment of the present invention. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0068] Figure 1 FIG. 1 is a flow chart of a method for identifying circuit board welding faults based on machine vision according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0069] S1. Use structured light projection system and multi-camera imaging system to obtain images of the soldering area of ​​the circuit board, obtain the three-dimensional morphology information of the soldering area through the parallax principle, and combine the adaptive light source system based on the combination of dual-optical coaxial light source and multi-angle annular light source to optimize the imaging quality of the soldering area by adjusting the light source parameters in real time; perform distortion correction and image registration on the original images obtained by the structured light projection system and multi-camera imaging system, and use wavelet transform and morphological filtering algorithm to perform adaptive noise reduction and background equalization on the registered images to obtain the pre-processed soldering area image;

[0070] S2. A method combining a watershed algorithm based on graph theory and an adaptive threshold segmentation algorithm is used to automatically segment the preprocessed welding area image to obtain the region of interest, and the three-dimensional morphology information is used to accurately locate the defect area in the region of interest through morphological operations and edge detection algorithms; for the defect area after accurate positioning, the triangulation principle is used to reconstruct the defect area in three dimensions based on the multi-view images and structured light information obtained by the multi-camera imaging system to obtain high-precision three-dimensional point cloud data of the defect area; for the high-precision three-dimensional point cloud data of the reconstructed defect area, a multi-scale curvature feature descriptor is extracted, and the multi-scale curvature feature is converted into The descriptor is fused with the visual feature to form a multi-feature descriptor, wherein the visual feature includes geometric features, grayscale statistical features, texture features and three-dimensional morphology features; the multi-feature descriptor is input into a defect classification and recognition module based on ensemble learning, wherein the defect classification and recognition module adopts a stacking integration strategy to combine multiple heterogeneous classifiers into a strong classifier, wherein the heterogeneous classifiers include support vector machines, K nearest neighbors, and random forests, and the weights of the heterogeneous classifiers are dynamically adjusted through an adaptive boosting algorithm, and the multi-feature descriptors are classified by the strong classifier, and a confidence threshold is introduced to adaptively correct the classification results to obtain the type and severity of the welding defects;

[0071] S3. During the training process of the defect classification and recognition module, a data enhancement method based on physical simulation is used. By performing physical lighting simulation on the three-dimensional CAD model of the circuit board welding area, virtual solder joint images under different defect types and imaging conditions are generated to expand the training data set and improve the generalization ability of the defect classification and recognition module. The virtual solder joint images contain corresponding three-dimensional morphological information, which complements the real collected data to jointly construct a diverse training data set. According to the results of defect recognition, a high-precision motion control system is used, including a linear motor, a grating ruler and a motion control card, to achieve precise positioning and continuous scanning of the circuit board within the camera's field of view, and to accurately locate the detected defective area in combination with the three-dimensional morphological information, and to visualize the defective area in the form of color markings or heat maps to intuitively display the three-dimensional spatial distribution and severity of welding defects.

[0072] In an optional embodiment,

[0073] The steps of using a structured light projection system and a multi-camera imaging system to obtain images of the soldering area of ​​the circuit board, obtaining the three-dimensional morphology information of the soldering area through the parallax principle, and optimizing the imaging quality of the soldering area by adjusting the light source parameters in real time in combination with an adaptive light source system based on a dual-light path coaxial light source and a multi-angle annular light source include:

[0074] A multi-camera imaging system with a coaxial light source and a structured light projection system are used to collect multi-view images of the welding area through the multi-camera imaging system, and an image of the welding area surface projected with a structured light stripe pattern is obtained through the structured light projection system; the multi-camera imaging system includes a plurality of industrial cameras with different imaging viewing angles, and the optical axis of the structured light projection system forms a triangulation relationship with the optical axis of the multi-camera imaging system;

[0075] Build an adaptive light source system, which includes a group of coaxial light sources and a group of annular light sources; the coaxial light source adopts a focus-adjustable lens, and the size and projection direction of the output light spot can be dynamically adjusted by adjusting the focal length and incident angle of the focus-adjustable lens; the annular light source includes two groups of independently controlled LED annular arrays in inner and outer zones, and the light intensity at different positions in the welding area is adjusted by controlling the switch and brightness of LEDs in different zones;

[0076] Calibrate the multi-camera imaging system to obtain the internal and external parameters of each industrial camera; use the internal and external parameters to perform distortion correction on the acquired multi-view images, obtain the correspondence between images of different viewpoints through feature matching, calculate and optimize the homography transformation matrix of each viewpoint, and register the multi-view images;

[0077] Perform three-dimensional reconstruction on the registered image to obtain three-dimensional point cloud data of the solder joint surface, calculate the phase information of the projector pixel points according to the structured light fringe pattern, convert the phase information into three-dimensional coordinates in combination with the internal and external parameters, and calculate and obtain the high-density point cloud of the solder joint surface;

[0078] A three-dimensional model of the surface morphology of the welding area is established based on the three-dimensional point cloud data; the establishment of the three-dimensional model includes: splicing the three-dimensional point clouds obtained from different perspectives through coordinate transformation and rigid body registration algorithm to obtain complete three-dimensional morphology information of the welding area;

[0079] In the process of collecting images of the welding area, the lighting parameters of the coaxial light source and the annular light source are controlled through real-time calculation and feedback of image quality evaluation indicators, so as to adaptively optimize the imaging quality of the welding point; the image quality evaluation indicators include image contrast, brightness uniformity, and signal-to-noise ratio.

[0080] For example, this paper proposes a method for image acquisition and 3D reconstruction of the soldering area of ​​a circuit board. By combining a structured light projection system with a multi-camera imaging system, high-quality multi-view images and 3D morphology information of the soldering area are acquired. At the same time, an adaptive light source system based on a combination of a dual-light path coaxial light source and a multi-angle annular light source is introduced to optimize the imaging quality of the soldering area by adjusting the light source parameters in real time. The specific technical solution is as follows:

[0081] This paper adopts a multi-camera imaging system with a coaxial light source and a structured light projection system. The multi-camera imaging system includes four industrial cameras with different imaging angles. The optical axis of the structured light projection system and the optical axis of the multi-camera imaging system form a 30° triangulation relationship to ensure good imaging angle coverage of the welding area.

[0082] The adaptive light source system includes a coaxial light source and a ring light source. The coaxial light source uses two 50W halogen lamps and is equipped with a focus-adjustable lens. By adjusting the focal length (10mm-50mm) and the incident angle (0°-60°) of the lens, the diameter of the exit spot can be dynamically adjusted within the range of 10mm-100mm to meet the imaging requirements of solder joints of different sizes. The ring light source includes two groups of independently controlled LED ring arrays in the inner and outer zones. Each array consists of 120 high-brightness white light LEDs with ring diameters of 50mm and 80mm respectively. The switch and brightness (0-100%) of different partitioned LEDs are controlled by pulse width modulation (PWM) to achieve dynamic adjustment of the light intensity at different positions in the welding area and improve the uniformity of the light of the solder joint image.

[0083] The multi-camera imaging system is calibrated using a checkerboard calibration plate. The Zhang Zhengyou calibration method [1] is used to estimate the intrinsic parameter matrix (including focal length, principal point coordinates) and distortion coefficient (including radial distortion and tangential distortion) of each industrial camera. The intrinsic parameters obtained by calibration are used to correct the distortion of the acquired original image to eliminate the geometric distortion introduced by lens distortion. Then, the scale-invariant feature transform (SIFT) and random sample consensus algorithm (RANSAC algorithm) are used to achieve feature matching between images of different viewpoints, calculate the homography transformation matrix between each viewpoint, and optimize it using the Levenberg-Marquardt algorithm to control the matching error within 0.5 pixels. Finally, the optimized homography matrix is ​​used to align the multi-view images to achieve geometric alignment of the solder joint areas of different viewpoints.

[0084] Taking the QFN solder joints of a certain type of PCB circuit board as an example, the original image resolution captured by four industrial cameras is 1920×1200. After distortion correction and registration, the effective imaging area is 1600×1000. After registration, the coincidence accuracy of solder joints at different viewing angles is better than 1.5 pixels.

[0085] While acquiring multi-view welding images, a structured light projection system is used to project a sinusoidal fringe pattern onto the surface of the weld, and the fringe deformation image is captured synchronously. The fringe projection scheme is designed to be 3-frequency 3-phase shift, that is, 3 frequencies (1 / 128, 1 / 64, 1 / 32) of sinusoidal fringe patterns are projected, and each frequency is phase shifted 3 times (0°, 120°, 240°), and a total of 9 fringe images are obtained. Combined with the multi-frequency heterodyne unwrapping algorithm, the absolute phase of each projector pixel is solved.

[0086] According to the internal and external parameters of the projector, the absolute phase of the projector pixel is converted into spatial three-dimensional coordinates. Since the stripe pattern has a high projection density (projection points per inch> 300), high-density three-dimensional point cloud data of the solder joint surface can be obtained. Taking the above QFN package solder joint as an example, the solder joint size is about 2mm×3mm, and the number of reconstructed three-dimensional point clouds is about 5000, and the point cloud density is better than 10 points / mm².

[0087] The three-dimensional point clouds obtained from multiple perspectives are subjected to coordinate transformation and rigid body registration to achieve geometric splicing of point clouds from different perspectives in the same coordinate system. The coordinate transformation is based on the rotation matrix and translation vector solved by the external parameters of each camera, and the point cloud in the perspective coordinate system is unified to the world coordinate system. The rigid body registration adopts the iterative closest point (ICP algorithm), and through iterative closest point search and coordinate transformation, the registration error of point clouds from different perspectives is reduced to less than 10μm. The number of spliced ​​point clouds can reach more than 20,000, which fully describes the three-dimensional morphological characteristics of the solder joint area.

[0088] Based on the point cloud stitching, the point cloud library PCL is used to reconstruct the mesh of the 3D model. First, the moving least squares method is used to smooth and reduce noise on the point cloud, and the outliers are filtered out by the normal vector consistency criterion; then, the greedy projection triangulation algorithm is used to generate triangular facets from the point cloud to construct the mesh model of the solder joint surface; finally, Laplace smoothing is used to refine and smooth the mesh model to generate the final 3D morphology model of the solder joint.

[0089] Taking the QFN solder joint as an example, the reconstructed three-dimensional model of the solder joint contains about 8,000 triangular facets, which successfully characterizes the solder joint's three-dimensional contour, surface undulation and other geometric features, providing a morphological basis for subsequent defect identification.

[0090] In order to obtain high-quality solder joint images, an adaptive light source optimization strategy is introduced in the image acquisition process. Based on the real-time calculation of image quality evaluation indicators, the illumination parameters of the coaxial light source and the ring light source are controlled through closed-loop feedback to dynamically optimize the solder joint imaging quality. The image quality evaluation indicators used include:

[0091] Image contrast. Normalized variance is used as a measure of image contrast, and its calculation formula is:

[0092] Contrast = Var(I) / (Mean(I)) 2 ;

[0093] Where I is the image grayscale matrix, Var and Mean represent the variance and mean respectively. The greater the contrast, the richer the image grayscale levels.

[0094] Brightness uniformity. The standard deviation of the grayscale mean of the image blocks is used to measure the brightness uniformity. The calculation formula is:

[0095] Uniform = Std(BlockMean(I));

[0096] In the formula, BlockMean represents the grayscale mean calculated for the image blocks, and Std represents the standard deviation. The block size is 20×20. The smaller the uniformity, the more balanced the image brightness distribution.

[0097] Signal-to-noise ratio. The signal-to-noise ratio is measured by the ratio of the image average grayscale to the noise standard deviation. The calculation formula is:

[0098] SNR = Mean(I) / Std(Noise(I));

[0099] In the formula, Noise means estimating and extracting the noise of the image. The larger the signal-to-noise ratio, the better the image quality.

[0100] In the process of light source optimization, the three quality evaluation indicators of the collected image are calculated in real time and compared with the preset target values ​​to obtain the quality evaluation deviation. The deviation value is converted into the light source adjustment amount through the PID control algorithm to control the spot size of the coaxial light source and the partition brightness of the annular light source, thus realizing the closed-loop optimization of the imaging quality.

[0101] After 20 iterative optimizations, the contrast of the QFN solder joint image increased from 1.2 to above 2.5, the brightness uniformity decreased from 15 to less than 6, the signal-to-noise ratio increased from 25 to above 40, and the quality difference between viewing angles was reduced to less than 10%, greatly improving the consistency and stability of image quality.

[0102] This paper proposes a method for image acquisition and three-dimensional reconstruction of the soldering area of ​​a circuit board. A multi-camera imaging system with coaxial light sources and a structured light projection system are combined to obtain high-quality multi-view images of the solder joint area through chessboard calibration, image distortion correction, and registration. The high-density three-dimensional point cloud of the solder joint surface is reconstructed by using the principle of structured light fringe projection decoding and triangulation, and the three-dimensional morphology model of the solder joint is generated by point cloud splicing and gridding. An adaptive light source system consisting of a dual-optical coaxial light source and a multi-angle annular light source is introduced. The solder joint imaging quality is dynamically optimized through image quality evaluation feedback and closed-loop control of light source parameters. The solder joint images acquired and reconstructed by this method have obvious advantages in resolution, viewing angle coverage, and illumination consistency. The obtained three-dimensional model of the solder joint accurately depicts the spatial characteristics of the solder joint morphology, laying a good data foundation for subsequent defect recognition. In addition, the adaptive light source optimization strategy effectively improves the stability of image quality and reduces the quality difference between viewing angles, creating favorable conditions for achieving robust and high-precision defect recognition.

[0103] In an optional embodiment,

[0104] The steps of performing distortion correction and image registration on the original images acquired by the structured light projection system and the multi-camera imaging system, and performing adaptive noise reduction and background equalization processing on the registered images using wavelet transform and morphological filtering algorithms to obtain the preprocessed welding area image include:

[0105] The original welding image obtained by the structured light projection system and the multi-camera imaging system is subjected to distortion correction. The intrinsic parameter matrix and distortion coefficient obtained by camera calibration are used to establish a mapping relationship between pixel coordinates and normalized image plane coordinates to eliminate radial distortion and tangential distortion introduced by lens distortion, and obtain a distortion-corrected image.

[0106] Perform feature matching and homography transformation estimation on the distortion-corrected image, extract the scale-invariant feature points of the image and calculate their descriptors, use nearest neighbor matching and Lowe's ratio test to screen reliable matching point pairs, use random sampling consistency algorithm to eliminate false matches, and combine with Levenberg-Marquardt algorithm to optimize the homography matrix for fine registration, perform projection transformation and resample the image, and achieve spatial alignment of images from different perspectives;

[0107] The registered image is decomposed by wavelet, and the Daubechies wavelet basis is used to decompose the image in multiple scales to obtain low-frequency approximate coefficients and high-frequency detail coefficients in the horizontal, vertical and diagonal directions; the Bayesian shrinkage criterion is used to adaptively estimate the optimal threshold of each sub-band coefficient, and the soft threshold processing method is used to adaptively shrink the high-frequency detail coefficient to suppress image noise;

[0108] The low-frequency approximate coefficients obtained by wavelet decomposition are transformed by morphology top-hat transformation, and the structural elements are used to extract the areas in the image that are brighter than the neighborhood, highlighting the uneven background illumination components; the top-hat transformation results are then subjected to morphological closing operation to fill small dark areas and smooth the background; the closing operation result is subtracted from the low-frequency approximate coefficients to obtain the low-frequency component after illumination correction;

[0109] The high-frequency detail coefficients processed by adaptive threshold and the low-frequency approximate coefficients after background correction are used for wavelet reconstruction to obtain the welding area image after noise reduction and background equalization.

[0110] For example, in order to improve the accuracy and reliability of subsequent weld defect identification, the acquired original welding image needs to be preprocessed, mainly including distortion correction, image registration, adaptive noise reduction and background equalization. The specific technical solution is as follows:

[0111] Due to the manufacturing process limitations of the lens in the imaging system, the original image generally has radial distortion and tangential distortion, which is mainly manifested as barrel distortion or pincushion distortion, and the straight line appears curved in the image. In order to eliminate the influence of lens distortion, the original image is distorted using the intrinsic parameter matrix and distortion coefficients obtained by camera calibration.

[0112] The correction principle is to establish the mapping relationship between the image pixel coordinates and the normalized image plane coordinates. Assuming the pixel coordinates are (u, v) and the normalized image plane coordinates are (x, y), the mapping process is divided into three steps:

[0113] The first step is to convert the pixel coordinates (u, v) into physical coordinates (x', y'), that is, to eliminate the effects of scale and offset. The x' component of the physical coordinates is equal to the pixel coordinate u minus the principal point horizontal coordinate u0 divided by the focal length f x , the y' component is equal to the pixel coordinate v minus the principal point ordinate v0 divided by the focal length f y .

[0114] The second step is to establish the mapping relationship between the physical coordinates (x', y') and the normalized coordinates (x, y) after distortion correction. The x-coordinate after distortion correction is equal to x' multiplied by the radial distortion polynomial, plus x'y' multiplied by twice the tangential distortion coefficient p1, plus x' squared multiplied by the tangential distortion coefficient p2. The y-coordinate after distortion correction is equal to y' multiplied by the radial distortion polynomial, plus x'y' multiplied by the tangential distortion coefficient p1, plus y' squared multiplied by twice the tangential distortion coefficient p2. The radial distortion polynomial is 1 plus the radial distortion coefficient k1 multiplied by r squared, plus k2 multiplied by r to the fourth power, plus k3 multiplied by r to the sixth power, where r is the square root of the sum of the squares of x' and y'.

[0115] The third step is to use the reverse mapping method to resample the space and obtain the distortion-corrected image. Taking the nearest neighbor interpolation as an example, for each pixel (u, v) of the corrected image, the corresponding physical coordinates (x', y') in the original image are obtained by inverting the above equations, and then the corresponding pixel coordinates (u', v') of the original image are obtained by inverting the relationship in the first step, and the grayscale of the (u', v') pixel is assigned to (u, v), completing the resampling and eliminating the distortion.

[0116] Since the cameras in a multi-camera imaging system have different viewing angles, the images of solder joints obtained are misaligned in space. To achieve pixel-level defect detection, images from different viewing angles need to be accurately registered. This paper adopts a feature-based image registration method to achieve sub-pixel registration accuracy through feature extraction, feature matching, mismatch elimination, and registration transformation optimization.

[0117] First, the SIFT (Scale Invariant Feature Transform) algorithm is used to extract local feature points of the image and calculate the 128-dimensional feature descriptor. Then, the kd tree structure is used to achieve the nearest neighbor matching of the feature points. For each feature point in the query image, the two feature points with the closest Euclidean distance in the reference image are found. If the ratio of the closest distance to the second closest distance is less than the empirical threshold of 0.6, the matching pair is accepted, otherwise it is considered to be a point that cannot be reliably matched.

[0118] To further eliminate false matches, the RANSAC algorithm is used to estimate the homography matrix between images. Four pairs of matching points are randomly sampled, the corresponding homography matrix is ​​calculated, and all matching points are transformed using the matrix. The geometric distance between the transformed matching points is calculated. If the distance is less than the threshold of 1.5 pixels, the pair of matching points is considered to be an inlier, otherwise it is an outlier. Iterate multiple random samplings to find the homography matrix with the largest number of inliers as the current optimal estimate.

[0119] Considering the limited accuracy of the homography matrix estimated by RANSAC, the result is used as the initial value, and the Levenberg-Marquardt (LM) iterative optimization algorithm is used to further improve the registration accuracy. Based on the principle of minimizing the reprojection error, the homography matrix is ​​optimized by least squares estimation. The goal is to minimize the sum of the squares of the Euclidean distances between the feature points of the reference image after the homography matrix transformation and the corresponding feature points of the query image. When the objective function changes less than 10-6, the iteration is terminated.

[0120] Finally, the optimized homography matrix is ​​used to perform projection transformation and bilinear interpolation resampling on the query image to complete the image registration process. Taking two QFN solder joint images with a size of 1800×1100 as an example, about 150 pairs of SIFT feature matches are extracted during the registration process, and the proportion of RANSAC inliers is more than 85%. The residual of the homography matrix after LM optimization is less than 0.5 pixels, achieving accurate registration.

[0121] Due to the complex imaging environment, high-frequency noise pollution is common in solder joint images. Wavelet transform can be used to achieve adaptive denoising of images. Firstly, the image is decomposed into 5-level wavelets using Daubechies9 / 7 wavelet basis to obtain a low-frequency approximate coefficient map and 5×3 high-frequency detail coefficient maps, which correspond to the edge texture features in the horizontal, vertical and diagonal directions respectively.

[0122] The Bayesian threshold estimation is used to adaptively calculate the optimal threshold for high-frequency detail coefficients. Assuming that the wavelet coefficients follow a generalized Gaussian distribution and the noise follows a Gaussian distribution, the Bayesian threshold is equal to the noise variance divided by the standard deviation of the wavelet coefficients. The noise variance is estimated by dividing the median of the absolute value of the wavelet coefficients at each scale by the square of 0.6745.

[0123] The high-frequency detail coefficients and Bayesian thresholds are soft-thresholded, that is, the coefficients with absolute values ​​less than the threshold are set to zero, and those greater than the threshold are subtracted from the threshold, which can effectively suppress the amplitude of the noise coefficient. After the image is reconstructed by wavelet, the high-frequency noise is significantly removed, and the texture edge is preserved.

[0124] Taking the QFN solder joint image after the above registration as an example, the Bayesian thresholds at five scales are 12.5, 17.8, 24.6, 31.2, and 38.5, respectively, and the noise variance estimation value is 5.1. After soft threshold processing, the signal-to-noise ratio is increased from 21.5 to 32.4, and the mean square error is reduced from 85 to 47, and the image noise is effectively suppressed.

[0125] Affected by the light source of the imaging system, the background brightness distribution of the solder joint image is often uneven, showing a phenomenon of bright in the middle and dark around, which requires background correction. Traditional histogram equalization is prone to introduce noise amplification and detail loss. This paper uses morphological top-hat transformation and closing operation to perform background equalization on the wavelet low-frequency approximation coefficients.

[0126] First, a 31×31 disk structure element is used to perform a morphological top-hat operation on the low-frequency approximate coefficient map, that is, the image is opened with the structure element, and then the result of the opening operation is subtracted from the original image to obtain a bright area larger than the structure element. The top-hat transform is equivalent to a high-pass filter, which highlights the local background brightness changes.

[0127] Then, the top-hat transformation result is subjected to morphological closing operation using the same structural element, that is, dilation followed by erosion, which can fill the dark areas in the image that are smaller than the structural element and smooth the background brightness distribution. Finally, the low-frequency approximate coefficient map is used to subtract the closing operation result to correct the uneven background illumination and obtain a low-frequency component with uniform brightness.

[0128] Taking the QFN solder joint image as an example, after top hat transformation and closing operation, the background brightness uniformity is improved by 32%, and the dynamic range is compressed to less than 0.25. The corrected low-frequency approximation coefficients and the high-frequency detail coefficients processed by noise reduction in Section 3.3.3 are reconstructed by wavelet to obtain the preprocessed solder joint image.

[0129] In this paper, a comprehensive image preprocessing method is proposed to solve the problems of distortion, perspective misalignment, noise pollution and background unevenness in the original image of the solder joint. The image distortion correction is achieved by using coordinate mapping and resampling based on intrinsic parameters and distortion coefficients; sub-pixel level precise registration is achieved by SIFT feature extraction, RANSAC outlier removal and LM optimization; the image is decomposed into multiple scales by wavelet transform, and the high-frequency detail coefficient is soft-threshold denoising is performed by adaptively estimating the Bayesian threshold; the background illumination equalization is achieved by using morphological top-hat transform and closing operation for the wavelet low-frequency approximation coefficient. After the above preprocessing, the geometric distortion of the solder joint image is corrected, the images of different perspectives are accurately aligned, and the noise interference and background interference are effectively suppressed, which lays a good image foundation for the subsequent accurate detection of solder joint defects.

[0130] In an optional embodiment,

[0131] The method combining the graph-based watershed algorithm and the adaptive threshold segmentation algorithm is used to automatically segment the preprocessed welding area image. The steps of obtaining the region of interest include:

[0132] The preprocessed welding area image is mapped into an undirected weighted graph, the nodes of which correspond to image pixels, the edges connect adjacent pixels, and the edge weight is the negative exponential function of the grayscale difference of adjacent pixels. The undirected weighted graph is recursively divided using the minimum spanning tree algorithm, and multiple connected branches are obtained by removing the edges with weights less than the increasing threshold. Each connected branch corresponds to a watershed area, and the watershed areas are merged. The average grayscale difference and boundary shape difference of adjacent areas are calculated, and the areas where both are less than the given threshold are merged into new watershed areas.

[0133] The grayscale histogram of the preprocessed welding area image is calculated and smoothed. The global threshold is adaptively determined by maximizing the inter-class variance using the maximum inter-class variance method, and the welding area image is binarized using the global threshold. Pixels with grayscale values ​​greater than the threshold are segmented into foreground areas, and pixels with grayscale values ​​less than or equal to the threshold are segmented into background areas.

[0134] Perform logical AND operations on the watershed area and the foreground area obtained by threshold segmentation to extract the common foreground area, and perform morphological opening and closing operations on the foreground area to fill small holes, remove small spots, and smooth the area boundaries;

[0135] The connected areas are extracted from the foreground area after morphological processing, and the regions of interest corresponding to the welding points are screened out according to the area, perimeter and circularity of the region to achieve automatic segmentation of the welding area image.

[0136] For example, in order to further narrow the scope of solder joint defect detection, it is necessary to extract the region of interest (ROI) corresponding to the solder joint from the preprocessed soldering area image. Traditional methods such as threshold segmentation and region growing are difficult to adapt to the situation where the solder joint area has a complex background and large grayscale differences, and are prone to over-segmentation or under-segmentation. This paper proposes a method that combines the watershed algorithm of graph theory and adaptive threshold segmentation, which can effectively overcome the above problems and realize automatic and accurate segmentation of solder joint ROI.

[0137] The watershed algorithm is a classic image segmentation method based on topological theory. Its basic idea is to regard the image as a terrain surface, the grayscale value corresponds to the terrain height, the watershed line corresponds to the ridge, and different watershed areas correspond to different valley basins. The traditional watershed algorithm based on gradient or distance transformation is easily disturbed by noise and over-segmentation occurs. This paper adopts a watershed variant based on graph theory, obtains the watershed area through recursive partitioning of the minimum spanning tree, and introduces a region merging strategy to effectively suppress the over-segmentation phenomenon.

[0138] First, the preprocessed M×N welding area image is mapped into an undirected weighted graph G=(V,E). The node set V of the graph corresponds to the image pixels, with a total of M×N nodes. The edge set E between the nodes includes three directions: horizontal, vertical and diagonal, corresponding to the connection relationship between the 8-neighborhood pixels. The edge weight w is defined as the negative exponential function of the difference between the grayscale values ​​I of the two pixels, that is, the edge weight w is equal to the negative power of e (the base of the natural logarithm) (the square of the grayscale difference between the two pixels divided by 2 times the square of the grayscale difference sensitivity factor), where the grayscale difference sensitivity factor σ takes an empirical value of 5.0. This weight setting makes the edge weight between pixel pairs with greater grayscale differences smaller.

[0139] The undirected weighted graph G is recursively partitioned using the Kruskal minimum spanning tree algorithm. The algorithm traverses all edges in order of edge weight from small to large. For the edge currently traversed, if the two nodes connected to it have not been merged into any connected branch, the connected branch where the two nodes are located is merged; otherwise, the edge is skipped and the next edge is traversed. When the edge weight is greater than a certain threshold λ, the traversal ends, and the graph is divided into multiple connected branches, each of which is a watershed region. By gradually increasing the threshold λ, the recursive partitioning of the watershed region can be achieved until only one complete region is left.

[0140] On the basis of watershed recursive partitioning, a regional merging strategy is further introduced. For two adjacent watershed regions R1 and R2, their average grayscale values ​​m1 and m2, as well as the mean grayscale difference d of pixels along their common boundary are calculated respectively. The similarity measure s between regions is defined as the weighted average of the average grayscale difference between regions and the grayscale difference of the boundary, with the weight α being 0.5. The smaller the s value, the more similar the grayscale and boundary shapes of the two regions are. When s is less than a given threshold (such as 15), it is considered that R1 and R2 can be merged into a new watershed region. Repeat the above process until no more regions can be merged.

[0141] In order to further extract the foreground area corresponding to the solder joint, an adaptive threshold segmentation method is used based on the watershed segmentation. Traditional fixed threshold segmentation is difficult to adapt to different solder joint images, while adaptive threshold segmentation can automatically determine the optimal segmentation threshold according to the grayscale distribution of the image, and has better robustness. This paper uses the maximum inter-class variance method (Otsu) to determine the global threshold, and uses morphological processing to optimize the segmentation results.

[0142] First, calculate the grayscale histogram of the preprocessed solder joint image and use Gaussian smoothing to process the histogram. Assuming the image grayscale range is 0~L-1, the histogram contains L grayscale levels. The number of pixels corresponding to the i-th grayscale level is n i , the total number of pixels is N, then the probability of the i-th grayscale appearing is p i Equal to n i Divide by N.

[0143] Assuming that a certain threshold t divides the histogram into two parts, corresponding to the foreground and background, then the foreground pixel ratio ω0 is equal to the grayscale from 0 to t for p i Sum, the foreground average grayscale μ0 is equal to the grayscale from 0 to t multiplied by p i The sum is then divided by ω0; the background pixel ratio ω1 is equal to the grayscale ratio from t+1 to L-1 to p i The sum is also equal to 1 minus ω0, and the background average grayscale μ1 is equal to the grayscale from t+1 to L-1 multiplied by p i Take the sum and divide by ω1.

[0144] The inter-class variance g between the foreground and background is equal to ω0 times (μ0 minus the average grayscale μ of the entire image) squared plus ω1 times (μ1 minus the average grayscale μ of the entire image) squared. The optimal global threshold t It is the threshold t that maximizes the inter-class variance g, and the value range of t is 0 to L-1.

[0145] Using the optimal threshold t Binarize the solder joint image, greater than t Pixels with a value less than or equal to t are segmented as foreground. The pixels are segmented as background.

[0146] Taking the QFN solder joint image as an example, its grayscale histogram has two main peaks in the range of 200~230, corresponding to the solder joint and the background. The optimal global threshold t is calculated by the maximum inter-class variance method. The value is 219, which is basically between the two peaks. The threshold segmentation classifies 83% of the pixels as background and 17% of the pixels as foreground, and preliminarily separates the solder joint area.

[0147] On the basis of watershed segmentation and threshold segmentation, the region of interest (ROI) of the solder joint is finally extracted through morphological processing and connected region analysis.

[0148] First, the watershed region R watershed And the foreground area R obtained by threshold segmentation others Perform logical AND operation to extract the common area R of the two candidate , as candidate solder joint areas.

[0149] Then, for the candidate region R candidate Perform morphological opening and closing operations. The opening operation first corrodes and then expands, which can remove small white areas and disconnect small connections; the closing operation first expands and then corrodes, which can fill small black holes and smooth the area contour. After processing, the morphologically optimized area R is obtained. morph In this paper, a 3×3 rectangular structural element is selected for morphological processing, which can effectively eliminate small interference areas such as tin slag, while retaining the main solder joint area.

[0150] Finally, from R morph Extract 8 connected regions from the image. Let the area of ​​the region be A, the perimeter be P, and the circularity C be defined as 4 times π multiplied by A divided by the square of P. The value range of C is 0~1, and the closer to 1, the more circular the region shape is. Generally, the area of ​​the solder joint area is not too small, the ratio of perimeter to area is moderate, and the shape is relatively circular. Therefore, the area A can be greater than the given threshold A. min , perimeter area ratio A / P in [R min ,R max ], the circularity C is greater than the given threshold C min The area most likely to correspond to the solder joint is selected based on the above conditions as the region of interest (ROI) of the solder joint. min =30 pixels, R min =0.8,R max =1.2,C min =0.6.

[0151] Taking the QFN solder joint image as an example, after the logical AND operation, 5 candidate regions are obtained, with an area of ​​25~1200 pixels and a perimeter of 18~260 pixels. After morphological processing, the small area of ​​tin slag is eliminated, and the candidate regions are reduced to 3. Through area, perimeter, and circularity screening, an area of ​​920 pixels, a perimeter of 110 pixels, and a circularity of 0.72 is finally determined as the solder joint ROI, realizing the automatic segmentation and extraction of the solder joint area.

[0152] This paper proposes a solder joint ROI extraction method that combines graph theory watershed and adaptive threshold segmentation. First, the solder joint image is mapped to an undirected weighted graph, and the watershed region is obtained by recursive partitioning of the minimum spanning tree and region merging, which effectively suppresses over-segmentation; then the maximum inter-class variance method is used to adaptively determine the global threshold and perform foreground and background segmentation on the image; finally, the final solder joint region of interest is obtained by logical AND operation of the watershed region and the threshold segmentation result, and after morphological processing and screening of the area, perimeter and circularity of the connected region. This method overcomes the shortcomings of traditional watershed that is easy to over-segment and threshold segmentation is not robust enough, and can automatically adapt to different solder joint images, with stable and reliable segmentation effect.

[0153] In an optional embodiment,

[0154] The steps of using 3D morphology information and achieving accurate positioning of defect areas within the region of interest through morphological operations and edge detection algorithms include:

[0155] The disparity map corresponding to the region of interest is obtained by stereo matching of multi-view images, and the disparity map is converted into a three-dimensional point cloud according to the camera calibration parameters. The three-dimensional point cloud is denoised and smoothed, and the point cloud surface is reconstructed using the moving least squares method to obtain a smooth three-dimensional morphology of the region of interest.

[0156] The three-dimensional morphology of the region of interest is projected back to the two-dimensional plane to obtain a depth image, and the depth image is subjected to morphological top-hat transformation to extract the region that is more prominent than the neighboring region, and the top-hat transformation result is binarized and morphologically closed to obtain a preliminary defect candidate region;

[0157] Perform geometric feature analysis on the defect candidate area, set feature thresholds based on prior knowledge of area, aspect ratio, and circularity, and filter out areas that do not meet the defect characteristics to obtain refined defect areas;

[0158] Calculate the gradient magnitude map and gradient direction map of the image of the region of interest, use the edge detection operator to perform edge detection to obtain a binary edge map, and perform morphological refinement on the binary edge map to obtain an edge skeleton line with a single pixel width;

[0159] The refined defect area and the edge skeleton line are logically ANDed to extract the precise boundary of the defect area, and the precise boundary of the defect area is subjected to geometric feature analysis and smoothing to remove small protrusions and depressions to achieve precise positioning of the defect area.

[0160] For example, based on the extraction of the ROI of the solder joint, the three-dimensional morphology information of the solder joint is further obtained by using multi-view stereo vision, and the precise positioning of the defective area is achieved through morphological operations and edge detection algorithms. The main steps include:

[0161] Images of the solder joint ROI are collected from the left and right perspectives, and the disparity information between the two images is calculated using the stereo matching algorithm. This paper adopts an improved semi-global matching (SGM) algorithm, introduces an adaptive summation path and a hierarchical cost aggregation strategy, and improves the accuracy and efficiency of disparity calculation. The matching window size is 11×11, the disparity search range is 0~64 pixels, the matching cost function uses Census transform + Hamming distance, and the smoothing term parameters P1 and P2 are 20 and 100, respectively.

[0162] After obtaining the disparity map, the disparity value is back-projected into three-dimensional space by using the internal and external parameters of the binocular camera system through the principle of triangulation to obtain dense three-dimensional point cloud data on the solder joint surface. For a corresponding point, its coordinates in the left and right images are (x l , y l ) and (x r , y r ), the parallax is the difference d between the two x coordinates. Assuming the camera baseline length is B, the focal length of the left camera is f, the three-dimensional coordinates (X, Y, Z) of the point can be obtained according to the following relationship:

[0163] The X coordinate is equal to (x l Subtract the left camera optical center c x ) multiplied by the baseline length B and then divided by the parallax d;

[0164] The Y coordinate is equal to (y l Subtract the left camera optical center c y ) multiplied by the baseline length B and then divided by the parallax d;

[0165] The Z coordinate is equal to the focal length f multiplied by the baseline length B divided by the parallax d.

[0166] In this paper, a 9×7 chessboard calibration plate is used to calibrate the binocular camera system. The baseline length B is 75 mm, the focal length f is 16 mm, the distortion parameters k1 is -0.12, and k2 is 0.034.

[0167] The reconstructed 3D point cloud data is denoised and smoothed, and outliers are removed by statistical filtering. Then the local surface of the point cloud is reconstructed by the moving least square method based on the normal vector. The order of the fitting parameter is 2, and the standard deviation of the Gaussian weight of the weighted average is 2.5. Finally, a smooth 3D point cloud of the solder joint ROI is obtained.

[0168] To facilitate subsequent processing, the 3D point cloud of the solder joint ROI is projected back to a 2D plane to obtain a depth map with the same size as the original ROI image. The Z value of the point cloud is used as the depth information during projection. The larger the Z value, the farther the point is from the camera. The Z value is mapped to a range of 0 to 255 to form a grayscale image, which is called a depth map.

[0169] The depth map is subjected to morphological top-hat transformation to extract the convex areas in the local area that are higher than the average height of the neighborhood as potential defect candidate areas. The top-hat transformation is the result of subtracting the morphological opening operation from the original image, and the opening operation uses a 5×5 circular structure element. After the transformation, the grayscale value of the convex defect area is higher (such as greater than 150), while the grayscale value of the flat normal area is lower (such as less than 50).

[0170] The top-hat transformation result is segmented by adaptive threshold, and pixels with grayscale values ​​higher than threshold T are regarded as defect candidate areas. Threshold T is the weighted average of the grayscale mean and maximum value of the top-hat image, and the weight is 0.6. After segmentation, a binary image of the defect candidate area is obtained. In order to eliminate small discrete candidate areas, the binary image is further morphologically closed, and a 3×3 cross-shaped structural element is used to connect adjacent candidate areas to form a preliminary defect area.

[0171] Since the top-hat transform is sensitive to height changes, some non-defective raised areas will still be extracted, which need to be further screened based on the prior features of the defects. Analysis shows that the real defect area generally has a moderate area (such as 50-500 pixels), high compactness (the ratio of the area to the area of ​​the circumscribed rectangle is greater than 0.6) and circularity (circularity is defined as 4 times π times the area divided by the square of the perimeter, greater than 0.5). Therefore, by setting feature thresholds such as area, compactness, and circularity, candidate areas that are unlikely to be defects can be eliminated to obtain refined defect areas.

[0172] The seven candidate areas of QFN solder joints were screened by features, with the area threshold set as [80,400], the compactness threshold set as 0.65, and the circularity threshold set as 0.55. Finally, three refined defect areas were obtained.

[0173] Based on the refined defect area, the defect edge is further precisely located to accurately segment the defect area. First, the original ROI image is subjected to Sobel operator edge detection, and the gradient amplitude map is threshold segmented to obtain a binary edge map. The threshold is 1.5 times the mean value of the gradient amplitude. Then, the morphological thinning algorithm is used to extract the skeleton of the binary edge map to obtain an edge skeleton line with a single pixel width.

[0174] The refined defect area is logically ANDed with the edge skeleton line to obtain the precise edge of the defect area. Finally, the defect edge is contour tracked and smoothed to remove small concave and convex fluctuations to obtain the final precise boundary of the defect area.

[0175] Sobel edge detection was performed on the QFN solder joint ROI, with a threshold of 22. The edge skeleton line was obtained by refinement, and the intersection with the three refined defect areas was obtained to obtain three defect edge curves. The Douglas-Peucker algorithm was applied to the defect edge for polygonal approximation, with an approximate threshold of 1.5 pixels, and the polygon vertices were fitted with a B-spline curve to obtain a smooth defect edge. The final defect areas were 105, 82, and 126 pixels, respectively.

[0176] This paper proposes a method for accurately locating defective areas based on the three-dimensional morphological information of solder joints. First, the three-dimensional morphology of the region of interest (ROI) of the solder joint is obtained by binocular stereo vision, and the surface height information of the solder joint is obtained by point cloud denoising and smoothing. Then, the morphological top-hat transformation of the depth map is performed to extract the raised area as the defect candidate area, and the candidate area is refined according to geometric features such as area, compactness, and circularity. Finally, the defect edge is accurately located by edge detection and skeleton extraction, and the automatic segmentation of the defect area is realized. This method overcomes the limitation of traditional two-dimensional image segmentation that is easily disturbed by factors such as illumination and noise, makes full use of the three-dimensional morphological characteristics of the solder joint, and improves the accuracy and robustness of defect location. Good results have been achieved in actual QFN solder joint cases.

[0177] In an optional embodiment,

[0178] For the defect area after precise positioning, the defect area is three-dimensionally reconstructed using the triangulation principle according to the multi-view images and structured light information obtained by the multi-camera imaging system to obtain high-precision three-dimensional point cloud data of the defect area; for the reconstructed high-precision three-dimensional point cloud data of the defect area, a multi-scale curvature feature descriptor is extracted, and the multi-scale curvature feature descriptor is fused with the visual feature to form a multi-feature descriptor, the steps include:

[0179] The structured light sensor is used to obtain the three-dimensional data of the welding spot area. The structured light projector and multiple cameras are arranged around the welding workpiece. The corresponding relationship between the fringe code and the spatial coordinates is established by projecting the coded fringe pattern and capturing its deformed image.

[0180] Perform 2D feature matching on multi-view images, use scale-invariant feature transformation algorithm to extract key points of multi-view images and calculate their descriptors, and find the corresponding pairs of points with the same name between different viewpoints through the nearest neighbor distance ratio matching strategy;

[0181] Combined with the camera calibration parameters, the three-dimensional coordinates of the same-name points in the defect area are calculated using the triangulation principle, and the welding area is reconstructed in three dimensions. The outliers caused by mismatching are eliminated through statistical filtering methods to obtain high-precision three-dimensional point cloud data of the defect area.

[0182] For the high-precision three-dimensional point cloud of the reconstructed defect area, the principal component analysis method is used to estimate the normal vector and curvature information of each point, and the curvature mean and curvature variance in the point neighborhood are extracted and spliced ​​to form a multi-scale curvature feature descriptor;

[0183] The multi-scale curvature feature descriptor is fused with the visual feature descriptor, and the multi-feature descriptor of the defect area is obtained by feature splicing.

[0184] For example, in order to obtain the three-dimensional morphology information of the precisely located solder joint defect area, this paper uses a multi-camera imaging system and a structured light sensor to perform three-dimensional reconstruction of the defect area. The multi-view image provides the texture features of the defect, while the structured light provides the depth information of the defect. The combination of the two can obtain a high-precision three-dimensional point cloud model of the defect. The main steps are as follows:

[0185] Multiple cameras (e.g., 3) are arranged at different viewing angles around the welded workpiece to simultaneously obtain multi-view color images of the defective area. Considering the influence of the metallic gloss on the weld surface on the imaging quality, a short exposure time (e.g., 5ms) and a polarizing filter are selected to suppress highlights.

[0186] For each defect area image of each perspective, the scale-invariant feature transform (SIFT) algorithm is used to extract its key point features. Taking the first perspective as an example, 53 SIFT key points are detected in the solder joint defect ROI (such as a 100×100 pixel area), and the key point scale varies from 2 to 5 pixels. A 128-dimensional SIFT descriptor vector is calculated for each key point, which describes the statistical histogram of the gradient distribution of the key point neighborhood. The key points and descriptors of the three perspective images are calculated respectively.

[0187] SIFT keypoints are matched between images of different viewpoints to find corresponding pairs of points with the same name. The nearest neighbor distance ratio (NNDR) strategy is adopted, that is, the ratio of the Euclidean distance between the candidate matching point and the nearest neighbor and the second nearest neighbor is compared. When the ratio is less than the threshold (such as 0.8), it is considered a reliable match. 23 pairs of points with the same name are matched between view 1 and view 2, 19 pairs are matched between view 1 and view 3, and 21 pairs are matched between view 2 and view 3. Points with less than two matches across viewpoints are eliminated, and finally 15 stable SIFT points with the same name corresponding to three viewpoints are obtained.

[0188] The internal and external parameters of the three cameras are pre-calibrated. The internal parameters include focal length, optical center and radial distortion coefficient, and the external parameters include rotation matrix R and translation vector T. The 6×8 checkerboard points on the calibration plate are used as control points, and each camera takes 20 calibration images at different angles. The internal parameter estimation result is focal length f x Approximately equal to f y It is approximately equal to 2000, the radial distortion coefficient k1 is approximately equal to -0.1, k2 is approximately equal to 0.01, the relative rotation angles of the external parameters are estimated to be (10°, 8°, 15°), and the relative translation is (150mm, 120mm, 200mm).

[0189] The SIFT points of the defect area are triangulated using the calibration parameters to calculate their three-dimensional spatial coordinates. i The image point coordinates (x i1 ,y i1 )(x i2 ,y i2 )(x i3 ,y i3 ), combining the three camera projection equations:

[0190] Among them (R j ,T j ) is the external parameter of the j-th camera, K is the camera internal parameter, and sj is the scale factor.

[0191] Solve the three-dimensional point P by the least squares method i The coordinates (X i ,Y i ,Z i ). The 15 points with the same name are triangulated respectively to obtain the initial sparse three-dimensional point cloud of the defect area.

[0192] At the same time as the multi-view camera, the structured light sensor is used to obtain the depth information of the defect area, which is combined with the visual features to obtain a denser and more accurate three-dimensional point cloud. The coded stripe pattern is projected on the defect area and the stripe image of its deformation on the solder joint surface is captured. Due to the small size of the solder joint area, the projection window is set to 100×100 pixels. The decoded image obtains the phase value of each pixel, unpacks the phase into a continuous phase, and combines the absolute stripe number to establish a mapping between the pixel position and the projected stripe number.

[0193] For a point P on the solder joint surface, its position in the fringe image is (u, v). Through window interpolation, its fringe number is determined to be n+φ, where n is an integer fringe number and φ is the sub-pixel fringe phase. Then, the pixel position corresponding to the point on a certain column of the projector is inversely calculated based on the fringe number. Assuming the horizontal resolution of the projector is W, the pixel coordinates of P on the projector are marked as (n+φ, n / W), together with its imaging point coordinates (u, v), forming a matching pair ((n+φ, n / W), (u, v)).

[0194] Substitute the matching pixel coordinates into the pinhole imaging model of the projector and camera respectively, and by establishing the imaging equations of the projector and camera and solving them simultaneously, the coordinates of the three-dimensional point P (X P ,Y P ,Z P ):

[0195] Where (u p ,v p ) and (u c ,v c ) are the imaging coordinates of P on the projector and camera image planes, respectively, (u 0p ,v 0p ) and (u 0c ,v 0c ) are the principal point coordinates of the projector and camera, (f xp ,f yp ) and (f xc ,f yc ) is the focal length of the projector and camera, (T x ,T y ) is the baseline length between the projector and the camera. The structured light system is calibrated with a checkerboard grid in advance, and the baseline length T x Approximately 100mm.

[0196] Through the above steps, a 100×100 3D point cloud of the defect area is obtained, with a total of about 10,000 points and a point spacing of about 0.1 mm. Further remove outliers with discontinuous depth, align the structured light point cloud with the sparse point cloud obtained by visual triangulation, refine the 3D model, and finally obtain a high-precision 3D reconstruction result of the defect area.

[0197] On the basis of 3D point cloud reconstruction, the defect morphology is further characterized from the perspective of geometric features, and the curvature features of the point cloud are extracted. The principal component analysis (PCA) method is used to estimate the normal vector and curvature of each point. Specifically, for a point p in the point cloud, all points within its radius r neighborhood are taken to form a 3×k coordinate matrix, and the matrix is ​​decomposed by PCA to obtain three eigenvalues ​​λ1≥λ2≥λ3. Then the normal vector N at point p is the eigenvector corresponding to the minimum eigenvalue, and the curvature K can be obtained through the eigenvalue:

[0198] The curvature K is calculated by dividing the minimum eigenvalue λ3 by the sum of the three eigenvalues ​​λ1+λ2+λ3.

[0199] The curvature K reflects the degree of curvature of the local point cloud. The larger the curvature, the steeper the shape. Select the neighborhood radius at different scales (such as r = 1, 2, 3, 5 mm), calculate the curvature value of the defect point cloud at each scale, and further calculate the mean curvature K in each scale. m and curvature variance K v :

[0200] Curvature mean K m Equal to the curvature value K of all points in the neighborhood of this scale i The sum is divided by the number of points k; the curvature variance K v Equal to the curvature K of each point i and mean K m The sum of the squares of the differences is divided by the number of points k.

[0201] Where k is the number of points in the neighborhood of this scale, K i is the curvature value of the i-th point. m and K v Arrange them in order of scale, and you get a multi-scale curvature feature descriptor with increasing scale:

[0202] The multi-scale curvature feature descriptor F is expressed as [K m1 ,K v1 ,K m2 ,K v2 ,...,K mn ,K vn ], where the numbers 1 to n represent the order of scales from small to large.

[0203] This descriptor characterizes the statistical distribution characteristics of the point cloud curvature from details to the whole.

[0204] The above multi-scale curvature feature descriptor is fused with the original visual texture feature to obtain a multi-feature descriptor of the defect area. The visual feature selects the SIFT descriptor, which remains invariant under transformations such as image scale and rotation. The 128-dimensional descriptor vector of each SIFT key point is concatenated with the 8-dimensional curvature feature vector of its corresponding three-dimensional point to obtain a 136-dimensional multi-feature descriptor. This descriptor combines the information of two-dimensional texture and three-dimensional shape to more comprehensively characterize the defect characteristics from different angles. In the defect recognition and classification tasks, it will show better performance.

[0205] Taking QFN solder joints as an example, 15 136-dimensional defect multi-feature descriptor vectors are finally obtained. Compared with single visual or shape features, the fused multi-features are more robust in defect similarity measurement and matching, providing high-quality feature input for subsequent defect quantitative analysis.

[0206] In an optional embodiment,

[0207] The multi-feature descriptor is input into a defect classification and recognition module based on ensemble learning. The defect classification and recognition module adopts a stacking integration strategy to combine multiple heterogeneous classifiers into a strong classifier. The heterogeneous classifiers include support vector machines, K nearest neighbors, and random forests. The weights of the heterogeneous classifiers are dynamically adjusted through an adaptive boosting algorithm. The multi-feature descriptors are classified using a strong classifier, and a confidence threshold is introduced to adaptively correct the classification results. The steps of obtaining the type and severity of welding defects include:

[0208] Construct a set of heterogeneous classifiers including support vector machine, K nearest neighbor and random forest, use multi-feature descriptors to train each heterogeneous classifier separately, and optimize the kernel function parameters and penalty coefficient of support vector machine, the optimal number of neighbors of K nearest neighbor classifier, and the number of decision trees and maximum tree depth parameters of random forest classifier through grid search and cross validation;

[0209] The trained heterogeneous classifiers are combined into a strong classifier using an adaptive boosting algorithm. The sample weights are adjusted iteratively so that the samples misclassified by the previous round of classifiers gain attention in the next round. The weight coefficient of each heterogeneous classifier is calculated based on the classification error rate to construct a weighted decision function for the strong classifier.

[0210] The soft classification confidence vector is calculated for the defect type given by the strong classifier, and the maximum confidence is compared with the set confidence threshold. When the maximum confidence is less than the threshold, the classification result of the sample is considered unreliable. The defect type of the sample is re-predicted by the K-nearest neighbor algorithm based on the weighted Euclidean distance of the multi-feature descriptor of the sample and the known defect sample, and the classification result of the welding defect after adaptive correction is updated; the geometric features of the area and length of the defect area are mapped into severity levels to obtain the quantitative evaluation results of the welding defects.

[0211] For example, this paper proposes a welding defect classification and identification method based on ensemble learning. By fusing multiple heterogeneous classifiers to build a strong classifier, and introducing a confidence adaptive correction mechanism, it can achieve accurate identification of welding defect types and quantitative evaluation of severity. The main technical steps are as follows:

[0212] Support Vector Machine (SVM), K-Nearest Neighbor (KNN) and Random Forest (RF) were selected as heterogeneous classifiers to construct a classifier set. Each classifier was trained using multiple feature descriptors and the classifier parameters were optimized to improve performance.

[0213] For the SVM classifier, the radial basis kernel function (RBF) was used to determine the optimal penalty coefficient C and kernel function parameter γ through grid search and 5-fold cross validation. The search range was between 2 to the negative 5th power and 2 to the 15th power, and γ was between 2 to the negative 15th power and 2 to the 5th power, with a step size of 2 to the 2nd power. Finally, the optimal parameters C=512 and γ=0.5 were obtained, at which time the average accuracy of the SVM classifier on the validation set was 92.5%.

[0214] For the KNN classifier, the optimal number of neighbors K is selected through cross-validation. The candidate value set of K is {1, 3, 5, 7, 9, 11, 13, 15}, and the optimal K value is finally determined to be 7. At this time, the average accuracy of the KNN classifier on the validation set is 89.8%.

[0215] For the RF classifier, the number of decision trees ntree and the maximum tree depth max are optimized through grid search and cross validation. depth Two key parameters. The search range of ntree is {10,50,100,200,300,500}, max depth The search range is {5,10,15,20,25,30}. Finally, the optimal parameters ntree=200,max depth =20, at this time the average accuracy of the RF classifier on the validation set is 94.1%.

[0216] The three heterogeneous classifiers are integrated into a strong classifier using the Adaptive Boosting (AdaBoost) algorithm. AdaBoost constructs a weighted strong classification decision function by iteratively training multiple weak classifiers and dynamically adjusting the training sample weights and classifier weights.

[0217] The algorithm first initializes the weight of each training sample to 1 divided by the total number of samples N. Then it iterates for T rounds, performing the following steps in each round t:

[0218] Use the current sample weight distribution to train a base classifier h t (x), make predictions on the training set.

[0219] Calculate h t (x) Classification error rate ε on the training set t ε t Equal to h t (x) The sum of the weights of misclassified samples.

[0220] Calculate h t The weight coefficient α of (x) t α t It is equal to one-half times (1 minus ε t ) divided by ε t The natural logarithm of .

[0221] Update the sample weight distribution for the next iteration. t (x) correctly classified samples, weight multiplied by negative α of e t For misclassified samples, the weight is multiplied by α of e t Power.

[0222] The weights are normalized so that the sum of all sample weights is 1.

[0223] After T rounds of iteration, we get T base classifiers {h1(x),h2(x),...,h T (x)} and the corresponding weight coefficients {α1,α2,...,α T}. The decision function of the strong classifier H(x) is:

[0224] H(x) is equal to the sign function acting on T base classifiers α t Multiply by h t Weighted sum of (x)

[0225] The sign function predicts the sample as positive if the weighted sum is greater than 0, otherwise it is predicted as negative. The sum sign represents the weighted sum of T base classifiers.

[0226] To further improve the classification reliability, a confidence adaptive correction mechanism is introduced for the prediction results given by the strong classifier. For the test sample x, let H(x) output its soft classification confidence vector P(x) = [p1, p2,..., p C , where C is the number of defect categories, and p i is the predicted probability that x belongs to the i-th type of defect.

[0227] Calculate the maximum confidence value p max in P(x), and compare it with the preset confidence threshold θ. If pm ax is greater than or equal to θ, it is considered that the current classification result is credible and directly output; if p max is less than θ, it is considered that the classification result is unreliable and needs to be adaptively corrected.

[0228] On the basis of accurately discriminating the defect type, further quantitatively evaluate the severity of the defect. The severity reflects the impact of the defect on the quality of the welded joint and needs to be determined according to geometric features such as the size and shape of the defect.

[0229] Specifically, geometric parameters such as the area S, length L, width W, height H, and volume V are extracted for the defect area. Among them, S is obtained by counting the defect pixels after binarization, L and W are respectively the length and width of the maximum circumscribed rectangle of the defect area, H is the maximum height of the defect in the direction perpendicular to the welded surface, and V is obtained by cumulative calculation of the three-dimensional defect point cloud.

[0230] According to industry standards, determine the judgment thresholds for the severity of various types of defects. Taking porosity defects as an example, referring to the ISO5817 standard, according to the ratio of the pore diameter d (taking the larger value of L and W) to the weld thickness T, the porosity defects are divided into 5 severity levels from 1 to 5:

[0231] Pore diameter d severity level:

[0232] d less than or equal to 0.1T Level 1 (acceptable);

[0233] 0.1T < d less than or equal to 0.2T Level 2 (minor defect);

[0234] 0.2T < d less than or equal to 0.3T Level 3 (medium defect);

[0235] 0.3T < d less than or equal to 0.4T Level 4 (severe defect);

[0236] d greater than 0.4T Level 5 (extremely severe defect);

[0237] For other types of defects such as lack of penetration, overlap, crack, and undercut, also refer to the corresponding industry standards to give the geometric parameter thresholds under different severity levels.

[0238] After determining the defect type, the geometric parameters of the defect area are extracted and compared with the severity threshold to determine its severity level and achieve quantitative evaluation of the defect.

[0239] In summary, the adaptive confidence correction integrated learning classification method, combined with the severity determination under industry standards, achieves accurate identification of welding defect types and quantitative evaluation of severity, providing a quantitative basis for subsequent defect statistical analysis and quality control. In practical applications, the classifier can perform online learning and incremental training based on the dynamic update of defect samples, continuously improving classification performance and generalization ability.

[0240] Figure 2 FIG. 1 is a schematic diagram of the structure of a circuit board welding fault identification system based on machine vision according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0241] The first unit is used to obtain images of the soldering area of ​​the circuit board using a structured light projection system and a multi-camera imaging system, obtain the three-dimensional morphology information of the soldering area through the parallax principle, and optimize the imaging quality of the soldering area by adjusting the light source parameters in real time in combination with an adaptive light source system based on a dual-optical coaxial light source and a multi-angle annular light source; perform distortion correction and image registration on the original images obtained by the structured light projection system and the multi-camera imaging system, and use wavelet transform and morphological filtering algorithms to perform adaptive noise reduction and background equalization on the registered images to obtain the pre-processed soldering area images;

[0242] The second unit is used to automatically segment the pre-processed welding area image by combining the watershed algorithm based on graph theory and the adaptive threshold segmentation algorithm to obtain the region of interest, and accurately locate the defect area in the region of interest by using the three-dimensional morphology information and morphological operations and edge detection algorithms; for the defect area after accurate positioning, the triangulation principle is used to reconstruct the defect area in three dimensions based on the multi-view images and structured light information obtained by the multi-camera imaging system to obtain high-precision three-dimensional point cloud data of the defect area; for the high-precision three-dimensional point cloud data of the reconstructed defect area, a multi-scale curvature feature descriptor is extracted, and the multi-scale curvature feature descriptor is converted into the multi-scale curvature feature descriptor. The feature descriptor is fused with the visual feature to form a multi-feature descriptor, wherein the visual feature includes geometric features, grayscale statistical features, texture features and three-dimensional morphology features; the multi-feature descriptor is input into a defect classification and recognition module based on ensemble learning, wherein the defect classification and recognition module adopts a stacking integration strategy to combine multiple heterogeneous classifiers into a strong classifier, wherein the heterogeneous classifiers include support vector machines, K nearest neighbors and random forests, and the weights of the heterogeneous classifiers are dynamically adjusted through an adaptive boosting algorithm, and the multi-feature descriptors are classified by the strong classifier, and a confidence threshold is introduced to adaptively correct the classification results, thereby obtaining the type and severity of the welding defects;

[0243] The third unit is used to use a data enhancement method based on physical simulation during the training process of the defect classification and recognition module. By performing physical lighting simulation on the three-dimensional CAD model of the circuit board welding area, virtual solder joint images under different defect types and imaging conditions are generated to expand the training data set and improve the generalization ability of the defect classification and recognition module. The virtual solder joint images contain corresponding three-dimensional morphological information, which complements the real collected data to jointly construct a diverse training data set. According to the results of defect recognition, a high-precision motion control system is used, including a linear motor, a grating ruler and a motion control card, to achieve precise positioning and continuous scanning of the circuit board within the camera's field of view, accurately locate the detected defective area in combination with the three-dimensional morphological information, and visualize the defective area in the form of color markings or heat maps to intuitively display the three-dimensional spatial distribution and severity of welding defects.

[0244] According to a third aspect of the embodiments of the present invention,

[0245] An electronic device is provided, comprising:

[0246] processor;

[0247] a memory for storing processor-executable instructions;

[0248] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0249] A fourth aspect of the embodiments of the present invention is:

[0250] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0251] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0252] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A circuit board welding fault identification method based on machine vision, characterized in that: include: The image of the soldering area of ​​the circuit board is acquired by using a structured light projection system and a multi-camera imaging system. The three-dimensional morphology information of the soldering area is acquired by using the parallax principle. The imaging quality of the soldering area is optimized by adjusting the light source parameters in real time in combination with an adaptive light source system based on a dual-optical coaxial light source and a multi-angle annular light source. The original images acquired by the structured light projection system and the multi-camera imaging system are subjected to distortion correction and image registration. The registered images are subjected to adaptive noise reduction and background equalization processing by using wavelet transform and morphological filtering algorithms to obtain the pre-processed soldering area image. A method combining a watershed algorithm based on graph theory and an adaptive threshold segmentation algorithm is used to automatically segment the preprocessed welding area image to obtain the region of interest. The 3D morphology information is used and morphological operations and edge detection algorithms are used to accurately locate the defect area in the region of interest. For the accurately located defect area, the triangulation principle is used to reconstruct the defect area based on the multi-view images and structured light information obtained by a multi-camera imaging system to obtain high-precision 3D point cloud data of the defect area. For the reconstructed high-precision 3D point cloud data of the defect area, a multi-scale curvature feature descriptor is extracted and the multi-scale curvature feature descriptor is converted into a pixel value. The descriptor is fused with the visual feature to form a multi-feature descriptor, and the visual feature includes geometric features, grayscale statistical features, texture features and three-dimensional morphology features; the multi-feature descriptor is input into a defect classification and recognition module based on ensemble learning, and the defect classification and recognition module adopts a stacking integration strategy to combine multiple heterogeneous classifiers into a strong classifier, and the heterogeneous classifiers include support vector machines, K nearest neighbors, and random forests. The weights of the heterogeneous classifiers are dynamically adjusted through an adaptive boosting algorithm, and the multi-feature descriptors are classified by the strong classifier. At the same time, a confidence threshold is introduced to adaptively correct the classification results to obtain the type and severity of the welding defects; In the training process of the defect classification and recognition module, a data enhancement method based on physical simulation is used. By performing physical lighting simulation on the three-dimensional CAD model of the circuit board welding area, virtual solder joint images under different defect types and imaging conditions are generated to expand the training data set and improve the generalization ability of the defect classification and recognition module. The virtual solder joint images contain corresponding three-dimensional morphological information, which complements the real collected data to jointly construct a diverse training data set. According to the results of defect recognition, a high-precision motion control system is used, including a linear motor, a grating ruler and a motion control card, to achieve precise positioning and continuous scanning of the circuit board within the camera's field of view, accurately locate the detected defective area in combination with the three-dimensional morphological information, and visualize the defective area in the form of color markings or heat maps to intuitively display the three-dimensional spatial distribution and severity of welding defects.

2. The circuit board welding fault identification method based on machine vision according to claim 1 is characterized in that: The steps of using a structured light projection system and a multi-camera imaging system to obtain images of the soldering area of ​​the circuit board, obtaining the three-dimensional morphology information of the soldering area through the parallax principle, and optimizing the imaging quality of the soldering area by adjusting the light source parameters in real time in combination with an adaptive light source system based on a dual-light path coaxial light source and a multi-angle annular light source include: A multi-camera imaging system with a coaxial light source and a structured light projection system are used to collect multi-view images of the welding area through the multi-camera imaging system, and an image of the welding area surface projected with a structured light stripe pattern is obtained through the structured light projection system; the multi-camera imaging system includes a plurality of industrial cameras with different imaging viewing angles, and the optical axis of the structured light projection system forms a triangulation relationship with the optical axis of the multi-camera imaging system; An adaptive light source system is constructed, wherein the adaptive light source system includes a group of coaxial light sources and a group of annular light sources; the coaxial light source adopts a focus-adjustable lens, and the size and projection direction of the output light spot are dynamically adjustable by adjusting the focal length and incident angle of the focus-adjustable lens; the annular light source includes two groups of independently controlled LED annular arrays in inner and outer zones, and the light intensity at different positions of the welding area is adjusted by controlling the switch and brightness of LEDs in different zones; Calibrate the multi-camera imaging system to obtain the internal and external parameters of each industrial camera; use the internal and external parameters to perform distortion correction on the collected multi-view images, obtain the correspondence between images of different viewpoints through feature matching, calculate and optimize the homography transformation matrix of each viewpoint, and align the multi-view images; Perform three-dimensional reconstruction on the registered image to obtain three-dimensional point cloud data of the solder joint surface, calculate the phase information of the projector pixel points according to the structured light fringe pattern, convert the phase information into three-dimensional coordinates in combination with the internal and external parameters, and calculate and obtain a high-density point cloud of the solder joint surface; A three-dimensional model of the surface morphology of the welding area is established based on the three-dimensional point cloud data; the establishment of the three-dimensional model includes: splicing the three-dimensional point clouds obtained from different perspectives through coordinate transformation and rigid body registration algorithm to obtain complete three-dimensional morphology information of the welding area; In the process of collecting images of the welding area, the lighting parameters of the coaxial light source and the annular light source are controlled through real-time calculation and feedback of image quality evaluation indicators, so as to adaptively optimize the welding point imaging quality; the image quality evaluation indicators include image contrast, brightness uniformity, and signal-to-noise ratio.

3. The circuit board welding fault identification method based on machine vision according to claim 1 is characterized in that: The steps of performing distortion correction and image registration on the original images acquired by the structured light projection system and the multi-camera imaging system, and performing adaptive noise reduction and background equalization processing on the registered images using wavelet transform and morphological filtering algorithms to obtain the preprocessed welding area image include: The original welding image obtained by the structured light projection system and the multi-camera imaging system is subjected to distortion correction. The intrinsic parameter matrix and distortion coefficient obtained by camera calibration are used to establish a mapping relationship between pixel coordinates and normalized image plane coordinates to eliminate radial distortion and tangential distortion introduced by lens distortion, and obtain a distortion-corrected image. Perform feature matching and homography transformation estimation on the distortion-corrected image, extract the scale-invariant feature points of the image and calculate their descriptors, use nearest neighbor matching and Lowe's ratio test to screen reliable matching point pairs, use random sampling consistency algorithm to eliminate false matches, and combine with Levenberg-Marquardt algorithm to optimize the homography matrix for fine registration, perform projection transformation and resample the image, and achieve spatial alignment of images from different perspectives; The registered image is decomposed by wavelet, and the multi-scale decomposition of the image is performed using the Dobesi wavelet basis to obtain the low-frequency approximate coefficients and the high-frequency detail coefficients in the horizontal, vertical and diagonal directions; the optimal threshold of each sub-band coefficient is adaptively estimated using the Bayesian shrinkage criterion, and the high-frequency detail coefficient is adaptively shrunk using the soft threshold processing method to suppress image noise; The low-frequency approximate coefficients obtained by wavelet decomposition are subjected to morphological top-hat transformation, and structural elements are used to extract areas in the image that are brighter than the neighborhood, highlighting the uneven background illumination components; the top-hat transformation results are then subjected to morphological closing operations to fill small dark areas and smooth the background; the closing operation results are subtracted from the low-frequency approximate coefficients to obtain the low-frequency components after illumination correction; The high-frequency detail coefficients processed by adaptive threshold and the low-frequency approximate coefficients after background correction are used for wavelet reconstruction to obtain the welding area image after noise reduction and background equalization.

4. The circuit board welding fault identification method based on machine vision according to claim 1 is characterized in that: The method combining the graph-based watershed algorithm and the adaptive threshold segmentation algorithm is used to automatically segment the preprocessed welding area image. The steps of obtaining the region of interest include: The preprocessed welding area image is mapped into an undirected weighted graph, the nodes of the undirected weighted graph correspond to image pixels, the edges connect adjacent pixels, and the edge weight is the negative exponential function of the grayscale difference of adjacent pixels. The undirected weighted graph is recursively divided using the minimum spanning tree algorithm, and multiple connected branches are obtained by removing the edges with weights less than the increasing threshold. Each connected branch corresponds to a watershed area, and the watershed areas are merged. The average grayscale difference and boundary shape difference of adjacent areas are calculated, and the areas where both are less than the given threshold are merged into new watershed areas. The grayscale histogram of the preprocessed welding area image is calculated and smoothed. The global threshold is adaptively determined by maximizing the inter-class variance using the maximum inter-class variance method, and the welding area image is binarized using the global threshold. Pixels with grayscale values ​​greater than the threshold are segmented into foreground areas, and pixels with grayscale values ​​less than or equal to the threshold are segmented into background areas. Perform logical AND operations on the watershed area and the foreground area obtained by threshold segmentation to extract the common foreground area, and perform morphological opening and closing operations on the foreground area to fill small holes, remove small spots, and smooth the area boundaries; The connected areas are extracted from the foreground area after morphological processing, and the regions of interest corresponding to the welding points are screened out according to the area, perimeter and circularity of the region to realize automatic segmentation of the welding area image.

5. The circuit board welding fault identification method based on machine vision according to claim 1 is characterized in that: The steps of using 3D morphology information and achieving accurate positioning of defect areas within the region of interest through morphological operations and edge detection algorithms include: The disparity map corresponding to the region of interest is obtained by stereo matching of multi-view images, and the disparity map is converted into a three-dimensional point cloud according to the camera calibration parameters. The three-dimensional point cloud is denoised and smoothed, and the point cloud surface is reconstructed using the moving least squares method to obtain a smooth three-dimensional morphology of the region of interest. The three-dimensional morphology of the region of interest is projected back to the two-dimensional plane to obtain a depth image, and the depth image is subjected to morphological top-hat transformation to extract the region that is more prominent than the neighboring region, and the top-hat transformation result is binarized and morphologically closed to obtain a preliminary defect candidate region; Perform geometric feature analysis on the defect candidate area, set feature thresholds based on prior knowledge of area, aspect ratio, and circularity, and filter out areas that do not meet the defect characteristics to obtain refined defect areas; Calculate the gradient magnitude map and gradient direction map of the image of the region of interest, use the edge detection operator to perform edge detection to obtain a binary edge map, and perform morphological refinement on the binary edge map to obtain an edge skeleton line with a single pixel width; The refined defect area and the edge skeleton line are logically ANDed to extract the precise boundary of the defect area, and the precise boundary of the defect area is subjected to geometric feature analysis and smoothing to remove small protrusions and depressions to achieve precise positioning of the defect area.

6. The circuit board welding fault identification method based on machine vision according to claim 1 is characterized in that: For the defect area after precise positioning, the defect area is three-dimensionally reconstructed using the triangulation principle according to the multi-view images and structured light information obtained by the multi-camera imaging system to obtain high-precision three-dimensional point cloud data of the defect area; for the reconstructed high-precision three-dimensional point cloud data of the defect area, a multi-scale curvature feature descriptor is extracted, and the multi-scale curvature feature descriptor is fused with the visual feature to form a multi-feature descriptor, the steps include: The structured light sensor is used to obtain the three-dimensional data of the welding spot area. The structured light projector and multiple cameras are arranged around the welding workpiece. The corresponding relationship between the fringe code and the spatial coordinates is established by projecting the coded fringe pattern and capturing its deformed image. Perform 2D feature matching on multi-view images, use scale-invariant feature transformation algorithm to extract key points of multi-view images and calculate their descriptors, and find corresponding pairs of points with the same name between different viewpoints through nearest neighbor distance ratio matching strategy; Combined with the camera calibration parameters, the three-dimensional coordinates of the same-name points in the defect area are calculated using the triangulation principle, and the welding area is reconstructed in three dimensions. The outliers caused by mismatching are eliminated through statistical filtering methods to obtain high-precision three-dimensional point cloud data of the defect area. For the high-precision three-dimensional point cloud of the reconstructed defect area, the principal component analysis method is used to estimate the normal vector and curvature information of each point, extract the curvature mean and curvature variance in the point neighborhood, and splice them to form a multi-scale curvature feature descriptor; The multi-scale curvature feature descriptor is fused with the visual feature descriptor, and the multi-feature descriptor of the defect area is obtained by feature splicing.

7. The circuit board welding fault identification method based on machine vision according to claim 1 is characterized in that ,The multi-feature descriptor is input into the defect classification and recognition module based on ensemble learning. The defect classification and recognition module adopts a stacking integration strategy to combine multiple heterogeneous classifiers into a strong classifier. The heterogeneous classifiers include support vector machines, K nearest neighbors, and random forests. The weights of the heterogeneous classifiers are dynamically adjusted through an adaptive boosting algorithm. The strong classifier is used to classify the multi-feature descriptors, and a confidence threshold is introduced to adaptively correct the classification results. The steps of obtaining the type and severity of welding defects include: Construct a set of heterogeneous classifiers including support vector machine, K nearest neighbor and random forest, use multi-feature descriptors to train each heterogeneous classifier separately, and optimize the kernel function parameters and penalty coefficient of support vector machine, the optimal number of neighbors of K nearest neighbor classifier, and the number of decision trees and maximum tree depth parameters of random forest classifier through grid search and cross-validation; The trained heterogeneous classifiers are combined into a strong classifier using an adaptive boosting algorithm. The sample weights are adjusted iteratively so that the samples misclassified by the previous round of classifiers gain attention in the next round. The weight coefficient of each heterogeneous classifier is calculated based on the classification error rate to construct a weighted decision function for the strong classifier. The soft classification confidence vector is calculated for the defect type given by the strong classifier, and the maximum confidence is compared with the set confidence threshold. When the maximum confidence is less than the threshold, the classification result of the sample is considered unreliable. The defect type of the sample is re-predicted by the K-nearest neighbor algorithm based on the weighted Euclidean distance of the multi-feature descriptor of the sample and the known defect sample, and the classification result of the welding defect after adaptive correction is updated; the geometric features of the area and length of the defect area are mapped into severity levels to obtain the quantitative evaluation results of the welding defects.

8. A circuit board welding fault identification system based on machine vision, used to implement the circuit board welding fault identification method based on machine vision as described in any one of claims 1 to 7, characterized in that: include: The first unit is used to obtain images of the soldering area of ​​the circuit board using a structured light projection system and a multi-camera imaging system, obtain the three-dimensional morphology information of the soldering area through the parallax principle, and optimize the imaging quality of the soldering area by adjusting the light source parameters in real time in combination with an adaptive light source system based on a dual-light path coaxial light source and a multi-angle annular light source; perform distortion correction and image registration on the original images obtained by the structured light projection system and the multi-camera imaging system, and use wavelet transform and morphological filtering algorithms to perform adaptive noise reduction and background equalization on the registered images to obtain the pre-processed soldering area images; The second unit is used to automatically segment the pre-processed welding area image by combining a watershed algorithm based on graph theory and an adaptive threshold segmentation algorithm to obtain an area of ​​interest, and to accurately locate the defect area in the area of ​​interest by using three-dimensional morphology information and morphological operations and edge detection algorithms; for the accurately located defect area, the triangulation principle is used to perform three-dimensional reconstruction of the defect area based on the multi-view images and structured light information obtained by a multi-camera imaging system to obtain high-precision three-dimensional point cloud data of the defect area; for the high-precision three-dimensional point cloud data of the reconstructed defect area, a multi-scale curvature feature descriptor is extracted, and the multi-scale curvature feature descriptor is converted into a pixel value. The feature descriptor is fused with the visual feature to form a multi-feature descriptor, wherein the visual feature includes geometric features, grayscale statistical features, texture features and three-dimensional morphology features; the multi-feature descriptor is input into a defect classification and recognition module based on ensemble learning, wherein the defect classification and recognition module adopts a stacking integration strategy to combine multiple heterogeneous classifiers into a strong classifier, wherein the heterogeneous classifiers include support vector machines, K nearest neighbors and random forests, and the weights of the heterogeneous classifiers are dynamically adjusted through an adaptive boosting algorithm, and the multi-feature descriptors are classified by the strong classifier, and a confidence threshold is introduced to adaptively correct the classification results, thereby obtaining the type and severity of the welding defects; The third unit is used to use a data enhancement method based on physical simulation during the training process of the defect classification and recognition module. By performing physical lighting simulation on the three-dimensional CAD model of the circuit board welding area, virtual solder joint images under different defect types and imaging conditions are generated to expand the training data set and improve the generalization ability of the defect classification and recognition module. The virtual solder joint images contain corresponding three-dimensional morphological information, which complements the real collected data to jointly construct a diverse training data set. According to the results of defect recognition, a high-precision motion control system is used, including a linear motor, a grating ruler and a motion control card, to achieve precise positioning and continuous scanning of the circuit board within the camera's field of view, accurately locate the detected defective area in combination with the three-dimensional morphological information, and visualize the defective area in the form of color markings or heat maps to intuitively display the three-dimensional spatial distribution and severity of welding defects.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the circuit board welding fault identification method based on machine vision as described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the circuit board welding fault identification method based on machine vision described in any one of claims 1 to 7 is implemented.

Images