Electronic component welding defect visual detection method and system based on multispectral imaging

Abstract

The invention relates to an electronic component welding defect visual detection method and system based on multispectral imaging, and the method comprises the steps: obtaining a preset multiband sequence image, collected by a multispectral imaging terminal, of a to-be-detected welding region when a data collection instruction is received; constructing a three-dimensional morphology point cloud of the to-be-detected welding area based on the multi-band sequence image; extracting associated feature vectors related to the welding quality in the multi-band sequence image, and performing weighted fusion on associated features under different bands to generate a fusion feature map; performing space matching alignment on the fused feature map and the three-dimensional topography point cloud, and outputting geometric position information of the associated feature vector in the three-dimensional topography point cloud; identifying and judging whether defect features are formed or not; when it is judged that the defect features exist, defect types corresponding to the defect features are screened out, and quantitative information corresponding to the defect features is calculated. The method has the advantages that welding spot detection is more comprehensive, and welding spot defects are accurately analyzed.

Description

Technical Field

[0001] This application relates to the technical field of visual inspection, and in particular to a visual inspection method and system for welding defects in electronic components based on multispectral imaging. Background Technology

[0002] As electronic components become increasingly miniaturized and high-density, automated visual inspection of soldering quality faces severe challenges. Current mainstream inspection technologies primarily acquire surface morphology and color information of solder joints using visible light two-dimensional vision. However, due to a lack of penetration or insufficient feature contrast, defects such as cold solder joints, internal voids, and microcracks, which are internal or subsurface defects, are prone to being missed. Furthermore, current two-dimensional visual inspection only determines the presence or absence of defects and lacks the ability to quantify and analyze the causes of solder joint defects. Consequently, it is difficult to provide effective closed-loop feedback for subsequent process parameter optimization. Therefore, the detection accuracy and analytical depth of soldering defects need to be improved. Summary of the Invention

[0003] To make the inspection of welding points more comprehensive and to achieve accurate analysis of welding point defects, thereby providing effective and reliable improvement strategies for welding process parameters, this application provides a visual inspection method and system for welding defects in electronic components based on multispectral imaging.

[0004] The above-mentioned objective of this application is achieved through the following technical solution: A visual inspection method for welding defects in electronic components based on multispectral imaging includes the following steps: When a data acquisition command is received, a multi-band sequence image of the welding area to be tested is acquired by a preset multispectral imaging terminal. The multi-band sequence image includes visible light band, near-infrared band and short-wave infrared band. A three-dimensional topographic point cloud of the welding area to be tested is constructed based on the multi-band sequence images. Based on preset feature recognition rules, the associated feature vectors related to welding quality are extracted from the multi-band sequence images, and the associated features under different bands are weighted and fused to generate a fused feature map. By transforming coordinates, the fused feature map is spatially matched and aligned with the three-dimensional topography point cloud, and the geometric position information of the associated feature vector in the three-dimensional topography point cloud is output. Based on the distribution of the associated feature vectors and the corresponding geometric position information, identify and determine whether a defect feature has been formed; When the defect feature is determined to exist, based on a preset defect feature database, the defect type corresponding to the defect feature is selected and the quantitative information corresponding to the defect feature is calculated. The defect type and the quantitative information are then sent to the user terminal.

[0005] By adopting the above technical solution, the multispectral imaging terminal performs multiple image acquisitions of the solder joints in the welding area under test under different wavelengths of light, including the visible light band, near-infrared band, and short-wave infrared band. The visible light band can acquire the surface texture and color features of the solder joints, the near-infrared band can acquire the internal structural transmission features of the solder joints, and the short-wave infrared band can acquire the thermal distribution features of the solder joints. Image acquisition under different wavelengths can comprehensively capture the multi-dimensional features of the solder joints, achieving high-sensitivity detection of defects in traditional blind zones such as cold solder joints, internal voids, and microcracks, extending the detection range from the surface to the interior and interface. Furthermore, by constructing a three-dimensional topographic point cloud, misjudgments caused by two-dimensional projection are fundamentally avoided, enabling the detection of defects such as bridging, warping, and insufficient solder. The accuracy of defect identification is significantly improved, and the false alarm rate is reduced. Furthermore, associated feature vectors are integrated into each point in the point cloud, i.e., each geometric location information, fusing the features of solder joint morphology with those of solder joint spectral imaging. This makes solder joint defect identification more concrete and improves the accuracy of defect detection. Since the defect features of the solder joint are integrated with the geometric space display, the severity of the defect features can also be quantified through geometric calculations, such as the area and void ratio of solder ball voids, the size of the connections between connected solder joints, and the error amount of excess or insufficient solder in the solder joint. Finally, the defect type and the quantified information are sent to the user terminal, which helps staff to clearly understand the cause of the defect and the specific quantitative data of the defect, thereby facilitating the subsequent adjustment of welding process parameters.

[0006] Optionally, the step of constructing a three-dimensional topographic point cloud of the welding area to be tested based on the multi-band sequence images includes: The spectral imaging terminal pre-acquires static planar images of the welding area to be tested using visible light and near-infrared bands; The spectral imaging terminal performs a three-dimensional scan of the welding area under test using the visible light band, and performs phase constraint on the static planar image to generate a preliminary topographic point cloud. The spectral imaging terminal scans the welding area to be tested in the near-infrared band or the short-wave infrared band to determine whether there are abnormal point clusters in the preliminary morphology point cloud. If they exist, acquire the point cluster image corresponding to the location of the anomalous point cluster in the near-infrared band or short-wave infrared band, and map the point cluster image to the location of the anomalous point cluster to cover the anomalous point cluster.

[0007] By adopting the above technical solutions, such as using a digital grating projector to perform 3D scanning of the area to be soldered, and synchronously controlling the camera to capture each deformed grating image for phase calculation and coordinate parameter calibration, a preliminary morphological point cloud of the solder joint can be constructed. However, during the creation of the 3D point cloud on the solder joint surface, high reflectivity can easily lead to overexposure or uneven imaging in the visible light band, resulting in the loss of key textures and the appearance of holes or distortions in the point cloud. Therefore, the spectral imaging terminal scans the soldering area under test in the near-infrared or short-wave infrared bands. By comparing the grayscale values ​​of the images in different bands, it determines whether there are unreliable abnormal point clusters in the visible light band, i.e., lost areas caused by reflection. Furthermore, it maps the point cluster images of the lost areas in the near-infrared or short-wave infrared bands to the location of the lost areas in the visible light band, effectively identifying and repairing the failed areas in the creation of the 3D morphological point cloud caused by the high reflectivity of the solder, significantly improving the integrity and reliability of the 3D morphological point cloud.

[0008] Optionally, the step of extracting the associated feature vectors related to welding quality from the multi-band sequence images based on preset feature recognition rules, and weighting and fusing the associated features under different bands to generate a fused feature map includes: Denoising and phase registration of multi-band image sequences; By using a pre-defined feature extraction model, common feature vectors of different bands in multi-band sequence images are extracted in advance; The feature extraction model then extracts specific feature vectors corresponding to the visible light band, near-infrared band, and short-wave infrared band respectively; the common feature vector is bound to multiple specific feature vectors to obtain the associated feature vectors under different bands; By using an attention mechanism and preset labels, different weight values ​​are assigned to the associated feature vectors under different bands, and the associated feature vectors under different bands are then weighted and fused.

[0009] By adopting the above technical solution, in order to make the subsequent defect feature identification more accurate, before fusing spectral features and three-dimensional topographic point clouds, it is necessary to fuse the spectral features of each weld point in the welding area to be tested and set weight values. Since the specific feature vectors of weld points reflected under different bands are different, the visible light band focuses on appearance features, the near-infrared band focuses on sub-surface features of internal structural continuity, and the short-wave infrared band focuses on thermally related features such as thermal radiation. After binding with the basic common feature vector, the associated feature vector is obtained. In order to achieve the accuracy of defect judgment for different weld point positions during the detection process in different weld point positions in the welding area to be tested, an attention mechanism and preset labels are introduced. The preset labels represent different welding positions and assign different weight values ​​to specific feature vectors under different bands. This makes it easier to identify defects that are prone to occur at the weld point position when performing defect feature identification. For example, when detecting BGA type weld points, void defects are prone to occur inside. Therefore, the weight of specific feature vectors under the short-wave infrared band is higher than that under the visible light band.

[0010] Optionally, in the step of assigning different weight values ​​to the associated feature vectors under different bands through an attention mechanism and preset identifiers, the preset identifiers include solder joint type vectors. Different solder joint type vectors correspond to different weight ratios of the visible light band, near-infrared band, and short-wave infrared band. The feature extraction model uses an attention mechanism to identify the solder joint type of the welding area to be tested in order to adjust the weight ratio of the visible light band, near-infrared band, and short-wave infrared band.

[0011] By adopting the above technical solution, different types of weld points in the area to be welded can be intelligently identified, and the specific feature vectors under different bands can be dynamically adjusted during the defect feature identification process, thereby improving the accuracy and efficiency of defect feature identification.

[0012] Optionally, the step of identifying and determining whether a defect feature has been formed based on the distribution of the associated feature vector and the corresponding geometric position information includes: Based on the geometric location information, determine whether the associated feature vector at the location is abnormal. If it is abnormal, obtain the neighboring point group of the geometric location information in the three-dimensional topographic point cloud. Identify the range of anomalous point groups in neighboring point groups where abnormal associated feature vectors appear; When the range of the abnormal point group exceeds the preset range parameter, it is determined that the current solder joint has formed a defect feature.

[0013] By adopting the above technical solution, the spectral characteristics of the welding area to be tested and the three-dimensional topographic point cloud are fused. Each point in the three-dimensional topographic point cloud is bound to a specific feature vector under different bands, such as spectral reflectance / intensity values, three-dimensional spatial coordinates, local surface normal vectors, curvature, etc. in multiple bands. Therefore, points with abnormal associated feature vectors in the three-dimensional topographic point cloud are identified. Then, it is further identified whether the neighboring point groups also have abnormal associated feature vectors. The neighboring point groups are divided according to the location of the weld point. By judging the range of the abnormal point groups, it can be determined whether a defect feature has been formed. If the range of the abnormal point groups is smaller than the range parameter, it is not judged as a defect.

[0014] Optionally, the step of determining whether the associated feature vector at the current location is abnormal based on geometric location information, and if abnormal, obtaining the neighboring point group of the geometric location information in the three-dimensional topographic point cloud, includes: Determine the weld point to which the geometric position information belongs in the three-dimensional topographic point cloud, and identify the weld point type; Based on the type of solder joint, determine the distribution of associated feature vectors of all geometric position information in the solder joint; Based on the distribution of associated feature vectors of all geometric position information in the current solder joint, determine whether there are any points in the solder joint with abnormal associated feature vectors; If they exist, obtain the set of points associated with the anomaly in the 3D topographic point cloud, and use the points associated with the coordinates as the neighboring point group.

[0015] By adopting the above technical solution, to determine whether the associated feature vector at the geometric position information is normal, it is necessary to first determine the welding position and welding type to which the geometric position information belongs. For example, it may be located on the surface of the weld ball or between two weld points that should be isolated. Given the welding type, based on the distribution of the other associated feature vectors of the weld point, anomalies that are different from most other associated feature vectors can be found through feature learning via convolutional networks. These anomalies are then used as reference points for judging abnormal features to obtain the group of points in their vicinity, thus providing the initial basis for judging whether defect features exist.

[0016] Optionally, the step of selecting the defect type corresponding to the defect feature and calculating the quantitative information corresponding to the defect feature based on a preset defect feature database when the defect feature is determined to exist includes: When it is determined that the aforementioned characteristic defect exists, the abnormal geometric region formed by the range of abnormal point groups is identified. Based on the combination of abnormal geometric regions and corresponding associated feature vectors, the defect type of the current abnormal feature is matched from the preset defect feature database; The defect calculation method that matches the defect type is retrieved, and the quantitative information corresponding to the defect type is calculated based on the abnormal geometric region and the corresponding associated feature vector.

[0017] By adopting the above technical solution, after determining whether defect features exist, the combination of the contour shape shown by the range of the abnormal point group and the associated feature vectors can be used to find the defect type that matches the features from the preset defect feature database. For example, low near-infrared intensity and local depressions are highly likely to point to the defect type of internal voids. After clarifying the defect type, the brightness information of the defect can be calculated based on the parameters of the associated feature vectors by using the defect calculation method that is pre-matched with the defect type. For example, the calculation of void defects is to calculate the percentage of its actual volume in the three-dimensional topography point cloud to the theoretical solder ball volume, thereby realizing the quantitative calculation of the defect type.

[0018] The second objective of this invention is achieved through the following technical solution: A visual inspection system for welding defects in electronic components based on multispectral imaging, comprising: The acquisition module is used to acquire multi-band sequence images of the welding area to be tested acquired by a preset multispectral imaging terminal when a data acquisition command is received. The multi-band sequence images include visible light band, near-infrared band and short-wave infrared band. The point cloud construction module is used to construct a three-dimensional topographic point cloud of the welding area to be tested based on the multi-band sequence images; The feature fusion module is used to extract the associated feature vectors related to welding quality from the multi-band sequence images based on preset feature recognition rules, and to weight and fuse the associated features under different bands to generate a fused feature map. The spectrum and morphology combination module is used to spatially match and align the fused feature map with the three-dimensional morphology point cloud through coordinate transformation, and output the geometric position information of the associated feature vector in the three-dimensional morphology point cloud; The defect judgment module is used to identify and determine whether a defect feature has been formed based on the distribution of the associated feature vector and the corresponding geometric position information; The defect analysis module is used to, when the existence of the defect feature is determined, filter out the defect type corresponding to the defect feature based on a preset defect feature database, calculate the quantitative information corresponding to the defect feature, and send the defect type and the quantitative information to the user terminal.

[0019] The above-mentioned objective three of this application is achieved through the following technical solution: A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described visual inspection method for welding defects in electronic components based on multispectral imaging.

[0020] The fourth objective of this application is achieved through the following technical solution: A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described visual inspection method for welding defects in electronic components based on multispectral imaging.

[0021] In summary, this application includes at least one of the following beneficial technical effects: The multispectral imaging terminal acquires images of the solder joints in the welding area under test multiple times under different wavelengths of light, including visible, near-infrared, and short-wave infrared bands. This image acquisition under different wavelengths can comprehensively capture the multi-dimensional features of the solder joints, achieving high-sensitivity detection of defects in traditional blind zones such as cold solder joints, internal voids, and microcracks, extending the detection range from the surface to the interior and interface. Furthermore, by constructing a three-dimensional topographic point cloud, it fundamentally avoids misjudgments caused by two-dimensional projection, significantly improving the accuracy of judging defects such as bridging, warping, and insufficient solder, and reducing the false alarm rate. In addition, associated feature vectors are integrated into each geometric location information in the point cloud, fusing the features of the solder joint topography with the spectral imaging of the solder joint, making the identification of solder joint defects more concrete and improving the accuracy of defect detection. Since the defect features of the solder joint are integrated with the geometric space display, the severity of the defect features can also be quantified through geometric calculations. Finally, the defect type and the quantified information are sent to the user terminal, helping staff to clearly understand the cause of the defect and the specific quantitative data of the defect, thereby facilitating the subsequent adjustment of welding process parameters. To improve the accuracy of defect judgment at different weld point locations during the detection process in the welding area under test, an attention mechanism and preset labels are introduced. The preset labels represent different welding positions and assign different weight values ​​to specific feature vectors under different wavelengths. This makes it easier to identify defects that are prone to occur at the weld point locations when performing defect feature identification. Points with abnormal associated feature vectors in the 3D topographic point cloud are identified. Then, it is further identified whether the neighboring point groups also have abnormal associated feature vectors. The neighboring point groups are divided according to the location of the weld point. By judging the range of the abnormal point group, it can be determined whether a defect feature has been formed. If the range of the abnormal point group is smaller than the range parameter, it is not judged as a defect.

[0022] To determine whether the associated feature vector at the geometric location information is normal, it is necessary to first determine the welding position and welding type to which the geometric location information belongs. For example, it may be located on the surface of the weld ball or between two weld points that should be isolated. Given the welding type, based on the distribution of the other associated feature vectors of the weld point, anomalies that are different from most other associated feature vectors can be found through feature learning via convolutional networks. These anomalies can be used as reference points for judging abnormal features to obtain the group of points in their vicinity, thus providing the initial basis for judging whether defect features exist. Attached Figure Description

[0023] Figure 1 This is a flowchart of an embodiment of the visual inspection method for welding defects in electronic components based on multispectral imaging in this application; Figure 2 This is a schematic diagram illustrating the principle of three-dimensional topography point cloud construction in an embodiment of the visual detection method for welding defects in electronic components based on multispectral imaging in this application. Figure 3 This is a schematic diagram illustrating the quantitative calculation principle of defect types in an embodiment of the visual inspection method for welding defects of electronic components based on multispectral imaging in this application; Figure 4 This is a schematic block diagram of a computer device according to this application. Detailed Implementation

[0024] The following is in conjunction with the appendix Figure 1-4 This application will be described in further detail.

[0025] In the following embodiments, such as Figure 1 As shown, this application discloses a visual inspection method for welding defects in electronic components based on multispectral imaging, which specifically includes the following steps: S10: When a data acquisition command is received, acquire a multi-band sequence image of the welding area to be tested acquired by a preset multi-spectral imaging terminal. The multi-band sequence image includes visible light band, near-infrared band and short-wave infrared band. In this embodiment, the preset multispectral imaging terminal is a high-resolution scientific-grade CMOS camera equipped with a programmable filter wheel. The filter center wavelengths are 550nm for green light, 850nm for near-infrared light, and 1200nm for short-wave infrared light. It is used in conjunction with a highly uniform multi-band ring LED light source. In the multi-band sequence images, the wavelengths in the visible light band are 400nm-700nm, the near-infrared band is 700nm-1100nm, and the short-wave infrared band is 1100nm-1700nm, which can be selected according to the actual equipment and detection environment.

[0026] S20: Constructing a three-dimensional topographic point cloud of the welding area under test based on multi-band sequence images; In this embodiment, the three-dimensional topography point cloud refers to the generation of a three-dimensional topography point cloud model of each weld point in the welding area to be tested, which is pixel-level aligned with the multi-band sequence image.

[0027] Specifically, refer to Figure 2 Step S20 includes the following steps: S21: The spectral imaging terminal pre-acquires static planar images of the welding area to be tested through the visible light band and near-infrared band; S22: The spectral imaging terminal performs a three-dimensional scan of the welding area under test using the visible light band, and performs phase constraint on the static planar image to generate a preliminary topographic point cloud; S23: The spectral imaging terminal scans the welding area to be tested in the near-infrared band or short-wave infrared band to determine whether there are abnormal point clusters in the preliminary morphology point cloud. S24: If it exists, acquire the point group image corresponding to the location of the anomalous point group in the near-infrared band or short-wave infrared band, and map the point group image to the location of the anomalous point group to cover the anomalous point group.

[0028] In the acquisition of static planar images, a DLP projector is used to acquire static planar images of the area to be measured in the visible light band and near-infrared band respectively. These two images serve as reference views for subsequent registration and constraint.

[0029] The 3D scanning of the welding area under test involves projecting a specifically coded Gray code grating pattern and a phase-shifted sinusoidal grating pattern onto the area using a digital grating projector. Simultaneously, a structured light camera is controlled to capture each modulated deformed grating image using the visible light channel, either without filters or using a wide-band filter. A wrapping phase map is then calculated from the sinusoidal grating image using the phase-shifting method. Using the absolute order information provided by the Gray code image, the wrapping phase map is unwrapped to obtain a continuous absolute phase map Φ(x,y). The phase value Φ of each pixel corresponds one-to-one with its spatial height Z. By scanning a standard block with known heights beforehand, a precise mathematical model of the camera and projector system is established, yielding the transformation parameters from (x,y,Φ) to (X,Y,Z), and generating a preliminary topographic point cloud.

[0030] Regarding the identification of anomalous point clusters, the corner and edge information extracted under the visible light band is first registered with the corner and edge information under the near-infrared or short-wave infrared bands and projected onto the image coordinate system of the optical camera. This aligns the multi-band sequence images under the near-infrared or short-wave infrared bands with the preliminary topographic point cloud. Furthermore, because the near-infrared or short-wave infrared bands have stronger penetration and more uniform scattering of solder, highly reflective areas that are overexposed under visible light can exhibit uniform grayscale in the near-infrared image, containing effective image information. Therefore, by analyzing the local grayscale uniformity and gradient of the near-infrared image, areas where reconstruction failed due to reflection under visible light can be identified more accurately, generating a repair mask. This area is the anomalous point cluster in the preliminary topographic point cloud.

[0031] Further, point group images corresponding to the abnormal point groups in the near-infrared or short-wave infrared bands are mapped to the region of the preliminary topography point cloud to generate a repair mask by position recognition, covering the abnormal point groups, and performing point cloud smoothing and upsampling based on normal consistency. While preserving sharp edges, a final three-dimensional topography point cloud with a denser, smoother and noise-free surface is generated.

[0032] S30: Based on preset feature recognition rules, extract the associated feature vectors related to welding quality from multi-band sequence images, and weight and fuse the associated features under different bands to generate a fused feature map; In this embodiment, the associated feature vectors related to welding quality include common feature vectors across different wavebands and feature vectors from different wavebands. Feature recognition rules are used to denoise and correct non-uniformity of the feature vectors from different wavebands and extract common feature vectors. Furthermore, a deeper convolutional neural network is used to extract feature vectors from different wavebands and assign different weight values ​​to these feature vectors.

[0033] Specifically, step S30 includes the following steps: S31: Denoise and perform phase registration of multi-band image sequences; S32: By using a preset feature extraction model, common feature vectors of different bands in multi-band sequence images are extracted in advance; S33: The feature extraction model then extracts specific feature vectors corresponding to the visible light band, near-infrared band, and short-wave infrared band respectively; the common feature vector is bound to multiple specific feature vectors to obtain the associated feature vectors under different bands; S34: Through attention mechanism and preset labels, different weight values ​​are assigned to the associated feature vectors under different bands, and the associated feature vectors under different bands are weighted and fused.

[0034] Among them, a phase-correlation-based subpixel registration algorithm is adopted. Taking the visible light band with the richest information as the benchmark, nonlinear elastic transformation is performed on images of other bands in the near-infrared band and short-wave infrared band to achieve pixel-level precise alignment. Adaptive histogram normalization is performed on images of each band to eliminate global grayscale differences caused by differences in light intensity and sensor response, and to map data of different bands to a comparable feature value range.

[0035] The feature extraction model uses a shared shallow encoder to process all aligned different bands through the same lightweight 3-4 layer convolutional network. This network learns to extract basic common feature vectors, such as edges, corners, and patches. The feature extraction model then inputs the image of each band into its own independent deep encoder branch. Each branch is a deeper convolutional neural network that learns deep abstract features specific to that band and related to welding physics. For example, the visible light branch focuses on learning surface texture, color distribution, oxidation discoloration, and wetting corner contours, outputting specific feature vectors for the visible light band; the near-infrared branch focuses on learning subsurface features such as internal structural continuity, material density variations, void shadows, and microcrack scattering, outputting specific feature vectors for the near-infrared band; and the short-wave infrared branch focuses on learning thermally relevant features such as thermal radiation / reflection distribution and material composition differences, outputting specific feature vectors for the short-wave infrared band. The unique feature vectors corresponding to different bands are concatenated along their channel dimensions. A lightweight convolutional module then generates a spatial and channel-joint attention weight tensor W. This tensor has dimensions (C, H, W), where C is the total number of feature channels, and H and W are the spatial dimensions. An additional preset identifier is added, representing different solder joint type vectors. Different solder joint type vectors correspond to different weight ratios for the visible light, near-infrared, and short-wave infrared bands. The feature extraction model uses an attention mechanism to identify the solder joint type in the tested welding area and adjusts the weight ratios of the visible light, near-infrared, and short-wave infrared bands accordingly.

[0036] A 1x1 convolutional layer is used as a feature compressor and integrator to reduce the dimensionality and integrate the concatenated associated feature vectors, and finally output a unified multispectral fusion feature map.

[0037] S40: Through coordinate transformation, the fused feature map is spatially matched and aligned with the three-dimensional topography point cloud, and the geometric position information of the associated feature vector in the three-dimensional topography point cloud is output. In this embodiment, a perspective projection matrix is ​​established from 3D world coordinates to 2D image pixel coordinates. For each point in the point cloud, its sub-pixel level coordinates on the fused feature map are calculated using the perspective projection matrix. Bilinear interpolation is used to sample the associated feature vector corresponding to that point from the fused feature map, and this associated feature vector is attached as an attribute to the corresponding point in the 3D topographic point cloud. Thus, each point is upgraded to an enhanced point. Using a fixed-radius neighborhood method, neighboring nodes in 3D space are found for each enhanced point, and undirected edges are established between neighboring nodes, thereby constructing an attribute map that can express the 3D structure of the welding area, completing the matching and alignment between the fused feature map and the 3D topographic point cloud.

[0038] S50: Based on the distribution of the associated feature vectors and the corresponding geometric position information, identify and determine whether a defect feature has been formed; In this embodiment, geometric location information refers to the spectral reflectance / intensity values, three-dimensional spatial coordinates, local surface normal vectors, and curvature of each point in the three-dimensional topographic point cloud across multiple spectral bands. Defect features refer to the topographic features presented by the associated feature vectors and geometric location information fed back by different defect types.

[0039] Specifically, step S50 includes the following steps: S51: Based on the geometric position information, determine whether the associated feature vector at the location is abnormal. If it is abnormal, obtain the neighboring point group of the geometric position information in the three-dimensional topographic point cloud. S52: Identify the range of abnormal point groups in neighboring point groups where abnormal associated feature vectors appear; S53: When the range of abnormal points exceeds the preset range parameter, the current solder joint is judged to have formed a defect feature.

[0040] Among them, points with abnormal associated feature vectors in the three-dimensional topographic point cloud are identified, and it is further identified whether the neighboring point groups also have abnormal associated feature vectors. The neighboring point groups are divided according to the location of the weld point. By judging the range of the abnormal point group, it can be determined whether a defect feature has been formed. As the range of the abnormal point group expands, the confidence of the defect increases. If the range of the abnormal point group is smaller than the range parameter, it is not judged as a defect.

[0041] Furthermore, step S51 includes the following steps: S511: Determine the weld point to which the geometric position information belongs in the three-dimensional topography point cloud, and identify the weld point type of the weld point; S512: Based on the solder joint type, determine the distribution of associated feature vectors of all geometric position information in the solder joint; S513: Based on the distribution of associated feature vectors of all geometric position information in the current solder joint, determine whether there are any points in the solder joint with abnormal associated feature vectors; S514: If it exists, obtain the set of points associated with the anomaly in the 3D topographic point cloud, and take the points associated with the coordinates as the neighboring point group.

[0042] In this embodiment, an edge convolution module is used. For each point in the 3D topographic point cloud, it not only aggregates the attributes of neighboring nodes, but more importantly, calculates the relative geometric relationship between itself and its neighboring points and uses it as an edge feature. This enables the network to simultaneously perceive the local geometric structure and the local spectral distribution. Based on the weld type of the weld point where the set location information is located, it filters out significantly different associated feature vectors from the local geometric structure and the local spectral distribution. This allows the network to filter out points with associated feature vectors based on the distribution. Furthermore, graph attention pooling is introduced to adaptively filter out key region nodes most relevant to defects, such as the group of center points of suspected voided weld balls, thereby filtering out neighboring point groups.

[0043] S60: When a defect feature is determined to exist, based on a preset defect feature database, the defect type corresponding to the defect feature is selected and the quantitative information corresponding to the defect feature is calculated. The defect type and quantitative information are then sent to the user terminal.

[0044] Specifically, step S60 includes the following steps: S61: When it is determined that the aforementioned feature defect exists, identify the abnormal geometric region formed by the range of abnormal point groups; S62: Based on the combination of abnormal geometric regions and corresponding associated feature vectors, match the defect type of the current abnormal feature from the preset defect feature database; S63: Retrieve the defect calculation method that matches the defect type, and calculate the quantitative information corresponding to the defect type based on the abnormal geometric region and the corresponding associated feature vector.

[0045] In this embodiment, the determination of the defect type, for example, low near-infrared intensity combined with the abnormal geometric region being a local depression, indicates a high probability that the defect type is an internal void. A specific visible light color shift combined with the abnormal geometric region being an abnormally raised solder outline indicates a high probability that the defect type is a cold solder joint. Continuous short-wave infrared characteristics between adjacent solder joints combined with an abnormal geometric region acting as a bridging feature confirms a true bridging defect type.

[0046] Reference Figure 3 The calculation method of quantitative information is associated with the corresponding defect type. The calculation is performed through the defect quantification engine. For example, for void defects, the volume occupied by the point set classified as "void" is calculated by identifying the point set of nodes. The void rate is calculated as void volume / theoretical solder ball volume x 100%.

[0047] Bridging defects are identified by identifying a set of bridge-like points connecting two or more weld points that should be isolated, calculating the minimum cross-sectional area and minimum bridge height of the bridging region, and calculating the total volume of the bridging.

[0048] In the case of cold solder joints or incomplete solder joints, the system identifies the set of points on the surface of the solder joint that are predicted to be cold solder joints and have abnormal spectral characteristics. It then calculates the average curvature of the region and the average offset of the spectral characteristics of a specific band in that region.

[0049] In cases of insufficient or excessive solder, the total volume of the actual solder joint 3D point cloud is compared with the theoretical ideal volume obtained based on the CAD model or standard sample, and the percentage of volume deviation is calculated.

[0050] A report on the quantitative information of defects is generated and sent to the user. The report includes the quantitative values ​​of each defect, 3D visualization annotations, and suggestions on the correlation of process parameters.

[0051] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0052] In one embodiment, a visual inspection system for welding defects in electronic components based on multispectral imaging is provided. This system corresponds to the visual inspection method for welding defects in electronic components based on multispectral imaging described in the above embodiments. The multispectral imaging-based visual inspection system for welding defects in electronic components includes: The acquisition module is used to acquire multi-band sequence images of the welding area to be tested acquired by a preset multispectral imaging terminal when a data acquisition command is received. The multi-band sequence images include visible light band, near-infrared band and short-wave infrared band. The point cloud construction module is used to construct a three-dimensional topographic point cloud of the welding area to be tested based on the multi-band sequence images; The feature fusion module is used to extract the associated feature vectors related to welding quality from the multi-band sequence images based on preset feature recognition rules, and to weight and fuse the associated features under different bands to generate a fused feature map. The spectrum and morphology combination module is used to spatially match and align the fused feature map with the three-dimensional morphology point cloud through coordinate transformation, and output the geometric position information of the associated feature vector in the three-dimensional morphology point cloud; The defect judgment module is used to identify and determine whether a defect feature has been formed based on the distribution of the associated feature vector and the corresponding geometric position information; The defect analysis module is used to, when the existence of the defect feature is determined, filter out the defect type corresponding to the defect feature based on a preset defect feature database, calculate the quantitative information corresponding to the defect feature, and send the defect type and the quantitative information to the user terminal.

[0053] Optionally, point cloud building modules include: The static planar image acquisition submodule is used by the spectral imaging terminal to pre-acquire static planar images of the welding area to be tested in the visible light and near-infrared bands. The scanning submodule is used by the spectral imaging terminal to perform three-dimensional scanning of the welding area to be tested through the visible light band, and to perform phase constraint on the static planar image to generate a preliminary topographic point cloud. The anomaly cluster judgment submodule is used by the spectral imaging terminal to scan the welding area to be tested in the near-infrared band or short-wave infrared band to determine whether there are anomaly clusters in the preliminary topography point cloud. The mapping submodule is used to acquire, if present, point group images corresponding to the locations of the anomalous point groups in the near-infrared or short-wave infrared bands, and map the point group images to the locations of the anomalous point groups to cover them.

[0054] Optionally, the feature fusion module includes: The preprocessing submodule is used to denoise and perform phase registration of multi-band image sequences. The common feature extraction submodule is used to extract common feature vectors of different bands in multi-band sequence images in advance through a preset feature extraction model; The specific feature extraction submodule is used by the feature extraction model to extract specific feature vectors corresponding to the visible light band, near-infrared band, and short-wave infrared band respectively; and bind the common feature vector with multiple specific feature vectors to obtain the associated feature vectors under different bands. The weight setting submodule is used to assign different weight values ​​to the associated feature vectors under different bands through an attention mechanism and preset labels, and to perform weighted fusion of the associated feature vectors under different bands.

[0055] Optionally, the defect analysis module includes: The anomaly detection submodule is used to determine whether the associated feature vector at the current location is abnormal based on the geometric location information. If it is abnormal, the neighboring point group of the geometric location information in the three-dimensional topographic point cloud is obtained. The range identification submodule is used to identify the range of abnormal point groups in neighboring point groups where there are abnormal associated feature vectors; The defect determination submodule is used to determine that the current solder joint has formed a defect feature when the range of the abnormal point group exceeds the preset range parameter.

[0056] Optional, the exception detection submodule includes: The solder joint type determination unit is used to determine the solder joint to which the geometric position information belongs in the three-dimensional topographic point cloud, and to identify the solder joint type of the solder joint; The distribution unit is used to determine the distribution of associated feature vectors of all geometric position information in the solder joint based on the solder joint type. The anomaly detection unit is used to determine whether there are any points with abnormal associated feature vectors in the solder joint based on the distribution of associated feature vectors of all geometric position information in the current solder joint. The neighbor point group acquisition unit is used to acquire, if present, a set of points associated with the anomaly in the 3D topographic point cloud, and to take the points associated with the coordinates as the neighbor point group.

[0057] Optionally, the defect analysis module includes: The region determination submodule is used to identify the abnormal geometric region formed by the range of abnormal point groups when the aforementioned feature defects are determined to exist. The defect type determination submodule is used to match the defect type of the current abnormal feature from the preset defect feature database based on the combination of abnormal geometric regions and corresponding associated feature vectors. The quantization calculation submodule is used to retrieve the defect calculation method that matches the defect type, and calculate the quantization information corresponding to the defect type based on the abnormal geometric region and the corresponding associated feature vector.

[0058] Specific limitations regarding the visual inspection system for welding defects in electronic components based on multispectral imaging can be found in the limitations of the visual inspection method for welding defects in electronic components based on multispectral imaging mentioned above, and will not be repeated here. Each module in the aforementioned visual inspection system for welding defects in electronic components based on multispectral imaging can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0059] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a visual inspection method for welding defects in electronic components based on multispectral imaging.

[0060] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a visual inspection method for welding defects in electronic components based on multispectral imaging.

[0061] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements a visual inspection method for welding defects in electronic components based on multispectral imaging.

[0062] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0063] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0064] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A visual inspection method for welding defects in electronic components based on multispectral imaging, characterized in that: When a data acquisition command is received, a multi-band sequence image of the welding area to be tested is acquired by a preset multispectral imaging terminal. The multi-band sequence image includes visible light band, near-infrared band and short-wave infrared band. A three-dimensional topographic point cloud of the welding area to be tested is constructed based on the multi-band sequence images. Based on preset feature recognition rules, the associated feature vectors related to welding quality are extracted from the multi-band sequence images, and the associated features under different bands are weighted and fused to generate a fused feature map. By transforming coordinates, the fused feature map is spatially matched and aligned with the three-dimensional topography point cloud, and the geometric position information of the associated feature vector in the three-dimensional topography point cloud is output. Based on the distribution of the associated feature vectors and the corresponding geometric position information, identify and determine whether a defect feature has been formed; When the defect feature is determined to exist, based on a preset defect feature database, the defect type corresponding to the defect feature is selected and the quantitative information corresponding to the defect feature is calculated. The defect type and the quantitative information are then sent to the user terminal.

2. The method for visual inspection of welding defects in electronic components based on multispectral imaging according to claim 1, characterized in that, The step of constructing a three-dimensional topographic point cloud of the welding area to be tested based on the multi-band sequence images includes: The spectral imaging terminal pre-acquires static planar images of the welding area to be tested using visible light and near-infrared bands; The spectral imaging terminal performs a three-dimensional scan of the welding area under test using the visible light band, and performs phase constraint on the static planar image to generate a preliminary topographic point cloud. The spectral imaging terminal scans the welding area to be tested in the near-infrared band or the short-wave infrared band to determine whether there are abnormal point clusters in the preliminary morphology point cloud. If they exist, acquire the point cluster image corresponding to the location of the anomalous point cluster in the near-infrared band or short-wave infrared band, and map the point cluster image to the location of the anomalous point cluster to cover the anomalous point cluster.

3. The method for visual inspection of welding defects in electronic components based on multispectral imaging according to claim 1, characterized in that, The step of extracting the associated feature vectors related to welding quality from the multi-band sequence images based on preset feature recognition rules, and weighting and fusing the associated features under different bands to generate a fused feature map includes: Denoising and phase registration of multi-band image sequences; By using a pre-defined feature extraction model, common feature vectors of different bands in multi-band sequence images are extracted in advance; The feature extraction model then extracts specific feature vectors corresponding to the visible light band, near-infrared band, and short-wave infrared band respectively; the common feature vector is bound to multiple specific feature vectors to obtain the associated feature vectors under different bands; By using an attention mechanism and preset labels, different weight values ​​are assigned to the associated feature vectors under different bands, and the associated feature vectors under different bands are then weighted and fused.

4. The method for visual inspection of welding defects in electronic components based on multispectral imaging according to claim 3, characterized in that, In the step of assigning different weight values ​​to the associated feature vectors under different bands through the attention mechanism and preset labels, the preset labels include solder joint type vectors. Different solder joint type vectors correspond to different weight ratios of the visible light band, near-infrared band and short-wave infrared band. The feature extraction model uses the attention mechanism to identify the solder joint type of the welding area to be tested in order to adjust the weight ratio of the visible light band, near-infrared band and short-wave infrared band.

5. The method for visual inspection of welding defects in electronic components based on multispectral imaging according to claim 1, characterized in that, The step of identifying and determining whether a defect feature has been formed based on the distribution of the associated feature vector and the corresponding geometric position information includes: Based on the geometric location information, determine whether the associated feature vector at the location is abnormal. If it is abnormal, obtain the neighboring point group of the geometric location information in the three-dimensional topographic point cloud. Identify the range of anomalous point groups in neighboring point groups where abnormal associated feature vectors appear; When the range of the abnormal point group exceeds the preset range parameter, it is determined that the current solder joint has formed a defect feature.

6. The method for visual inspection of welding defects in electronic components based on multispectral imaging according to claim 5, characterized in that, The step of determining whether the associated feature vector at a given location is abnormal based on geometric location information, and if abnormal, obtaining the neighboring point group of the geometric location information in the 3D topographic point cloud, includes: Determine the weld point to which the geometric position information belongs in the three-dimensional topographic point cloud, and identify the weld point type; Based on the type of solder joint, determine the distribution of associated feature vectors of all geometric position information in the solder joint; Based on the distribution of associated feature vectors of all geometric position information in the current solder joint, determine whether there are any points in the solder joint with abnormal associated feature vectors; If they exist, obtain the set of points associated with the anomaly in the 3D topographic point cloud, and use the points associated with the coordinates as the neighboring point group.

7. The method for visual inspection of welding defects in electronic components based on multispectral imaging according to claim 6, characterized in that, The step of, when it is determined that the defect feature exists, filtering out the defect type corresponding to the defect feature based on a preset defect feature database and calculating the quantitative information corresponding to the defect feature, includes: When it is determined that the aforementioned characteristic defect exists, the abnormal geometric region formed by the range of abnormal point groups is identified. Based on the combination of abnormal geometric regions and corresponding associated feature vectors, the defect type of the current abnormal feature is matched from the preset defect feature database; The defect calculation method that matches the defect type is retrieved, and the quantitative information corresponding to the defect type is calculated based on the abnormal geometric region and the corresponding associated feature vector.

8. A visual inspection system for welding defects in electronic components based on multispectral imaging, characterized in that: The acquisition module is used to acquire multi-band sequence images of the welding area to be tested acquired by a preset multispectral imaging terminal when a data acquisition command is received. The multi-band sequence images include visible light band, near-infrared band and short-wave infrared band. The point cloud construction module is used to construct a three-dimensional topographic point cloud of the welding area to be tested based on the multi-band sequence images; The feature fusion module is used to extract the associated feature vectors related to welding quality from the multi-band sequence images based on preset feature recognition rules, and to weight and fuse the associated features under different bands to generate a fused feature map. The spectrum and morphology combination module is used to spatially match and align the fused feature map with the three-dimensional morphology point cloud through coordinate transformation, and output the geometric position information of the associated feature vector in the three-dimensional morphology point cloud; The defect judgment module is used to identify and determine whether a defect feature has been formed based on the distribution of the associated feature vector and the corresponding geometric position information; The defect analysis module is used to, when the existence of the defect feature is determined, filter out the defect type corresponding to the defect feature based on a preset defect feature database, calculate the quantitative information corresponding to the defect feature, and send the defect type and the quantitative information to the user terminal.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the visual inspection method for welding defects in electronic components based on multispectral imaging as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the visual inspection method for welding defects of electronic components based on multispectral imaging as described in any one of claims 1 to 7.

Images