Wafer packaging quality identification method and system based on image recognition

Abstract

The application relates to the technical field of image recognition, and discloses a wafer packaging quality identification method and system based on image recognition. A multispectral imaging unit is used to collect multispectral images of wafer packaging and perform fusion processing to obtain multispectral fusion images. An RGB-D unit is used to collect RGB color images of the wafer packaging and depth images of the wafer packaging. A three-dimensional reconstruction algorithm is used to perform fusion to generate three-dimensional images of the wafer packaging. Three-dimensional point cloud data is extracted from the three-dimensional images after preprocessing and is used to reconstruct a three-dimensional point cloud model. A mixed attention generation network is used to detect defect features in the three-dimensional point cloud model and output information of the defect features. The application can not only explain the detection results in detail, but also output targeted defect feature analysis and processing suggestions, thereby meeting the requirement of comprehensive evaluation of wafer packaging quality and improving the efficiency and accuracy of packaging quality evaluation.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to a wafer packaging quality identification method and system based on image recognition. Background Technology

[0002] In the semiconductor manufacturing industry, wafer production is a crucial step. Wafer packaging, as a protective measure for wafers during storage, transportation, and subsequent processing, directly affects the integrity and performance of the wafers. Traditional wafer packaging quality inspection relies heavily on manual visual inspection. Quality inspectors check the appearance of the wafer packaging with the naked eye to determine if the packaging material has cracks or scratches, and whether the wafers are placed neatly inside the packaging.

[0003] However, existing inspection methods have significant limitations. Manual inspection is inefficient, and with the booming development of the semiconductor industry and the continuous expansion of wafer production scale, the speed of manual inspection lags far behind the production pace. Furthermore, manual inspection is significantly affected by subjective factors; the experience and fatigue levels of different quality inspectors can lead to deviations in inspection results, making it difficult to guarantee accuracy and consistency. For example, some minor packaging defects may be difficult for humans to detect, allowing wafers with potential quality issues to enter subsequent production processes. Moreover, existing automated inspection methods are mostly based on contact measurement or simple sensor detection. Contact measurement may cause secondary damage to wafer packaging, affecting wafer quality, while simple sensor detection can only obtain limited packaging information and cannot comprehensively and accurately assess packaging quality. Summary of the Invention

[0004] This application provides a wafer packaging quality identification method and system based on image recognition. By using image recognition to identify wafer packaging quality, it achieves the requirement of comprehensive wafer packaging quality assessment and improves the efficiency and accuracy of wafer packaging quality assessment.

[0005] In a first aspect, this application provides a wafer packaging quality identification method based on image recognition, the wafer packaging quality identification method based on image recognition comprising:

[0006] Multispectral images of the wafer packaging are acquired using a multispectral imaging unit, and the multispectral images are fused to obtain a multispectral fused image.

[0007] The RGB color image and depth image of the wafer package are acquired using the RGB-D unit.

[0008] The multispectral fusion image, the RGB color image, and the depth image are fused using a 3D reconstruction algorithm to generate a 3D image of the wafer package.

[0009] The three-dimensional image is subjected to multimodal preprocessing, and three-dimensional point cloud data is extracted from the preprocessed three-dimensional image. Surface reconstruction is then performed on the three-dimensional point cloud data to generate a three-dimensional point cloud model.

[0010] The 3D point cloud model is input into a hybrid attention generation network, which detects defect features in the 3D point cloud model and outputs information about the defect features.

[0011] Secondly, this application provides an image recognition-based wafer packaging quality identification system for implementing the image recognition-based wafer packaging quality identification method, wherein the image recognition-based wafer packaging quality identification system includes:

[0012] A multispectral acquisition module is used to acquire multispectral images of the wafer package through a multispectral imaging unit, and to perform fusion processing on the multispectral images to obtain a multispectral fused image;

[0013] An RGB-D acquisition module is used to acquire RGB color images and depth images of the wafer package through an RGB-D unit.

[0014] A three-dimensional fusion module is used to fuse the multispectral fusion image, the RGB color image, and the depth image using a three-dimensional reconstruction algorithm to generate a three-dimensional image of the wafer package.

[0015] A three-dimensional reconstruction module is used to perform multimodal preprocessing on the three-dimensional image, extract three-dimensional point cloud data from the preprocessed three-dimensional image, and perform surface reconstruction on the three-dimensional point cloud data to generate a three-dimensional point cloud model.

[0016] A defect detection module is used to input the 3D point cloud model into a hybrid attention generation network, and to detect defect features in the 3D point cloud model and output the information of the defect features through the hybrid attention generation network.

[0017] This application has at least the following technical effects:

[0018] In the technical solution provided in this application, a multispectral imaging unit acquires multispectral images of the wafer package, and the multispectral images are fused to obtain a multispectral fused image. An RGB-D unit acquires RGB color images and depth images of the wafer package. A 3D reconstruction algorithm fuses the multispectral fused image, the RGB color image, and the depth image to generate a 3D image of the wafer package. The 3D image undergoes multimodal preprocessing, and 3D point cloud data is extracted from the preprocessed 3D image. Surface reconstruction is then performed on the 3D point cloud data to generate a 3D point cloud model. The 3D point cloud model is input into a hybrid attention generation network, which detects defect features in the 3D point cloud model and outputs the defect feature information. The multispectral imaging unit and RGB-D unit are used to acquire RGB color images and depth images of the wafer package. The D unit acquires multispectral images, RGB color images, and depth images of the wafer packaging. Multispectral images can capture spectral information invisible to the naked eye, reflecting subtle differences in packaging materials. RGB color images provide rich color and texture details, while depth images provide information about three-dimensional space. By fusing these images of different modalities through a 3D reconstruction algorithm, the generated 3D image of the wafer packaging contains more comprehensive information. Compared to a single image data source, this greatly improves the completeness of the description of wafer packaging features. In addition, the hybrid attention generation network detects defect features in the 3D point cloud model and outputs information about these defect features. This not only provides a detailed interpretation of the detection results but also outputs targeted defect feature analysis and processing suggestions. This application fulfills the requirements for comprehensive quality assessment of wafer packaging, improving the efficiency and accuracy of packaging quality assessment. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic flowchart of an embodiment of the wafer packaging quality identification method based on image recognition in this application.

[0021] Figure 2 This is a schematic diagram of an embodiment of the wafer packaging quality identification system based on image recognition in this application. Detailed Implementation

[0022] This application provides a wafer packaging quality identification method and system based on image recognition. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0023] For ease of understanding, the specific process of the embodiments of this application is described below. Please refer to [link / reference]. Figure 1 One embodiment of the wafer packaging quality identification method based on image recognition in this application includes:

[0024] Step S101: Acquire multispectral images of the wafer packaging using a multispectral imaging unit, and perform fusion processing on the multispectral images to obtain a multispectral fused image;

[0025] Step S102: Acquire RGB color images and depth images of the wafer package using the RGB-D unit;

[0026] Step S103: The multispectral fusion image, the RGB color image, and the depth image are fused using a 3D reconstruction algorithm to generate a 3D image of the wafer package;

[0027] Step S104: Perform multimodal preprocessing on the three-dimensional image, extract three-dimensional point cloud data from the preprocessed three-dimensional image, and perform surface reconstruction on the three-dimensional point cloud data to generate a three-dimensional point cloud model.

[0028] Step S105: Input the three-dimensional point cloud model into the hybrid attention generation network, and use the hybrid attention generation network to detect the defect features in the three-dimensional point cloud model and output the information of the defect features.

[0029] In one specific embodiment, the multispectral imaging unit includes an adjustable polarization angle ring light source array and a multispectral camera. During the acquisition process, the ring light source array surrounds the wafer package, and the polarization angle is adjusted according to actual needs. Different polarization angles can change the reflection and scattering characteristics of light on the surface of the wafer package, which helps to obtain more detailed material information. The multispectral camera is used to capture the light reflected by the wafer package to record image information in different bands. The combination of the adjustable polarization angle ring light source array and the multispectral camera can acquire multispectral images. The multispectral images contain the reflection characteristics of the wafer package under different spectra, providing a rich data foundation for subsequent analysis.

[0030] In one specific embodiment, the acquired multispectral images are subject to interference from various noises, such as electronic noise and environmental noise. Noise can affect the quality of multispectral images and the accuracy of subsequent analysis. Therefore, wavelet transform algorithm is used to denoise the multispectral images. The wavelet transform algorithm can decompose the multispectral images into sub-bands of different frequencies. By thresholding the wavelet coefficients, the high-frequency coefficients corresponding to noise can be effectively removed while retaining the main features of the multispectral images. Ideally, the multispectral images are first decomposed into wavelet coefficients of different scales and directions. Then, according to certain threshold rules, the high-frequency wavelet coefficients are filtered and adjusted. Finally, the processed wavelet coefficients are reconstructed into the denoised multispectral image through inverse wavelet transform.

[0031] In this embodiment, the image of each band of the denoised multispectral image reflects the reflection intensity of the wafer packaging at a specific wavelength. In order to more accurately analyze the material characteristics of the wafer packaging, the reflectance characteristics of each band are calculated, and then the pixel values ​​in each band image are compared with a known reference standard (such as a white board) to obtain the reflectance of each pixel in the corresponding band. The reflectance characteristic data of all bands are arranged in wavelength order to obtain the spectral reflectance curve. The spectral reflectance curve shows the reflectance variation of the wafer packaging at different wavelengths, which is convenient for identifying different materials and detecting defects.

[0032] In addition to spectral reflectance features, multispectral images also include edge and texture information. Edge detection and texture analysis are performed on the denoised multispectral images to extract multi-scale features. Edge detection algorithms are used to identify the position of object edges in the multispectral image. Then, by calculating the gradient changes of pixel gray values ​​in the multispectral image, points with larger gradient amplitudes are identified as edge points. Statistical algorithms are used to analyze texture, including the direction, frequency, and contrast features of texture in the multispectral image. Edge detection and texture analysis are performed at multiple scales. For example, subtle edges and textures are detected at small scales, while the overall structural features of the wafer packaging can be grasped at large scales. By extracting and fusing edge and texture features at different scales, the surface characteristics of the wafer packaging can be described more comprehensively.

[0033] Ideally, spectral reflectance curves and multi-scale features are fused. The resulting feature vector has a high dimension, but the presence of redundant information increases the computational burden of subsequent processing. Principal Component Analysis (PCA) is used to process the fused features. PCA transforms the feature vector linearly, projecting it onto a new orthogonal basis. This maximizes the variance of the feature vector data in the new coordinate system, retaining the principal components with higher variance and removing those with lower variance, thus achieving dimensionality reduction. PCA not only reduces data dimensionality and computational complexity but also removes noise and redundant information to some extent, improving data quality and analysis efficiency. The resulting multispectral fused image contains optimized and fused spectral and texture features, providing more comprehensive and accurate data for subsequent wafer packaging quality identification.

[0034] In one specific embodiment, a histogram equalization algorithm is used to enhance the contrast of the acquired RGB color image. The RGB color image consists of three color channels: red (R), green (G), and blue (B). Each channel has its corresponding pixel value distribution. The histogram equalization algorithm redistributes the pixel values ​​of the image to make the grayscale histogram of the image as uniformly distributed as possible throughout the grayscale range. In the RGB color image, the histogram equalization algorithm is performed on the red, green, and blue channels respectively. First, the frequency of different pixel values ​​in each channel is counted to construct a histogram. Then, according to a certain mapping rule, the original pixel values ​​are mapped to new pixel values, making the new histogram distribution more uniform. The image with low contrast and unclear details becomes brighter and darker after histogram equalization, and the details in the image are more clearly displayed. For example, some areas on the surface of a wafer packaging that were originally difficult to distinguish due to similar colors can be more clearly distinguished in terms of their boundaries and textures after contrast enhancement, which helps in the subsequent identification and analysis of the surface features of the packaging.

[0035] The depth image records the distance information from each point on the wafer packaging surface to the camera. However, during the acquisition process, various noise interferences will generate some noise points. These noise points will affect the accuracy of the depth information. For noise points in the depth image, the pixel values ​​often differ greatly from the surrounding normal pixel values. The median filtering algorithm can replace the abnormal values ​​of the noise points with the normal pixel values ​​in the neighborhood, thereby effectively removing noise, smoothing the depth image, and making the depth image information more accurate and reliable.

[0036] Furthermore, for cases where there are missing regions in the depth image, i.e., the depth values ​​of some pixels are lost, the nearest neighbor interpolation algorithm can be used to fill in these missing regions. For example, in a two-dimensional depth image matrix, if the depth value of a certain pixel is missing, the nearest pixel is found by calculating its distance to surrounding pixels with known depth values ​​(such as Euclidean distance), and its depth value is copied to the missing pixel. Through the nearest neighbor interpolation algorithm, the integrity of the depth image can be restored to a certain extent, providing more complete data for subsequent 3D reconstruction and analysis based on depth information.

[0037] The 3D reconstruction algorithm transforms the depth values ​​of a depth image into 3D spatial coordinates. By analyzing the depth value of each pixel in the depth image, the X, Y, and Z coordinates of the corresponding pixel in 3D space are obtained. A set of several 3D coordinate points constitutes a point cloud surface, displaying the 3D geometry of the wafer packaging and providing a spatial framework for subsequent fusion of other image information. Multispectral fused images and RGB color images are projected onto the point cloud surface. The normalized vegetation index (NDI) of the multispectral image and the texture features of the RGB image are extracted, respectively, providing material and texture information for subsequent analysis. Principal component analysis (PCA) is used to fuse and reduce the NDI and texture features, removing redundant information and improving the representativeness and processing efficiency of the features. Using the Phong lighting model, considering ambient light, diffuse light, and specular light, the illumination intensity is calculated and fused with color information. A realistic 3D image is generated by the rendering engine, showcasing the overall appearance of the wafer packaging.

[0038] In one specific embodiment, multimodal preprocessing is performed on the 3D image, including geometric correction, denoising, data normalization, and multimodal information fusion enhancement, which improves image quality and comprehensive information expression capability. Then, 3D point cloud data is extracted from the preprocessed 3D image, and a 3D point cloud model is generated through registration and surface reconstruction. The 3D point cloud model is then optimized by smoothing and hole filling. Finally, the 3D point cloud model is input into a hybrid attention generation network. First, multimodal feature precoding and compression are performed to obtain point cloud feature vectors. The generator of the hybrid attention generation network generates defect features, and the discriminator of the hybrid attention generation network detects and judges the defect feature information and outputs it, providing information basis for subsequent analysis and processing.

[0039] In one specific embodiment, the process of performing step S101 may specifically include the following steps:

[0040] (1) Multispectral images of the wafer package are acquired by the adjustable polarization angle ring light source array in the multispectral imaging unit and the multispectral camera in the multispectral imaging unit;

[0041] (2) The multispectral image is denoised using a wavelet transform algorithm;

[0042] (3) Calculate the band reflectance characteristics of the denoised multispectral image to obtain the spectral reflectance curve;

[0043] (4) Perform edge detection and texture analysis on the band images of the denoised multispectral image and extract multi-scale features;

[0044] (5) The spectral reflectance curve and the multi-scale features are fused together and the multispectral fusion image is generated by principal component analysis dimensionality reduction algorithm.

[0045] In one specific embodiment, the multispectral imaging unit includes an adjustable polarization angle ring light source array and a multispectral camera. During the acquisition process, the ring light source array surrounds the wafer package, and the polarization angle is adjusted according to actual needs. Different polarization angles can change the reflection and scattering characteristics of light on the surface of the wafer package, which helps to obtain more detailed material information. The multispectral camera is used to capture the light reflected by the wafer package to record image information in different bands. The combination of the adjustable polarization angle ring light source array and the multispectral camera can acquire multispectral images. The multispectral images contain the reflection characteristics of the wafer package under different spectra, providing a rich data foundation for subsequent analysis.

[0046] In one specific embodiment, the acquired multispectral images are subject to interference from various noises, such as electronic noise and environmental noise. Noise can affect the quality of multispectral images and the accuracy of subsequent analysis. Therefore, wavelet transform algorithm is used to denoise the multispectral images. The wavelet transform algorithm can decompose the multispectral images into sub-bands of different frequencies. By thresholding the wavelet coefficients, the high-frequency coefficients corresponding to noise can be effectively removed while retaining the main features of the multispectral images. Ideally, the multispectral images are first decomposed into wavelet coefficients of different scales and directions. Then, according to certain threshold rules, the high-frequency wavelet coefficients are filtered and adjusted. Finally, the processed wavelet coefficients are reconstructed into the denoised multispectral image through inverse wavelet transform.

[0047] In this embodiment, the image of each band of the denoised multispectral image reflects the reflection intensity of the wafer packaging at a specific wavelength. In order to more accurately analyze the material characteristics of the wafer packaging, the reflectance characteristics of each band are calculated, and then the pixel values ​​in each band image are compared with a known reference standard (such as a white board) to obtain the reflectance of each pixel in the corresponding band. The reflectance characteristic data of all bands are arranged in wavelength order to obtain the spectral reflectance curve. The spectral reflectance curve shows the reflectance variation of the wafer packaging at different wavelengths, which is convenient for identifying different materials and detecting defects.

[0048] In addition to spectral reflectance features, multispectral images also include edge and texture information. Edge detection and texture analysis are performed on the denoised multispectral images to extract multi-scale features. Edge detection algorithms are used to identify the position of object edges in the multispectral image. Then, by calculating the gradient changes of pixel gray values ​​in the multispectral image, points with larger gradient amplitudes are identified as edge points. Statistical algorithms are used to analyze texture, including the direction, frequency, and contrast features of texture in the multispectral image. Edge detection and texture analysis are performed at multiple scales. For example, subtle edges and textures are detected at small scales, while the overall structural features of the wafer packaging can be grasped at large scales. By extracting and fusing edge and texture features at different scales, the surface characteristics of the wafer packaging can be described more comprehensively.

[0049] Ideally, spectral reflectance curves and multi-scale features are fused. The resulting feature vector has a high dimension, but the presence of redundant information increases the computational burden of subsequent processing. Principal Component Analysis (PCA) is used to process the fused features. PCA transforms the feature vector linearly, projecting it onto a new orthogonal basis. This maximizes the variance of the feature vector data in the new coordinate system, retaining the principal components with higher variance and removing those with lower variance, thus achieving dimensionality reduction. PCA not only reduces data dimensionality and computational complexity but also removes noise and redundant information to some extent, improving data quality and analysis efficiency. The resulting multispectral fused image contains optimized and fused spectral and texture features, providing more comprehensive and accurate data for subsequent wafer packaging quality identification.

[0050] In one specific embodiment, the process of performing step S102 may specifically include the following steps:

[0051] (1) Enhance the contrast of the RGB color image by using a histogram equalization algorithm;

[0052] (2) Remove noise points in the depth image by median filtering algorithm and fill in the missing regions in the depth image by nearest neighbor interpolation algorithm.

[0053] In one specific embodiment, a histogram equalization algorithm is used to enhance the contrast of the acquired RGB color image. The RGB color image consists of three color channels: red (R), green (G), and blue (B). Each channel has its corresponding pixel value distribution. The histogram equalization algorithm redistributes the pixel values ​​of the image to make the grayscale histogram of the image as uniformly distributed as possible throughout the grayscale range. In the RGB color image, the histogram equalization algorithm is performed on the red, green, and blue channels respectively. First, the frequency of different pixel values ​​in each channel is counted to construct a histogram. Then, according to a certain mapping rule, the original pixel values ​​are mapped to new pixel values, making the new histogram distribution more uniform. The image with low contrast and unclear details becomes brighter and darker after histogram equalization, and the details in the image are more clearly displayed. For example, some areas on the surface of a wafer packaging that were originally difficult to distinguish due to similar colors can be more clearly distinguished in terms of their boundaries and textures after contrast enhancement, which helps in the subsequent identification and analysis of the surface features of the packaging.

[0054] Depth images record the distance information from various points on the wafer packaging surface to the camera. However, during acquisition, various noise interferences can generate noise points, affecting the accuracy of the depth information. For noise points in depth images, pixel values ​​often differ significantly from surrounding normal pixel values. Median filtering algorithms can replace the abnormal values ​​of noise points with normal pixel values ​​in the neighborhood, effectively removing noise, smoothing the depth image, and making the depth image information more accurate and reliable. Furthermore, for cases where there are missing regions in the depth image, i.e., some pixels have lost depth values, nearest neighbor interpolation algorithms can fill these missing regions. For example, in a two-dimensional depth image matrix, if a pixel's depth value is missing, by calculating its distance to surrounding pixels with known depth values ​​(such as Euclidean distance), the nearest pixel is found, and its depth value is copied to the missing pixel. Through nearest neighbor interpolation, the integrity of the depth image can be restored to a certain extent, providing more complete data for subsequent 3D reconstruction and analysis based on depth information.

[0055] In one specific embodiment, the process of performing step S103 may specifically include the following steps:

[0056] (1) The depth image is used to generate a point cloud surface using the three-dimensional reconstruction algorithm;

[0057] (2) Project the multispectral fused image and the RGB color image onto the point cloud surface and extract the normalized vegetation index of the multispectral image and the texture features of the RGB color image;

[0058] (3) The normalized vegetation index and the texture features are fused using principal component analysis dimensionality reduction algorithm to generate dimensionality-reduced fused feature data;

[0059] (4) The point cloud surface and the dimension-reduced fusion feature data are rendered using the Phong lighting model to generate a three-dimensional image of the wafer package.

[0060] In one specific embodiment, the step of rendering the point cloud surface and the dimensionality-reduced fusion feature data using the Phong lighting model to generate a 3D image of the wafer package specifically includes:

[0061] (1) The data of the point cloud surface is converted into a three-dimensional mesh model using the Poisson reconstruction algorithm;

[0062] (2) Associate the dimensionality reduction and fusion feature data with the three-dimensional mesh model to generate a three-dimensional geometric composite model;

[0063] (3) Calculate the ambient light intensity, diffuse light intensity and specular light intensity of the Phong illumination model, and add the ambient light intensity, diffuse light intensity and specular light intensity to obtain the total illumination intensity;

[0064] (4) The total illumination intensity is fused with the color information in the dimension reduction fusion feature data to obtain a mixed color value. The mixed color value is then rendered with the three-dimensional geometric composite model through the rendering engine to generate a three-dimensional image of the wafer package.

[0065] Specifically, the depth value of each pixel in the depth image is converted into coordinates in three-dimensional space through a 3D reconstruction algorithm. Ideally, the X, Y, and Z coordinates of each pixel in the depth image are calculated in three-dimensional space using a triangulation algorithm. Several 3D coordinate points are combined to form a point cloud surface. The point cloud surface intuitively presents the three-dimensional geometry of the wafer packaging, which helps to more accurately reconstruct the true three-dimensional shape of the wafer packaging.

[0066] By projecting multispectral fused images and RGB color images onto the generated point cloud surface, each pixel in the 2D image corresponds to a point on the point cloud surface, achieving alignment of different modal image information in 3D space. Combining the information from the multispectral fused images and RGB color images with the geometric information of the point cloud surface allows subsequent analysis to comprehensively consider multiple features. Among these features, the normalized vegetation index and texture characteristics are helpful in detecting defects that are difficult to detect with the naked eye, such as tiny scratches and material inhomogeneities on the wafer packaging surface, thus improving the accuracy of defect detection.

[0067] Next, the extracted normalized vegetation index and texture features are used to construct a high-dimensional feature vector. Since the high-dimensional feature vector has a high dimension and contains redundant information, it will increase the computational complexity of subsequent processing. The PCA dimensionality reduction algorithm transforms the high-dimensional feature vector into dimensionality-reduced fused feature data by performing a linear transformation. Specifically, the covariance matrix of the high-dimensional feature vector is calculated, and then the eigenvalues ​​and eigenvectors of the covariance matrix are solved. The eigenvector with the larger eigenvalue is selected as the new basis, and the original feature vector is projected onto the new basis to obtain the dimensionality-reduced fused feature data. This realizes the fusion and dimensionality reduction of the normalized vegetation index and texture features. The dimensionality-reduced fused feature data retains the main information of the original features, is more compact and representative, and helps to improve the accuracy of analysis.

[0068] In this embodiment, the Phong lighting model includes ambient light, diffuse light, and specular light. Ambient light is a uniform illumination whose intensity is determined by the ambient light coefficient and ambient light color. Diffuse light simulates the scattering effect of light on a rough surface. For each point on the point cloud surface, the intensity of diffuse light is calculated based on the normal vector and the direction of the light source. The intensity of diffuse light is proportional to the cosine of the incident angle. Specular light simulates the reflection effect of light on a smooth surface. The intensity of specular light is calculated based on the observation direction and the direction of the reflected light. The intensity of specular light is related to the angle between the reflected light and the observation direction. The total illumination intensity is obtained by adding the intensities of ambient light, diffuse light, and specular light. Then, the total illumination intensity is fused with the color information in the dimensionality reduction and fusion feature data. The rendering engine then colors each point on the point cloud surface to generate a 3D image of the wafer package with realistic lighting effects. The generated 3D image has realistic lighting effects, which are more in line with human visual perception. It allows for more intuitive observation and analysis of the appearance and structure of the wafer package. In addition, appropriate lighting effects can highlight defects on the surface of the wafer package, such as dents and protrusions, which helps to more accurately identify and locate defects, thereby improving the efficiency and accuracy of quality inspection.

[0069] In one specific embodiment, the multimodal preprocessing of the three-dimensional image includes:

[0070] (1) Perform geometric correction on the three-dimensional image, and perform noise reduction processing on the geometrically corrected three-dimensional image using a filtering algorithm;

[0071] (2) Normalize the data of the denoised three-dimensional image so that the data of different dimensions are in the same quantity range, and fuse and enhance the multiple modal information of the three-dimensional image.

[0072] In one specific embodiment, the step of extracting three-dimensional point cloud data from the preprocessed three-dimensional image and performing surface reconstruction on the three-dimensional point cloud data to generate a three-dimensional point cloud model includes:

[0073] (1) Extract the three-dimensional point cloud data using the corresponding algorithm according to the representation of the preprocessed three-dimensional image;

[0074] (2) The three-dimensional point cloud data is registered using the iterative nearest point algorithm;

[0075] (3) The registered three-dimensional point cloud data is used to perform surface reconstruction by solving the Poisson equation to generate the three-dimensional point cloud model;

[0076] (4) The three-dimensional point cloud model is smoothed by the Laplace smoothing algorithm and the holes in the three-dimensional point cloud model are filled by the hole filling algorithm.

[0077] Among them, geometric correction restores the geometry of the 3D image to normal, providing a foundation for subsequent accurate analysis. Denoising processing improves the clarity and quality of the 3D image, reduces the interference of noise on feature extraction and analysis, and provides more accurate identification and analysis of information in the 3D image.

[0078] 3D images contain data in different dimensions, such as depth and color information. Normalization maps data from different dimensions to the same range of values, such as [0, 1]. By calculating the maximum and minimum values, each data point in the 3D image undergoes a linear transformation. Furthermore, 3D images include multimodal information, such as depth, color, and texture. Integrating this multimodal information achieves fusion enhancement, making each modality complementary. Normalization ensures comparability of data from different dimensions, improving accuracy. Multimodal information fusion enhancement fully utilizes various information sources, providing more comprehensive image features and enhancing the understanding and analysis of 3D image content.

[0079] Furthermore, based on the representation of the preprocessed 3D image, 3D point cloud data is extracted using corresponding algorithms. For depth maps, the depth value of each pixel is converted into coordinates in 3D space. For RGB-D images, in addition to depth information, color information is also associated with 3D coordinates, thus achieving accurate extraction of 3D point cloud data. This can convert information in 2D images into a set of points in 3D space, providing data support for constructing 3D point cloud models.

[0080] In one specific embodiment, the Iterative Closest Point (ICP) algorithm is used to register 3D point cloud data. The ICP algorithm iteratively searches for the optimal transformation between two sets of point clouds, minimizing the distance between corresponding points. Specifically, it first finds corresponding points in both sets of point clouds, then calculates the transformation matrix, transforms one set of point clouds, and then searches for corresponding points again, repeating this process until convergence. Point cloud registration aligns point cloud data acquired from different viewpoints or at different times to the same coordinate system, providing complete and accurate point cloud data for subsequent surface reconstruction, making the reconstructed model more accurate and coherent.

[0081] Next, the registered 3D point cloud data is reconstructed by solving the Poisson equation. An implicit function is constructed based on the normal vector information of the point cloud. The specific form of this implicit function is determined by solving the Poisson equation. Then, the isosurface is extracted to obtain the 3D point cloud model. Poisson surface reconstruction can generate a continuous and smooth surface based on the point cloud data, accurately restore the geometry of the object, and generate a high-quality 3D point cloud model.

[0082] Ideally, the Laplacian smoothing algorithm is used to calculate the average value of the neighboring vertices of each vertex in the 3D point cloud model. This adjusts the vertex positions, making the surface of the 3D point cloud model smoother. It reduces noise and irregularities on the surface, improving the quality of the 3D point cloud model. The hole-filling algorithm is used to fill holes in the 3D point cloud model. For example, based on the point cloud information around the hole, the point cloud data inside the hole is estimated through interpolation or fitting, thus filling the hole. The Laplacian smoothing algorithm makes the surface of the 3D point cloud model smoother, conforming to the surface characteristics of real objects. Hole filling eliminates voids in the 3D point cloud model, making it more complete and improving its usability and visualization.

[0083] In one specific embodiment, the process of executing step S105 may specifically include the following steps:

[0084] (1) The three-dimensional point cloud model is pre-encoded with multimodal features and compressed using an autoencoder to generate point cloud feature vectors;

[0085] (2) The point cloud feature vector is input into the generator of the hybrid attention generation network to generate the defect features, and the discriminator of the hybrid attention generation network detects and judges the information of the defect features;

[0086] (3) Construct a knowledge graph based on the information of the defect features;

[0087] (4) After the hybrid attention generation network outputs the detection results of the defect features, the detection results are interpreted through the knowledge graph, and the defect feature analysis and processing suggestions are output through the graph reasoning algorithm.

[0088] Specifically, the process involves constructing a knowledge graph based on the information of the defect features; after the hybrid attention generation network outputs the detection results of the defect features, the detection results are interpreted through the knowledge graph, and a graph reasoning algorithm is used to output analysis and processing suggestions for the defect features, including:

[0089] (1) Extract the entities and the relationships between them from the information of the defect features;

[0090] (2) Construct the knowledge graph by using the entities and relationships through a graph database;

[0091] (3) Map the detection results of the defect features output by the hybrid attention generation network to entities and relations in the knowledge graph, and perform matching queries in the knowledge graph to obtain an explanation of the detection results;

[0092] (4) The analysis results are obtained by analyzing and summarizing the defect features through the graph reasoning algorithm;

[0093] (5) Based on the analysis results, extract the corresponding processing measures from the knowledge graph and generate processing suggestions.

[0094] Specifically, a 3D point cloud model contains multiple modal information, such as shape, color, and texture. Multimodal feature precoding extracts and encodes features from multiple modal information, converting features from different modalities into a unified feature representation. Then, an autoencoder is used to compress the encoded features. Multimodal feature precoding integrates multiple types of information, making the features more representative. The compression operation of the autoencoder reduces data dimensionality and computational complexity, while retaining key information, which helps with efficient subsequent processing.

[0095] Next, the point cloud feature vectors are input into the generator of the hybrid attention generation network. The generator learns and generates possible defect features based on the input feature vectors, while the discriminator detects and judges the generated defect features, determining whether they are real defects and their type, location, and other information. The generator and discriminator are continuously optimized through adversarial training. The generator generates more realistic defect features, while the discriminator strives to accurately distinguish between real and generated features. The hybrid attention generation network can effectively mine defect features in 3D point cloud models. Through adversarial learning between the generator and discriminator, the accuracy and robustness of defect detection are improved, enabling the detection of various types of potential defects.

[0096] Furthermore, entities are identified from defect feature information, such as different types of defects and components of wafer packaging. At the same time, the relationships between these entities are determined, for example, scratches appear on the outer shell. Entities and relationships are extracted and stored in a graph database. The graph database uses nodes to represent entities and edges to represent relationships, constructing an intuitive knowledge graph that clearly shows the associations of defect-related information. The knowledge graph structures the defect feature information, which not only facilitates subsequent queries and reasoning but also integrates various defect-related knowledge, providing a foundation for interpreting detection results and providing processing suggestions.

[0097] In one specific embodiment, the defect feature detection results output by the hybrid attention generation network are mapped to entities and relationships in a knowledge graph. For example, if the detection result shows the presence of scratches, entities and relationships related to "scratches" are searched in the knowledge graph for matching queries, thereby obtaining a detailed explanation of the detection result, such as the possible causes and effects of the scratches. In addition, the defect features are analyzed in depth through graph reasoning algorithms. Based on the relationships and rules in the knowledge graph, the graph reasoning algorithm infers and summarizes the defect information to obtain more comprehensive analysis results, such as the development trend of defects and other problems that may be caused. Finally, based on the analysis results, corresponding processing measures are extracted from the knowledge graph. For example, if it is a minor scratch, the knowledge graph may record grinding and repair processing measures. These processing measures are organized into processing suggestions and output. The knowledge graph provides interpretability for the detection results and can give the causes and effects of the defects. The analysis and processing suggestions generated by the graph reasoning algorithm provide a scientific and reasonable basis for decision-making, which helps to deal with wafer packaging defects in a timely and effective manner, and improve production quality and efficiency.

[0098] The image recognition-based wafer packaging quality identification method in the embodiments of this application has been described above. The image recognition-based wafer packaging quality identification system in the embodiments of this application is described below. The image recognition-based wafer packaging quality identification method can be applied to the image recognition-based wafer packaging quality identification system. Please refer to [link to relevant documentation]. Figure 2 The image recognition-based wafer packaging quality identification system includes:

[0099] A multispectral acquisition module 201 is used to acquire multispectral images of the wafer package through a multispectral imaging unit, and to perform fusion processing on the multispectral images to obtain a multispectral fused image.

[0100] RGB-D acquisition module 202, the RGB-D acquisition module 202 is used to acquire RGB color images and depth images of the wafer package through RGB-D units;

[0101] The three-dimensional fusion module 203 is used to fuse the multispectral fusion image, the RGB color image and the depth image through a three-dimensional reconstruction algorithm to generate a three-dimensional image of the wafer package;

[0102] The three-dimensional reconstruction module 204 is used to perform multimodal preprocessing on the three-dimensional image, extract three-dimensional point cloud data from the preprocessed three-dimensional image, and perform surface reconstruction on the three-dimensional point cloud data to generate a three-dimensional point cloud model.

[0103] The defect detection module 205 is used to input the three-dimensional point cloud model into the hybrid attention generation network, detect the defect features in the three-dimensional point cloud model through the hybrid attention generation network, and output the information of the defect features.

[0104] In this embodiment, a multispectral imaging unit acquires multispectral images of the wafer package, and the multispectral images are fused to obtain a multispectral fused image. An RGB-D unit acquires RGB color images and depth images of the wafer package. A 3D reconstruction algorithm fuses the multispectral fused image, the RGB color image, and the depth image to generate a 3D image of the wafer package. The 3D image undergoes multimodal preprocessing, and 3D point cloud data is extracted from the preprocessed 3D image. Surface reconstruction is then performed on the 3D point cloud data to generate a 3D point cloud model. The 3D point cloud model is input into a hybrid attention generation network, which detects defect features in the 3D point cloud model and outputs the defect feature information. The multispectral imaging unit and the RGB-D unit are used in this process. Multispectral images, RGB color images, and depth images of the wafer packaging are acquired separately. Multispectral images can capture spectral information invisible to the naked eye, reflecting subtle differences in packaging materials. RGB color images provide rich color and texture details, and depth images provide information on three-dimensional spatial dimensions. These images of different modalities are fused through a 3D reconstruction algorithm, and the generated 3D image of the wafer packaging contains more comprehensive information. Compared with a single image data source, this greatly improves the completeness of the description of wafer packaging features. In addition, the hybrid attention generation network is used to detect defect features in the 3D point cloud model and output the defect feature information. This not only provides a detailed interpretation of the detection results but also outputs targeted defect feature analysis and processing suggestions. This application realizes the requirement for comprehensive evaluation of wafer packaging quality and improves the efficiency and accuracy of packaging quality evaluation.

[0105] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0106] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An image recognition-based wafer package quality identification method, characterized by, The wafer packaging quality identification method based on image recognition comprises: A multispectral imaging unit is used to acquire a multispectral image of the wafer packaging, and the multispectral image is fused to obtain a multispectral fusion image; An RGB-D unit is used to acquire an RGB color image of the wafer packaging and a depth image of the wafer packaging; A three-dimensional reconstruction algorithm is used to fuse the multispectral fusion image, the RGB color image and the depth image to generate a three-dimensional image of the wafer packaging, which comprises: the three-dimensional reconstruction algorithm is used to generate a point cloud surface from the depth image; the multispectral fusion image and the RGB color image are projected onto the point cloud surface, and the normalized vegetation index of the multispectral image and the texture features of the RGB color image are extracted; a principal component analysis dimension reduction algorithm is used to fuse the normalized vegetation index and the texture features to generate reduced dimension fusion feature data; a Phong lighting model is used to render the point cloud surface and the reduced dimension fusion feature data to generate the three-dimensional image of the wafer packaging; The three-dimensional image is preprocessed in multiple modes, three-dimensional point cloud data is extracted from the preprocessed three-dimensional image, and a three-dimensional point cloud model is generated by surface reconstruction of the three-dimensional point cloud data; The three-dimensional point cloud model is input into a hybrid attention generation network, and the hybrid attention generation network is used to detect defect features in the three-dimensional point cloud model and output information of the defect features.

2. The image recognition-based wafer package quality identification method according to claim 1, characterized in that, The multispectral imaging unit is used to acquire a multispectral image of the wafer packaging, and the multispectral image is fused to obtain a multispectral fusion image, which comprises: An annular light source array with adjustable polarization angle in the multispectral imaging unit and a multispectral camera in the multispectral imaging unit are used to acquire the multispectral image of the wafer packaging; A wavelet transform algorithm is used to denoise the multispectral image; The band reflectance features of the denoised multispectral image are calculated to obtain a spectral reflectance curve; Edge detection and texture analysis are performed on the band image of the denoised multispectral image, and multiscale features are extracted; The spectral reflectance curve and the multiscale features are fused, and a principal component analysis dimension reduction algorithm is used to generate the multispectral fusion image.

3. The image recognition-based wafer package quality identification method according to claim 1, characterized in that, After the RGB-D unit is used to acquire the RGB color image of the wafer packaging and the depth image of the wafer packaging, the following steps are included: A histogram equalization algorithm is used to enhance the contrast of the RGB color image; A median filter algorithm is used to remove noise points in the depth image, and a nearest neighbor interpolation algorithm is used to complete the missing areas in the depth image.

4. The image recognition-based wafer package quality identification method according to claim 1, characterized in that, The Phong lighting model is used to render the point cloud surface and the reduced dimension fusion feature data to generate the three-dimensional image of the wafer packaging, which comprises: A Poisson reconstruction algorithm is used to convert the data of the point cloud surface into a three-dimensional mesh model; The reduced dimension fusion feature data is associated with the three-dimensional mesh model to generate a three-dimensional geometric composite model; computing ambient light intensity, diffuse light intensity and specular light intensity of the Phong lighting model, and adding the ambient light intensity, the diffuse light intensity and the specular light intensity to obtain total light intensity; fusing the total light intensity with color information in the reduced dimension fusion feature data to obtain mixed color values, and rendering the mixed color values and the three-dimensional geometric composite model through a rendering engine to generate a three-dimensional image of the wafer package.

5. The image recognition-based wafer package quality identification method according to claim 1, characterized in that, The multi-modal preprocessing of the three-dimensional image comprises: performing geometric correction on the three-dimensional image, and performing denoising processing on the three-dimensional image after geometric correction through a filtering algorithm; normalizing data of the three-dimensional image after denoising processing to make data of different dimensions in the same quantity interval, and fusing and enhancing various modal information of the three-dimensional image.

6. The image recognition-based wafer package quality identification method according to claim 5, wherein, The three-dimensional point cloud data is extracted from the preprocessed three-dimensional image, and a three-dimensional point cloud model is generated by surface reconstruction on the three-dimensional point cloud data, comprising: extracting the three-dimensional point cloud data according to the representation of the preprocessed three-dimensional image using a corresponding algorithm; aligning the three-dimensional point cloud data based on an iterative closest point algorithm; generating the three-dimensional point cloud model by surface reconstruction by solving Poisson equation on the three-dimensional point cloud data after alignment; smoothing the three-dimensional point cloud model by Laplace smoothing algorithm and filling holes in the three-dimensional point cloud model by hole filling algorithm.

7. The image recognition-based wafer package quality identification method according to claim 1, wherein, The three-dimensional point cloud model is input into a hybrid attention generation network, the defect features in the three-dimensional point cloud model are detected by the hybrid attention generation network, and the information of the defect features is output, comprising: The three-dimensional point cloud model is pre-encoded by a multi-modal feature and compressed by a self-encoder to generate a point cloud feature vector; The point cloud feature vector is input into the generator of the hybrid attention generation network to generate the defect features, and the discriminator of the hybrid attention generation network detects and judges the information of the defect features; constructing a knowledge graph according to the information of the defect features; After the hybrid attention generation network outputs the detection result of the defect features, the detection result is explained through the knowledge graph, and the defect feature analysis and processing suggestion is output through the graph reasoning algorithm.

8. The image recognition-based wafer package quality identification method according to claim 7, characterized in that, The knowledge graph is constructed according to the information of the defect features, and after the hybrid attention generation network outputs the detection result of the defect features, the detection result is explained through the knowledge graph, and the defect feature analysis and processing suggestion is output through the graph reasoning algorithm, comprising: extracting entities and relationships between the entities from the information of the defect features; constructing the knowledge graph through a graph database by the entities and the relationships; mapping the detection result of the defect features output by the hybrid attention generation network to the entities and the relationships in the knowledge graph, and performing matching query in the knowledge graph to obtain the explanation of the detection result; analyzing and summarizing the defect features through the graph reasoning algorithm to obtain an analysis result; According to the analysis result, corresponding treatment measures are extracted from the knowledge graph and a treatment suggestion is generated.

9. An image recognition-based wafer package quality identification system for implementing the image recognition-based wafer package quality identification method according to any one of claims 1-8, characterized in that, The wafer packaging quality identification system based on image recognition comprises: A multispectral acquisition module is configured to acquire a multispectral image of the wafer packaging through a multispectral imaging unit and perform fusion processing on the multispectral image to obtain a multispectral fusion image. An RGB-D acquisition module is configured to acquire an RGB color image of the wafer packaging and a depth image of the wafer packaging through an RGB-D unit. A three-dimensional fusion module is configured to perform fusion on the multispectral fusion image, the RGB color image and the depth image through a three-dimensional reconstruction algorithm to generate a three-dimensional image of the wafer packaging. A three-dimensional reconstruction module is configured to perform multi-modal preprocessing on the three-dimensional image, extract three-dimensional point cloud data from the preprocessed three-dimensional image and perform surface reconstruction on the three-dimensional point cloud data to generate a three-dimensional point cloud model. A defect detection module is configured to input the three-dimensional point cloud model into a hybrid attention generation network, detect defect features in the three-dimensional point cloud model through the hybrid attention generation network and output information of the defect features.

Images