Wafer packaging quality identification method and system based on image identification

Abstract

The invention relates to the technical field of image recognition, and discloses a wafer package quality recognition method and system based on image recognition, and the method comprises the steps: collecting a multispectral image of a wafer package through a multispectral imaging unit, and carrying out the fusion processing of the multispectral image, and obtaining a multispectral fusion image; the method comprises the following steps: acquiring an RGB color image of a wafer package and a depth image of the wafer package through an RGB-D unit, fusing through a three-dimensional reconstruction algorithm to generate a three-dimensional image of the wafer package, extracting three-dimensional point cloud data from the preprocessed three-dimensional image, and reconstructing to generate a three-dimensional point cloud model; the defect features in the three-dimensional point cloud model are detected through the mixed attention generation network, and information of the defect features is output, the detection result can be explained in detail, targeted defect feature analysis and processing suggestions can be output, the requirement for comprehensive evaluation of the wafer packaging quality is met, and the wafer packaging quality evaluation efficiency is improved. And the high efficiency and accuracy of packaging quality evaluation are improved.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a wafer packaging quality recognition method and system based on image recognition. Background Art

[0002] In the semiconductor manufacturing industry, wafer production is a critical step. Wafer packaging, a protective method for wafers during storage, transportation, and subsequent processing, has a direct impact on the integrity and performance of the wafers. Traditional wafer packaging quality inspection relies heavily on manual visual inspection. Inspectors visually inspect the packaging to determine if there are any cracks or scratches in the packaging material, and whether the wafers are properly positioned within the packaging.

[0003] However, existing inspection methods have significant limitations. Manual inspection is inefficient, and with the booming semiconductor industry and the ever-expanding scale of wafer production, manual inspection lags far behind the production pace. Furthermore, manual inspection is significantly affected by subjective factors. The experience and fatigue of different quality inspectors can lead to deviations in inspection results, making it difficult to ensure accuracy and consistency. For example, subtle packaging defects can be difficult for humans to detect, allowing wafers with quality risks to enter subsequent production processes. Furthermore, some existing automated inspection methods rely on contact measurement or simple sensor detection. Contact measurement can cause secondary damage to wafer packaging, affecting wafer quality, while simple sensor detection can only obtain limited packaging information and cannot comprehensively and accurately assess packaging quality. Summary of the Invention

[0004] The present application provides a wafer packaging quality identification method and system based on image recognition, which identifies wafer packaging quality through image recognition, realizes the requirement of comprehensive evaluation of wafer packaging quality, and improves the efficiency and accuracy of wafer packaging quality evaluation.

[0005] In a first aspect, the present application provides a wafer packaging quality identification method based on image recognition, the wafer packaging quality identification method based on image recognition comprising: Collecting a multispectral image of the wafer package by a multispectral imaging unit, and fusing the multispectral image to obtain a multispectral fused image; Acquire an RGB color image of the wafer package and a depth image of the wafer package through an RGB-D unit; fusing the multispectral fusion image, the RGB color image, and the depth image using a three-dimensional reconstruction algorithm to generate a three-dimensional image of the wafer package; Performing multimodal preprocessing on the three-dimensional image, extracting three-dimensional point cloud data from the preprocessed three-dimensional image, and performing surface reconstruction on the three-dimensional point cloud data to generate a three-dimensional point cloud model; The three-dimensional point cloud model is input into a hybrid attention generation network, and the hybrid attention generation network is used to detect defect features in the three-dimensional point cloud model and output information about the defect features.

[0006] In a second aspect, the present application provides a wafer packaging quality identification system based on image recognition, which is used to implement the wafer packaging quality identification method based on image recognition. The wafer packaging quality identification system based on image recognition includes: a multispectral acquisition module, configured to acquire a multispectral image of the wafer package through a multispectral imaging unit, and fuse the multispectral image to obtain a multispectral fused image; an RGB-D acquisition module, the RGB-D acquisition module being configured to acquire an RGB color image of the wafer package and a depth image of the wafer package through an RGB-D unit; a three-dimensional fusion module, configured to fuse the multispectral fusion image, the RGB color image, and the depth image using a three-dimensional reconstruction algorithm to generate a three-dimensional image of the wafer package; a three-dimensional reconstruction module, configured to perform multimodal preprocessing on the three-dimensional image, extract three-dimensional point cloud data from the preprocessed three-dimensional image, and perform surface reconstruction on the three-dimensional point cloud data to generate a three-dimensional point cloud model; A defect detection module is used to input the three-dimensional point cloud model into a hybrid attention generation network, detect defect features in the three-dimensional point cloud model through the hybrid attention generation network, and output information about the defect features.

[0007] This application has at least the following technical effects: In the technical solution provided by the present application, a multispectral image of the wafer package is collected by a multispectral imaging unit, and the multispectral image is fused to obtain a multispectral fused image, an RGB color image of the wafer package and a depth image of the wafer package are collected by an RGB-D unit, the multispectral fused image, the RGB color image and the depth image are fused by a 3D reconstruction algorithm to generate a 3D image of the wafer package, the 3D image is multimodally preprocessed, and 3D point cloud data is extracted from the preprocessed 3D image and surface reconstruction is performed on the 3D point cloud data to generate a 3D point cloud model, the 3D point cloud model is input into a hybrid attention generation network, and the defect features in the 3D point cloud model are detected by the hybrid attention generation network and the information of the defect features is output, wherein, with the help of the multispectral imaging unit and the RGB- The D unit collects multispectral images, RGB color images and depth images of the wafer packaging respectively. The multispectral image can capture spectral information that is invisible to the naked eye and reflect the subtle differences in the packaging materials. The RGB color image provides rich color and texture details, and the depth image provides information of the three-dimensional spatial dimension. These images of different modalities are fused through the three-dimensional reconstruction algorithm. The generated three-dimensional image of the wafer packaging contains more comprehensive information. Compared with a single image data source, the completeness of the description of the wafer packaging features is greatly improved. In addition, the hybrid attention generation network is used to detect the defect features in the three-dimensional point cloud model and output the information of the defect features. It can not only provide a detailed explanation of the detection results, but also output targeted defect feature analysis and processing suggestions. This application realizes the requirements for comprehensive evaluation of wafer packaging quality and improves the efficiency and accuracy of packaging quality evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0009] Figure 1 This is a flow chart of an embodiment of a wafer packaging quality identification method based on image recognition in an embodiment of the present application; Figure 2 This is a structural diagram of an embodiment of a wafer packaging quality identification system based on image recognition in an embodiment of the present application. DETAILED DESCRIPTION

[0010] The embodiments of the present application provide a method and system for wafer packaging quality identification based on image recognition. The terms "first," "second," "third," "fourth," and so on (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way are interchangeable where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatus.

[0011] For ease of understanding, the specific process of the embodiment of the present application is described below. Figure 1 In the embodiment of the present application, an embodiment of a wafer packaging quality identification method based on image recognition includes: Step S101: collecting a multispectral image of the wafer package by a multispectral imaging unit, and fusing the multispectral image to obtain a multispectral fused image; Step S102: collecting an RGB color image and a depth image of the wafer package through an RGB-D unit; Step S103: fusing the multispectral fusion image, the RGB color image, and the depth image using a three-dimensional reconstruction algorithm to generate a three-dimensional image of the wafer package; Step S104: performing multimodal preprocessing on the three-dimensional image, extracting three-dimensional point cloud data from the preprocessed three-dimensional image, and performing surface reconstruction on the three-dimensional point cloud data to generate a three-dimensional point cloud model; Step S105: input the three-dimensional point cloud model into a hybrid attention generation network, detect defect features in the three-dimensional point cloud model through the hybrid attention generation network, and output information about the defect features.

[0012] In a specific embodiment, the multispectral imaging unit includes a ring-shaped light source array with an adjustable polarization angle and a multispectral camera. During the acquisition process, the ring-shaped light source array surrounds the wafer package, and the polarization angle is adjusted according to actual needs. Different polarization angles can change the reflection and scattering characteristics of light on the surface of the wafer package, which helps to obtain more detailed material information. The multispectral camera is used to capture light reflected by the wafer package to record image information in different bands. The combination of the ring-shaped light source array with an adjustable polarization angle and the multispectral camera can capture multispectral images. The multispectral images contain the reflection characteristics of the wafer package under different spectra, providing a rich data basis for subsequent analysis.

[0013] In a specific embodiment, the collected multispectral image has the problem of being interfered by various noises, such as electronic noise and environmental noise. The noise will affect the quality of the multispectral image and the accuracy of subsequent analysis. Therefore, a wavelet transform algorithm is used to denoise the multispectral image. The wavelet transform algorithm can decompose the multispectral image into sub-bands of different frequencies. By performing threshold processing on the wavelet coefficients, the high-frequency coefficients corresponding to the noise can be effectively removed while retaining the main features of the multispectral image. Optimally, the multispectral image is first subjected to wavelet decomposition to obtain wavelet coefficients of different scales and directions. Then, according to certain threshold rules, the high-frequency wavelet coefficients are screened and adjusted. Finally, the processed wavelet coefficients are reconstructed into a denoised multispectral image through inverse wavelet transform.

[0014] In this embodiment, each band of the denoised multispectral image reflects the reflectance intensity of the wafer package at a specific wavelength. To more accurately analyze the material properties of the wafer package, the reflectance characteristics of each band are calculated. The pixel values in each band image are then compared with a known reference standard (such as a whiteboard) to obtain the reflectance of each pixel in the corresponding band. The reflectance characteristic data of all bands are arranged in order of wavelength to obtain a spectral reflectance curve. The spectral reflectance curve shows the changes in the reflectance of the wafer package at different wavelengths, facilitating the identification of different materials and defect detection.

[0015] In addition to spectral reflectance features, the band images of multispectral images also include edge and texture information. Edge detection and texture analysis are performed on the denoised multispectral images to extract multi-scale features. The edge detection algorithm is used to identify the location of the edge of the object in the multispectral image. The gradient change of the pixel grayscale value in the multispectral image is then calculated to find the points with the largest gradient amplitude as the edge point. The texture is analyzed using a statistical algorithm, including the direction, frequency and contrast characteristics of the texture in the multispectral image. Edge detection and texture analysis are performed at multiple scales. For example, subtle edges and textures can be detected at a small scale, while the overall structural characteristics of the wafer package can be grasped at a large scale. By extracting and fusing edge and texture features at different scales, the surface characteristics of the wafer package can be more comprehensively described.

[0016] The best approach is to fuse the spectral reflectance curve and multi-scale features. The dimension of the fused feature vector is high, but the presence of redundant information will increase the computational complexity of subsequent processing. The fused features are processed using the principal component analysis (PCA) dimensionality reduction algorithm. The PCA dimensionality reduction algorithm performs a linear transformation on the feature vector and projects it onto a new set of orthogonal bases, so that the feature vector data has the largest variance in the new coordinate system, retains the principal components with larger variance, and removes the components with smaller variance, thereby achieving dimensionality reduction of the feature vector data. The PCA dimensionality reduction algorithm can not only reduce the dimension of the data and reduce the computational complexity, but also remove noise and redundant information to a certain extent, thereby improving the data quality and analysis efficiency. The multispectral fusion image finally generated contains optimized and fused spectral and texture features, providing more comprehensive and accurate data for subsequent wafer packaging quality identification.

[0017] In one specific embodiment, a histogram equalization algorithm is used to perform contrast enhancement on a captured RGB color image. An RGB color image consists of three color channels: red (R), green (G), and blue (B), each with its corresponding pixel value distribution. The histogram equalization algorithm redistributes the image pixel values so that the image's grayscale histogram is as evenly distributed as possible across the entire grayscale range. Within the RGB color image, the histogram equalization algorithm is performed on each of the three channels: the red, green, and blue. First, the frequency of occurrence of different pixel values in each channel is counted to construct a histogram. Then, according to a specific mapping rule, the original pixel values are mapped to new pixel values, resulting in a more uniform distribution of the new histogram. In an image with originally low contrast and unclear details, after histogram equalization, bright areas are brighter and dark areas are darker, resulting in clearer details within the image. For example, some areas on the surface of a wafer package that were originally similar in color and difficult to distinguish can now have their boundaries and textures more clearly distinguished after contrast enhancement, facilitating subsequent recognition and analysis of package surface features.

[0018] Among them, the depth image records the distance information from each point on the wafer packaging surface to the camera. However, during the acquisition process, it will be interfered by various noises and produce some noise points. These noise points will affect the accuracy of the depth information. For the noise points in the depth image, the pixel values are often quite different from the surrounding normal pixel values. The median filtering algorithm can replace the abnormal values of the noise points with the normal pixel values in the neighborhood, thereby effectively removing noise and smoothing the depth image, making the depth image information more accurate and reliable.

[0019] In addition, for the case where there are some missing areas in the depth image, that is, the depth values of some pixels are lost, these missing areas are filled by the nearest neighbor interpolation algorithm. For example: in a two-dimensional depth image matrix, if the depth value of a pixel is missing, by calculating its distance to the surrounding pixels with known depth values (such as Euclidean distance), the nearest pixel is found and its depth value is copied to the missing pixel. Through the nearest neighbor interpolation algorithm, the integrity of the depth image can be restored to a certain extent, providing more complete data for subsequent three-dimensional reconstruction and analysis based on depth information. The 3D reconstruction algorithm converts the depth values of the depth image into 3D spatial coordinates. By analyzing the depth value of each pixel in the depth image, the corresponding X, Y, and Z coordinates in 3D space are derived. A collection of these 3D coordinate points forms a point cloud surface, revealing the 3D geometry of the wafer package and providing a spatial framework for subsequent integration with other image information. The multispectral fusion image and the RGB color image are projected onto the point cloud surface. The normalized vegetation index of the multispectral image and the texture features of the RGB image are extracted, respectively, providing material and texture information for subsequent analysis. The normalized vegetation index and texture features are fused and reduced using the principal component analysis dimensionality reduction algorithm to remove redundant information, improve feature representation, and enhance processing efficiency. The Phong illumination model comprehensively considers ambient light, diffuse reflection, and specular reflection, calculates light intensity, and fuses it with color information. The rendering engine then generates a realistic 3D image, showcasing the entire wafer package.

[0020] In a specific embodiment, multimodal preprocessing is performed on the three-dimensional image, including geometric correction, denoising, data normalization, and multimodal information fusion enhancement, which improves the image quality and comprehensive information expression capability. Then, three-dimensional point cloud data is extracted from the preprocessed three-dimensional image, and a three-dimensional point cloud model is generated through alignment and surface reconstruction. The three-dimensional point cloud model is then smoothed and optimized for hole filling. Finally, the three-dimensional point cloud model is input into a hybrid attention generation network, and multimodal feature precoding and compression are first performed to obtain a point cloud feature vector. Defect features are generated by the generator of the hybrid attention generation network, and the discriminator of the hybrid attention generation network detects and judges the defect feature information and outputs it, providing an information basis for subsequent analysis and processing.

[0021] In a specific embodiment, the process of executing step S101 may specifically include the following steps: (1) collecting a multispectral image of the wafer package by using a polarization-angle-adjustable annular light source array in the multispectral imaging unit and a multispectral camera in the multispectral imaging unit; (2) performing denoising on the multispectral image by using a wavelet transform algorithm; (3) calculating the band reflectance characteristics of the multispectral image after denoising to obtain a spectral reflectance curve; (4) performing edge detection and texture analysis on the band images of the multispectral image after denoising and extracting multi-scale features; (5) Fusing the spectral reflectance curve with the multi-scale features and generating the multi-spectral fusion image through a principal component analysis dimensionality reduction algorithm.

[0022] In a specific embodiment, the multispectral imaging unit includes a ring-shaped light source array with an adjustable polarization angle and a multispectral camera. During the acquisition process, the ring-shaped light source array surrounds the wafer package, and the polarization angle is adjusted according to actual needs. Different polarization angles can change the reflection and scattering characteristics of light on the surface of the wafer package, which helps to obtain more detailed material information. The multispectral camera is used to capture light reflected by the wafer package to record image information in different bands. The combination of the ring-shaped light source array with an adjustable polarization angle and the multispectral camera can capture multispectral images. The multispectral images contain the reflection characteristics of the wafer package under different spectra, providing a rich data basis for subsequent analysis.

[0023] In a specific embodiment, the collected multispectral image has the problem of being interfered by various noises, such as electronic noise and environmental noise. The noise will affect the quality of the multispectral image and the accuracy of subsequent analysis. Therefore, a wavelet transform algorithm is used to denoise the multispectral image. The wavelet transform algorithm can decompose the multispectral image into sub-bands of different frequencies. By performing threshold processing on the wavelet coefficients, the high-frequency coefficients corresponding to the noise can be effectively removed while retaining the main features of the multispectral image. Optimally, the multispectral image is first subjected to wavelet decomposition to obtain wavelet coefficients of different scales and directions. Then, according to certain threshold rules, the high-frequency wavelet coefficients are screened and adjusted. Finally, the processed wavelet coefficients are reconstructed into a denoised multispectral image through inverse wavelet transform.

[0024] In this embodiment, each band of the denoised multispectral image reflects the reflectance intensity of the wafer package at a specific wavelength. To more accurately analyze the material properties of the wafer package, the reflectance characteristics of each band are calculated. The pixel values in each band image are then compared with a known reference standard (such as a whiteboard) to obtain the reflectance of each pixel in the corresponding band. The reflectance characteristic data of all bands is arranged in order of wavelength to produce a spectral reflectance curve. The spectral reflectance curve shows the changes in the reflectance of the wafer package at different wavelengths, facilitating the identification of different materials and defect detection.

[0025] In addition to spectral reflectance features, the band images of multispectral images also include edge and texture information. Edge detection and texture analysis are performed on the denoised multispectral images to extract multi-scale features. The edge detection algorithm is used to identify the location of the edge of the object in the multispectral image. The gradient change of the pixel grayscale value in the multispectral image is then calculated to find the points with the largest gradient amplitude as the edge point. The texture is analyzed using a statistical algorithm, including the direction, frequency and contrast characteristics of the texture in the multispectral image. Edge detection and texture analysis are performed at multiple scales. For example, subtle edges and textures can be detected at a small scale, while the overall structural characteristics of the wafer package can be grasped at a large scale. By extracting and fusing edge and texture features at different scales, the surface characteristics of the wafer package can be more comprehensively described.

[0026] The best approach is to fuse the spectral reflectance curve and multi-scale features. The dimension of the fused feature vector is high, but the presence of redundant information will increase the computational complexity of subsequent processing. The fused features are processed using the principal component analysis (PCA) dimensionality reduction algorithm. The PCA dimensionality reduction algorithm performs a linear transformation on the feature vector and projects it onto a new set of orthogonal bases, so that the feature vector data has the largest variance in the new coordinate system, retains the principal components with larger variance, and removes the components with smaller variance, thereby achieving dimensionality reduction of the feature vector data. The PCA dimensionality reduction algorithm can not only reduce the dimension of the data and reduce the computational complexity, but also remove noise and redundant information to a certain extent, thereby improving the data quality and analysis efficiency. The multispectral fusion image finally generated contains optimized and fused spectral and texture features, providing more comprehensive and accurate data for subsequent wafer packaging quality identification.

[0027] In a specific embodiment, the process of executing step S102 may specifically include the following steps: (1) enhancing the contrast of the RGB color image by using a histogram equalization algorithm; (2) removing noise points in the depth image by a median filtering algorithm and completing the missing areas in the depth image by a nearest neighbor interpolation algorithm.

[0028] In one specific embodiment, a histogram equalization algorithm is used to perform contrast enhancement on a captured RGB color image. An RGB color image consists of three color channels: red (R), green (G), and blue (B), each with its corresponding pixel value distribution. The histogram equalization algorithm redistributes the image pixel values so that the image's grayscale histogram is as evenly distributed as possible across the entire grayscale range. Within the RGB color image, the histogram equalization algorithm is performed on each of the three channels: the red, green, and blue. First, the frequency of occurrence of different pixel values in each channel is counted to construct a histogram. Then, according to a specific mapping rule, the original pixel values are mapped to new pixel values, resulting in a more uniform distribution of the new histogram. In an image with originally low contrast and unclear details, after histogram equalization, bright areas are brighter and dark areas are darker, resulting in clearer details within the image. For example, some areas on the surface of a wafer package that were originally similar in color and difficult to distinguish can now have their boundaries and textures more clearly distinguished after contrast enhancement, facilitating subsequent recognition and analysis of package surface features.

[0029] The depth image records the distance information from each point on the wafer packaging surface to the camera. However, during the acquisition process, various noise interferences will generate some noise points, which will affect the accuracy of the depth information. For noise points in the depth image, the pixel values are often significantly different from the normal pixel values in the surrounding area. The median filtering algorithm can replace the outlier values of the noise points with the normal pixel values in the neighborhood, effectively removing the noise and smoothing the depth image, making the depth image information more accurate and reliable. In addition, when there are some missing areas in the depth image, that is, the depth values of some pixels are missing, these missing areas are filled using the nearest neighbor interpolation algorithm. For example, in a two-dimensional depth image matrix, if the depth value of a pixel is missing, the distance between it and the surrounding pixels with known depth values (such as the Euclidean distance) is calculated to find the nearest pixel and copy its depth value to the missing pixel. The nearest neighbor interpolation algorithm can restore the integrity of the depth image to a certain extent, providing more complete data for subsequent 3D reconstruction and analysis based on depth information. In a specific embodiment, the process of executing step S103 may specifically include the following steps: (1) generating a point cloud surface from the depth image using the three-dimensional reconstruction algorithm; (2) Projecting the multispectral fusion image and the RGB color image onto the point cloud surface and extracting the normalized vegetation index of the multispectral image and the texture features of the RGB color image; (3) fusing the normalized vegetation index and the texture feature through a principal component analysis dimensionality reduction algorithm to generate dimensionality reduction fusion feature data; (4) Rendering the point cloud surface and the dimensionality reduction fusion feature data using a Phong illumination model to generate a three-dimensional image of the wafer package.

[0030] In a specific embodiment, rendering the point cloud surface and the dimensionality reduction fusion feature data using a Phong illumination model to generate a three-dimensional image of the wafer package specifically includes: (1) converting the point cloud surface data into a three-dimensional mesh model through a Poisson reconstruction algorithm; (2) Associating the dimension reduction fusion feature data with the three-dimensional grid model to generate a three-dimensional geometric composite model; (3) calculating the ambient light intensity, diffuse reflection light intensity, and specular reflection light intensity of the Phong illumination model, and adding the ambient light intensity, the diffuse reflection light intensity, and the specular reflection light intensity to obtain a total illumination intensity; (4) Fusing the total light intensity with the color information in the dimensionality reduction fusion feature data to obtain a mixed color value, and rendering the mixed color value and the three-dimensional geometric composite model through a rendering engine to generate a three-dimensional image of the wafer package.

[0031] Specifically, a 3D reconstruction algorithm is used to convert the depth value of each pixel in the depth image into coordinates in 3D space. Optimally, a triangulation algorithm is used to calculate the X, Y, and Z coordinates of each pixel in the depth image. Several 3D coordinate points are combined to form a point cloud surface. The point cloud surface intuitively presents the 3D geometric shape of the wafer package, helping to more accurately reconstruct the true 3D form of the wafer package.

[0032] By projecting the multispectral fusion image and the RGB color image onto the generated point cloud surface, each pixel in the 2D image corresponds to a point on the point cloud surface, achieving three-dimensional alignment of image information from different modalities. Combining the information from the multispectral fusion image and the RGB color image with the geometric information of the point cloud surface allows subsequent analysis to comprehensively consider multiple features. The Normalized Difference Vegetation Index and texture features are particularly useful for detecting defects that are difficult to detect with the naked eye, such as tiny scratches and uneven materials on wafer packaging, thereby improving the accuracy of defect detection.

[0033] Next, the extracted normalized vegetation index and texture features are used to form a high-dimensional feature vector. Since the high-dimensional feature vector has a high dimension and contains redundant information, it will increase the computational complexity of subsequent processing. The PCA dimensionality reduction algorithm converts the high-dimensional feature vector into dimensionality-reduced fusion feature data by performing a linear transformation on the high-dimensional feature vector. Specifically, the covariance matrix of the high-dimensional feature vector is calculated, and then the eigenvalues and eigenvectors of the covariance matrix are solved. The eigenvector with a larger eigenvalue is selected as the new basis, and the original eigenvector is projected onto the new basis to obtain the reduced-dimensionality fusion feature data. This realizes the fusion and dimensionality reduction of the normalized vegetation index and texture features. The reduced-dimensionality fusion feature data retains the main information of the original features, is more compact and representative, and helps to improve the accuracy of analysis.

[0034] In this embodiment, the Phong lighting model includes ambient light, diffuse light, and specular light. Ambient light is a uniform lighting whose intensity is determined by the ambient light coefficient and the ambient light color. Diffuse light simulates the scattering effect of light on a rough surface. For each point on the point cloud surface, the intensity of the diffuse light is calculated based on the normal vector and the direction of the light source. The intensity of the diffuse light is proportional to the cosine value of the incident angle. Specular light simulates the reflection effect of light on a smooth surface. The intensity of the specular light is calculated based on the observation direction and the direction of the reflected light. The intensity of the specular light is related to the angle between the reflected light and the observation direction. The intensities of ambient light, diffuse light, and specular light are added together to obtain the total light intensity. This total light intensity is then fused with the color information in the dimensionality-reduced fusion feature data. Each point on the point cloud surface is colored using a rendering engine to generate a three-dimensional image of the wafer package with realistic lighting effects. The generated three-dimensional image has realistic lighting effects that are more consistent with the human eye's visual perception, enabling more intuitive observation and analysis of the appearance and structure of the wafer package. Furthermore, appropriate lighting effects can highlight surface defects on the wafer package, such as dents and bumps, helping to more accurately identify and locate defects and improve the efficiency and accuracy of quality inspection.

[0035] In a specific embodiment, the performing multimodal preprocessing on the three-dimensional image includes: (1) performing geometric correction on the three-dimensional image, and performing denoising on the geometrically corrected three-dimensional image by using a filtering algorithm; (2) Normalizing the denoised three-dimensional image data so that data of different dimensions are in the same quantity range, and fusing and enhancing the multiple modal information of the three-dimensional image.

[0036] In a specific embodiment, extracting three-dimensional point cloud data from the pre-processed three-dimensional image and performing surface reconstruction on the three-dimensional point cloud data to generate a three-dimensional point cloud model includes: (1) extracting the three-dimensional point cloud data using a corresponding algorithm according to the representation of the preprocessed three-dimensional image; (2) registering the three-dimensional point cloud data based on an iterative closest point algorithm; (3) performing surface reconstruction on the registered three-dimensional point cloud data by solving the Poisson equation to generate the three-dimensional point cloud model; (4) Smoothing the three-dimensional point cloud model using a Laplace smoothing algorithm and filling holes in the three-dimensional point cloud model using a hole filling algorithm.

[0037] Among them, geometric correction restores the geometric shape of the three-dimensional image to normal, providing a basis for subsequent accurate analysis. De-noising processing improves the clarity and quality of the three-dimensional image, reduces the interference of noise on feature extraction and analysis, and provides more accurate identification and analysis of information in the three-dimensional image.

[0038] 3D images contain data of different dimensions, such as depth and color. Normalization maps these different dimensional data to the same numerical range, such as [0, 1]. Each data point in the 3D image is linearly transformed by calculating its maximum and minimum values. Furthermore, 3D images contain information from multiple modalities, such as depth, color, and texture. Integrating these modalities for fusion enhancement allows each modality to complement the other. Normalization makes data of different dimensions comparable, improving accuracy. Multimodal information fusion enhancement fully utilizes this diverse information, providing more comprehensive image features and enhancing understanding and analysis of 3D image content.

[0039] In addition, three-dimensional point cloud data is extracted through corresponding algorithms based on the representation of the preprocessed three-dimensional image. For the depth map, the depth value of each pixel is converted into coordinates in the three-dimensional space. For the RGB-D image, in addition to the depth information, the color information is also associated with the three-dimensional coordinates, thereby achieving accurate extraction of three-dimensional point cloud data and converting the information in the two-dimensional image into a point set in the three-dimensional space, providing data support for constructing a three-dimensional point cloud model.

[0040] In one specific embodiment, 3D point cloud data is registered using the Iterative Closest Point (ICP) algorithm. The ICP algorithm iteratively searches for the optimal transformation between two point clouds, minimizing the distance between corresponding points. Specifically, the algorithm first finds corresponding points in the two point clouds, then calculates the transformation matrix, transforms one point cloud, and then re-finds corresponding points, repeating this process until convergence. Point cloud registration aligns point cloud data collected from different perspectives or at different times into the same coordinate system, providing complete and accurate point cloud data for subsequent surface reconstruction, making the reconstructed model more accurate and coherent.

[0041] Next, the surface of the registered three-dimensional point cloud data is reconstructed by solving the Poisson equation. An implicit function is constructed based on the normal vector information of the point cloud. The specific form of this implicit function is determined by solving the Poisson equation. Then, the isosurface is extracted to obtain a three-dimensional point cloud model. Poisson surface reconstruction can generate a continuous and smooth surface based on the point cloud data, accurately restore the geometric shape of the object, and generate a high-quality three-dimensional point cloud model.

[0042] Optimally, the Laplace smoothing algorithm calculates the average value of the neighboring vertices of each vertex in the 3D point cloud model and adjusts the vertex positions to make the 3D point cloud model surface smoother. This can reduce noise and irregularities on the surface of the 3D point cloud model and improve the quality of the 3D point cloud model. The hole filling algorithm is used to fill holes in the 3D point cloud model. For example, based on the point cloud information around the hole, the point cloud data within the hole is estimated through interpolation or fitting to fill the hole. The Laplace smoothing algorithm makes the surface of the 3D point cloud model smoother and more consistent with the surface characteristics of the actual object. Hole filling eliminates holes in the 3D point cloud model, making the 3D point cloud model more complete and improving the usability and visualization of the 3D point cloud model.

[0043] In a specific embodiment, the process of executing step S105 may specifically include the following steps: (1) Pre-encoding the three-dimensional point cloud model with multimodal features and compressing it through an autoencoder to generate a point cloud feature vector; (2) The point cloud feature vector is input into the generator of the hybrid attention generation network to generate the defect feature, and the discriminator of the hybrid attention generation network detects and judges the information of the defect feature; (3) Constructing a knowledge graph based on the information of the defect characteristics; (4) After the hybrid attention generation network outputs the detection result of the defect feature, the detection result is interpreted through the knowledge graph, and the defect feature analysis and processing suggestions are output through the graph reasoning algorithm.

[0044] The process of constructing a knowledge graph based on the defect feature information and interpreting the defect feature detection results through the knowledge graph after the hybrid attention generation network outputs the defect feature detection results is performed, and outputting defect feature analysis and processing suggestions through a graph reasoning algorithm specifically includes: (1) Extracting entities and relationships between the entities from the information of the defect features; (2) constructing the knowledge graph by using a graph database for the entities and the relationships; (3) Mapping the detection results of the defect features output by the hybrid attention generation network to entities and relationships in the knowledge graph, and performing matching queries in the knowledge graph to obtain an explanation of the detection results; (4) Analyzing and summarizing the defect characteristics through the graph reasoning algorithm to obtain analysis results; (5) Based on the analysis results, corresponding treatment measures are extracted from the knowledge graph and treatment suggestions are generated.

[0045] Specifically, the three-dimensional point cloud model contains multiple modal information, such as shape, color, and texture. Multimodal feature precoding will extract and encode the features of multiple modal information, and convert the features of different modalities into a unified feature representation. Then, the autoencoder is used to compress the encoded features. Multimodal feature precoding integrates multiple information to make the features more representative. The compression operation of the autoencoder reduces the data dimension and reduces the computational complexity, while retaining key information, which helps in subsequent efficient processing.

[0046] Next, the point cloud feature vector is input into the generator of the hybrid attention generation network. The generator learns and generates possible defect features based on the input feature vector, and the discriminator detects and judges the generated defect features to determine whether they are real defects and the type, location and other information of the defects. The generator and discriminator are continuously optimized through adversarial training. The generator generates more realistic defect features, and the discriminator strives to accurately distinguish between real and generated features. The hybrid attention generation network can effectively mine the defect features in the three-dimensional point cloud model. Through the adversarial learning of the generator and the discriminator, the accuracy and robustness of defect detection are improved, and various types of potential defects can be detected.

[0047] In addition, entities are identified from the information of defect features, such as different types of defects and components of wafer packaging. At the same time, the relationships between these entities are determined, for example: scratches appear on the outer shell. The extracted entities and relationships are stored in a graph database. The graph database uses nodes to represent entities and edges to represent relationships. An intuitive knowledge graph is constructed to clearly display the association of defect-related information. The knowledge graph structures the information of defect features, which not only facilitates subsequent queries and reasoning, but also integrates various defect-related knowledge, providing a basis for interpreting detection results and providing processing suggestions.

[0048] In one specific embodiment, the defect feature detection results output by the hybrid attention generation network are mapped to entities and relationships in a knowledge graph. For example, if the detection result indicates the presence of a scratch, entities and relationships related to "scratch" are searched in the knowledge graph and matched. This results in a detailed explanation of the detection result, such as the possible causes and impacts of the scratch. Furthermore, the defect features are deeply analyzed using a graph reasoning algorithm. Based on the relationships and rules in the knowledge graph, the graph reasoning algorithm infers and summarizes the defect information, obtaining more comprehensive analysis results, such as the development trend of the defect and other possible problems. Finally, based on the analysis results, corresponding treatment measures are extracted from the knowledge graph. For example, if the scratch is minor, the knowledge graph may record the treatment measure of polishing and repair. These treatment measures are organized into treatment suggestions and output. The knowledge graph provides explainability for the detection results and can provide the causes and impacts of the defect. The analysis and treatment suggestion generation of the graph reasoning algorithm provide a scientific and reasonable basis for decision-making, which helps to timely and effectively address wafer packaging defects and improve production quality and efficiency.

[0049] The above describes the wafer packaging quality recognition method based on image recognition in the embodiment of the present application. The following describes the wafer packaging quality recognition system based on image recognition in the embodiment of the present application. The wafer packaging quality recognition method based on image recognition can be applied to the wafer packaging quality recognition system based on image recognition. Figure 2 , the wafer packaging quality recognition system based on image recognition includes: A multispectral acquisition module 201 is configured to acquire a multispectral image of the wafer package through a multispectral imaging unit and fuse the multispectral image to obtain a multispectral fused image; An RGB-D acquisition module 202 is configured to acquire an RGB color image of the wafer package and a depth image of the wafer package through an RGB-D unit; A three-dimensional fusion module 203 is configured to fuse the multispectral fusion image, the RGB color image, and the depth image using a three-dimensional reconstruction algorithm to generate a three-dimensional image of the wafer package; A three-dimensional reconstruction module 204 is configured to perform multimodal preprocessing on the three-dimensional image, extract three-dimensional point cloud data from the preprocessed three-dimensional image, and perform surface reconstruction on the three-dimensional point cloud data to generate a three-dimensional point cloud model; The defect detection module 205 is used to input the three-dimensional point cloud model into the hybrid attention generation network, detect the defect features in the three-dimensional point cloud model through the hybrid attention generation network, and output the information of the defect features.

[0050] In the embodiment of the present application, a multispectral image of the wafer package is collected by a multispectral imaging unit, and the multispectral image is fused to obtain a multispectral fused image, an RGB-D unit is used to collect an RGB color image of the wafer package and a depth image of the wafer package, and the multispectral fused image, the RGB color image and the depth image are fused by a three-dimensional reconstruction algorithm to generate a three-dimensional image of the wafer package, the three-dimensional image is multimodally preprocessed, and three-dimensional point cloud data is extracted from the preprocessed three-dimensional image and surface reconstruction is performed on the three-dimensional point cloud data to generate a three-dimensional point cloud model, the three-dimensional point cloud model is input into a hybrid attention generation network, and the defect features in the three-dimensional point cloud model are detected by the hybrid attention generation network and the information of the defect features is output, wherein, with the help of the multispectral imaging unit and the RGB-D unit Multispectral images, RGB color images and depth images of wafer packaging are collected separately. Multispectral images can capture spectral information that is invisible to the naked eye and reflect subtle differences in packaging materials. RGB color images provide rich color and texture details, and depth images provide information on three-dimensional spatial dimensions. These images of different modalities are fused through a three-dimensional reconstruction algorithm. The generated three-dimensional image of the wafer packaging contains more comprehensive information. Compared with a single image data source, the completeness of the description of wafer packaging features is greatly improved. In addition, the hybrid attention generation network is used to detect defect features in the three-dimensional point cloud model and output information on the defect features. It can not only provide a detailed explanation of the detection results, but also output targeted defect feature analysis and processing suggestions. This application realizes the requirement for comprehensive evaluation of wafer packaging quality and improves the efficiency and accuracy of packaging quality evaluation.

[0051] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0052] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A wafer packaging quality identification method based on image recognition, characterized in that: The wafer packaging quality identification method based on image recognition includes: Collecting a multispectral image of the wafer package by a multispectral imaging unit, and fusing the multispectral image to obtain a multispectral fused image; Acquire an RGB color image of the wafer package and a depth image of the wafer package through an RGB-D unit; fusing the multispectral fusion image, the RGB color image, and the depth image using a three-dimensional reconstruction algorithm to generate a three-dimensional image of the wafer package; Performing multimodal preprocessing on the three-dimensional image, extracting three-dimensional point cloud data from the preprocessed three-dimensional image, and performing surface reconstruction on the three-dimensional point cloud data to generate a three-dimensional point cloud model; The three-dimensional point cloud model is input into a hybrid attention generation network, and the hybrid attention generation network is used to detect defect features in the three-dimensional point cloud model and output information about the defect features.

2. The wafer packaging quality identification method based on image recognition according to claim 1, characterized in that: The multispectral image of the wafer package is collected by a multispectral imaging unit, and the multispectral image is fused to obtain a multispectral fused image, including: Acquire a multispectral image of the wafer package by using a polarization-angle-adjustable annular light source array in the multispectral imaging unit and a multispectral camera in the multispectral imaging unit; Performing denoising on the multispectral image by using a wavelet transform algorithm; Calculating the band reflectance characteristics of the multispectral image after denoising to obtain a spectral reflectance curve; Performing edge detection and texture analysis on the band images of the multispectral image after denoising and extracting multi-scale features; The spectral reflectance curve is fused with the multi-scale features and the multi-spectral fusion image is generated by a principal component analysis dimensionality reduction algorithm.

3. The wafer packaging quality identification method based on image recognition according to claim 1, characterized in that: After collecting the RGB color image and the depth image of the wafer package by the RGB-D unit, the method includes: enhancing the contrast of the RGB color image by using a histogram equalization algorithm; Noise points in the depth image are removed by a median filtering algorithm, and missing areas in the depth image are completed by a nearest neighbor interpolation algorithm.

4. The wafer packaging quality identification method based on image recognition according to claim 1, characterized in that: The step of fusing the multispectral fusion image, the RGB color image, and the depth image to generate a three-dimensional image of the wafer package by a three-dimensional reconstruction algorithm includes: Generating a point cloud surface from the depth image using the three-dimensional reconstruction algorithm; Projecting the multispectral fusion image and the RGB color image onto the point cloud surface and extracting the normalized vegetation index of the multispectral image and the texture features of the RGB color image; fusing the normalized vegetation index and the texture feature through a principal component analysis dimensionality reduction algorithm to generate dimensionality reduction fusion feature data; The point cloud surface and the dimensionality reduction fusion feature data are rendered using a Phong illumination model to generate a three-dimensional image of the wafer package.

5. The wafer packaging quality identification method based on image recognition according to claim 4, characterized in that: The step of rendering the point cloud surface and the dimensionality reduction fusion feature data using a Phong illumination model to generate a three-dimensional image of the wafer package includes: Converting the point cloud surface data into a three-dimensional mesh model using a Poisson reconstruction algorithm; Associating the dimension reduction fusion feature data with the three-dimensional grid model to generate a three-dimensional geometric composite model; Calculating the ambient light intensity, diffuse reflection light intensity, and specular reflection light intensity of the Phong illumination model, and adding the ambient light intensity, the diffuse reflection light intensity, and the specular reflection light intensity to obtain a total illumination intensity; The total light intensity is fused with the color information in the dimensionality reduction fusion feature data to obtain a mixed color value, and the mixed color value and the three-dimensional geometric composite model are rendered by a rendering engine to generate a three-dimensional image of the wafer package.

6. The wafer packaging quality identification method based on image recognition according to claim 1, characterized in that: The performing multimodal preprocessing on the three-dimensional image includes: Performing geometric correction on the three-dimensional image, and performing denoising on the three-dimensional image after geometric correction using a filtering algorithm; The denoised three-dimensional image data is normalized so that data of different dimensions are in the same quantity range, and multiple modal information of the three-dimensional image is fused and enhanced.

7. The wafer packaging quality identification method based on image recognition according to claim 6, characterized in that: The step of extracting three-dimensional point cloud data from the pre-processed three-dimensional image and performing surface reconstruction on the three-dimensional point cloud data to generate a three-dimensional point cloud model comprises: Extracting the three-dimensional point cloud data using a corresponding algorithm according to the representation of the pre-processed three-dimensional image; Registering the three-dimensional point cloud data based on an iterative closest point algorithm; Performing surface reconstruction on the registered three-dimensional point cloud data by solving the Poisson equation to generate the three-dimensional point cloud model; The three-dimensional point cloud model is smoothed by a Laplace smoothing algorithm, and holes in the three-dimensional point cloud model are filled by a hole filling algorithm.

8. The wafer packaging quality identification method based on image recognition according to claim 1, characterized in that: Inputting the three-dimensional point cloud model into a hybrid attention generation network, detecting defect features in the three-dimensional point cloud model through the hybrid attention generation network and outputting information about the defect features, includes: Performing multimodal feature precoding on the three-dimensional point cloud model and compressing it through an autoencoder to generate a point cloud feature vector; The point cloud feature vector is input into the generator of the hybrid attention generation network to generate the defect feature, and the discriminator of the hybrid attention generation network detects and judges the information of the defect feature; Constructing a knowledge graph based on the information of the defect characteristics; After the hybrid attention generation network outputs the detection results of the defect features, the detection results are interpreted through the knowledge graph, and the defect feature analysis and processing suggestions are output through the graph reasoning algorithm.

9. The wafer packaging quality identification method based on image recognition according to claim 8, characterized in that: The method constructs a knowledge graph based on the information of the defect features; after the hybrid attention generation network outputs the detection results of the defect features, the detection results are interpreted through the knowledge graph, and the defect feature analysis and processing suggestions are output through the graph reasoning algorithm, including: Extracting entities and relationships between the entities from information about the defect features; Construct the knowledge graph using a graph database based on the entities and the relationships; Mapping the detection results of the defect features output by the hybrid attention generation network to entities and relationships in the knowledge graph, and performing a matching query in the knowledge graph to obtain an explanation of the detection results; Analyzing and summarizing the defect characteristics by the graph reasoning algorithm to obtain analysis results; Based on the analysis results, corresponding processing measures are extracted from the knowledge graph and processing suggestions are generated.

10. A wafer packaging quality identification system based on image recognition, used to implement the wafer packaging quality identification method based on image recognition according to any one of claims 1 to 9, characterized in that: The wafer packaging quality recognition system based on image recognition includes: a multispectral acquisition module, configured to acquire a multispectral image of the wafer package through a multispectral imaging unit, and fuse the multispectral image to obtain a multispectral fused image; an RGB-D acquisition module, the RGB-D acquisition module being configured to acquire an RGB color image of the wafer package and a depth image of the wafer package through an RGB-D unit; a three-dimensional fusion module, configured to fuse the multispectral fusion image, the RGB color image, and the depth image using a three-dimensional reconstruction algorithm to generate a three-dimensional image of the wafer package; a three-dimensional reconstruction module, configured to perform multimodal preprocessing on the three-dimensional image, extract three-dimensional point cloud data from the preprocessed three-dimensional image, and perform surface reconstruction on the three-dimensional point cloud data to generate a three-dimensional point cloud model; A defect detection module is used to input the three-dimensional point cloud model into a hybrid attention generation network, detect defect features in the three-dimensional point cloud model through the hybrid attention generation network, and output information about the defect features.

Images