Die bonder image acquisition system based on FPGA
The interpolation direction is judged through three gradient operators in the FPGA image acquisition system, and the image color is restored, which solves the problem of image acquisition molar pattern phenomenon and improves the positioning accuracy of the solid crystal machine.
Patent Information
- Application Number
- CN202510509695.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-22
- Publication Date
- 2025-07-25
AI Technical Summary
In the prior art, molar patterns are easily generated during image acquisition, which affects the packaging accuracy of the crystal-solid machine.
The image acquisition system based on FPGA is adopted, and the original image data is converted into digital signals through the decoding module. Three different gradient operators are used to judge the interpolation direction of the image edge, restore the colors of the green, red and blue channels, reduce the molar pattern phenomenon, and improve the image positioning accuracy.
It improves the accuracy of image acquisition and the overall positioning accuracy of the solid crystal machine, reduces the possibility of errors in interpolation direction judgment, and enhances the robustness of the algorithm.
Smart Images
Figure CN120378575A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and specifically to an image acquisition system of a die bonder based on FPGA. Background Art
[0002] Digital image technology has been fully developed and is of irreplaceable importance in fields such as industrial production, aerospace, and fire safety. As the integrated circuit industry that China is currently focusing on developing, chip packaging in its production process is a crucial step, and the required precision and packaging effect are also continuously improving. As one of the main devices in the production process, the precision in the control process of the die bonder determines the quality of the packaging effect. The common structural parts of a die bonder include: a pick-up and push mechanism, a dispensing mechanism, a swing arm mechanism, a die bonding mechanism, etc. Currently, most manufacturers will adopt different methods to improve the precision in the control process of the die bonder. For example, an image processing module is added to assist in positioning the position of the chip to be packaged, and at this time, some image processing algorithms are involved, such as deep learning and other algorithms. The image acquisition here needs to use a camera module. Therefore, the accuracy of image acquisition will also affect the equipment precision of the die bonder to a certain extent.
[0003] Currently, most image acquisitions are carried out using image sensors. Most manufacturers usually use a single CMOS image sensor as the source of image data. The original image data output by the camera is in RAW format. In this format of image, each pixel point has only one of the three colors: green, red, and blue. It is necessary to perform image processing through image processing technology to output an RGB format image suitable for the human eye structure, that is, the three colors of each pixel point are complete. This process is called the demosaicing technology, which is usually completed through a designed interpolation algorithm. In this process, moiré patterns are likely to occur. Seriously, it will cause errors in the subsequent image processing process, affect the actual effects of other subsequent algorithms, and ultimately lead to a decrease in the packaging precision of the die bonder. Therefore, before processing the acquired image using an image processing algorithm, it is an extremely important step to perform color reconstruction on the image. The present invention improves this process. Summary of the Invention
[0004] In view of the above technical problems, the present invention provides an image acquisition system of a die bonder based on FPGA, which realizes the color reconstruction of the original image acquired for the packaged chip, reduces the moiré pattern phenomenon in the demosaicing process, and improves the overall image positioning precision of the die bonder.
[0005] To achieve the above object, the present invention provides the following technical solutions: An image acquisition system for a die bonder based on FPGA, including an image acquisition module for acquiring original image data of packaged chips, a decoding module for decoding the original image information into digital signals processed by FPGA, an FPGA image processing module for processing the decoded digital signals and performing color reconstruction to output image data, a buffer module for buffering the reconstructed image data, and an encoding module for encoding and outputting the reconstructed image data; the original image data is in Bayer format image.
[0006] The FPGA image processing module performs color reconstruction through a demosaicing algorithm, and determines the interpolation direction of the image edge for one or more combined states of smooth regions, texture-dense regions, and texture-non-dense regions through three different gradient operators, and first restores the green channel color to guide the red channel color and the blue channel color.
[0007] Preferably, the decoding module uses a decoding chip or a written decoding module to decode the original image information into digital signals processed by FPGA.
[0008] Preferably, the encoding module uses an encoding chip or a written encoding module to encode and output the image data after reconstructed color.
[0009] Preferably, the image acquisition module uses a CMOS image sensor to acquire the original image data of the packaged chips and transmits the original image data to FPGA.
[0010] Preferably, the FPGA image processing module builds a 5×5 data window through the FIFO in FPGA.
[0011] Preferably, the 5×5 data window uses 5 FIFOs to cache the decoded digital signals to form 5 rows of data, and respectively outputs them to the shift register. The input and output data bit width of the data window is 8bit, and the cache depth is 2048 data.
[0012] Preferably, the three different gradient operators determine the image edge interpolation direction as follows:
[0013] Set a flag bit for all gradient operators;
[0014] The first gradient operator: In the B ij Approximate the horizontal red-green color difference and the vertical color difference respectively according to the color difference law within the 3×3 region:
[0015] Horizontal direction:
[0016]
[0017] Vertical direction:
[0018]
[0019] Compare ΔH1 and ΔV1 and operate on the flag bit;
[0020] The second gradient operator: At the interpolation point B ij Within the 5×5 area, estimate the changing trend of the missing green component by the changing trend of a blue component, and calculate the second-order differential of the blue component:
[0021] Horizontal direction:
[0022] ΔH2 = |2B(i,j) - B(i,j - 2) - B(i,j + 2)|
[0023] Vertical direction:
[0024] ΔV2 = |2B(i,j) - B(i - 2,j) - B(i + 2,j)|
[0025] Compare ΔH2 and ΔV2 and operate on the flag bit;
[0026] The third gradient operator: At the interpolation point B ij Within the 5×5 area, use the upper and lower adjacent pixels parallel to the horizontal direction in the first-order differential method and the left and right adjacent pixels parallel to the vertical direction in the first-order differential method to estimate the changing trend of the interpolation point through the local gradient changing trend, that is:
[0027] Horizontal direction:
[0028]
[0029] Vertical direction:
[0030]
[0031] Compare ΔH3 and ΔV3 and operate on the flag bit;
[0032] Comprehensively recover the missing green channel component at the interpolation point according to the values of the flag bits of the above three gradient operators.
[0033] Preferably, the red channel color and the blue color are recovered as follows:
[0034] Recover the missing red component at G3:
[0035]
[0036] Where are the recovered green components at B1 and B2;
[0037] Recover the missing blue component on the green channel:
[0038]
[0039] Among them are the restored green components at B1 and B2;
[0040] Restoration of the red component on the blue channel:
[0041] The missing red components on B are distributed on the two diagonals of B. According to the color difference law:
[0042]
[0043] Among them, are the restored green components at point B and at R1, R2, R3, and R4 respectively;
[0044] Restoration of the missing blue component on the red channel:
[0045]
[0046] Among them, are the restored green components at point R and at B1, B2, B3, and B4 respectively.
[0047] Advantages of the present invention: The image acquisition module acquires the original image data of the packaged chip. The original image data is converted into a digital signal by the decoding module and sent to the FPGA image processing module. The FPGA image processing module reconstructs the image. Three different ladder operators are used to judge the interpolation direction of the image edge. First, the green component is restored to guide the red component and the blue component, and a complete color image is output. The interpolation direction is comprehensively determined by three different gradient operators, reducing the possibility of misjudging the interpolation direction, improving the robustness of the algorithm, improving the accuracy of image color reconstruction, and improving the overall image positioning accuracy of the subsequent die bonder. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] The drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation to the present invention. In the drawings:
[0049] Figure 1 is a schematic diagram of the simple structure of the die bonder image acquisition system based on FPGA proposed by the present invention.
[0050] Figure 2 is a schematic diagram of the bottom view structure of the die bonder image acquisition system based on FPGA proposed by the present invention.
[0051] Figure 3 is a schematic diagram of the structure of the 7×7 template Bayer format diagram of the present invention.
[0052] Figure 4 Schematic diagram of the 3×3 template Bayer format image structure of the present invention.
[0053] Figure 5 Schematic diagram of the 5×5 template Bayer format image structure of the present invention.
[0054] Figure 6 Schematic diagram of the 5×5 template Bayer format image structure of the present invention.
[0055] Figure 7 Schematic diagram of the 4 Bayer image array structures of the present invention. Detailed implementation manners
[0056] In order to make the technical means, creative features, achieved purposes and functions of the present invention easy to understand, the present invention will be further described below in conjunction with specific embodiments and drawings. However, the following embodiments are only the preferred embodiments of the present invention, not all of them. Based on the embodiments in the implementation manners, other embodiments obtained by those skilled in the art without creative efforts all fall within the protection scope of the present invention.
[0057] Please refer to Figure 1-7 , an image acquisition system of a die bonder based on FPGA, and the image acquisition module for encapsulating chips is carried out through the image acquisition system installed on the die bonder. Among them, the image acquisition end can use a CMOS image sensor to collect the original image data, and can obtain the original image information of the chip to be encapsulated in real time.
[0058] Decoding module, which decodes the obtained original image information. The decoding operation is usually carried out through a dedicated decoding chip or a written decoding module. In order to save costs, the present invention uses a written decoding module for decoding, and finally obtains a digital signal that can be processed by the FPGA.
[0059] FPGA image processing module, this module implements the image color reconstruction method designed by the present invention, processes the obtained original image information within data windows of different sizes, calculates the three gradient operators respectively, judges the possible interpolation directions, performs color reconstruction on the original image, and finally outputs the image data after reconstructed color. The input and output signals of the image data in this module are judged by the frame signal of the image. The image data entering this unit is decoded by the previous decoding module and is the original image data stream with the correct data structure;
[0060] The cache module caches the image data after color reconstruction. It consists of a front-end FIFO and DDR3. The front-end FIFO caches the processed image data output by the image processing module and provides it for DDR3 to read. DDR3 reads 128-bit data each time and then outputs 128-bit data to the back-end FIFO at one time. The back-end FIFO then outputs the image data for display according to the read signal sent by the image display end. The read and write clock domains of the front-end and back-end and DDR3 can be different.
[0061] The encoding module encodes and outputs the image data after color reconstruction. Similar to the decoding module, an encoding chip or an encoding module can be used for encoding. Similarly, in order to save costs, an encoding module is written here for encoding.
[0062] The image display module is usually not required. The image data processed by the present invention can continue to serve other algorithms. The image display module set here is to better facilitate the operator to perform real-time image monitoring and be able to detect problems in the packaging process in a timely manner.
[0063] The algorithm for image color reconstruction includes the following steps:
[0064] Overall design idea: Comprehensively judge the possible interpolation directions of the image through three different gradient operators designed, and restore the green, red, and blue components of the image along the most likely image direction, and finally output a complete color image.
[0065] The image sensor in the image acquisition module usually outputs a Bayer format image. In the Bayer format image, the sampling frequency of the green component is twice that of the red and blue components, containing more image information. Therefore, most interpolation methods first interpolate the green channel, and then restore the red and blue components according to the restored green component as a guide. Most interpolation errors are caused by incorrect interpolation direction judgment or incorrect interpolation values, and the restoration effect of the green channel directly affects the subsequent restoration of the red and blue channels. Therefore, the present invention focuses on judging the interpolation direction of the green channel to attach Figure 1 the Bayer format image as an example.
[0066] Green channel color reconstruction:
[0067] The algorithm of the present invention is based on the correlation between the image color channels. According to the color ratio law, in a small smooth area, the color values of each part of an image change uniformly. Then, the intensity ratio of the image color channels can also be regarded as a constant value. This theory conforms to the regional characteristics of the scene and is called the color ratio constant law. That is, if point (u, v) and point (x, y) are two adjacent points, there is:
[0068]
[0069] In practical applications, to avoid the situation where the numerator is zero, a similar color difference rule is usually used, that is:
[0070] R(u, v) - G(u, v) ≈ R(x, y) - G(x, y)
[0071] The three color channels have similar change trends along the edge direction of the regional image and have relatively close values. According to this change trend, the present invention designs a new gradient operator, combines and improves the first-order differential algorithm based on the gradient, and finally comprehensively considers the possible interpolation directions of the regional image based on the results of the three gradient judgments.
[0072] Taking Figure 4 the Bayer format image template as an example, restore the missing G-channel component at point B i，j Traditional interpolation algorithms usually can only detect the pixel gradient changes at intervals of one pixel point, and it is often difficult to detect the changes of adjacent pixels. Eventually, in areas with relatively complex image textures, the judgment of the image edge direction is incorrect, resulting in poor corresponding color restoration effects. Therefore, the present invention proposes a new gradient operator and sets a flag bit for all gradient operators. When the horizontal gradient ΔH is less than the vertical gradient ΔV, the flag bit is incremented by one. Finally, the value of the flag bit is used to judge the possible interpolation direction. The following introduces the specific calculation methods of the three gradient operators:
[0073] The first operator: It is observed that in the 3×3 area ( i，j ) around the interpolation point B Figure 4 there is more red-channel information and green-channel information, which helps to judge the edge information of adjacent pixels. Since within a small area, the change of the same color channel value along the image edge direction should be smooth, according to the color difference rule, the horizontal red-green color difference and the vertical red-green color difference are approximated respectively:
[0074] Horizontal direction:
[0075]
[0076] Vertical direction:
[0077]
[0078] Compare ΔH1 and ΔV1 and operate on the flag bit.
[0079] The second operator: As described above, the three color components should have similar change trends along the image edge direction. Therefore, in the 5×5 area ( i，j ) around the interpolation point BFigure 5 ) to estimate the missing green component change trend of the interpolation point based on the blue component change trend. Calculate the second-order differential of the blue component as follows:
[0080] Horizontal direction:
[0081] ΔH2 = |2B(i,j) - B(i,j - 2) - B(i,j + 2)|
[0082] Vertical direction:
[0083] ΔV2 = |2B(i,j) - B(i - 2,j) - B(i + 2,j)|
[0084] Compare ΔH2 and ΔV2, and operate on the flag bit.
[0085] The third operator: Based on the method of the present invention to finally judge the image edge direction by decision-making. It is found in actual tests that when using the second-order differential method or the adaptive method to perform gradient judgment in some complex texture regions, the correct rate of interpolation direction judgment does not increase significantly compared with the first-order differential method, and even decreases in some cases. Therefore, the present invention improves the first-order differential method based on gradient, and makes full use of the green component information of the interpolation point B i，j in the 3×3 region ( Figure 5 ). When calculating the horizontal gradient, use the upper and lower adjacent pixels parallel to the horizontal direction in the first-order differential method. When calculating the vertical gradient, use the left and right adjacent pixels parallel to the vertical direction in the first-order differential method. Finally, estimate the change trend of the interpolation point through the local gradient change trend, that is:
[0086] Horizontal direction:
[0087]
[0088] Vertical direction:
[0089]
[0090] Compare ΔH3 and ΔV3, and operate on the flag bit.
[0091] After calculating the above three gradients, restore the missing green channel component of the interpolation point according to the value of the flag bit, taking Figure 6 restoring the green component at B5 as an example:
[0092]
[0093] When restoring the missing green component at the red component and when restoring the missing green component in other types of Bayer image arrays, a similar calculation method as the above formula is adopted, which will not be elaborated here. The traditional single gradient calculation method often misestimates the interpolation direction in the edge direction. Therefore, the present invention adopts a method of jointly judging multiple gradients. Gradient 1 ensures the judgment of the interpolation direction of adjacent pixels in the texture-dense area, and gradients 2 and 3 have better judgment effects in the smooth area and the area with less dense texture. The comprehensive determination of the interpolation direction by three different gradient operators reduces the possibility of misjudging the interpolation direction, improves the robustness of the algorithm, improves the accuracy of image color reconstruction, and improves the overall image positioning accuracy of the subsequent die bonder.
[0094] The following is the color reconstruction of the missing red and blue components.
[0095] Red and blue color reconstruction:
[0096] The restoration of the red and blue channels mainly requires restoring the missing red and blue components on the green channel, as well as the missing red component on the blue channel and the missing blue component on the red channel. Taking the Bayer image template of Figure 7 as an example.
[0097] Red and blue color reconstruction on the green channel:
[0098] After the green channel is completely restored, using the color difference law described above, the missing red and blue components on the green channel are restored with the green component as the guide. When the distribution of the green channel is as in Figure 7 (a) and (b), the methods of restoring the red and blue components are similar. Taking Figure 7 (a) as an example, the missing red component at G3 is restored as follows:
[0099]
[0100] where are the restored green components at R1 and R2. The restoration of the missing blue component on the green channel is similar to the above formula. Taking Figure 7 (a) as an example, there is:
[0101]
[0102] where are the restored green components at B1 and B2.
[0103] Restoration of the blue (red) component on the red (blue) channel:
[0104] Taking Figure 7 (d) as an example, to restore the missing red component on B, the red components closest to point B are distributed on two diagonals. Still using the color difference law, there is:
[0105]
[0106] Among them, are the restored green components at point B and at R1, R2, R3, and R4 respectively. For the restoration of the missing blue components on the red channel, it is similar to the above formula. Taking Figure 7 (c) as an example, there is:
[0107]
[0108] Among them, are the restored green components at point R and at B1, B2, B3, and B4 respectively.
[0109] So far, the core part of the present invention, the color restoration of the image, is completed. Next is the construction of the overall image acquisition system.
[0110] The overall block diagram of the FPGA-based system is as shown in Figure 2 shown.
[0111] The image acquisition module is mainly composed of a CMOS image sensor. The original image data collected is sent into the main control FPGA chip through a differential interface, and then processed by the image processing module. The processed image data is cached and then output through a differential port to the image display module.
[0112] The specific steps to implement the image color reconstruction module using FPGA:
[0113] The data processing of the present invention needs to be carried out within 3×3 and 5×5 windows respectively. Therefore, it is only necessary to build a data window with a size of 5×5. The construction of the data window is carried out through the FIFO in the FPGA. Five FIFOs are used to cache the data transmitted by the acquisition module to form 5 rows of data. These 5 rows of data are then output to the shift register respectively, and finally a data window with a size of 5×5 is formed. The input and output data bit width is 8bit, and the cache depth is 2048 data. The overall design is synchronous. During the whole processing process, it is carried out in the 5×5-sized data window constructed by the FIFO and shift register in the FPGA, and the system design is carried out from top to bottom using Verilog language.
[0114] The cache module consists of DDR3 and the front and rear FIFOs. The read / write bit width of DR3 is 128 bits, and the data depth is the same as that of the front FIFO. It is necessary to read 128-bit data from the FIFO at a time. The cache module mainly caches the processed image data output from the image color restoration module. After being cached by the front FIFO, it is read into DDR3. Before finally being output to the image display module, it also needs to be cached by the rear FIFO, encoded, and finally the processed image data is output through the transmission protocol.
[0115] The image display module can transmit image data according to different transmission protocols. This invention mainly uses the cameralink protocol for transmission.
[0116] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and the descriptions in the specification are only the preferred examples of the present invention and do not limit the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of the present invention claimed is defined by the appended claims and their equivalents.
Claims
1. A die bonder image acquisition system based on FPGA, characterized in that, It includes an image acquisition module for acquiring the original image data of the packaged chip, a decoding module for decoding the original image information into a digital signal processed by FPGA, an FPGA image processing module for processing the decoded digital signal and performing color reconstruction to output image data, a caching module for caching the reconstructed image data, and an encoding module for encoding and outputting the reconstructed image data; the original image data is a Bayer format image. The FPGA image processing module performs color reconstruction through a demosaicing algorithm, and determines the interpolation direction of the image edge for one or more combined states of the smooth area, the texture-dense area, and the texture-non-dense area through three different gradient operators, and first restores the green channel color to guide the red channel color and the blue channel color.
2. The image acquisition system of a die bonder based on FPGA according to claim 1, characterized in that: The decoding module uses a decoding chip or a written decoding module to decode the original image information into a digital signal processed by FPGA.
3. The image acquisition system of a die bonder based on FPGA according to claim 1, characterized in that: The encoding module uses an encoding chip or a written encoding module to encode and output the image data after reconstructed color.
4. A die bonder image acquisition system based on FPGA according to claim 1, characterized in that: The image acquisition module uses a CMOS image sensor to acquire the original image data of the packaged chip and transmits the original image data to FPGA.
5. The image acquisition system of a die bonder based on FPGA according to claim 1, wherein: The FPGA image processing module builds a 5×5 data window through the FIFO in FPGA.
6. The image acquisition system of a die bonder based on FPGA according to claim 5, characterized in that: The 5×5 data window uses 5 FIFOs to cache the decoded digital signal to form 5 rows of data, and outputs them to the shift register respectively. The input and output data bit width of the data window is 8bit, and the cache depth is 2048 data.
7. The image acquisition system of a die bonder based on FPGA according to claim 6, wherein: The interpolation directions of the image edge determined by the three different gradient operators are as follows: Set the flag bit for all gradient operators; The first gradient operator: In B ij Approximate the horizontal red-green color difference and the vertical color difference respectively according to the color difference rule within the 3×3 region: Horizontal direction: Vertical direction: Compare ΔH1 and ΔV1, and operate on the flag bit; The second gradient operator: At the interpolation point B ij Within the 5×5 region, the change trend of the blue component is used to estimate the change trend of the missing green component to be interpolated, and the second-order differential of the blue component is calculated: Horizontal direction: ΔH2 = |2B(i,j)-B(i,j - 2)-B(i,j + 2)| Vertical direction: ΔV2 = |2B(i,j)-B(i - 2,j)-B(i + 2,j)| Compare ΔH2 and ΔV2, and operate on the flag bit; The third gradient operator: At the interpolation point B ij In the 5×5 area, use the upper and lower adjacent pixels parallel to the horizontal direction in the first-order differential method and the left and right adjacent pixels parallel to the vertical direction in the first-order differential method to estimate the change trend of the interpolation point through the local gradient change trend, that is: Horizontal direction: Vertical direction: Compare ΔH3 and ΔV3, and operate on the flag bit; Comprehensively restore the missing green channel component at the interpolation point according to the values of the flag bits of the above three gradient operators.
8. An image acquisition system of a die bonder based on FPGA according to claim 7, characterized in that: The restoration of the red channel color and the blue color is as follows: Restore the missing red component at G3: Among them are the restored green components at B1 and B2; Restore the missing blue component on the green channel: Among them are the restored green components at B1 and B2; Restore the red component on the blue channel: The missing red components on B are distributed on the two diagonals of B. According to the color difference law: Among them, are the restored green components at point B and at R1, R2, R3, and R4 respectively; Restore the missing blue component on the red channel: Among them, are the restored green components at point R and points B1, B2, B3, and B4 respectively.