Optical alignment system and method for chip patch processing

Abstract

The invention relates to the technical field of chip mounting processing, and discloses an optical alignment system and method for chip mounting processing, and the method comprises the steps: collecting a target chip image and a target substrate image in a mounting process through an upper visual field camera disposed above a suction nozzle assembly and a lower visual field camera disposed above a substrate carrying platform; chip edge accurate coordinates in the target chip image are extracted, and reference mark accurate coordinates in the target substrate image are extracted; solving a chip mounting process compensation parameter matrix of an upper and lower camera-suction nozzle-substrate four-coordinate system in the chip mounting process based on the chip edge accurate coordinate and the reference mark accurate coordinate; calculating an X-Y-theta three-dimensional alignment compensation vector in the chip mounting process according to the mounting process compensation parameter matrix; on the basis of the X-Y-theta three-dimensional alignment compensation vector, the optimal mounting track parameter is calculated, an execution instruction sequence is generated, the problem of error accumulation caused by traditional independent calibration is effectively solved, and the alignment precision is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of chip bonding processing, and particularly to an optical alignment system and method for chip bonding processing. Background Art

[0002] With the development of miniaturization and high density of electronic products, the alignment accuracy requirements for chip bonding processing are continuously increasing, and traditional monocular vision alignment can no longer meet the high-precision mounting requirements. Existing optical alignment methods generally have problems such as manual selection of registration parameters for feature recognition clustering, complex coordinate calibration and error accumulation, and difficulty in balancing alignment speed and accuracy. Especially in the production process, due to the error accumulation caused by the separate calibration of each optical component, the actual mounting accuracy deviation often exceeds the process allowable range, and the production efficiency is low. In addition, traditional methods have limited ability to process chip and substrate marks with different shape features, and are prone to failure when dealing with situations such as uneven surface illumination and unclear features.

[0003] The optical alignment system in current chip bonding processing usually calibrates the upper and lower cameras - nozzle - substrate as independent components one by one, lacking an overall system error model, resulting in the inability to effectively compensate for the errors between subsystems. This separate calibration method not only increases the complexity of calibration operations but also causes accuracy loss. At the same time, most traditional alignment algorithms use single feature extraction methods, which are difficult to meet the recognition requirements of different types of marks. Especially in the high-speed mounting process, the collaborative optimization problem between alignment calculation and trajectory planning has not been effectively solved. Summary of the Invention

[0004] The present invention provides an optical alignment system and method for chip bonding processing. The present invention effectively solves the problem of error accumulation caused by traditional separate calibration and significantly improves the alignment accuracy.

[0005] In a first aspect, the present invention provides an optical alignment method for chip bonding processing, and the optical alignment method for chip bonding processing includes: Collecting the target chip image and the target substrate image during the bonding process through an upper field-of-view camera configured above the nozzle assembly and a lower field-of-view camera configured above the substrate stage; Extracting the precise coordinates of the chip edge in the target chip image and the precise coordinates of the reference mark in the target substrate image; Based on the precise coordinates of the chip edge and the precise coordinates of the reference mark, solving the chip bonding process compensation parameter matrix of the four-coordinate system of the upper and lower cameras - nozzle - substrate during the bonding process; Calculating the X-Y-θ three-dimensional alignment compensation vector during the chip bonding process according to the chip bonding process compensation parameter matrix; Based on the X-Y-θ three-dimensional alignment compensation vector, calculate the optimal placement trajectory parameters and generate an execution instruction sequence.

[0006] In a second aspect, the present invention provides an optical alignment system for chip placement processing. The optical alignment system for chip placement processing includes: An acquisition module, configured to acquire a target chip image and a target substrate image during the placement process through an upper vision camera disposed above the nozzle assembly and a lower vision camera disposed above the substrate stage; An extraction module, configured to extract the precise coordinates of the chip edge in the target chip image and extract the precise coordinates of the fiducial mark in the target substrate image; A solution module, configured to solve the placement process compensation parameter matrix of the upper and lower camera-nozzle-substrate four-coordinate system during the placement process based on the precise coordinates of the chip edge and the precise coordinates of the fiducial mark; A calculation module, configured to calculate the X-Y-θ three-dimensional alignment compensation vector during the chip placement process according to the placement process compensation parameter matrix; A generation module, configured to calculate the optimal placement trajectory parameters and generate an execution instruction sequence based on the X-Y-θ three-dimensional alignment compensation vector.

[0007] In the technical solution provided by the present invention, by constructing an overall vision error model of the upper and lower camera-nozzle-substrate four-coordinate system, the problem of error accumulation caused by traditional separate calibration is effectively solved, and the alignment accuracy is significantly improved; the use of spatial parallel processing and image pyramid multi-scale processing technologies greatly improves the feature extraction and matching speed; a special feature extraction algorithm is designed for fiducial marks of different shapes, enhancing the system's recognition ability for various marks; the chip placement process is modeled as a real-time convex optimization problem and solved through a sequential quadratic programming algorithm, achieving an effective balance between placement speed and accuracy; automatic calculation of alignment parameters is realized, eliminating the manual intervention link, improving production consistency and reliability; through a multi-level feature processing strategy and a backup algorithm mechanism, the robustness of the system in a complex production environment is enhanced, adapting to various chip placement processing scenarios.

[0008] Other features and advantages of the present invention will be described in the subsequent description, and, in part, will be obvious from the description, or will be understood by implementing the present invention. The objectives and other advantages of the present invention are achieved and obtained by the structures specifically pointed out in the description, the claims, and the drawings.

[0009] To make the above objectives, features, and advantages of the present invention more obvious and understandable, the following specific preferred embodiments are given, and in conjunction with the accompanying drawings, the detailed description is as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 Schematic diagram of an embodiment of the optical alignment method for chip placement processing in an embodiment of the present invention; Figure 2 Schematic diagram of an embodiment of the optical alignment system for chip placement processing in an embodiment of the present invention. Detailed implementation manners

[0011] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Apparently, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0012] The terms "including" and "having" and any variations thereof mentioned in the embodiments of the present invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device including a series of steps or units is not limited to the listed steps or units, but optionally further includes other unlisted steps or units, or optionally further includes other steps or units inherent to these processes, methods, products, or devices.

[0013] For ease of understanding of this embodiment, first, a detailed introduction is given to an optical alignment method for chip placement processing disclosed in the embodiments of the present invention. As Figure 1 shown, this method includes the following steps: 101. Collect the target chip image and the target substrate image during the placement process through the upper vision camera configured above the nozzle assembly and the lower vision camera configured above the substrate stage; It can be understood that the execution subject of the present invention can be an optical alignment system for chip placement processing, or a terminal or a server. Specifically, no limitation is made here. The embodiments of the present invention are described by taking the server as the execution subject as an example.

[0014] Specifically, the upper vision camera configured above the nozzle assembly is equipped with an annular light source. The function of this light source is to provide uniform illumination, reduce the influence of shadows and reflections on the chip image acquisition, and at the same time set the first exposure time and the first gain value to ensure that the acquired chip image has appropriate brightness and contrast. Similarly, the lower vision camera configured above the substrate stage is equipped with a flexible surface light source. This light source provides uniform and soft illumination, reduces the specular reflection or local overexposure on the substrate surface, and sets the second exposure time and the second gain value to optimize the acquisition quality of the substrate image and ensure that the features of the fiducial marks are clearly presented. Through this step, the initial chip image and the initial substrate image are obtained respectively. The upper vision camera and the lower vision camera are synchronously triggered by a high-speed image acquisition card, so that the two cameras acquire images at the same moment, thereby avoiding coordinate errors caused by time delay. The initial chip image and the initial substrate image are transmitted to the image processing system through a high-speed data channel to form a continuous data stream, ensuring fast response speed for image analysis and processing, so as to meet the requirements of high-speed chip mounter. The initial chip image and the initial substrate image are subjected to grayscale processing to remove color information, reduce computational complexity, and highlight the structural information of the image, making the subsequent feature extraction more efficient and stable. The grayscale images are the first chip image and the first substrate image. The first chip image and the first substrate image are subjected to histogram equalization processing to enhance the contrast, making the chip edge features and the fiducial marks on the substrate clearer, and obtaining the second chip image and the second substrate image. The second chip image and the second substrate image are subjected to Gaussian filtering and edge enhancement processing. The role of Gaussian filtering is to reduce the noise in the image, smooth the image, and at the same time maintain the main structural features to prevent the subsequent edge detection from being affected by random noise. The edge enhancement processing is used to highlight the boundary information of the chip edge and the fiducial marks, improve the accuracy of edge detection, and ensure the accuracy of coordinate extraction. Through the preprocessing steps, the target chip image and the target substrate image are obtained.

[0015] 102. Extract the precise coordinates of the chip edge in the target chip image and extract the precise coordinates of the fiducial marks in the target substrate image; Specifically, perform Canny edge detection on the target chip image. The Canny algorithm has strong anti-noise ability and can extract clear and continuous edge information to obtain the initial edge map of the chip. Perform morphological processing on the initial edge map of the chip. Use morphological dilation and erosion operations to effectively remove isolated noise points and enhance the connectivity of broken edges, obtaining a chip edge map with noise removed and edges connected. Apply the probabilistic Hough transform algorithm to the chip edge map with noise removed and edges connected to extract straight line segments. The probabilistic Hough transform has higher computational efficiency and anti-noise ability compared to the traditional Hough transform, and can quickly extract straight line segments from the edge map to form a set of line segments of the chip edge. Group the direction of the chip edge line segment set. Cluster them according to the direction angle of the line segments, and divide the line segments representing the four sides of the chip into four different direction groups. For each group of edge line segments with direction clustering, perform straight line fitting respectively. The straight line fitting uses the least squares method to obtain a mathematical expression that accurately describes the four sides of the chip. Each side is represented by a straight line equation, effectively reducing the edge information error caused by noise or defects. Calculate the coordinates of the four intersection points where the four straight lines intersect pairwise based on the mathematical expressions of the four sides of the chip, thereby determining the coordinates of the four corner points of the chip. These intersection points are the accurate edge coordinates of the chip. At the same time, in order to obtain the accurate coordinates of the fiducial marks in the target substrate image, use the adaptive local threshold segmentation method to dynamically adjust the threshold according to the brightness change of the local image, so as to accurately segment the fiducial marks under complex lighting conditions. Combine with the multi-scale feature extraction method to enhance the detection accuracy of the fiducial marks. The multi-scale feature extraction can identify the mark shapes of different sizes and calculate the center coordinates of the fiducial marks through the feature matching algorithm to obtain the accurate coordinates of the fiducial marks.

[0016] Apply adaptive binarization processing to the target substrate image, dynamically adjust the binarization threshold according to the local brightness change of the image, so as to ensure that the fiducial marks can still be clearly segmented under different lighting conditions and reduce the false detection caused by uneven lighting. Through this processing, a binarized substrate image is obtained, making the fiducial marks form an obvious contrast with the background. Perform connected component labeling on the binarized substrate image, label all white pixel regions through the connected component analysis algorithm, and extract the set of connected components containing the candidate fiducial marks. Perform shape classification processing on the set of connected components of the candidate fiducial marks. By calculating the geometric features of the connected components, including area, aspect ratio, perimeter, roundness, edge complexity, etc., classify the candidate regions according to the fiducial mark type, and obtain the classified candidate regions of the fiducial marks. Since the fiducial marks have different shapes, such as circular, cross-shaped or rectangular, perform multi-scale feature extraction on the classified candidate regions of the fiducial marks. Through the scale space analysis method, extract the contour features of the fiducial marks at different scales, so as to ensure that no matter how the size of the marks changes, their shapes and boundaries can be accurately identified, and the accurate contour features of each type of mark are obtained. Based on the sub-pixel accurate position coordinates of each fiducial mark, adopt the sub-pixel edge detection method, such as the gradient interpolation-based or sub-pixel fitting algorithm, to improve the accuracy of edge detection to the sub-pixel level, and obtain the sub-pixel accurate position coordinates of each fiducial mark. Based on the sub-pixel accurate position coordinates of each fiducial mark, establish a fiducial mark topology map. By calculating the relative position relationship and topological connection between each mark point, analyze the arrangement pattern of the fiducial marks on the substrate, and construct a local coordinate system with the mark center as the origin and the X-axis parallel to the main axis of the substrate, so as to establish a stable coordinate reference system. After constructing the local coordinate system, through coordinate transformation, map the coordinates of all fiducial mark points to the global substrate coordinate system to ensure that the positions of all fiducial marks are consistent with the global coordinates of the substrate, and obtain the accurate coordinates of the fiducial marks.

[0017] 103. Based on the accurate coordinates of the chip edge and the accurate coordinates of the fiducial marks, solve the patch process compensation parameter matrix of the four-coordinate system of the upper and lower cameras-nozzle-substrate during the patching process; Specifically, an upper camera coordinate system is established based on the imaging space of the upper vision camera. This coordinate system describes the corresponding relationship between the chip image captured by the upper camera and the actual chip in space. At the same time, a lower camera coordinate system is established based on the imaging space of the lower vision camera to characterize the mapping relationship between the substrate image captured by the lower camera and the substrate coordinates. A chip coordinate system is established based on the geometric characteristics of the chip and the pick-and-place center of the nozzle, and a substrate coordinate system is established based on the geometric structure of the substrate and the fiducial point information, obtaining the initial mathematical representations of the complete four spatial coordinate systems. Based on the initial mathematical representations of the four spatial coordinate systems, transformation matrices between the coordinate systems are constructed to describe the conversion relationships between each coordinate system. A transformation matrix between the upper camera and the chip is established, which describes how the pixel coordinates on the imaging plane of the upper camera are mapped to the actual physical coordinates in the chip coordinate system; a transformation matrix between the lower camera and the substrate is established, which is used to calculate the projection relationship of the fiducial marks on the substrate in the lower camera coordinate system; a transformation matrix between the nozzle and the substrate is established, which is used to describe the spatial movement relationship of the nozzle during the mounting process and map it to the substrate coordinate system, forming the mathematical model expression of the vision system. According to the mathematical model expression of the overall vision system, an overall vision error model is constructed, which is used to describe the system deviation caused by factors such as camera distortion, calibration error, and mechanical error. To solve this error model, its parameters are expanded, expressing the complex error relationship as a set of nonlinear equations, and this set of equations is solved by numerical calculation methods to obtain the initial solution of the high-dimensional nonlinear optimization problem. Since this optimization problem involves multiple variables and is mutually coupled, it is difficult to converge or has a high computational complexity using the traditional gradient descent method. Therefore, a hierarchical Levenberg-Marquardt optimization strategy is designed, which effectively combines the advantages of gradient descent and Newton's method to improve the optimization convergence speed and computational stability. In the specific implementation process of the Levenberg-Marquardt optimization strategy, a two-stage iterative solution mode is adopted to reduce the influence of variable coupling on the optimization process. In the first stage, the lower camera parameters are fixed, and only the upper camera and nozzle parameters are optimized, thus ensuring the stability of the preliminary optimization and at the same time improving the alignment accuracy between the upper camera coordinate system and the chip coordinate system; in the second stage, the upper camera parameters are fixed, and only the lower camera and nozzle parameters are optimized, thus further optimizing the mounting accuracy and reducing the error accumulation caused by nozzle deviation. During the entire iterative process, an adaptive step size control strategy is adopted to make the optimization process improve the computational efficiency while ensuring convergence, avoid local optimal traps, and obtain the accurate transformation parameters between each coordinate system. According to the accurate transformation parameters between each coordinate system, the system reprojection error is calculated and globally optimized to improve the accuracy and stability of the system. The calculation of the reprojection error is based on the optimized transformation parameters, back-projecting all fiducial points and detection points, calculating the deviation between the actual coordinates and the theoretical coordinates, and optimizing the error through the least squares fitting method to make the final coordinate transformation relationship more accurate.After global optimization, a chip mounting process compensation parameter matrix is finally obtained. This matrix is used to correct the spatial error during the chip mounting process, improve the accuracy and consistency of chip mounting, and thus ensure that the chip can be accurately placed at the target position on the substrate.

[0018] 104. Calculate the X-Y-θ three-dimensional alignment compensation vector during the chip mounting process according to the chip mounting process compensation parameter matrix; Specifically, perform grid area division on the real-time chip image and the real-time substrate image, that is, divide the entire image into several small grid areas, so as to improve the efficiency of feature extraction and reduce the situation of missing local features. After completing the grid division, in order to accelerate the calculation, assign independent threads to each grid area, so that the feature extraction of different areas is carried out simultaneously, and thus quickly extract feature points through spatial parallel processing, and obtain the initial feature point set after parallel processing. Perform feature enhancement on the initial feature point set after parallel processing to improve the accuracy of subsequent matching. The key to feature enhancement lies in using the multi-scale analysis method, that is, extracting feature points at different scales and normalizing these feature points to eliminate the influence of scale change on feature matching. Through this process, a multi-scale enhanced feature point set is obtained, thereby enhancing the stability of the feature points and enabling them to maintain good matching effects under different shooting angles and different lighting conditions. Based on the multi-scale enhanced feature point set, perform FAST corner extraction on the real-time chip image. The FAST corner detection algorithm is an efficient corner detection method that quickly extracts key feature points in the image and generates a set of descriptors for the chip feature points and the substrate feature points. In order to improve the matching efficiency, construct a KD-tree index structure for the set of descriptors of the chip feature points and the substrate feature points. The KD-tree is a data structure used for fast retrieval of high-dimensional data, which can quickly find the nearest neighbor matching points in a large-scale data set and obtain a set of effective matching point pairs. Calculate the spatial transformation relationship based on the set of effective matching point pairs. In order to ensure the calculation accuracy, use the singular value decomposition method to solve the optimal rigid transformation matrix. The calculation steps include: denote the chip feature point set as E, the corresponding substrate feature point set as F, then calculate the centroids of E and F, and perform data decentralization processing to eliminate the influence of coordinate offset on the calculation. Construct the covariance matrix H = E'F T, where \(T\) represents the transpose of the matrix. Subsequently, the covariance matrix \(H\) is subjected to singular value decomposition to obtain the optimal rotation matrix and translation vector. The rotation matrix represents the rotation error that occurs during the chip placement process, while the translation vector reflects the offset error of the chip in the X-Y plane. Finally, the spatial transformation relationship between the chip and the substrate is determined. According to the spatial transformation relationship, the displacement in the X-Y plane and the rotation angle \(\theta\) are parsed and extracted from the transformation matrix to generate a registration error vector, which describes the deviation between the current actual position of the chip and the target position of the substrate. Based on the chip placement process compensation parameter matrix, the registration error vector is transformed from the camera coordinate system to the machine coordinate system to ensure that the calculation results are consistent with the device control coordinate system. The X-Y-\(\theta\) three-dimensional registration compensation vector is obtained.

[0019] 105. Based on the X-Y-\(\theta\) three-dimensional registration compensation vector, calculate the optimal chip placement trajectory parameters and generate an execution instruction sequence.

[0020] Specifically, a kinematic model is established based on the chip mounting process. This model is used to express the entire trajectory change of the chip from picking to mounting, including the dynamic evolution of parameters such as translation, rotation, and acceleration. The kinematic model is based on the theory of robot kinematics and combines the physical structure characteristics of the mounter. The motion trajectory of the nozzle in different coordinate systems is described by a coordinate transformation matrix, forming an expression of the mounting trajectory in the motion space. Based on this motion trajectory expression, a convex optimization objective function is constructed. The core of this objective function is to balance the mounting speed and mounting accuracy simultaneously. Among them, speed optimization mainly focuses on minimizing the motion time, while accuracy optimization focuses on minimizing the terminal position error and rotation angle error. To ensure that the optimization problem has a global optimal solution during the calculation process and can be efficiently solved in practical engineering, a mathematical optimization model with convex characteristics is constructed so that the objective function can converge to the optimal solution among all feasible solutions, thereby ensuring the optimality of the mounting trajectory. Motion constraint conditions are added to the convex optimization mathematical model, and a series of constraint conditions are added to the optimization problem to ensure the feasibility of the motion trajectory. For example, during the chip mounting process, the speed, acceleration, and angular velocity of the mounting head are physically limited by the mechanical system. At the same time, it is ensured that the chip will not slide or become unstable due to excessive acceleration during the entire motion process. Therefore, speed, acceleration, and force control constraints are imposed on the convex optimization mathematical model, and considering the maximum load capacity and dynamic response characteristics of the mounting equipment, the original unconstrained optimization problem is transformed into an optimization problem with boundary constraints. To improve the feasibility of the optimization calculation, the continuous optimization problem is discretized, that is, the trajectory is segmented on the time axis, and a discrete optimization method is used to solve the entire mounting path, and an optimized planning model is obtained. This model can provide the optimal motion trajectory reference during the actual mounting process. The sequential quadratic programming algorithm is applied to the planning model. The basic principle of the sequential quadratic programming algorithm is: at each moment, a quadratic approximation of the objective function is constructed, and at the same time, the constraint conditions are linearly approximated, and the optimal solution is continuously approximated by iteratively solving a series of quadratic programming subproblems. In each iteration process, the sequential quadratic programming algorithm will calculate the gradient information of the current optimization variable and update the optimization direction to continuously reduce the value of the objective function, thereby gradually converging to the optimal solution. To improve the convergence speed and ensure that the calculation process is completed within a finite time, a convergence condition is set, that is, when the change amount of the objective function is less than a preset threshold, the iteration process terminates, and finally the optimal mounting trajectory parameters are obtained. These parameters are used in the mounting control system to guide the motion path of the nozzle. According to the optimal mounting trajectory parameters, the mounting force control parameters are dynamically calculated in combination with the chip size. Since chips of different sizes have different sensitivities to pressure during mounting, the force control strategy is adjusted during the mounting process to ensure that the chip will not be damaged due to excessive pressure, and at the same time, ensure sufficient adhesion force to make the chip stably adhere to the substrate surface.By calculating the force on the real-time computing chip and dynamically adjusting the force control parameters according to the change of the placement trajectory, it is ensured that the entire placement process is both safe and accurate. The calculated trajectory information and force control parameters are transmitted to the PID controller to achieve closed-loop control. The PID controller dynamically adjusts the placement process according to the information fed back by the real-time sensor, so as to ensure that the chip is smoothly placed on the substrate according to the calculated optimal trajectory and compensate for the offset caused by environmental interference or mechanical error. A complete execution instruction sequence is obtained, and this instruction sequence is used to drive the actuator of the mounter, so as to ensure that the chip placement process can be completed under the conditions of the optimal path, the optimal speed and the optimal accuracy, and realize high-efficiency and high-precision chip placement processing.

[0021] In the embodiment of the present invention, by constructing an overall vision error model of the upper and lower cameras - nozzle - substrate four-coordinate system, the problem of error accumulation caused by traditional separate calibration is effectively solved, and the alignment accuracy is significantly improved; the spatial parallel processing and image pyramid multi-scale processing technologies are adopted to greatly improve the feature extraction and matching speed; special feature extraction algorithms are designed for fiducial marks of different shapes to enhance the system's recognition ability for various marks; the chip placement process is modeled as a real-time convex optimization problem and solved by the sequential quadratic programming algorithm to achieve an effective balance between placement speed and accuracy; automatic calculation of alignment parameters is realized, the manual intervention link is eliminated, and the production consistency and reliability are improved; through the multi-level feature processing strategy and the backup algorithm mechanism, the robustness of the system in a complex production environment is enhanced, and it can adapt to various chip placement processing scenarios.

[0022] In a specific embodiment, the process of executing step 101 may specifically include the following steps: Configure an annular light source on the upper vision camera arranged above the nozzle assembly and set the first exposure time and the first gain value, configure a flexible surface light source on the lower vision camera arranged above the substrate stage and set the second exposure time and the second gain value to obtain an initial chip image and an initial substrate image; Synchronously trigger the upper vision camera and the lower vision camera through a high-speed image acquisition card and transmit the initial chip image and the initial substrate image to obtain an image data stream; Perform grayscale processing on the initial chip image and the initial substrate image to obtain a first chip image and a first substrate image, and perform histogram equalization processing on the first chip image and the first substrate image to obtain a second chip image and a second substrate image; Perform Gaussian filtering and edge enhancement processing on the second chip image and the second substrate image to obtain a target chip image and a target substrate image.

[0023] Specifically, an upper vision camera is configured above the nozzle assembly, and an annular light source is configured for it. The annular light source can effectively reduce the influence of shadows and ensure the uniformity of the image. At the same time, appropriate exposure time and gain value are set. The first exposure time ( ), which determines the response time of the camera sensor to light, and the first gain value ( ), which is used to adjust the brightness of the image. A lower vision camera is configured above the substrate stage, and a flexible surface light source is configured for it. The flexible light source provides uniform illumination and avoids quality fluctuations of the image caused by uneven illumination. Similarly, the second exposure time and the second gain value are adjusted according to the characteristics of the lower vision camera. Through the collaborative work of the upper and lower vision cameras, an initial chip image and an initial substrate image are obtained, which respectively represent the image data of the target chip and the target substrate. The image data of the target chip and the target substrate are synchronously collected and transmitted. A high-speed image acquisition card is used to synchronously trigger the upper and lower vision cameras to ensure the consistency of the two images in time and space. The purpose of synchronous triggering is to avoid mismatching of image data due to time delay. The image acquisition card transmits these two initial images to form an image data stream, and each frame of the image in the data stream contains the image information captured at a specific time point. The initial images are grayscale processed to convert the color images into single grayscale value images to simplify subsequent processing steps. The grayscale formula uses the weighted average method, that is, the red, green, and blue components of each pixel are weighted and summed through a certain weight to obtain the grayscale value of the pixel. Assuming that the RGB value of a certain pixel in the original image is , and , then the grayscale value of this pixel is calculated by the following formula: ; where, is the pixel value of the image after grayscale processing, , and are the pixel values of the red, green, and blue components respectively. By grayscale processing each pixel, the initial chip image and the initial substrate image are respectively converted into grayscale images to obtain the first chip image and the first substrate image. The images are histogram equalized to improve the contrast of the images and make the brightness distribution of the images more uniform. Histogram equalization adjusts the grayscale value distribution of the image so that the grayscale level distribution of the output image is as uniform as possible. Assuming that the grayscale value of the original grayscale image is , and its grayscale histogram is , then the grayscale value after equalization is calculated by the following formula: ; where, Indicates the grayscale value The new value after equalization, is the total number of pixels in the image, is the gray level The number of pixels. Through this process, the second chip image and the second substrate image are obtained. Gaussian filtering and edge enhancement processing are performed on the second chip image and the second substrate image. The purpose of Gaussian filtering is to remove noise in the image and smooth image details, thereby highlighting the edge information of the object. Gaussian filtering is a linear smoothing filter, and its filter kernel function is: ; where, is the value of the Gaussian filter at coordinate , is the standard deviation of the Gaussian distribution, which controls the width of the filter. Through Gaussian filtering, the noise and details in the image are reduced, making the image smoother. Edge enhancement processing is performed to highlight the edge information in the image, enhance the contour of the object, and make subsequent edge detection and feature extraction more obvious. The edge enhancement method enhances the gradient change in the image through a convolution operator, such as the Sobel operator or the Laplacian operator. The edge-enhanced image highlights the edge features of the chip and the substrate. Through the above steps, the target chip image and the target substrate image are finally obtained.

[0024] In a specific embodiment, the process of performing step 102 may specifically include the following steps: Perform Canny edge detection on the target chip image to obtain an initial chip edge map, and perform morphological processing on the initial chip edge map to obtain a chip edge map with noise removed and edges connected; Apply the probabilistic Hough transform algorithm to the chip edge map with noise removed and edges connected to extract line segments, obtaining a set of chip edge line segments; Group the chip edge line segments by direction to obtain four groups of edge line segments representing the four sides of the chip, and perform linear fitting on the four groups of edge line segments grouped by direction respectively to obtain the mathematical expressions of the four sides of the chip; Calculate the coordinates of the four intersection points of the four lines intersecting pairwise based on the mathematical expressions of the four sides of the chip to obtain the precise coordinates of the chip edge; Perform adaptive local threshold segmentation and multi-scale feature extraction on the target substrate image to obtain the precise coordinates of the fiducial mark.

[0025] Specifically, perform Canny edge detection on the target chip image to extract the edge information in the chip image. The Canny edge detection algorithm processes through multiple stages. It smooths the image by Gaussian filtering to reduce the influence of noise, then calculates the gradient magnitude and direction of the image, and finally uses non-maximum suppression and hysteresis thresholding to accurately locate the edges. Assume the gradient magnitude of the image is and its gradient direction is , then the core steps of the Canny algorithm are described as: ; ; Among them, and are the gradients of the image in the and directions respectively. The Canny algorithm generates the initial chip edge map through these steps. Perform morphological processing on the initial chip edge map to remove noise and connect the edges. Morphological processing includes erosion and dilation operations. Among them, the dilation operation can connect the broken parts of the edges, while erosion removes small noise and unnecessary details. After morphological processing, a chip edge map with noise removed and edges connected is obtained. Use the probabilistic Hough transform algorithm for straight line segment extraction. The Hough transform maps the straight lines in the image space to the parameter space and finds the parameters most likely to represent the straight lines through the accumulation in the parameter space. Assume a straight line in the image space is represented as , where is the distance from the straight line to the origin, and is the angle of the straight line. By performing probabilistic Hough transform on each edge point in the chip edge map, a parameter space is generated, and the set of chip edge line segments is extracted. Perform direction grouping on the set of chip edge line segments, cluster the edge line segments according to their directions, and obtain four groups of direction-clustered edge line segments representing the four sides of the chip. The core idea of direction grouping is to group the edge line segments according to the direction of the line segments, so that the direction difference of the line segments in each group is the smallest. For example, use the k-means clustering algorithm to group the directions of each line segment. In this way, the edge line segments in the image are classified according to their relative directions, and the sets of edge line segments of the four sides of the chip are obtained respectively. For each group of direction-clustered edge line segments, use the least squares method for straight line fitting to obtain the mathematical expressions of the four sides of the chip. According to the endpoint coordinates of the edge line segments, the best-fitting straight line is obtained by minimizing the fitting error. Assume the endpoint coordinates of a certain edge line segment are and , then the equation of the fitting straight line is expressed as: ; Among them, is the slope of the straight line, is the intercept of the straight line. Solved by the least squares method, minimizing the fitting error: ; Finally, the mathematical expressions for each side are obtained, represented as the equations of four straight lines. These straight line equations represent the four sides of the chip. Based on the equations of these four straight lines, the coordinates of the four intersection points where the four straight lines intersect pairwise are calculated to obtain the exact coordinates of the chip edge. Assume the equations of two straight lines are respectively: ; ; By solving the linear system of these two equations, the intersection point coordinates are obtained: ; ; By calculating the intersection points of each pair of straight lines, the coordinates of the four intersection points on the chip edge are finally obtained, and these intersection points are the exact coordinates of the chip edge. For the target substrate image, adaptive local threshold segmentation is performed. Different thresholds are used for segmentation in different regions to better adapt to the uneven illumination of the substrate image. In each local region, by calculating the local average or median of the region, an adaptive threshold is set to segment the image and obtain a binary substrate image. The markers in the substrate image are extracted by multi-scale feature extraction, extracting features at different scales to cope with the different sizes of the target markers. In the extraction process, the scale space method is used, and features at different scales are extracted through multiple smoothing and Gaussian blur processes. By this method, the markers on the substrate are effectively extracted, and the exact coordinates of the reference markers are obtained. For example, during the chip bonding process, if there is a circular marker on the target substrate, after adaptive local threshold segmentation, the circular marker will be extracted as a binary image region. And through the multi-scale feature extraction method, the circular marker is recognized at different scales to ensure that the position of the marker can be accurately extracted even if the size of the marker changes greatly.

[0026] In a specific embodiment, the process of performing adaptive local threshold segmentation and multi-scale feature extraction on the target substrate image to obtain the exact coordinates of the reference markers may specifically include the following steps: Apply adaptive binarization processing to the target substrate image to obtain a binary substrate image, and perform connected component labeling on the binary substrate image to obtain a set of connected components containing candidate reference markers; Perform shape classification processing on the set of connected components of the candidate reference markers to obtain the classified candidate regions of the reference markers, and perform multi-scale feature extraction on the classified candidate regions of the reference markers to obtain the exact contour features of each type of marker; Locate the exact contour features of each type of mark with sub-pixel accuracy to obtain the sub-pixel level exact position coordinates of each reference mark; Based on the sub-pixel level exact position coordinates of each reference mark, establish a reference mark topology map, calculate the relative position relationship and topological connection between each mark point, construct a local coordinate system with the mark center as the origin and the X-axis parallel to the main axis of the substrate, and map each mark point into the global substrate coordinate system through coordinate transformation to obtain the exact coordinates of the reference mark.

[0027] Specifically, perform adaptive binarization on the target substrate image. Adaptive binarization is used to solve the problem of uneven image illumination. Adaptive binarization dynamically selects different thresholds according to the brightness of the local area of the image. For each pixel point , calculate the mean or median of the local window where the point is located, and compare this local statistic with the threshold to determine whether the pixel point is the foreground. The set threshold varies according to the statistical information of the local window, and the binarization operation is implemented in the following way: ; where, is the pixel value of the target substrate image at point , is the window size, is the radius of the window. In this way, the illumination change of the local area is processed to obtain a clear binarized substrate image. Perform connected component labeling on the binarized substrate image to identify the connected components in the image and label them as different regions. By traversing each pixel in the image and labeling adjacent pixels, if two pixels belong to the same object (i.e., they are connected), they are labeled as the same class. Connected component labeling is implemented by depth-first search or breadth-first search algorithms to obtain a set of connected components containing candidate reference marks. For each connected component of the candidate reference mark, perform shape classification processing. Filter out the regions that meet the reference mark characteristics according to the morphological features of the marked region (such as area, aspect ratio, roundness, etc.). For example, by calculating the aspect ratio of the circumscribed rectangle of each connected component, where and are the width and height of the rectangle respectively. If this value is close to 1, it is determined as a circular or nearly circular mark; calculate the roundness of the shape, where is the area of the region, is the perimeter of the region. Based on these shape features, the candidate marked regions are classified into different types to screen out possible fiducial marks. The classified candidate regions of fiducial marks enter the multi-scale feature extraction stage. The goal of multi-scale feature extraction is to extract the accurate contour features of the marks at different scales to ensure accurate recognition regardless of the size change of the marks. The multi-scale feature extraction methods include Gaussian pyramid and Laplacian pyramid. By performing different scale smoothing processes on the image, the edge information at different scales is extracted. Assume the image at a certain scale is represented as where is the scale factor. After scale conversion, the obtained edge features are represented by the following Difference of Gaussians (DoG) filtering: ; where is the Gaussian filter, and are different scale factors. Through this processing, the edge information at different scales is extracted to obtain more accurate contour features. On this basis, the accurate contour features of each type of mark are located at sub-pixel accuracy. Through interpolation technology, the positions of the feature points in the image are accurately located. By interpolation methods such as bilinear interpolation or cubic interpolation, more accurate position coordinates than the original pixels are obtained. For example, applying a cubic interpolation function near the edge point to obtain the accurate position coordinates at the sub-pixel level: ; where represents the kernel function of cubic interpolation, where and are the coordinate offsets relative to the target pixel point . Through such processing, the positioning accuracy of the fiducial marks is ensured to reach the sub-pixel level. Based on the accurate position coordinates at the sub-pixel level of each fiducial mark, a fiducial mark topology map is constructed. The role of the topology map is to describe the relative position relationship and topological connection between each mark point. By calculating the distance and angle between each pair of mark points, the topological relationship between the mark points is constructed. Let the distance and the included angle between the mark points be respectively: ; ; By calculating the relative position relationships between all the marked points, a topological graph of the fiducial marks is constructed, where each node represents a fiducial mark and the edges represent the relative position relationships between them. A local coordinate system with the mark center as the origin and the X-axis parallel to the main axis of the substrate is constructed. For each fiducial mark, its center serves as the origin of this local coordinate system, and the X-axis direction is determined by calculating the angular relationships between the marked points. Through coordinate transformation, each marked point is transformed from the local coordinate system to the global substrate coordinate system. Assume the coordinates in the local coordinate system are , and the coordinates in the global substrate coordinate system are , then the local coordinate system is transformed to the global coordinate system through the following rotation and translation transformations: ; where is the rotation angle between the local coordinate system and the global coordinate system, and is the position of the origin of the local coordinate system in the global coordinate system. Through these processing steps, the precise position coordinates of the fiducial marks are obtained.

[0028] In a specific embodiment, the process of executing step 103 may specifically include the following steps: Based on the imaging space of the upper vision camera, an upper camera coordinate system is established, based on the imaging space of the lower vision camera, a lower camera coordinate system is established, based on the chip, a chip coordinate system is established, and based on the substrate, a substrate coordinate system is established, obtaining the initial mathematical representations of the four space coordinate systems; Based on the initial mathematical representations of the four space coordinate systems, transformation matrices between the coordinate systems are constructed, including the transformation matrix between the upper camera and the chip, the transformation matrix between the lower camera and the substrate, and the transformation matrix between the nozzle and the substrate, obtaining the mathematical model expression of the overall vision system; According to the mathematical model expression of the overall vision system, an overall vision error model is constructed, and the overall vision error model is expanded according to the parameters to obtain the initial solution of the high-dimensional non-linear optimization problem; For the high-dimensional non-linear optimization problem, a hierarchical Levenberg-Marquardt optimization strategy is designed, adopting a two-stage iterative solution mode: in the first stage, the lower camera parameters are fixed and the upper camera and nozzle parameters are optimized; in the second stage, the upper camera parameters are fixed and the lower camera and nozzle parameters are optimized, and the iterative process is controlled by an adaptive step size to obtain the precise transformation parameters between the coordinate systems; According to the precise transformation parameters between the coordinate systems, the system reprojection error is calculated and globally optimized to obtain the patch process compensation parameter matrix.

[0029] Specifically, establish the upper field of view camera coordinate system, the lower field of view camera coordinate system, the chip coordinate system, and the substrate coordinate system. Through the mathematical expressions of these coordinate systems and the construction of transformation matrices, provide data support for subsequent visual error correction and compensation. When establishing the upper field of view camera coordinate system, consider the origin, direction of the upper field of view camera imaging space, and the internal and external parameters of the camera. When setting the camera coordinate system, set the imaging plane of the camera as -plane, and set the -axis along the camera optical axis direction as the optical axis direction of the camera. The transformation relationship between the camera coordinate system and the world coordinate system is represented by a rotation matrix and a translation vector : ; Among them, is the representation of a point in the camera coordinate system, and is the representation of this point in the world coordinate system. The transformation matrix of the upper field of view camera is the and in the above formula, and its purpose is to map the points in the world coordinate system to the imaging space of the upper field of view camera. Similarly, the establishment of the lower field of view camera coordinate system is based on the setting of the camera imaging space. Among them, the rotation matrix and transformation matrix of the lower field of view camera coordinate system are represented as and . In order to uniformly process the establishment of the substrate coordinate system, the selection of the substrate coordinate system is usually to select a local coordinate system on the substrate surface and calibrate the position of the substrate through relevant external sensors. The chip coordinate system is calibrated through the physical feature points of the chip to ensure that the coordinate system is aligned with the actual position of the chip. Finally, four spatial coordinate systems are obtained: the upper camera coordinate system, the lower camera coordinate system, the chip coordinate system, and the substrate coordinate system. Construct the transformation matrix between the coordinate systems according to the initial mathematical representations of these coordinate systems. These transformation matrices include the conversion relationships between the camera coordinate system and the chip coordinate system, the lower camera coordinate system and the substrate coordinate system, and the nozzle coordinate system and the substrate coordinate system. By combining the transformation matrix between the upper field of view camera and the chip, the transformation matrix between the lower field of view camera and the substrate, and the transformation matrix between the nozzle and the substrate, the mathematical model of the overall vision system is obtained: ; ; ; Through these transformation matrices, a complete transformation from the upper vision camera coordinate system to the substrate coordinate system is achieved. Based on the mathematical model of the overall vision system, an overall vision error model is constructed. The key to the error model lies in accurately measuring the matching accuracy between coordinate systems. For example, assuming that the transformation error between the chip coordinate system and the substrate coordinate system of a point is expressed as , the error model is described by the following formula: ; According to this error model, the errors between coordinate systems are expanded to obtain an initial solution to a high-dimensional non-linear optimization problem. In this process, by introducing an error function, the objective function of the optimization problem is expressed as: ; where represents the error of the th point, and are the transformation parameters to be optimized. Then, a hierarchical Levenberg-Marquardt optimization strategy is designed for the high-dimensional non-linear optimization problem. The Levenberg-Marquardt algorithm combines the Gauss-Newton method and the gradient descent method, and it balances the two optimization methods by controlling the size of the parameter . In the two-stage iterative solution mode of the algorithm, in the first stage, the camera parameters are fixed, and the transformation parameters between the upper camera and the nozzle are optimized. The specific optimization process is represented by the following update formula: ; where is the Jacobian matrix, is the error vector, is the Levenberg-Marquardt correction parameter, and is the identity matrix. By continuously adjusting the parameter , an optimization process from rough to fine is achieved. In the second stage, the upper camera parameters are fixed, and the transformation parameters between the lower camera and the nozzle are optimized. The optimization formula is similar to that in the first stage. Through step-by-step optimization, accurate transformation parameters between coordinate systems are obtained, and the optimization of the entire vision system is completed. According to the accurate transformation parameters, the reprojection error of the system is calculated and globally optimized. The reprojection error is expressed as: ; By minimizing the reprojection error, the optimal patch process compensation parameter matrix is finally obtained.

[0030] In a specific embodiment, the process of executing step 104 may specifically include the following steps: Perform grid region division on the real-time chip image and the real-time substrate image, assign independent threads to each grid region, and extract feature points through spatial parallel processing to obtain an initial feature point set after parallel processing; Perform feature enhancement on the initial feature point set after parallel processing to obtain a multi-scale enhanced feature point set; Based on the multi-scale enhanced feature point set, perform FAST corner extraction on the real-time chip image to obtain a descriptor set of chip feature points and substrate feature points, and construct a KD tree index structure for the descriptor set of chip feature points and substrate feature points to obtain a set of effective matching point pairs; Based on the set of effective matching point pairs, calculate the spatial transformation relationship through the singular value decomposition method; the specific steps are as follows: Denote the chip feature point set as E and the corresponding substrate feature point set as F, calculate the centroid of the two point sets and perform a centering process, and construct the covariance matrix H = E'T (where T represents the transpose of the matrix). T Perform singular value decomposition on the covariance matrix H to obtain the rotation matrix and translation vector, and obtain the spatial transformation relationship; According to the spatial transformation relationship, parse and extract the X-Y plane displacement and the rotation angle θ, generate a registration error vector, and based on the patch process compensation parameter matrix, convert the registration error vector from the camera coordinate system to the machine coordinate system to obtain an X-Y-θ three-dimensional registration compensation vector.

[0031] Specifically, perform grid division on the real-time acquired image, divide the entire image area into multiple small grids, and perform independent feature extraction within each grid unit. Set the image size as , divided into grid regions, and the size of each grid region is defined as , , and assign the entire feature extraction task to multiple independent threads for parallel processing. Within each grid, use a feature point detection algorithm to extract key feature points to form an initial feature point set. The feature point set obtained by parallel processing is denoted as , where represents the feature points extracted within the th grid region. The purpose of parallel computing is to reduce the time complexity of feature point extraction, from reduced to , while ensuring that feature points are evenly distributed throughout the image range to improve the reliability of matching. Perform feature enhancement on the initial feature point set to enhance key features at different scales in the image. The feature enhancement method uses a scale space method, that is, detect the response of the same feature points at different scales and select the points that are most stable to scale changes. Let the scale parameter be , and the feature point set at different scales is defined as: ; Among them, represents the set of feature points extracted at scale . After scale normalization, a multi-scale enhanced feature point set is obtained. Based on the multi-scale enhanced feature point set, the FAST corner extraction algorithm is performed on the real-time chip image to detect fast-changing corner features. The FAST corner detection calculates the response value of the corner based on the pixel brightness change. For a certain pixel , the neighborhood pixel set is defined, and the brightness change is calculated: ; Among them, is the brightness value at the neighborhood pixel , is the central pixel brightness value. If exceeds the set threshold, then this point is considered a corner. Through this method, the corner feature point sets of the chip image and the substrate image are obtained, and the descriptor sets of the feature points are calculated. Let the chip feature point set be , and the substrate feature point set be . The descriptor sets are respectively expressed as: ; To improve the efficiency of feature point matching, a tree-shaped index structure is used for searching, and a KD-tree index structure is constructed. The KD-tree uses a space partitioning method to transform high-dimensional data into a tree structure, improving the matching efficiency. The process of constructing the KD-tree is as follows: Let the feature point have coordinates in the -dimensional space. Then the steps for constructing the KD-tree are: Select the dimension such that the variance of the data in this dimension is the largest; Select the median as the partitioning point in the dimension, and divide the data into left and right subtrees; Recursively construct the KD-tree until all data points are included in the leaf nodes. After the KD-tree is constructed, the nearest neighbor search algorithm is used to find the set of matching point pairs : ; Based on the set of matching point pairs, calculate the spatial transformation relationship between the chip and the substrate. Calculate the centroids of the chip feature point set and the substrate feature point set : ; ; Perform a centering process on the feature point set and calculate the covariance matrix: ; Perform singular value decomposition on the covariance matrix : ; wherein, and are the left singular vector matrix and the right singular vector matrix respectively, is a diagonal matrix. The rotation matrix is calculated as follows: ; The translation vector is calculated as follows: ; Obtain the spatial transformation relationship from the chip to the substrate, including the rotation matrix and the translation vector . Based on this spatial transformation relationship, analyze and extract the displacement and rotation angle of the chip in the X-Y plane : ; wherein, and are the components of the translation vector on the X-axis and Y-axis respectively, and are the elements of the rotation matrix respectively. According to the patch process compensation parameter matrix, convert the calculated alignment error vector from the camera coordinate system to the machine coordinate system. Assume the patch process compensation matrix is , then the conversion relationship is: ; Finally, obtain the X-Y- three-dimensional alignment compensation vector.

[0032] In a specific embodiment, the process of performing step 105 may specifically include the following steps: Establish a kinematic model based on the chip mounting process to obtain the expression of the mounting trajectory in the motion space; Construct a convex optimization objective function based on the expression of the mounting trajectory in the motion space, balance the mounting speed and accuracy respectively, and obtain a convex optimization mathematical model; Add motion constraint conditions to the convex optimization mathematical model to obtain a constrained optimization problem, and apply boundary conditions to the constrained optimization problem to convert the continuous optimization problem into a discrete optimization problem to obtain a planning model; Apply the sequential quadratic programming algorithm to the planning model, construct a quadratic approximation of the objective function and a linear approximation of the constraint conditions at each moment, and iteratively solve the quadratic programming sub-problem. Set the convergence condition as the change of the objective function being less than the preset target value to obtain the optimal mounting trajectory parameters; According to the optimal placement trajectory parameters, dynamically calculate the placement force control parameters in combination with the chip size, and transfer the trajectory information and force control parameters to the PID controller for closed-loop control to obtain the execution instruction sequence.

[0033] Specifically, establish a kinematic model of the chip placement process, where the movement of the nozzle is represented as a trajectory function that changes with time. Assume that the pose state of the nozzle includes position and rotation angle , then the trajectory of the nozzle in the motion space is expressed as: ; where, represents the position and attitude vector of the nozzle at time , and is the translation coordinate in the three-dimensional space, is the rotation angle around the vertical axis. The kinematic equation of the nozzle is controlled by the velocity and acceleration : ; where, is the Jacobian matrix of the system, is the control input, including the linear velocity and angular velocity. Considering the dynamic characteristics of the placement process, optimize the placement trajectory to complete it within a finite time while ensuring accuracy. Therefore, based on the kinematic trajectory equation, construct an optimization objective function to balance the placement speed and accuracy, and establish a convex optimization mathematical model. When designing the objective function, simultaneously consider minimizing the motion time and maximizing the placement accuracy, and achieve it by constructing a trade-off function: ; where, is the optimization objective, is the weight parameter, is the motion speed, is the placement error, is the total time of the placement process. The significance of this objective function is to make the nozzle complete the placement in the shortest time while minimizing the final placement error. In order to make the optimization problem meet the actual physical constraints, add motion constraint conditions, such as the maximum speed, maximum acceleration of the nozzle and the condition of avoiding collisions during the placement process. Let the maximum speed be , and the maximum acceleration be , then the constraint conditions are expressed as: ; ; At the same time, in order to ensure the stability of the chip during the placement process, add a constraint on the rotation angle change rate : ; Convert the continuous optimization problem into a constrained optimization problem and discretize it to make it suitable for numerical solution. Through discretization, the time interval is divided into small time steps, and the length of each time step is . Then the trajectory planning is written as: ; where represents the pose state at the -th discrete time step, and are the velocity and acceleration respectively. After discretizing the optimization objective function and constraints, a planning model is obtained, and this model is solved by the sequential quadratic programming algorithm. During the solution process, an iterative method is adopted. At each moment, a quadratic approximation of the objective function and a linear approximation of the constraints are constructed, transforming the problem into multiple quadratic optimization sub-problems. Assume that the optimization variable at the current moment is , and its increment is solved by the following optimization problem: ; The constraint conditions are: ; where is the Hessian matrix of the quadratic optimization problem, is the gradient vector, is the constraint matrix, is the constraint vector. This optimization problem is solved by quadratic programming and iterated at each time step until the change in the objective function is less than the set convergence threshold. After calculating the optimal placement trajectory parameters, combined with the chip size, the placement force control parameters are dynamically calculated. Let the area of the chip be , and the required placement pressure be . Then the pressure calculation formula is: ; where is the chip mass, is the gravitational acceleration. The placement force control parameters are used to ensure that the chip will not be damaged or displaced due to uneven force during the placement process. The calculated trajectory information and force control parameters are transmitted to the PID controller for closed-loop control. The output of the PID controller is determined by the error signal , and the calculation formula is as follows: ; where is the proportional gain, is the integral gain, is the differential gain. The PID controller adjusts the movement of the nozzle according to the feedback to make the chip mounting process stable and accurate. An execution instruction sequence is generated, including parameters such as the movement path, rotation angle, and mounting pressure of the nozzle, and the chip mounting task is executed through the control system to ensure that the chip is accurately mounted on the substrate, improving the mounting accuracy and stability.

[0034] In this embodiment, according to the optimal mounting trajectory parameters, the mounting force control parameters are dynamically calculated in combination with the chip size, and the trajectory information and force control parameters are transmitted to the PID controller for closed-loop control to obtain an execution instruction sequence, including: constructing a segmented control model for different stages of the chip soldering process, identifying the dynamic characteristic differences between the chip's air movement stage and the contact mounting stage, establishing a full drive control model for the air movement stage, and establishing a nonholonomic system control model with contact constraints for the contact mounting stage to obtain the segmented control system expression of the chip soldering process; establishing a state space equation set for the segmented control system expression, representing the chip mounting process as , where x is the system state vector, including position, velocity, and attitude parameters, u is the control input vector, d(t) is the external disturbance vector, and f(x) and g(x) are respectively the state-related nonlinear function matrices, obtaining a nonlinear system model considering external disturbances; for the nonlinear system model, design two segmented stable controllers u 1 (x) and u 2 (x), which are respectively applicable to the air movement stage and the contact mounting stage, where u 1 (x) optimizes the dynamic response speed, u 2 (x) ensures precise positioning and appropriate contact force, and verifies that each of the two controllers satisfies the Lyapunov stability condition within its action region, obtaining a set of segmented stable controllers; based on the set of segmented stable controllers, the controller extension method is used to construct an intermediate auxiliary controller u m (x,J), where J∈[0,1] is a smooth transition parameter, satisfying the conditions u m (x,0)=u 1 (x) and u m (x,1)=u 2 (x), and at the same time ensuring is continuous on J∈[0,1], and the continuous smooth transition of the two-stage controller is realized through the exponential weighting function ( is a positive constant for adjusting smoothness), obtaining a fully continuous and smooth composite controller; for the composite controller, an extended state observer is constructed to estimate the system state vector and the external disturbance vector , and the dynamic equation of the observer is , where z is an auxiliary variable, β is the observer gain matrix, and the observation error is quickly suppressed through high-gain feedback to obtain a state estimation result with enhanced anti-interference ability. Based on the state estimation result with enhanced anti-interference ability, the compound controller is corrected for interference compensation to form the final control law. , where is the generalized inverse matrix of g(x), which effectively suppresses various external interferences during the mounting process to obtain a robust mounting control strategy. The robust mounting control strategy is converted into an instruction sequence for each driving axis of the nozzle, and interpolation calculations are performed on the trajectories and force control parameters of the X, Y, Z, and θ four motion axes. The sampling frequency is set to 1 kHz, and a real-time error compensation mechanism is added to each axis motion instruction. When the position error exceeds the set threshold, the remaining trajectory is recalculated based on the current state to obtain a complete execution sequence including position, speed, acceleration, and force control instructions. Closed-loop control is implemented according to the complete execution sequence, and a dynamic weight PID control strategy is adopted to adaptively adjust the PID parameters according to the mounting stage, with key optimization of the force control accuracy during the contact stage. Combining the feedforward control of the motion trajectory planning, an end-point positioning accuracy of ±1 μm and a force control accuracy of ±0.1 N are achieved, and an accurate chip mounting action is obtained.

[0035] The optical alignment method for chip soldering and mounting in the embodiments of the present invention has been described above. Next, the optical alignment system for chip soldering and mounting in the embodiments of the present invention will be described. Please refer to Figure 2 , an embodiment of the optical alignment system for chip soldering and mounting in the embodiments of the present invention includes: An acquisition module 201, configured to acquire the target chip image and the target substrate image during the mounting process through an upper vision camera disposed above the nozzle assembly and a lower vision camera disposed above the substrate stage; An extraction module 202, configured to extract the precise coordinates of the chip edge in the target chip image and extract the precise coordinates of the fiducial mark in the target substrate image; A solution module 203, configured to solve the mounting process compensation parameter matrix of the four-coordinate system of the upper and lower cameras - nozzle - substrate based on the precise coordinates of the chip edge and the precise coordinates of the fiducial mark; A calculation module 204, configured to calculate the X - Y - θ three-dimensional alignment compensation vector during the chip mounting process according to the mounting process compensation parameter matrix; A generation module 205, configured to calculate the optimal mounting trajectory parameters and generate an execution instruction sequence based on the X - Y - θ three-dimensional alignment compensation vector.

[0036] Through the collaborative cooperation of the above-mentioned various components, by constructing an overall vision error model of the upper and lower cameras - suction nozzle - substrate four-coordinate system, the problem of error accumulation caused by traditional separate calibration is effectively solved, and the alignment accuracy is significantly improved; by adopting spatial parallel processing and image pyramid multi-scale processing technologies, the feature extraction and matching speed are greatly improved; a dedicated feature extraction algorithm is designed for fiducial marks of different shapes, enhancing the system's recognition ability for various marks; the chip mounting process is modeled as a real-time convex optimization problem and solved by a sequential quadratic programming algorithm, achieving an effective balance between mounting speed and accuracy; the automatic calculation of alignment parameters is realized, eliminating the manual intervention link, and improving production consistency and reliability; through a multi-level feature processing strategy and a backup algorithm mechanism, the robustness of the system in a complex production environment is enhanced, adapting to various chip placement processing scenarios.

[0037] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described system, system, and unit can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0038] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0039] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An optical alignment method for chip placement processing, characterized in that: The method comprises: The target chip image and the target substrate image in the patch process are collected by an upper field camera configured above the nozzle assembly and a lower field camera configured above the substrate stage; Extracting precise coordinates of chip edges in the target chip image, and extracting precise coordinates of reference marks in the target substrate image; Based on the precise coordinates of the chip edge and the precise coordinates of the reference mark, solving the patch process compensation parameter matrix of the upper and lower camera-nozzle-substrate four-coordinate system during the patch process; Calculating the XY-θ three-dimensional alignment compensation vector in the chip mounting process according to the chip mounting process compensation parameter matrix; Based on the XY-θ three-dimensional alignment compensation vector, the optimal placement trajectory parameters are calculated and an execution instruction sequence is generated.

2. The optical alignment method for chip mounting processing according to claim 1, characterized in that: The method of collecting the target chip image and the target substrate image during the patch process by using an upper field of view camera configured above the nozzle assembly and a lower field of view camera configured above the substrate carrier stage includes: An upper field of view camera disposed above the nozzle assembly is provided with an annular light source and a first exposure time and a first gain value are set, and a lower field of view camera disposed above the substrate stage is provided with a flexible surface light source and a second exposure time and a second gain value are set, to obtain an initial chip image and an initial substrate image; The upper field of view camera and the lower field of view camera are synchronously triggered by a high-speed image acquisition card, and the initial chip image and the initial substrate image are transmitted to obtain an image data stream; Performing grayscale processing on the initial chip image and the initial substrate image to obtain a first chip image and a first substrate image, and performing histogram equalization processing on the first chip image and the first substrate image to obtain a second chip image and a second substrate image; Gaussian filtering and edge enhancement processing are performed on the second chip image and the second substrate image to obtain a target chip image and a target substrate image.

3. The optical alignment method for chip mounting processing according to claim 1, characterized in that: The step of extracting precise coordinates of the chip edge in the target chip image and extracting precise coordinates of the reference mark in the target substrate image includes: Performing Canny edge detection on the target chip image to obtain an initial chip edge map, and performing morphological processing on the initial chip edge map to obtain a chip edge map with noise removed and edges connected; Applying a probabilistic Hough transform algorithm to the chip edge image with noise removed and edges connected to extract straight line segments, thereby obtaining a chip edge segment set; Directionally grouping the chip edge line segment set to obtain four groups of directionally clustered edge line segments representing the four edges of the chip respectively, and performing straight line fitting on the four groups of directionally clustered edge line segments respectively to obtain mathematical expressions of the four edges of the chip; Based on the mathematical expressions of the four sides of the chip, the coordinates of four intersection points where two or three straight lines intersect each other are calculated to obtain the precise coordinates of the chip edge; Adaptive local threshold segmentation and multi-scale feature extraction are performed on the target substrate image to obtain precise coordinates of the reference mark.

4. The optical alignment method for chip mounting processing according to claim 3, characterized in that: The step of performing adaptive local threshold segmentation and multi-scale feature extraction on the target substrate image to obtain precise coordinates of the reference mark includes: Applying adaptive binarization processing to the target substrate image to obtain a binarized substrate image, and performing connected region labeling on the binarized substrate image to obtain a connected region set including candidate reference labels; Performing shape classification processing on the connected region set of the candidate fiducial markers to obtain classified fiducial marker candidate regions, and performing multi-scale feature extraction on the classified fiducial marker candidate regions to obtain accurate contour features of each type of marker; Performing sub-pixel precision positioning on the precise contour features of each type of mark to obtain the sub-pixel level precise position coordinates of each reference mark; Based on the sub-pixel precise position coordinates of each reference mark, a reference mark topology map is established, the relative position relationship and topological connection between each mark point are calculated, and a local coordinate system is constructed with the mark center as the origin and the X-axis parallel to the main axis of the substrate. Each mark point is mapped to the global substrate coordinate system through coordinate transformation to obtain the precise coordinates of the reference mark.

5. The optical alignment method for chip mounting processing according to claim 1, characterized in that: The method of solving the patch process compensation parameter matrix of the upper and lower camera-nozzle-substrate four-coordinate system during the patch process based on the precise coordinates of the chip edge and the precise coordinates of the reference mark includes: An upper camera coordinate system is established based on the imaging space of the upper field of view camera, a lower camera coordinate system is established based on the imaging space of the lower field of view camera, a chip coordinate system is established based on the chip, and a substrate coordinate system is established based on the substrate, to obtain initial mathematical representations of four space coordinate systems; Based on the initial mathematical representation of the four spatial coordinate systems, the transformation matrices between the coordinate systems are constructed, including the transformation matrix between the upper camera and the chip, the transformation matrix between the lower camera and the substrate, and the transformation matrix between the nozzle and the substrate, to obtain the mathematical model expression of the overall visual system; According to the mathematical model expression of the overall visual system, an overall visual error model is constructed, and the overall visual error model is expanded according to parameters to obtain an initial solution to the high-dimensional nonlinear optimization problem; Aiming at the high-dimensional nonlinear optimization problem, a hierarchical Levenberg-Marquardt optimization strategy is designed, and a two-stage iterative solution mode is adopted: in the first stage, the lower camera parameters are fixed, and the upper camera and nozzle parameters are optimized; in the second stage, the upper camera parameters are fixed, and the lower camera and nozzle parameters are optimized. The iterative process is controlled by adaptive step size to obtain accurate transformation parameters between various coordinate systems; According to the precise transformation parameters between the coordinate systems, the system reprojection error is calculated and globally optimized to obtain the patch process compensation parameter matrix.

6. The optical alignment method for chip mounting processing according to claim 1, characterized in that: The step of calculating the XY-θ three-dimensional alignment compensation vector in the chip mounting process according to the chip mounting process compensation parameter matrix includes: The real-time chip image and the real-time substrate image are divided into grid areas, and an independent thread is assigned to each grid area. Feature points are extracted through spatial parallel processing to obtain an initial feature point set after parallel processing. Performing feature enhancement on the initial feature point set after the parallel processing to obtain a multi-scale enhanced feature point set; Based on the multi-scale enhanced feature point set, FAST corner point extraction is performed on the real-time chip image to obtain a descriptor set of chip feature points and substrate feature points, and a KD tree index structure is constructed for the descriptor set of chip feature points and substrate feature points to obtain a valid matching point pair set; Based on the effective matching point pair set, the spatial transformation relationship is calculated by the singular value decomposition method; the specific steps are: the chip feature point set is recorded as E, the corresponding substrate feature point set is recorded as F, the centroid of the two point sets is calculated and decentralized, and the covariance matrix H=E'F is constructed. T , T represents the transpose of the matrix, and the covariance matrix H is subjected to singular value decomposition to obtain the rotation matrix and translation vector, and the spatial transformation relationship is obtained; According to the spatial transformation relationship, the XY plane displacement and the rotation angle θ are analyzed and extracted to generate an alignment error vector, and the alignment error vector is converted from the camera coordinate system to the machine coordinate system based on the patch process compensation parameter matrix to obtain an XY-θ three-dimensional alignment compensation vector.

7. The optical alignment method for chip mounting processing according to claim 1, characterized in that: The method of calculating the optimal placement trajectory parameters and generating an execution instruction sequence based on the XY-θ three-dimensional alignment compensation vector includes: A kinematic model is established based on the chip mounting process to obtain the mounting trajectory expression in the motion space; Based on the placement trajectory expression in the motion space, a convex optimization objective function is constructed to balance the placement speed and accuracy, thereby obtaining a convex optimization mathematical model; Adding motion constraints to the convex optimization mathematical model to obtain a constrained optimization problem, and applying boundary conditions to the constrained optimization problem to transform the continuous optimization problem into a discrete optimization problem to obtain a planning model; Applying a sequential quadratic programming algorithm to the planning model, constructing a quadratic approximation of the objective function and a linear approximation of the constraint conditions at each moment, solving the quadratic programming sub-problems iteratively, setting a convergence condition that the objective function change is less than a preset target value, and obtaining the optimal placement trajectory parameters; According to the optimal mounting trajectory parameters, the mounting force control parameters are dynamically calculated in combination with the chip size, and the trajectory information and force control parameters are transmitted to the PID controller for closed-loop control to obtain an execution instruction sequence.

8. An optical alignment system for chip placement processing, characterized in that: A system for performing an optical alignment method for chip bonding processing according to any one of claims 1 to 7, the system comprising: An acquisition module is used to acquire a target chip image and a target substrate image during the patch process through an upper field of view camera configured above the nozzle assembly and a lower field of view camera configured above the substrate carrier; An extraction module, used to extract the precise coordinates of the chip edge in the target chip image and the precise coordinates of the reference mark in the target substrate image; A solution module, used for solving the patch process compensation parameter matrix of the upper and lower camera-nozzle-substrate four-coordinate system during the patch process based on the precise coordinates of the chip edge and the precise coordinates of the reference mark; A calculation module, used for calculating the XY-θ three-dimensional alignment compensation vector in the chip mounting process according to the chip mounting process compensation parameter matrix; A generation module is used to calculate the optimal placement trajectory parameters and generate an execution instruction sequence based on the XY-θ three-dimensional alignment compensation vector.

Images