Full-viewpoint linkage suction nozzle micro-control method and system for patch machine material position and posture registration

Abstract

This invention discloses a full-viewpoint linkage nozzle micro-control method and system for material pose registration in a chip mounter. It obtains multi-source isomorphic point cloud data by acquiring depth maps using a full-viewpoint camera array. The multi-source isomorphic point cloud data is then fused and registered using the ICP algorithm. Based on the prior position and morphological information of the main objects in the chip mounter scene, the fused and registered point cloud data is iteratively optimized. Based on the iteratively optimized point cloud data, an iterative state control algorithm is used to model the pose registration process in the transfer step of chip eutectic bonding. According to the chip mounter material pose registration model, the entire process of pose registration for the chip mounter material is performed. This solves the technical problem of achieving fast, high-precision real-time control of the material gripping, transfer, and placement process in micron-level chip mounters. While ensuring the stability of the material transfer process in laser chip bonding, it increases the controllable process and improves the material transfer speed by reducing imperceptible points.

Description

Technical Field

[0001] This invention mainly relates to the field of chip manufacturing technology, specifically a full-view linkage nozzle micro-control method and system for material pose registration in a chip mounter. Background Technology

[0002] Semiconductor chips are widely used components in the electronics and information industry. Semiconductor chip production lines are primarily fully automated, large-scale production lines. The automation and high speed of the chip manufacturing process effectively reduce production costs. However, semiconductor chip processing requires high precision, which significantly impacts product quality. In the eutectic bonding process, the chip must be precisely placed on the eutectic solder within the eutectic temperature range. An unstable oxide layer on the solder surface is removed through friction. Subsequently, at the eutectic temperature, the solid phase between the chip and solder forms a liquid phase, achieving chip bonding. During this process, the relative positional accuracy between the chip and solder must be at the micrometer level. Excessive deviation will significantly alter the semiconductor chip's heat generation, thermal conductivity, and resistivity, leading to reduced reliable operating time, accelerated overheating, and a significant decrease in the lifespan and stability of electronic devices.

[0003] Chip eutectic bonding primarily involves two concurrent operations: eutectic bonding and material transfer. Eutectic bonding mainly includes heating and cooling steps, while material transfer requires the continuous long-distance movement of both the chip and substrate, encompassing three steps: grasping, transferring, and placing. Each step requires movement to a non-blind spot, pausing, and attitude calibration at the transition points to minimize process fluctuations caused by pose errors. The cumbersome material transfer process results in a significantly longer time commitment than eutectic bonding, severely limiting eutectic bonding efficiency. Therefore, achieving a rapid and automated material transfer method while maintaining high precision will effectively improve chip eutectic bonding efficiency and reduce product costs.

[0004] Chinese patent CN113013068B discloses a chip transfer method to improve transfer yield. The method utilizes a transfer structure equipped with a pressure detection element. The method involves a transfer mechanism picking up a chip and moving it a preset distance towards an adapter plate. Upon reaching the adapter plate, the pressure of the pressure detection element is checked against a preset value to determine if the chip is aligned with its target position. If a deviation exists, the transfer mechanism is adjusted; otherwise, the chip is aligned and soldered. This method, using pressure detection to determine chip position, significantly reduces the cost of the chip transfer device. However, this method is limited by the preset distance and pressure settings. Poor parameter settings, due to the lack of closed-loop adjustment, will lead to a significant decrease in chip yield. Furthermore, when the chip soldering or placement environment changes, the preset parameters prevent corresponding adjustments, resulting in a lack of adaptive adjustment capabilities. Therefore, the method has extremely poor stability and is unsuitable for high-precision chip placement processes.

[0005] Chinese patent CN107768285B discloses a method and system for picking up, transferring, and bonding semiconductor chips. The proposed method first uses a camera system to check the chip picking position in a vertical aiming line configuration. Then, a rotating arm moves two picking heads attached to each end of the arm to the picking position to pick up the chip. Next, the chip is moved above the alignment camera, and the chip's current orientation is obtained from the camera's image data. Based on this orientation, the rotating arm is used to adjust the chip's orientation to the target orientation. Finally, the chip is moved based on the difference between its current and target orientations and placed onto a carrier. Simultaneously, during the chip transfer process, a chip heating process is executed to ensure the chip reaches a set temperature before bonding with the carrier, preventing thermal shock. This method and system uses ordinary optical images to obtain the chip picking and placement positions, achieving rapid chip picking and placement. However, this method is limited by the detection method and separates the transfer process from the orientation adjustment process, increasing the overall working time and significantly increasing the time spent on chip soldering and mounting. There is still room for improvement. Summary of the Invention

[0006] This invention provides a full-viewpoint linkage nozzle micro-control method and system for material pose registration in a pick-and-place machine, which solves the technical problem of difficulty in achieving fast and high-precision real-time control of material gripping, transfer and placement processes in micron-level pick-and-place machines.

[0007] To solve the above-mentioned technical problems, the present invention proposes a full-viewpoint linkage nozzle micro-control method for material pose registration in a pick-and-place machine, comprising:

[0008] The full-view camera array acquires depth maps and obtains multi-source isomorphic point cloud data based on the depth maps. The full-view camera array is used to perform full-scene pose tracking of materials in the chip mounter.

[0009] The ICP algorithm is used to fuse and register multi-source isomorphic point cloud data to obtain single point cloud data.

[0010] Based on the prior position and shape information of the main objects in the patch scenario, the single point cloud data is iteratively optimized. The main objects in the patch scenario include chips, nozzles and substrates, and the substrates include carrier boards and chutes.

[0011] Based on the optimized point cloud data after iterative optimization, an iterative state control algorithm is used to model the pose registration process in the transfer step of chip eutectic bonding, and obtain the material pose registration model of the chip mounter.

[0012] Based on the material pose registration model of the pick and place machine, the pose registration of the pick and place machine materials is performed throughout the entire process.

[0013] Furthermore, based on the prior location and shape information of the main objects in the patch scene, iterative optimization of single-point cloud data includes:

[0014] Morphological segmentation methods are used to coarsely locate the main objects in the patch scene.

[0015] Based on the coarse positioning results, the single point cloud data is encoded.

[0016] A potential function is constructed to describe the physical model of the patch scene, where the specific formula of the potential function is:

[0017]

[0018] Where P' represents the optimized point cloud data, U1(P'), U2(P'), U3(P'), and U4(P') are the carrier plate constraint, chute constraint, nozzle lower surface constraint, and mean smoothing term corresponding to the point cloud P', respectively, and λ is the preset smoothing bias. Let B be the indicator function, B be the encoding result of encoding a single point cloud data, and δ be the prior distribution parameter description of the patch scene.

[0019] Based on the potential function, the gradient descent method is used to iteratively optimize the single point cloud data. The specific iterative optimization formula is as follows:

[0020]

[0021]

[0022] in, and These represent the (k+1)th and kth iteration results of the point cloud data P' at time t, respectively. This represents the result of the 0th iteration of the point cloud data P' at time t+1. This represents the derivative of U(P',B,δ), where Δ t P represents the displacement of a single point cloud data P from time t to time t+1.

[0023] Furthermore, the calculation formulas for the carrier plate constraint, chute constraint, nozzle lower surface constraint, and mean smoothing term are as follows:

[0024]

[0025] Where p' is any point in the point cloud P', (x',y',z') represents the position information of point p', U1(p'), U2(p'), U3(p'), and U4(p') are the constraints of the carrier plate, chute, nozzle lower surface, and mean smoothing term corresponding to point p', respectively, (L1,W1,H1) and (L2,W2,H2) represent the length, width, and height descriptions of the carrier plate and chute, respectively, (X1,Y1,Z1) and (X2,Y2,Z2) represent the geometric center positions of the carrier plate and chute, respectively, and (X3,Y3,Z3) is the center position of the nozzle lower surface. Let σ1, σ2, σ3, and σ4 be the pre-defined first, second, third, and fourth control coefficients, respectively, and mean(·) denotes the 8-nearest neighbor mean function. and These represent taking the partial derivatives with respect to x', y', and z', respectively.

[0026] Furthermore, the specific formula for the material pose registration model of the pick-and-place machine is as follows:

[0027]

[0028] Where L(t,k) and L(t,k+1) are the material poses sampled at time t for the kth and k+1th times, respectively, u(t,k) is the control quantity sampled at time t for the kth time, f(·) is the nonlinear relationship of the pose through control transformation, g(·) is the control signal selection strategy, K represents the total number of iterations between any two time points, and Q(P',t) and Q(P',t+1) represent the expected poses of the material at time t and t+1 based on the nozzle motion trajectory, respectively.

[0029] Furthermore, based on the pick-and-place machine material pose registration model, the entire process of pose registration for the pick-and-place machine materials includes:

[0030] Obtaining a finite time interval The tracking error within the range is specifically calculated using the following formula:

[0031]

[0032] Construct a transition matrix to convert the material pose registration model of the pick-and-place machine into:

[0033]

[0034]

[0035]

[0036]

[0037] in, and Let be the transition matrices before and after the update, respectively. Let e(t,K) and e(t+i,K) represent the error signals at time t and time t+i, respectively, for the Kth iteration. This represents the partial derivative with respect to |e(t+i,K)|, where u(t,i) is the control quantity sampled at time t and i-th. The initial value is set to J is the Jacobian matrix constructed based on the mouthpiece kinematic model, D is the relationship between the input control quantity and the change in mouthpiece joint angle, and N is a preset positive integer.

[0038] Based on the converted material pose registration model of the pick and place machine, the pose registration of the pick and place machine materials is performed throughout the entire process.

[0039] Furthermore, before fusing and registering multi-source isomorphic point cloud data based on the ICP algorithm, the following steps are also included:

[0040] Preprocessing of multi-source isomorphic point cloud data.

[0041] Furthermore, preprocessing of multi-source isomorphic point cloud data includes:

[0042] Three-dimensional erosion, bilateral filtering, and downsampling operations are performed on multi-source isomorphic point cloud data.

[0043] The present invention provides a full-viewpoint linkage nozzle micro-control system for material pose registration in a pick-and-place machine, comprising:

[0044] The present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the full-view linkage nozzle micro-control method for material pose registration in a pick-and-place machine provided by the present invention.

[0045] This invention proposes a full-viewpoint linkage nozzle micro-control method and system for material pose registration in a chip mounter. It acquires depth maps using a full-viewpoint camera array and obtains multi-source isomorphic point cloud data based on these maps. The multi-source isomorphic point cloud data is then fused and registered using the ICP algorithm. Based on the prior position and morphological information of the main objects in the chip mounter scene, the fused and registered point cloud data is iteratively optimized. Based on the iteratively optimized point cloud data, an iterative state control algorithm is used to model the pose registration process in the transfer step of chip eutectic bonding, obtaining a chip mounter material pose registration model. Based on this model, the entire process of pose registration for the chip mounter material is performed. This solves the technical problem of achieving rapid, high-precision, real-time control of the micron-level material gripping, transfer, and placement process in chip mounters. While ensuring the stability of the material transfer process in laser chip bonding, it increases the controllable process by reducing imperceptible points, thereby improving the material transfer speed and achieving cost reduction and efficiency improvement. Furthermore, it can acquire perceptible scene data throughout the entire process, ensuring a seamless material transfer path distribution for end-to-end control and reducing material transfer time overhead. Based on alternating open-loop and closed-loop control, and incorporating iterative control features to update the control process, it achieves high-precision path tracking throughout the entire process, improving control accuracy.

[0046] The beneficial effects of this invention specifically include:

[0047] (1) This invention designs a multi-source isomorphic cloud registration method based on a full-view camera array, constructs a high-precision point cloud scene of the entire process of eutectic patch, realizes full-process tracking of eutectic patch material transfer process, and realizes material attitude acquisition of the entire process.

[0048] (2) This invention solves the problem of nozzle position control during the material gripping, transfer and placement process of the pick and place machine by designing a full-view linkage nozzle micro-control method for material position registration of the pick and place machine, which increases the accuracy of the pick and place process, reduces the transfer process time and provides stable initial material conditions for subsequent chip bonding processes.

[0049] (3) This invention proposes an iterative state control algorithm, which solves the problem of real-time control of open-loop and closed-loop alternation in material transfer, ensuring the consistency of the long process of chip transfer and improving the stability of the process. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the overall structure of the all-view camera array according to Embodiment 2 of the present invention;

[0051] Figure 2 This is a flowchart of the full-view linkage nozzle micro-control method for material pose registration in a pick-and-place machine according to Embodiment 2 of the present invention;

[0052] Figure 3This is a flowchart of the multi-source isomorphic point cloud registration method according to Embodiment 2 of the present invention;

[0053] Figure 4 This is a schematic diagram of the iterative state control algorithm according to Embodiment 2 of the present invention;

[0054] Figure 5 This is a structural block diagram of the full-view linkage nozzle micro-control system for material position and orientation registration in a chip mounter according to an embodiment of the present invention.

[0055] Figure label:

[0056] 10. Memory; 20. Processor. Detailed Implementation

[0057] To facilitate understanding of the present invention, the present invention will be described more fully and in detail below with reference to the accompanying drawings and preferred embodiments, but the scope of protection of the present invention is not limited to the following specific embodiments.

[0058] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings, but the present invention can be implemented in many different ways as defined and covered by the claims.

[0059] Example 1

[0060] The present invention provides a full-viewpoint linkage nozzle micro-control method for material pose registration in a pick-and-place machine, comprising:

[0061] Step S101: Obtain a depth map based on the full-view camera array, and obtain multi-source isomorphic point cloud data based on the depth map. The full-view camera array is used to perform full-scene pose tracking of the materials in the pick-and-place machine.

[0062] Step S102: Perform fusion and registration of multi-source isomorphic point cloud data based on the ICP algorithm.

[0063] Step S103: Based on the prior position and shape information of the main objects in the patch scene, the point cloud data after fusion and registration is iteratively optimized. The main objects in the patch scene include chips, nozzles and substrates, and the substrates include carrier boards and chutes.

[0064] Step S104: Based on the iteratively optimized point cloud data, an iterative state control algorithm is used to model the pose registration process in the transfer step of chip eutectic bonding, and obtain the material pose registration model of the chip mounter.

[0065] Step S105: Perform full-process pose registration of the pick-and-place machine materials according to the pick-and-place machine material pose registration model.

[0066] The present invention provides a full-viewpoint linkage nozzle micro-control method for material pose registration in a chip mounter. This method acquires a depth map based on a full-viewpoint camera array and obtains multi-source isomorphic point cloud data from the depth map. It then fuses and registers the multi-source isomorphic point cloud data using the ICP algorithm. Based on the prior position and morphological information of the main objects in the chip mounter scene, iterative optimization is performed on the fused and registered point cloud data. Based on the iteratively optimized point cloud data, an iterative state control algorithm is used to model the pose registration process in the transfer step of chip eutectic bonding, obtaining a chip mounter material pose registration model. Based on this model, the entire process of pose registration for the chip mounter material is performed. This solves the technical problem of achieving rapid, high-precision, real-time control of the micron-level chip mounter material gripping, transfer, and placement process. While ensuring the stability of the material transfer process in laser chip bonding, it increases the controllable process by reducing imperceptible points, thereby improving the material transfer speed and achieving cost reduction and efficiency improvement. Furthermore, it can acquire perceptible scene data throughout the entire process, ensuring a seamless material transfer path distribution for end-to-end control and reducing material transfer time overhead. Based on alternating open-loop and closed-loop control, and incorporating iterative control features to update the control process, it achieves high-precision path tracking throughout the entire process, improving control accuracy.

[0067] Example 2

[0068] This invention provides a full-viewpoint linkage nozzle micro-control method for material pose registration in a pick-and-place machine. Based on a designed full-viewpoint camera array, it solves the problem of difficulty in achieving fast and high-precision real-time control of material gripping, transfer, and placement in micron-level pick-and-place machines due to the small size of the material, numerous blind spots, and low positioning accuracy.

[0069] like Figure 1 As shown, the material transport process of a pick-and-place machine can generally be broken down into three steps: gripping, transferring, and placing. During material transport, full-scene pose tracking of the material using a full-view camera array enables end-to-end pose registration. To integrate real-time sensing data from the full-view camera array and construct high-precision point cloud data for the entire material transport scene, providing information support for subsequent material pose tracking, this invention proposes a multi-source isomorphic point cloud registration optimization method based on the prior characteristics of multi-source isomorphic camera array data, and an iterative state control method for material registration.

[0070] Define the number of point clouds acquired by the full-view camera array as C, and define the point cloud data as P. c ={p0,p1,...,p n} c Where n represents the number of points in the point cloud data, and without loss of generality, any collection P c The location of the camera set is The algorithm flow is as follows:

[0071] Step 1: To improve the quality of point cloud data, morphological erosion, bilateral filtering, and the multi-nearest neighbor de-growth far-end downsampling proposed in this embodiment are sequentially applied to each point cloud data. Then, the center of the upper surface of the chip carrier platform is set as the origin. Based on the camera's intrinsic and extrinsic parameters and its position relative to the world coordinate system, the coordinate systems of each point cloud are transformed to the same coordinate system. Specifically, multi-nearest neighbor de-growth far-end downsampling involves performing a downsampling operation on point cloud data that satisfies the following inequality:

[0072]

[0073] Where Dis(·,·) calculates the Euclidean distance between two points, Max(·) calculates the maximum value in the set, and p near For P c The point cloud data closest to p.

[0074] Step 2: Based on the ICP algorithm, fuse and register the multi-source isomorphic point cloud data set into a single point cloud data P = {p0, p1, ..., p...} n}

[0075] Step 3: For the main objects in the patch scenario, such as chips, nozzles, and substrates (represented as U-shaped cuboids, with the main body being the carrier board and the recessed middle part being the channel), based on significant differences in color and position, automatic coarse localization can be achieved through morphological segmentation. Based on this coarse localization, P = {p0, p1, ..., p...} n Point cloud data encoding B = {b0, b1, ..., b} n}, where b i =0 indicates that it follows the distribution of the carrier plate, b i =1 indicates that it follows a chute distribution, b i =2 indicates that it follows the distribution on the lower surface of the nozzle, b i =-1 indicates that it is a point cloud of other parts.

[0076] Step 4: Based on the potential function calculation method in the proposed variational constraint point cloud iterative sampling method and the encoding obtained in Step 3, perform potential function constraint calculation on the point cloud corresponding to the main objects of each patch scene to obtain the conformity description of each point in the point cloud to the prior distribution.

[0077] Step 5: Based on the proposed variational constraint point cloud iterative sampling method, the gradient of the potential function result of the point cloud data is obtained, and gradient descent is performed to obtain the optimized point cloud P' through single-step iterative optimization, so that it conforms to the prior distribution and improves the positioning accuracy.

[0078] Step 6: Based on the current point cloud data P', a greedy algorithm is used to construct the shortest transfer trajectory Q without blind spots. Specifically, the nozzle movement trajectory Q is obtained through a greedy algorithm based on the current sampled optimized point cloud P': the movement trajectory cannot pass through imperceptible points and obstacles, and a reachable space is constructed based on the point cloud data P'; based on minimizing the movement trajectory and control signal changes, an arbitrary optimal trajectory is found within the reachable space using a greedy algorithm.

[0079] Step 7: Based on the proposed iterative state control algorithm, obtain the deviation e between the current pose and the expected transfer trajectory, select the strategy g(·) through the control signal, and query the transfer matrix. Select the optimal control signal u to perform micro-control of the nozzle and adjust the nozzle position.

[0080] Step 8: Is the process complete? If completed, end. If not completed, determine if the camera array has acquired new point cloud data. If yes, proceed to step 9; otherwise, return to step 4.

[0081] Step 9: Add the deviation e between the current pose and the expected transfer trajectory to the cumulative error. When the cumulative error increases more than N=5 times, update the transfer matrix based on the gradient descent method. And clear the accumulated error.

[0082] Step 10: Return to Step 1.

[0083] Based on the above method, registration and fusion of multi-source isomorphic point cloud data for a full-view camera array and full-process pose tracking of materials can be achieved.

[0084] Among them, the multi-source isomorphic point cloud registration method based on a full-view camera array includes the aforementioned point cloud optimization method and the proposed variational constraint point cloud iterative sampling method. The variational constraint point cloud iterative sampling method constructs prior distribution constraints based on the prior position and shape information of the main objects in the patch scene, thereby performing iterative sampling optimization on the existing point cloud. Let the sampled point cloud during the iterative sampling process be P'={p'0,p1',...,p' n Let it follow a distribution. A probabilistic graphical model can then be constructed to describe the iterative sampling process, namely:

[0085]

[0086] in U(·) is the potential function constructed based on the prior distribution, δ is the parameter description of the prior distribution of the patch scene, and k is the number of iterations. This will be explained in detail below.

[0087] The chip substrate is known to be U-shaped, with its main body being the carrier plate and the central recessed portion being the chute. Therefore, it can be simply described as the complement of the carrier plate (L1, W1, H1, X1, Y1, Z1) and the chute (L2, W2, H2, X2, Y2, Z2). Here, (L1, W1, H1, L2, W2, H2) represent the device's geometric parameters, and (X1, Y1, Z1, X2, Y2, Z2) represent the device's geometric center position. The nozzle, excluding the lower surface... Apart from this, other parts do not participate in the patch physical process, where (X3, Y3, Z3) is the center position of the lower surface. Let be the normal vector of the lower surface.

[0088] Typically, point cloud data errors originate from two sources: random errors caused by environmental interference and systematic errors caused by equipment. Random errors usually exhibit dispersion around the true value distribution. Although random errors exist at individual points in the point cloud data, the overall pose parameters of each device obtained through weighted summation can eliminate these random errors. Therefore, the geometric positions of each major object can be obtained by weighted summation of the point cloud sets of each device. That is, based on the encoding obtained in step 3, the point cloud corresponding to each device is located, and weighted summation is performed to obtain the geometric center of the device. Furthermore, based on the device information, a relatively accurate δ = (L1, L2, W1, W2, H1, H2, X1, Y1, Z1, X2, Y2, Z2, X3, Y3, Z3, ... A simplified, high-precision description of the patch scene is performed. Here, δ represents the prior distribution parameters of the patch scene, specifically the pose description of the main objects in the patch scene. Furthermore, a potential function U(P',B,δ) is constructed to describe the physical model of the patch scene, enabling a relatively accurate evaluation of the point cloud quality. This is achieved by fusing singular point calibration information and full-view camera array information to sample the required high-precision point cloud. U(P',B,δ) is constructed as follows:

[0089]

[0090] Where P' represents the optimized point cloud data, U1(P'), U2(P'), U3(P'), and U4(P') are the carrier plate constraint, chute constraint, nozzle lower surface constraint, and mean smoothing term corresponding to the point cloud P', respectively, and λ is the preset smoothing bias. This is an indicator function.

[0091] Furthermore, the constraints on each point p' in the point cloud P' are described:

[0092]

[0093] Wherein, mean(·) represents the mean of the parameters corresponding to the point and its 8 nearest neighbors. The 8 nearest neighbors can be quickly obtained by converting point cloud data into a depth map. Here, p' is any point in point cloud P', (x',y',z') represents the position information of point p', U1(p'), U2(p'), U3(p'), and U4(p') are the constraints of the carrier plate, chute, nozzle lower surface, and mean smoothing term corresponding to point p', respectively. (L1,W1,H1) and (L2,W2,H2) represent the length, width, and height of the carrier plate and chute, respectively. (X1,Y1,Z1) and (X2,Y2,Z2) represent the geometric center positions of the carrier plate and chute, respectively. (X3,Y3,Z3) is the center position of the nozzle lower surface. Let σ1, σ2, σ3, and σ4 be the pre-defined first, second, third, and fourth control coefficients, respectively. and These represent taking the partial derivatives with respect to x', y', and z', respectively.

[0094] Furthermore, the iterative formula can be given:

[0095]

[0096] Where k = 1, 2, 3...K represents the number of repeated sampling iterations, and the value of K is variable, depending on the computer's iteration execution speed. Since there is a certain time interval between camera array captures, the following iterative formula should be followed in actual use:

[0097]

[0098]

[0099] Where t = 1, 2, 3, ..., ∞ represents the time when the camera takes a picture. Let represent the result of the k-th iteration of the point cloud at time t. Equation 6 is the formula for iteratively updating the point cloud at time t, and Equation 7 is the formula for updating from time t to t+1. t P represents the displacement of the un-iterated point cloud P from time t to time t+1. For changes in device position, point cloud alignment can be performed based on the encoding B, and the displacement Δ of P at each point on time t can be obtained through interpolation. t (P), thereby updating the point cloud data in the time and iteration dimensions. It should be noted that during alignment, the point cloud P is converted into a depth map to accelerate alignment.

[0100] The variational constraint point cloud iterative sampling method proposed above effectively utilizes the prior distribution to achieve self-elimination of random errors, and minimizes the introduction of additional errors caused by prior distribution errors through gradient calculation. This significantly improves pose localization accuracy.

[0101] An iterative state control algorithm for material pose registration is used to ensure the stable and efficient operation of nozzle pose registration in the material transfer process.

[0102] In any given process, based on the iterative update of the point cloud P' described earlier, a trajectory distribution Q(P',t) can be planned. Without loss of generality, let the position of the nozzle at any given time be L(t) = [x(t),y(t),z(t),θ1(t),θ2(t)], and then we can define:

[0103]

[0104] in, The expected pose of the material at time t based on its trajectory. This refers to random disturbances caused by the combined effects of sensory errors and mechanical errors. The control method proposed in this invention aims to integrate the aforementioned sensory registration method to reduce random disturbances.

[0105] Assuming the number of point cloud iterations within the camera sequence shooting interval T is K, the pose registration process of the transfer step can be modeled as follows:

[0106]

[0107] Where L(t,k) is the material pose sampled at time t for the kth time, u(t,k) is the control quantity sampled at time t for the kth time, f(·) is the nonlinear relationship of pose through control transformation, and g(·) is the control signal selection strategy.

[0108] For the above process, the task of the control algorithm is to construct accurate nonlinear relationships f(·) and g(·) within a finite time interval. Within a range that allows the output L(t,k) to achieve the desired pose. Complete tracking. Therefore, the tracking error in this embodiment is defined as:

[0109]

[0110] In conventional control methods, for open-loop control problems, a kinematic model is typically constructed to obtain the nonlinear relationship f(·). However, it should be noted that f(·) is strongly correlated with the equipment state. When the equipment experiences wear or aging, the control accuracy will significantly decrease, and this deviation is unacceptable in the material transfer process of a pick-and-place machine. This deviation is usually reduced by constructing a closed-loop control, but in the Kth iteration, the camera array does not re-capture images, making it impossible to construct a feedback control process. Therefore, to improve equipment accuracy, a real-time update method for f(·) should be constructed based on the control process.

[0111] To simplify f(·) and reduce the computational cost of the update process, we can construct a transition matrix.

[0112]

[0113] Where D represents the relationship between the input control quantity (i.e., torque) and the change in the nozzle joint angle, J is the Jacobian matrix constructed based on the nozzle kinematic model, representing the partial derivative relationship between the nozzle tip and the change in angle, and the initial state of the transition matrix. It is easy to prove that:

[0114]

[0115]

[0116] Furthermore, a control signal selection strategy g(·) is constructed. To simplify the control process, g(·) is defined as:

[0117]

[0118] Furthermore, the specific errors of this method can be described, and an error signal can be constructed:

[0119]

[0120] Furthermore, based on the error signal t, the transition matrix can be constructed using the gradient descent method. Real-time updates:

[0121]

[0122] in, and These are the transition matrices before and after the update, respectively.

[0123] Based on equation (15), closed-loop information can be realized after each time t. The numerical update is used to reduce the perception and random disturbances caused by system errors when the dynamic model gradually deviates from the real device, thereby improving the control accuracy. However, in order to reduce the computational overhead, the cumulative gradient descent method is used to construct the matrix update formula, and the cumulative amount N=5 is designed.

[0124]

[0125] This embodiment achieves a full-process control method that alternates between open and closed loops by using open-loop control based on the transfer matrix and updating the transfer matrix based on the closed-loop signal. Compared with traditional control methods, this method maintains non-consistent control based on the transfer matrix during periods when the closed-loop signal is unavailable, achieving high-precision tracking and maintenance of the transfer path, effectively meeting the high-precision requirements of chip eutectic bonding.

[0126] The following will be combined with the appendix Figure 2The overall method of the embodiments of the present invention will be further described in detail.

[0127] Step S201: Based on a full-view camera array, multiple depth maps are acquired using the binocular imaging principle, and then converted into a point cloud map set {P1,...,P...}. C}

[0128] Step S202, for the point cloud set {P1,...,P...} C The point cloud position error caused by random interference is reduced by sequentially performing three-dimensional erosion, bilateral filtering, and downsampling operations.

[0129] Step S203, for the point cloud set {P1,...,P...} C Based on the camera's intrinsic and extrinsic parameters and the relative position in the world coordinate system, the ICP algorithm is used to fuse multi-source isomorphic point cloud data to construct a single point cloud P to achieve a comprehensive description of the spatial distribution of the patch scene.

[0130] Step S204: The positions of various parts of the point cloud data are initially determined using positional and chromaticity information. Specifically, for the substrate position, it should be located near the top of the substrate support platform, and the chromaticity should conform to the preset information of the corresponding product; for the chip position, it should be located near the top of the chip support platform, and the chromaticity should conform to the preset information of the corresponding product; for the lower surface of the nozzle, it should be located at the bottom of the robotic arm. Subsequently, corresponding codes are constructed for the point clouds that conform to the above information.

[0131] Step S205: Calculate the potential function of each point based on the potential function calculation method in the iterative sampling of variational constraint point cloud, and use the gradient descent method to optimize each point to obtain the optimized point cloud P', so that the point cloud data moves in a direction that conforms to the prior distribution.

[0132] Step S206: Based on point cloud data P', locate the material position and the nozzle position, and plan the optimal path Q based on a greedy algorithm with prerequisites such as shortest distance and no obstruction.

[0133] Step S207: Obtain the current error e based on the path Q and the current pose L, output the control signal using the iterative state control method, and perform the corresponding transition matrix. Update operation.

[0134] Step S208: If the camera acquires new point cloud data, return to step S202; otherwise, if the transfer operation is not completed, return to step S205; if the transfer operation is completed, proceed to step S209.

[0135] Step S209: End the nozzle micro-control section and automatically pick up / place materials through several preset open-loop control programs.

[0136] The following will be combined with the appendix Figure 3The multi-source isomorphic point cloud registration method of the present invention will be described in further detail.

[0137] Step S201: Based on a full-view camera array, multiple depth maps are acquired using the binocular imaging principle, and then converted into a point cloud map set {P1,...,P...}. C}

[0138] Step S202, for the point cloud set {P1,...,P...} C The point cloud position error caused by random interference is reduced by sequentially performing three-dimensional erosion, bilateral filtering, and downsampling operations.

[0139] Step S203, for the point cloud set {P1,...,P...} C Based on the camera's intrinsic and extrinsic parameters and the relative position in the world coordinate system, the ICP algorithm is used to fuse multi-source isomorphic point cloud data to construct a single point cloud P to achieve a comprehensive description of the spatial distribution of the patch scene.

[0140] Step S204: The positions of various parts of the point cloud data are initially determined using positional and chromaticity information. Specifically, for the substrate position, it should be located near the top of the substrate support platform, and the chromaticity should conform to the preset information of the corresponding product; for the chip position, it should be located near the top of the chip support platform, and the chromaticity should conform to the preset information of the corresponding product; for the lower surface of the nozzle, it should be located at the bottom of the robotic arm. Subsequently, corresponding codes are constructed for the point clouds that conform to the above information.

[0141] Step S305: Based on the encoding, locate the point cloud corresponding to each major scene object, and calculate the potential function of each point in the point cloud to quantify the degree of deviation of each point from the prior distribution.

[0142] Step S306: Perform gradient descent on the potential function of each point along the spatial direction to obtain the optimization vector, and then add it to the original point cloud after adjusting the weights to optimize the point cloud distribution.

[0143] Step S307: If the camera array acquires data again, the process ends; otherwise, return to step S305.

[0144] The following will be combined with the appendix Figure 4 The iterative state control method of the present invention will be described in further detail.

[0145] Step S401: Obtain the current expected pose based on the current motion transfer trajectory Q.

[0146] Step S402: Calculate the deviation e between the expected pose and the current pose L(t,k-1).

[0147] Step S403, based on the deviation and transition matrix A control signal selection strategy g(·) is adopted, with minimizing the current deviation e as the control objective, to find and output the optimal control signal u(t,k).

[0148] Step S404, based on u(t,k) and Calculate the expected displacement and correct L(t,k-1) to obtain the expected next pose L(t,k).

[0149] Step S405: Repeat steps 1-4 until k = K.

[0150] Step S406: Calculate the deviation based on the difference between L(t+1,0) acquired by the camera array and the expected pose, and add the deviation to the cumulative deviation.

[0151] Step S407: When the accumulated deviation is sufficient, perform gradient descent based on the accumulated deviation to update the transition matrix with minimizing the accumulated deviation as the optimization objective.

[0152] The present invention provides a full-viewpoint linkage nozzle micro-control method for material pose registration in a chip mounter. This method acquires a depth map based on a full-viewpoint camera array and obtains multi-source isomorphic point cloud data from the depth map. It then fuses and registers the multi-source isomorphic point cloud data using the ICP algorithm. Based on the prior position and morphological information of the main objects in the chip mounter scene, iterative optimization is performed on the fused and registered point cloud data. Based on the iteratively optimized point cloud data, an iterative state control algorithm is used to model the pose registration process in the transfer step of chip eutectic bonding, obtaining a chip mounter material pose registration model. Based on this model, the entire process of pose registration for the chip mounter material is performed. This solves the technical problem of achieving rapid, high-precision, real-time control of the micron-level chip mounter material gripping, transfer, and placement process. While ensuring the stability of the material transfer process in laser chip bonding, it increases the controllable process by reducing imperceptible points, thereby improving the material transfer speed and achieving cost reduction and efficiency improvement. Furthermore, it can acquire perceptible scene data throughout the entire process, ensuring a seamless material transfer path distribution for end-to-end control and reducing material transfer time overhead. Based on alternating open-loop and closed-loop control, and incorporating iterative control features to update the control process, it achieves high-precision path tracking throughout the entire process, improving control accuracy.

[0153] Specifically, the key improvements in this embodiment include:

[0154] (1) Key improvement point one: It provides a brand-new approach to material pose registration for the chip mounter. It adopts a full-process open-loop and closed-loop alternating control method, which enhances the anti-interference ability of material pose registration and realizes stable and fast material transfer.

[0155] (2) Key improvement point two: Based on the full-view camera array, a multi-source isomorphic point cloud data registration method was proposed, which realized the high-precision construction of point clouds in the whole scene of chip mounting process and improved the working efficiency of chip mounting machine.

[0156] (3) Key improvement point three: An iterative state control algorithm is proposed, which realizes the alternating open and closed loop pose control and parameter adaptive update, has strong tracking ability of system hardware parameters, and improves the stability of the control method.

[0157] (4) Key improvement point four: Full scene perception method based on full view camera array and point cloud data registration and fusion method, and iterative state control algorithm based on full-process open-loop and closed-loop alternating control.

[0158] Reference Figure 5 The full-view linkage nozzle micro-control system for material pose registration in a pick-and-place machine proposed in this embodiment includes a memory 10, a processor 20, and a computer program stored in the memory 10 and run on the processor 20. When the processor 20 executes the computer program, it implements the steps of the full-view linkage nozzle micro-control method for material pose registration in a pick-and-place machine proposed in this embodiment.

[0159] The specific working process and working principle of the full-view linkage nozzle micro-control system for material position registration in the pick-and-place machine of this embodiment can be referred to the working process and working principle of the full-view linkage nozzle micro-control method for material position registration in the pick-and-place machine of this embodiment.

[0160] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A full-viewpoint linkage nozzle micro-control method for patch machine material position and posture registration, characterized in that, The method includes: The depth map is obtained based on the full-view camera array, and multi-source isomorphic point cloud data is obtained based on the depth map. The full-view camera array is used to perform full-scene pose tracking of the materials of the chip mounter. The multi-source isomorphic point cloud data is fused and registered based on the ICP algorithm to obtain single point cloud data. Based on the prior location and shape information of the main objects in the patch scene, iterative optimization of single-point cloud data is performed. The main objects in the patch scene include chips, nozzles, and substrates, and the substrate includes a carrier board and a chute. The iterative optimization of single-point cloud data based on the prior location and shape information of the main objects in the patch scene includes: Morphological segmentation methods are used to coarsely locate the main objects in the patch scene; Based on the coarse positioning results, the single point cloud data is encoded; Construct a potential function to describe the physical model of the patch scene, wherein the specific formula of the potential function is: ， in, To optimize point cloud data, , , and They are respectively point clouds The corresponding carrier plate constraints, chute constraints, nozzle lower surface constraints, and mean smoothing term are as follows: For the preset smooth bias, For indicator functions, The encoded result after encoding a single point cloud data. The prior distribution parameters for the patch scene are described, and the calculation formulas for the carrier plate constraint, chute constraint, nozzle lower surface constraint, and mean smoothing term are as follows: ， in, For point clouds any point in, For point Location information, They are respectively with point The corresponding carrier plate constraints, chute constraints, nozzle lower surface constraints, and mean smoothing term are as follows: and These represent the length, width, and height of the carrier plate and the chute, respectively. and These represent the geometric center positions of the carrier plate and the chute, respectively. The center position of the lower surface of the suction nozzle. Let be the normal vector of the lower surface of the suction nozzle. These are the pre-defined first, second, third, and fourth control coefficients, respectively. This represents finding the 8-nearest neighbor mean function. , and Representing respectively to , and Find the partial derivative; Based on the potential function, the gradient descent method is used to iteratively optimize the single point cloud data. The specific iterative optimization formula is as follows: ， ， in, and Representing point cloud data respectively In the The first moment Second and third The result of the second iteration Representing point cloud data In the The first moment The result of the second iteration Indicates to Find the derivative. Represents single-point cloud data In time Until time Displacement at point; Based on the optimized point cloud data after iterative optimization, an iterative state control algorithm is used to model the pose registration process in the transfer step of chip eutectic bonding, and obtain the material pose registration model of the chip mounter. Based on the material pose registration model of the pick and place machine, the pose registration of the pick and place machine materials is performed throughout the entire process.

2. The full-viewpoint linkage nozzle micro-control method for material pose registration in a pick-and-place machine according to claim 1, characterized in that, The specific formula for the material pose registration model of the chip mounter is as follows: ， in, and The first Time of the first Second and third The material pose of the second sample. For the first Time of the first Control quantity for the next sample This represents the nonlinear relationship between pose and control transformation. To select a strategy for control signals, This represents the total number of iterations between any two time points. and These represent the material at different times. and The expected pose is based on the movement trajectory of the suction nozzle.

3. The full-viewpoint linkage nozzle micro-control method for material pose registration in a pick-and-place machine according to claim 2, characterized in that, Based on the material pose registration model of the pick-and-place machine, the full-process pose registration of the pick-and-place machine materials includes: Obtaining a finite time interval The tracking error within the range is specifically calculated using the following formula: ； Construct a transition matrix to convert the pick-and-place machine material pose registration model into: ， in, and These are the transition matrices before and after the update, respectively. and They represent the first Time and At that moment, the Error signal of the next iteration Indicates to Find the partial derivative. For the first Time of the first Control quantity for the next sample The initial value is set to , The Jacobian matrix is ​​constructed based on the mouthpiece kinematic model. This relates the input control quantity to the change in nozzle joint angle. It is a preset positive integer; Based on the converted material pose registration model of the pick and place machine, the pose registration of the pick and place machine materials is performed throughout the entire process.

4. The full-viewpoint linkage nozzle micro-control method for material pose registration in a pick-and-place machine according to claim 3, characterized in that, Before fusing and registering the multi-source isomorphic point cloud data based on the ICP algorithm, the following steps are also included: Preprocessing of multi-source isomorphic point cloud data.

5. The full-viewpoint linkage nozzle micro-control method for material pose registration in a pick-and-place machine according to claim 4, characterized in that, Preprocessing of multi-source isomorphic point cloud data includes: Three-dimensional erosion, bilateral filtering, and downsampling operations are performed on multi-source isomorphic point cloud data.

6. A full-viewpoint linkage nozzle micro-control system for material position and orientation registration in a pick-and-place machine, the system comprising: The memory (10), the processor (20), and the computer program stored in the memory (10) and executable on the processor (20) are characterized in that the processor (20) implements the steps of the method according to any one of claims 1 to 5 when executing the computer program.

Images