Full-viewpoint collaborative eutectic chip mounter material attitude detection method and system

Through the full viewpoint camera array and position matrix construction technology, the problem of closed-loop control difficulty in material transfer of eutectic chip machines is solved, and the stability monitoring and feedback of material position is realized, and the performance and quality of chip packaging is improved.

CN119090956AActive Publication Date: 2024-12-06CENT SOUTH UNIV +1
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202410981974.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-22
Publication Date
2024-12-06
Estimated Expiration
2044-07-22

AI Technical Summary

Technical Problem

The existing eutectic chip machines cannot achieve closed-loop control during material transfer, resulting in the failure to fully guarantee the stability of the material position and posture, affecting the performance and quality of chip packaging.

Method used

The full viewpoint image of the material is obtained by using a full viewpoint camera array, and the pose matrix of the material is constructed through preprocessing, corner detection and parallax calculation. The pose matrix sequence is used for Gaussian fitting to obtain the pose change function, which is used to control material pickup. The device adjusts the position of the material.

Benefits of technology

It realizes comprehensive monitoring and feedback on the material transfer process, ensures the stability of the chip welding process, and improves the stability of the material position and packaging quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090956A_ABST
    Figure CN119090956A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of eutectic chip mounting, in particular to a full-viewpoint collaborative eutectic chip mounter material posture detection method and system. The method comprises the following steps: acquiring a full-viewpoint image of a material at the same moment by using a full-viewpoint camera array, and preprocessing and binarizing the full-viewpoint image to obtain a binary image; searching angular points in the binary image to obtain a main viewpoint image set; solving spatial positions of material vertexes by using the main viewpoint image and parallax in the main viewpoint image, and constructing a pose matrix of the material through the spatial positions of the material vertexes; obtaining pose matrixes at different moments to obtain a pose matrix sequence; gaussian fitting is carried out on the pose matrix sequence to obtain a pose change function of the material; the material picking device is controlled based on the position and posture change function to adjust the position and posture of the material in the transferring process, and the stability of the chip pasting and welding process is guaranteed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of eutectic chip placement, and in particular to a full-viewpoint coordinated eutectic chip placement machine material posture detection method and system. Background Art

[0002] Eutectic placement machines are usually used on SMT (surface mount technology) production lines, and are mainly used for precise welding of chips and substrates. It is necessary to control the adsorption and transfer equipment to separately adsorb the substrate and chip and transfer them to the welding area. In the process of transferring the chip to the welding area, it is necessary to adjust the chip according to its position and posture so that the chip can be accurately transferred to the welding position on the substrate for welding.

[0003] The accuracy of posture is crucial to the performance and quality of chip packaging. However, the current material transfer process is mostly open-loop control. There is little response to interference encountered during the transfer process, and closed-loop control of the transfer process cannot be achieved. The stability of the material posture during the transfer process cannot be fully guaranteed. Summary of the invention

[0004] The present invention provides a full-viewpoint coordinated eutectic placement machine material posture detection method and system, which provides the position posture information of the eutectic placement machine during the material transfer process and provides feedback signals for the control of the eutectic placement machine during the material transfer process.

[0005] In order to achieve the above object, the present invention is implemented by the following technical solutions:

[0006] In a first aspect, the present invention provides a method for detecting material posture of a eutectic bonding machine in a full-viewpoint collaborative manner, comprising:

[0007] Use a full-viewpoint camera array to obtain full-viewpoint images of the material at the same time, and pre-process the full-viewpoint images;

[0008] Binarizing the preprocessed full viewpoint image to obtain a binary image;

[0009] Find corner points in the binary image and obtain the corner point set of the binary image;

[0010] The original full viewpoint image of the binary image containing two or more corner points is used as the main viewpoint image to obtain a main viewpoint image set;

[0011] The full viewpoint image including four corner points in the main viewpoint image set is taken as the main viewpoint image;

[0012] The spatial positions of the material vertices are obtained by using the parallax between the main viewpoint image and the primary viewpoint image, and the material pose matrix is ​​constructed by the spatial positions of the material vertices;

[0013] Obtain the posture matrix at different times to obtain the posture matrix sequence; and perform Gaussian fitting on the posture matrix sequence to obtain the posture change function of the material;

[0014] The material picking device is controlled based on the posture change function to adjust the posture of the material during the transfer process.

[0015] In the second aspect, the present application provides a full-viewpoint collaborative eutectic placement machine material posture detection system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the full-viewpoint collaborative eutectic placement machine material posture detection method provided in the present application are implemented.

[0016] Beneficial effects:

[0017] The present invention uses a full-viewpoint camera array on the same horizontal line to comprehensively acquire image information of the material transfer work area; uses the obtained full-viewpoint image to acquire corner point information; and divides the full-viewpoint image into a main viewpoint image and a main viewpoint image according to the corner point information; uses the parallax in the main viewpoint image and other main viewpoint images to obtain the spatial position of the material vertices; constructs the material posture matrix through the spatial position of the material vertices; and then uses the posture matrix sequence to fit to obtain a function of the material posture changing in space over time. The posture change function can be used as feedback information to control the material picking device during the transfer process, adjust the material posture during the transfer process, and ensure the stability of the chip bonding process. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of a material posture detection method of a eutectic placement machine with full viewpoint coordination according to the present invention;

[0019] Figure 2 A schematic diagram of a full-viewpoint camera array according to an embodiment of the present invention;

[0020] Figure 3 A diagram showing the material posture of an embodiment of the present invention;

[0021] Figure 4 A schematic diagram of obtaining the spatial position of a corner point according to an embodiment of the present invention.

[0022] Reference numerals: U1 - full view camera; U2 - full view camera array; U3 - material table; U4 - workbench; U5 - material; U6 - part of the main view image in the first quadrant; U7 - part of the parallax image in the first quadrant. DETAILED DESCRIPTION

[0023] The technical solution of the present invention is described clearly and completely below. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] like Figure 1 As shown, the present application provides a full-viewpoint collaborative eutectic bonding machine material posture detection method, including:

[0025] S1: Use a full-viewpoint camera array to obtain a full-viewpoint image of the material at the same time, and pre-process the full-viewpoint image;

[0026] like Figure 2 As shown, the full-viewpoint cameras are of the same model, located on the same horizontal plane, and the distances between the full-viewpoint cameras are equal.

[0027] In one example, the preprocessing includes: denoising, enhancement processing, grayscale, and erosion operations.

[0028] S2: Binarize the preprocessed full viewpoint image to obtain a binary image;

[0029] S3: Find corner points in the binary image to obtain a set of corner points of the binary image;

[0030] In one example, first, a sliding window is used to detect corners of a binary image to obtain potential corner positions of the full viewpoint image; the potential corners satisfy the following relationship:

[0031]

[0032] in, Represents the full viewpoint image I i The binary image obtained after preprocessing and binarization; I i Indicates the full-view image acquired by the full-view camera with serial number i; i is the serial number of the full-view camera; d1∈{l,r} is the mark indicating left and right, l means left, r means right; d2∈{t,b} is the mark indicating up and down, t means up, b means down; (d1,d2) is the orientation mark, which is used to indicate the four orientations of upper left, lower left, upper right and lower right; Represents a binary image At the corner point at (d1, d2), the corner point is in the binary image The coordinates of the points in are (x,y); For binary images The corner strength of the point (x, y) in the (d1, d2) direction; Indicates the value corresponding to the point (c1, c2) in the sliding window at the position (d1, d2); s is the size measurement scale, the size of the sliding window is 2s+1; the step size of the sliding window is 1; Represents a binary image The pixel value at point (x,y);

[0033] Then, the potential corner points are screened to obtain the corner point set S i , the set of corner points satisfies the following relationship;

[0034]

[0035] Wherein, T is the set threshold.

[0036] S4: taking the original full viewpoint image of the binary image containing two or more corner points as the main viewpoint image to obtain a main viewpoint image set;

[0037] S5: taking the full viewpoint image including four corner points in the main viewpoint image set as the main viewpoint image;

[0038] S6: Calculate the spatial position of the material vertices using the parallax in the main viewpoint image and the primary viewpoint image, and construct the material pose matrix through the spatial position of the material vertices; Figure 3 It is a diagram showing the material posture;

[0039] In one example, the specific steps of S6 include:

[0040] S601: Obtaining depth information of material vertices;

[0041] When obtaining the depth of the left vertex of the material, the parallax image is the main viewpoint image I m The previous main viewpoint image I m-1 ; When obtaining the depth of the right vertex of the material, the parallax image is the main viewpoint image I m The next main viewpoint image I m+1 ;

[0042] like Figure 4 As shown, the depth d of the material vertex is obtained through the main viewpoint image and the parallax image, and the depth of the material vertex satisfies the following relationship:

[0043]

[0044] Wherein, b is the distance between two adjacent full-view camera lenses; f is the focal length of the full-view camera; d is the depth of the material vertex, which means the vertical distance from the material vertex corresponding to the corner point in the main view image to the full-view camera lens plane; y is the depth of the material vertex corresponding to the corner point in the main view image to the full-view camera lens plane; m is the coordinate of the corner point in the y-axis direction of the main viewpoint image; kis the coordinate of the corner point in the y-axis direction of the disparity image;

[0045] S602: Calculate the spatial coordinates of the material vertices in combination with the depth information of the material vertices;

[0046] The spatial coordinates (x, y, z) of the material vertices satisfy the following constraints:

[0047]

[0048] Among them, (x c ,y c ,y c ) is the optical center of the primary view camera; x m and m are the coordinates of the corner point in the primary view image;

[0049] S603: constructing a pose matrix of the material through the spatial coordinates of the vertices of the material;

[0050] The upper left vertex p of the material lt The coordinates in the primary viewpoint image are The coordinates in the disparity image are Combining formulas (3)-(4), we can get the upper left vertex p of the material: lt The spatial coordinates (x lt ,y lt ,z lt ); Similarly, according to the lower left vertex p of the material lb Coordinates in the primary view image and the coordinates in the disparity image Find the lower left vertex p of the material lt The spatial coordinates (x lb ,y lb ,z lb ); According to the upper right vertex p of the material rt Coordinates in the primary view image and the coordinates in the disparity image Find the upper right vertex p of the material lt The spatial coordinates (x rt ,y rt ,z rt ); According to the lower right vertex p of the material rb Coordinates in the primary view image and the coordinates in the disparity image Find the lower right vertex p of the material lt The spatial coordinates (x rb ,y rb ,z rb );

[0051] Select p lt 、plb 、p rt 、p rb The coordinates of any three of the four vertices are used as column vectors to construct the pose matrix Pose of the material. lt 、p lb 、p rb The coordinates of these three vertices are used as column vectors to construct the pose matrix Pose of the material, which is expressed as:

[0052]

[0053] S7: Obtain the posture matrix at different times to obtain a posture matrix sequence; and perform Gaussian fitting on the posture matrix sequence to obtain a posture change function of the material;

[0054] In a specific example, the posture change function F(t) of the material is expressed as:

[0055]

[0056] in, For x lt A function that changes with time t; similarly, the other elements of F(t) are functions of the corresponding elements of Pose that change with time t.

[0057] S8: Based on the posture change function, the material picking device is controlled, and the material picking device adjusts the posture of the material during the transfer process.

[0058] The present application also provides a full-viewpoint collaborative eutectic placement machine material posture detection system, the system comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor implementing the steps of the above method when executing the computer program. The full-viewpoint collaborative eutectic placement machine material posture detection system can implement the various embodiments of the above-mentioned full-viewpoint collaborative eutectic placement machine material posture detection method, and can achieve the same beneficial effects, which will not be described here.

[0059] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. A person skilled in the art can make many modifications and changes based on the concept of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be within the scope of protection of the present invention.

Claims

1. A full-viewpoint collaborative eutectic mounter material posture detection method, characterized in that: include: S1: using a full-viewpoint camera array to acquire full-viewpoint images of the material at the same time, and preprocessing the full-viewpoint images; S2: Binarizing the pre-processed full-viewpoint image to obtain a binary image; S3: searching for corner points in the binary image to obtain a set of corner points of the binary image; S4: taking the original full-viewpoint image of the binary image containing two or more corner points as the main viewpoint image to obtain a main viewpoint image set; S5: taking the full viewpoint image including four corner points in the main viewpoint image set as the main viewpoint image; S6: Calculate the spatial positions of the material vertices using the parallax between the primary viewpoint image and the main viewpoint image, and construct a pose matrix of the material based on the spatial positions of the material vertices; S7: Obtaining the posture matrix at different times to obtain a posture matrix sequence; and performing Gaussian fitting on the posture matrix sequence to obtain a posture change function of the material; S8: Controlling a material picking device based on the posture change function, wherein the material picking device adjusts the posture of the material during the transfer process.

2. The full-viewpoint collaborative eutectic mounter material posture detection method according to claim 1 is characterized in that: The pre-processing in S1 includes: denoising, enhancement processing, grayscale and corrosion operations.

3. The full-viewpoint collaborative eutectic mounter material posture detection method according to claim 1 is characterized in that: The S3 includes: First, corner detection is performed on the binary image using a sliding window to obtain the potential corner positions of the full-view image; the potential corner points satisfy the following relationship: in, Represents the full viewpoint image I i The binary image obtained after preprocessing and binarization; I i Represents the full-viewpoint image acquired by the full-viewpoint camera with serial number i; i is the serial number identifier of the full-viewpoint camera; d1∈{l,r} is the identifier for left and right; d2∈{t,b} is the identifier for up and down; (d1,d2) is the direction identifier, used to indicate the four directions of upper left, lower left, upper right and lower right; Represents a binary image At the corner point at (d1, d2), the corner point is in the binary image The coordinates of the points in are (x,y); For binary images The corner strength of the point (x, y) in the (d1, d2) direction; Represents the value of the point (c1, c2) in the sliding window at the position (d1, d2); s is the size scale, and the size of the sliding window is 2s+1; Represents a binary image The pixel value at point (x,y); Then, the potential corner points are screened to obtain the corner point set S i , the set of corner points satisfies the following relationship; Wherein, T is the set threshold.

4. The full-viewpoint collaborative eutectic mounter material posture detection method according to claim 3 is characterized by: The S6 includes: S601: Obtaining depth information of material vertices; When obtaining the depth of the left vertex of the material, the parallax image is the main viewpoint image I m The previous main viewpoint image I m-1 ; When obtaining the depth of the right vertex of the material, the parallax image is the main viewpoint image I m The next main viewpoint image I m+1 ; The depth d of the material vertex is obtained through the main viewpoint image and the parallax image. The depth of the material vertex satisfies the following relationship: Where b is the distance between two adjacent full-view camera lenses; f is the focal length of the full-view camera; d is the depth of the material vertex, which represents the vertical distance from the material vertex corresponding to the corner point in the main view image to the full-view camera lens plane; y m is the coordinate of the corner point in the y-axis direction of the main viewpoint image; d is the coordinate of the corner point in the y-axis direction of the disparity image; S602: Calculating the spatial coordinates of the material vertices based on the depth information of the material vertices; The spatial coordinates (x, y, z) of the material vertex satisfy the following constraints: Among them, (x c ,y c ,y c ) is the optical center of the primary view camera; x m and y m is the coordinate of the corner point in the primary viewpoint image; S603: Constructing a pose matrix of the material using the spatial coordinates of the material vertices; The upper left vertex p of the material lt The coordinates in the primary viewpoint image are The coordinates in the disparity image are Combining formulas (3)-(4), we can get the upper left vertex p of the material: lt The spatial coordinates (x lt ,y lt ,z lt ); Similarly, according to the lower left vertex p of the material lb Coordinates in the primary viewpoint image and the coordinates in the disparity image Find the lower left vertex p of the material lt The spatial coordinates (x lb ,y lb ,z lb ); According to the upper right vertex p of the material rt Coordinates in the primary viewpoint image and the coordinates in the disparity image Find the upper right vertex p of the material lt The spatial coordinates (x rt ,y rt ,z rt ); According to the lower right vertex p of the material rb Coordinates in the primary viewpoint image and the coordinates in the disparity image Find the lower right vertex p of the material lt The spatial coordinates (x rb ,y rb ,z rb ); Select p lt 、p lb 、p rt 、p rb The coordinates of any three of these four vertices are used as column vectors to construct the pose matrix of the material.

5. The full-viewpoint collaborative eutectic mounter material posture detection method according to claim 4 is characterized in that: In the S603, select p lt 、p lb 、p rb The coordinates of these three vertices are used as column vectors to construct the pose matrix of the material. The pose matrix Pose is expressed as:

6. The full-viewpoint collaborative eutectic mounter material posture detection method according to claim 5 is characterized by: The posture change function F(t) of the material in S7 is expressed as: in, is x lt A function that changes with time t; similarly, the other elements of F(t) are functions of the corresponding elements of Pose that change with time t.

7. A full-viewpoint collaborative eutectic mounter material posture detection system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

Citation Information

Patent Citations

  • Micro-part pose automatic alignment device and method

    CN108972557A

  • Woodworking furniture high-precision assembly system and method based on binocular parallax calculation

    CN114290016A

  • Fitting positioning method and device, equipment and storage medium

    CN114862963A

  • Labeling machine intelligent positioning method based on artificial intelligence

    CN115147411A

  • Full-viewpoint linkage suction nozzle micro-control method and system for material pose registration of chip mounter

    CN117409048A