A full-viewpoint cooperative eutectic die bonder material posture detection method and system

Abstract

The present application relates to the technical field of eutectic patch, and particularly relates to a full-viewpoint cooperative eutectic patch machine material posture detection method and system. The method uses a full-viewpoint camera array to obtain full-viewpoint images of the material at the same time, and performs preprocessing and binarization on the full-viewpoint images to obtain binary images; finds corners in the binary images to obtain a main viewpoint image set; uses the main viewpoint image and the parallax in the main viewpoint image to calculate the spatial position of the material vertex, and constructs a material pose matrix through the spatial position of the material vertex; obtains pose matrices at different times to obtain a pose matrix sequence; and performs Gaussian fitting on the pose matrix sequence to obtain a material pose change function; and controls a material picking device based on the pose change function to adjust the pose of the material in the transfer process, thereby ensuring the stability of the chip patch welding process.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of eutectic patch, and particularly to a full-viewpoint cooperative eutectic patch machine material posture detection method and system. BACKGROUND

[0002] The eutectic patch machine is usually used in the SMT (Surface Mount Technology) production line, and is mainly used for precisely welding a chip and a substrate. The eutectic patch machine needs to control the adsorption and transfer equipment to adsorb and transfer the two kinds of materials of the substrate and the chip to the welding area. In the process of transferring the chip to the welding area, the chip needs to be adjusted according to the position and posture of the chip so that the chip can be accurately transferred to the welding position on the substrate for welding.

[0003] The accuracy of the position and posture is crucial to the performance and quality of the chip packaging. At present, the transfer process of the material is mostly open-loop control, and there is little response to the interference encountered in the transfer process. The closed-loop control of the transfer process cannot be realized, and the stability of the material position and posture in the transfer process cannot be fully guaranteed. SUMMARY

[0004] The present application provides a full-viewpoint cooperative eutectic patch machine material posture detection method and system, which provides the position and posture information of the eutectic patch machine in the material transfer process, and provides a feedback signal for the control of the eutectic patch machine material transfer process.

[0005] In order to achieve the above-mentioned purpose, the present application realizes the technical scheme as follows:

[0006] In a first aspect, the present application provides a full-viewpoint cooperative eutectic patch machine material posture detection method, comprising:

[0007] acquiring full-viewpoint images of the material at the same time by using a full-viewpoint camera array, and pre-processing the full-viewpoint images;

[0008] obtaining a binary image by binarizing the pre-processed full-viewpoint images;

[0009] finding corner points in the binary image to obtain a corner point set of the binary image;

[0010] taking the original full-viewpoint image of the binary image containing two or more corner points as a main viewpoint image to obtain a main viewpoint image set;

[0011] taking the full-viewpoint image containing four corner points in the main viewpoint image set as a main viewpoint image;

[0012] calculating the spatial position of the material vertex by using the parallax in the main viewpoint image and the main viewpoint image, and constructing a material posture matrix by using the spatial position of the material vertex;

[0013] Obtaining the pose matrix at different time, obtaining a pose matrix sequence; and performing Gaussian fitting on the pose matrix sequence to obtain a pose change function of the material;

[0014] Controlling the material pickup device based on the pose change function to adjust the pose of the material in the transfer process.

[0015] In a second aspect, the present application provides a full-viewpoint collaborative eutectic chip mounter material posture detection system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the full-viewpoint collaborative eutectic chip mounter material posture detection method provided by the present application when executing the computer program.

[0016] Advantages:

[0017] The present application comprehensively acquires image information of a material transfer work area through a full-viewpoint camera array on the same horizontal straight line; acquires corner point information from the obtained full-viewpoint image; and divides the full-viewpoint image into a main viewpoint image and a main viewpoint image according to the corner point information; calculates the spatial position of the material vertex using the parallax in the main viewpoint image and other main viewpoint images; constructs a pose matrix of the material through the spatial position of the material vertex; and then performs fitting on the pose matrix sequence to obtain a function of the pose of the material changing with time in space. The pose change function can be used as feedback information to control the pickup device of the material in the transfer process, adjust the pose of the material in the transfer process, and ensure the stability of the chip mounting and welding process. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 The flowchart of the full-viewpoint collaborative eutectic chip mounter material posture detection method of the present application;

[0019] Figure 2 The schematic diagram of the full-viewpoint camera array of the embodiment of the present application;

[0020] Figure 3 The representation diagram of the material posture of the embodiment of the present application;

[0021] Figure 4 The schematic diagram of the corner point spatial position acquisition of the embodiment of the present application.

[0022] Reference signs: U1-full-viewpoint camera; U2-full-viewpoint camera array; U3-material table; U4-workbench; U5-material; U6-main viewpoint image in the first quadrant; U7-parallax image in the first quadrant. DETAILED DESCRIPTION

[0023] The following is a clear and complete description of the technical solutions of the present invention. It should be understood that the embodiments described are only a portion of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0024] like Figure 1 As shown, the present application provides a full-viewpoint collaborative eutectic placement machine material posture detection method, including:

[0025] S1: Use a full-viewpoint camera array to obtain full-viewpoint images of the material at the same time, and preprocess the full-viewpoint images;

[0026] like Figure 2 As shown, all full-viewpoint cameras are of the same model, located on the same horizontal plane, and the distances between the full-viewpoint cameras are equal.

[0027] In one example, the pre-processing includes: denoising, enhancement, grayscale, and erosion operations.

[0028] S2: Binarize the preprocessed full viewpoint image to obtain a binary image;

[0029] S3: Find corner points in the binary image to obtain a set of corner points of the binary image;

[0030] In one example, first, a sliding window is used to detect corners in a binary image to obtain the potential corner positions of the full-view image; the potential corners satisfy the following relationship:

[0031]

[0032] in, Represents the full viewpoint image I i The binary image obtained after preprocessing and binarization; I i Represents the full-viewpoint image acquired by the full-viewpoint camera with serial number i; i is the serial number identifier of the full-viewpoint camera; d1∈{l,r} is the identifier for left and right, l represents left, r represents right; d2∈{t,b} is the identifier for up and down, t represents up, b represents down; (d1,d2) is the direction identifier, used to indicate the four directions of upper left, lower left, upper right and lower right; Represents a binary image At the corner point at (d1, d2), the corner point is in the binary image The coordinates of the points in are (x,y); For binary images The corner strength of the point (x, y) in the (d1, d2) direction;

[0033] Represents the value of the point (c1, c2) in the sliding window at the position (d1, d2); s is the size scale, the size of the sliding window is 2s+1; the step size of the sliding window is 1; Represents a binary image The pixel value at point (x,y);

[0034] Then, the potential corner points are screened to obtain the corner point set S i , the set of corner points satisfies the following relationship;

[0035]

[0036] Where T is the set threshold.

[0037] S4: taking the original full-viewpoint image of the binary image containing two or more corner points as the main viewpoint image to obtain a main viewpoint image set;

[0038] S5: taking the full viewpoint image containing four corner points in the main viewpoint image set as the main viewpoint image;

[0039] S6: Calculate the spatial position of the material vertices using the parallax between the main viewpoint image and the primary viewpoint image, and construct the material pose matrix based on the spatial position of the material vertices; Figure 3 A diagram showing the material posture;

[0040] In one example, the specific steps of S6 include:

[0041] S601: Obtaining depth information of material vertices;

[0042] When obtaining the depth of the left vertex of the material, the parallax image is the main viewpoint image I m The previous main viewpoint image I m-1 ; When obtaining the depth of the right vertex of the material, the parallax image is the main viewpoint image I m The next main viewpoint image I m+1 ;

[0043] like Figure 4 As shown, the depth d of the material vertex is obtained through the main viewpoint image and the parallax image. The depth of the material vertex satisfies the following relationship:

[0044]

[0045] Where b is the distance between two adjacent full-view camera lenses; f is the focal length of the full-view camera; d is the depth of the material vertex, which represents the vertical distance from the material vertex corresponding to the corner point in the main view image to the full-view camera lens plane; y m is the coordinate of the corner point in the y-axis direction of the main viewpoint image; kis the coordinate of the corner point in the y-axis direction of the disparity image;

[0046] S602: Calculating the spatial coordinates of the material vertices based on the depth information of the material vertices;

[0047] The spatial coordinates (x, y, z) of the material vertex satisfy the following constraints:

[0048]

[0049] Among them, (x c ,y c ,y c ) is the optical center of the primary view camera; x m and y m is the coordinate of the corner point in the primary viewpoint image;

[0050] S603: Constructing a pose matrix of the material using the spatial coordinates of the material vertices;

[0051] The upper left vertex p of the material lt The coordinates in the primary viewpoint image are The coordinates in the disparity image are Combining formulas (3)-(4), we can get the upper left vertex p of the material: lt The spatial coordinates (x lt ,y lt ,z lt ); Similarly, according to the lower left vertex p of the material lb Coordinates in the primary viewpoint image and the coordinates in the disparity image Find the lower left vertex p of the material lb The spatial coordinates (x lb ,y lb ,z lb ); According to the upper right vertex p of the material rt Coordinates in the primary viewpoint image and the coordinates in the disparity image Find the upper right vertex p of the material rt The spatial coordinates (x rt ,y rt ,z rt ); According to the lower right vertex p of the material rb Coordinates in the primary viewpoint image and the coordinates in the disparity image Find the lower right vertex p of the material rb The spatial coordinates (x rb ,y rb ,z rb );

[0052] Select p lt 、plb 、p rt 、p rb The coordinates of any three vertices among these four vertices are used as column vectors to construct the pose matrix Pose of the material. In a specific example, select p lt 、p lb 、p rb The coordinates of these three vertices are used as column vectors to construct the pose matrix Pose of the material, which is expressed as:

[0053]

[0054] S7: Obtaining the posture matrix at different times to obtain a posture matrix sequence; and performing Gaussian fitting on the posture matrix sequence to obtain a posture change function of the material;

[0055] In a specific example, the posture change function F(t) of the material is expressed as:

[0056]

[0057] in, is x lt A function that changes with time t; similarly, the other elements of F(t) are functions of the corresponding elements of Pose that change with time t.

[0058] S8: Based on the posture change function, the material picking device is controlled, and the material picking device adjusts the posture of the material during the transfer process.

[0059] The present application also provides a full-viewpoint collaborative eutectic placement machine material posture detection system, the system comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method when executing the computer program. The full-viewpoint collaborative eutectic placement machine material posture detection system can implement various embodiments of the above-mentioned full-viewpoint collaborative eutectic placement machine material posture detection method and achieve the same beneficial effects, which will not be described in detail here.

[0060] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Persons skilled in the art may make numerous modifications and variations based on the concepts of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be within the scope of protection of the present invention.

Claims

1. A full-viewpoint collaborative eutectic mounter material posture detection method, characterized in that: include: S1: using a full-viewpoint camera array to acquire full-viewpoint images of the material at the same time, and preprocessing the full-viewpoint images; S2: Binarizing the pre-processed full-viewpoint image to obtain a binary image; S3: searching for corner points in the binary image to obtain a set of corner points of the binary image; S4: taking the original full-viewpoint image of the binary image containing two or more corner points as the main viewpoint image to obtain a main viewpoint image set; S5: taking the full viewpoint image including four corner points in the main viewpoint image set as the main viewpoint image; S6: Calculate the spatial positions of the material vertices using the parallax between the primary viewpoint image and the main viewpoint image, and construct a pose matrix of the material based on the spatial positions of the material vertices; S7: Obtaining the posture matrix at different times to obtain a posture matrix sequence; and performing Gaussian fitting on the posture matrix sequence to obtain a posture change function of the material; S8: controlling a material picking device based on the posture change function, wherein the material picking device adjusts the posture of the material during the transfer process; The S6 includes: S601: Obtaining depth information of material vertices; When obtaining the depth of the left vertex of the material, the parallax image is the main viewpoint image I m The previous main viewpoint image I m-1 ; When obtaining the depth of the right vertex of the material, the parallax image is the main viewpoint image I m The next main viewpoint image I m+1 ; The depth d of the material vertex is obtained through the main viewpoint image and the parallax image. The depth of the material vertex satisfies the following relationship: Where b is the distance between two adjacent full-view camera lenses; f is the focal length of the full-view camera; d is the depth of the material vertex, which represents the vertical distance from the material vertex corresponding to the corner point in the main view image to the full-view camera lens plane; y m is the coordinate of the corner point in the y-axis direction of the main viewpoint image; d is the coordinate of the corner point in the y-axis direction of the disparity image; S602: Calculating the spatial coordinates of the material vertices based on the depth information of the material vertices; The spatial coordinates (x, y, z) of the material vertex satisfy the following constraints: Among them, (x c ,y c ,y c ) is the optical center of the primary view camera; x m and y m is the coordinate of the corner point in the primary viewpoint image; S603: Constructing a pose matrix of the material using the spatial coordinates of the material vertices; The upper left vertex p of the material lt The coordinates in the primary viewpoint image are The coordinates in the disparity image are Combining formulas (1)-(2), we can get the upper left vertex p of the material: lt The spatial coordinates (x lt ,y lt ,z lt ); According to the lower left vertex p of the material lb Coordinates in the primary viewpoint image and the coordinates in the disparity image Find the lower left vertex p of the material lb The spatial coordinates (x lb ,y lb ,z lb ); According to the upper right vertex p of the material rt Coordinates in the primary viewpoint image and the coordinates in the disparity image Find the upper right vertex p of the material rt The spatial coordinates (x rt ,y rt ,z rt ); According to the lower right vertex p of the material rb Coordinates in the primary viewpoint image and the coordinates in the disparity image Find the lower right vertex p of the material rb The spatial coordinates (x rb ,y rb ,z rb ); Select p lt 、p lb 、p rt 、p rb The coordinates of any three of these four vertices are used as column vectors to construct the pose matrix of the material.

2. The full-viewpoint collaborative eutectic mounter material posture detection method according to claim 1 is characterized in that: The pre-processing in S1 includes: denoising, enhancement processing, grayscale and corrosion operations.

3. The full-viewpoint collaborative eutectic mounter material posture detection method according to claim 1 is characterized in that: The S3 includes: First, corner detection is performed on the binary image using a sliding window to obtain the potential corner positions of the full-view image; the potential corner points satisfy the following relationship: in, Represents the full viewpoint image I i The binary image obtained after preprocessing and binarization; I i Represents the full-viewpoint image acquired by the full-viewpoint camera with serial number i; i is the serial number identifier of the full-viewpoint camera; d1∈{l,r} is the identifier for left and right; d2∈{t,b} is the identifier for up and down; (d1,d2) is the direction identifier, used to indicate the four directions of upper left, lower left, upper right and lower right; Represents a binary image At the corner point at (d1, d2), the corner point is in the binary image The coordinates of the points in are (x,y); For binary images The corner strength of the point (x, y) in the (d1, d2) direction; Represents the value of the point (c1, c2) in the sliding window at the position (d1, d2); s is the size scale, and the size of the sliding window is 2s+1; Represents a binary image The pixel value at point (x,y); Then, the potential corner points are screened to obtain the corner point set S i , the set of corner points satisfies the following relationship; Wherein, T is the set threshold.

4. The full-viewpoint collaborative eutectic mounter material posture detection method according to claim 1 is characterized in that: In the S603, select p lt 、p lb 、p rb The coordinates of these three vertices are used as column vectors to construct the pose matrix Pose of the material. The pose matrix Pose is expressed as:

5. The full-viewpoint collaborative eutectic mounter material posture detection method according to claim 4 is characterized in that: The posture change function F(t) of the material in S7 is expressed as: in, is x lt A function that changes with time t; They are all functions of the corresponding elements of Pose changing with time t.

6. A full-viewpoint collaborative eutectic mounter material posture detection system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

Images