Patch alignment method, device and equipment based on visual imaging, medium and product

Obtain multiple viewing angle images through visual imaging technology and perform refocus detection, solving the problem of insufficient imaging effect and accuracy in the existing patch alignment technology, achieving high-precision patch alignment, reducing costs and avoiding optical Diffraction interference in alignment.

CN118969696AActive Publication Date: 2024-11-15ZHONGBEI UNIV
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411052776.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-11-15
Estimated Expiration
2044-08-01

AI Technical Summary

Technical Problem

The existing patch alignment technology has shortcomings in improving imaging effects and accuracy, especially the grating marking and digital grating methods based on moiré stripes will lead to image blurring and diffraction effects, affecting the alignment accuracy.

Method used

Using the patch alignment method based on visual imaging, by acquiring multiple viewing angles and refocusing, combining edge detection to calculate the rotation angle and center point coordinate difference of the calculation element and base, judge and adjust their position until it coincides, and use visual imaging Improve alignment accuracy.

Benefits of technology

The accuracy and imaging effect of patch alignment are improved, the cost is reduced, and diffraction interference in optical alignment is avoided, achieving low-cost and high-precision patch alignment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118969696A_ABST
    Figure CN118969696A_ABST
Patent Text Reader

Abstract

The invention discloses a patch alignment method, device and equipment based on visual imaging, a medium and a product, and relates to the technical field of patch alignment, an element and a base are shot at the same height from different visual angles to obtain multiple visual angle images, the multiple visual angle images are obtained in a visual imaging mode, and the imaging effect can be improved. And performing refocusing and edge detection based on the plurality of visual angle images, calculating to obtain a rotation angle difference and a center point coordinate difference between the element and the base, judging whether the element and the base are overlapped or not, controlling the element to act based on the rotation angle difference and the center point coordinate difference when the element and the base are not overlapped, and performing next overlapping judgment until the element and the base are overlapped. Therefore, clear images of the element and the base are obtained by means of better imaging effects of the multi-view-angle images in combination with refocusing, the coincidence judgment precision is further improved, and corresponding alignment guidance is given to improve the alignment precision of the element and the base.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of patch alignment technology, and in particular to a patch alignment method, device, equipment, medium and product based on visual imaging. Background Art

[0002] With the rapid development of micro-electromechanical systems and 3D integrated packaging, multifunctional high-performance devices, small-volume and low-power devices, high-temperature and high-voltage resistant devices, and miniaturized high-integration devices are the focus of current research in the packaging field. In the field of micro-packaging technology, the patch process is a key process. The patch process involves placing tiny components accurately in a predetermined position, that is, placing the components on the base, making the components and the base overlap, and firmly fixing them. In this process, any slight deviation may cause the device performance to degrade or fail. In order to achieve high-precision, high-efficiency, and high-quality patch effects, the accuracy of patch alignment must be continuously improved.

[0003] Yang Zhen et al. proposed a mechanical alignment device, which is simple to operate. The driven cam is rotated by controlling the rotary handle, and the four slides are pushed outward to move the four support columns outward by a certain distance. Then, the base to be bonded is placed on the support step, and the four separation gaskets are controlled to slide inward by a certain distance. Then, the component to be bonded is placed on the four separation gaskets, and the four slides are staggered by controlling the rotary handle, and returned inward under the pull of the return spring, so that the base and the component can be aligned. The mechanical alignment device can complete the alignment of the base and the component only through the rotation of the driven cam, the return of the return spring, and the sliding of the isolation gasket. It has the advantages of low equipment cost and high work efficiency. However, the accuracy of mechanical alignment is much lower than that of optical alignment.

[0004] FAN et al. used grating marks and digital gratings based on moiré fringes to significantly improve the deviation alignment accuracy of the upper and lower two pieces of product circles during the alignment process. The alignment equipment can control the error within 10nm, and at the same time can meet all the requirements of high-precision bonding. Its calculation effect and speed are better than most image processing software on the market. Therefore, the alignment equipment is more conducive to the subsequent bonding process. In addition, the designed grating marks are small enough to leave more space for the subsequent process of etching circuits on the product circle. However, the method based on moiré fringes grating marks and digital gratings requires reasonable control of the distance between the upper and lower two pieces of product circles, and the diffraction effect produced by them will cause image blur, reduce the imaging effect, and thus affect the alignment accuracy.

[0005] Based on this, there is an urgent need for a patch alignment technology that can improve imaging effects. Summary of the invention

[0006] The purpose of this application is to provide a patch alignment method, device, equipment, medium and product based on visual imaging, which can improve the imaging effect of multiple perspective images and further combine refocusing to improve the accuracy of patch alignment.

[0007] To achieve the above objectives, this application provides the following solutions:

[0008] In a first aspect, the present application provides a patch alignment method based on visual imaging, the patch alignment method based on visual imaging comprising:

[0009] Acquire multiple viewing angle images; the multiple viewing angle images are obtained by photographing the component and the base at the same height from different viewing angles, the component is located above the base, and the component and the base are the same size;

[0010] Refocusing is performed based on the multiple viewing angle images to obtain a first refocused image refocused to the plane where the top surface of the component is located and a second refocused image refocused to the plane where the top surface of the base is located;

[0011] Performing edge detection on the first refocused image to obtain a first edge image, and calculating a first rotation angle, first center point coordinates, height and width of the element based on the first edge image; the first edge image includes an edge of the element;

[0012] Performing edge detection on the second refocused image to obtain a second edge image, and calculating a second rotation angle and a second center point coordinate of the base based on the second edge image and the height and width of the component; the second edge image includes a portion of the edge of the base;

[0013] Calculating the difference between the first rotation angle and the second rotation angle to obtain a rotation angle difference, calculating the difference between the first center point coordinates and the second center point coordinates to obtain a center point coordinate difference, and determining whether the rotation angle difference and the center point coordinate difference are both 0, so as to determine whether the component and the base overlap;

[0014] If not, the component action is controlled based on the rotation angle difference and the center point coordinate difference, and the process returns to the step of "obtaining multiple viewing angle images".

[0015] Optionally, acquiring multiple perspective images specifically includes:

[0016] The camera is controlled to move at a constant speed along the slide rail, and during the constant speed movement, the camera is controlled to simultaneously shoot the component and the base at predetermined time intervals, and each shooting obtains a perspective image to obtain multiple perspective images; the camera is installed on the slide rail, and the slide rail is located above the component.

[0017] Optionally, refocusing is performed based on multiple viewing angle images to obtain a first refocused image refocused to a plane where the top surface of the component is located and a second refocused image refocused to a plane where the top surface of the base is located, specifically including:

[0018] A perspective image is randomly selected from multiple perspective images and recorded as the first perspective image;

[0019] For each second perspective image except the first perspective image among the multiple perspective images, respectively extract the first feature points in the first perspective image and the second feature points in the second perspective image, match all the first feature points and all the second feature points to obtain a plurality of matching feature point pairs, wherein the matching feature point pairs include a first feature point and a second feature point that match each other; for each of the matching feature point pairs, calculate the disparity between the first feature point and the second feature point in the matching feature point pair, and calculate the depth of each pixel in the first perspective image based on the disparity of all the matching feature point pairs;

[0020] For each pixel point in the first perspective image, based on the coordinates and depth of the pixel point and the depth of the plane where the top surface of the component is located, calculate the reprojection coordinates of the pixel point on the plane where the top surface of the component is located; the reprojection coordinates of all the pixel points on the plane where the top surface of the component is located constitute a first refocused image refocused to the plane where the top surface of the component is located;

[0021] For each pixel point in the first perspective image, the reprojected coordinates of the pixel point on the plane where the top surface of the base is located are calculated based on the coordinates and depth of the pixel point and the depth of the plane where the top surface of the base is located; the reprojected coordinates of all the pixel points on the plane where the top surface of the base is located constitute a second refocused image refocused to the plane where the top surface of the base is located.

[0022] Optionally, after acquiring multiple perspective images, the patch alignment method based on visual imaging further includes: controlling the camera to enter a sleep state.

[0023] Optionally, calculating the first rotation angle, the first center point coordinates, the height and the width of the component based on the first edge image specifically includes:

[0024] determining coordinates of four vertices of the element based on the first edge image;

[0025] Randomly select two adjacent vertices from the four vertices, and calculate a first rotation angle of the component based on the coordinates of the two adjacent vertices;

[0026] Calculate the average value of the coordinates of the four vertices to obtain the coordinates of the first center point of the component;

[0027] Randomly select two adjacent vertices along the height direction from the four vertices, calculate the distance between the two adjacent vertices along the height direction based on the coordinates of the two adjacent vertices along the height direction, and obtain the height of the component;

[0028] Two adjacent vertices along the width direction are randomly selected from the four vertices, and based on the coordinates of the two adjacent vertices along the width direction, the distance between the two adjacent vertices along the width direction is calculated to obtain the width of the component.

[0029] Optionally, calculating a second rotation angle and a second center point coordinate of the base based on the second edge image and the height and width of the component specifically includes:

[0030] determining coordinates of two adjacent vertices of the base based on the second edge image;

[0031] Calculating a second rotation angle of the base based on the coordinates of two adjacent vertices;

[0032] A vertex is randomly selected from two adjacent vertices as a selected vertex, and the coordinates of the second center point of the base are calculated based on the coordinates of the selected vertex, the second rotation angle, and the height and width of the component.

[0033] In a second aspect, the present application provides a patch alignment device based on visual imaging, the patch alignment device based on visual imaging comprising:

[0034] An image acquisition module is used to acquire multiple-view images; the multiple-view images are obtained by photographing the component and the base from different viewpoints at the same height, the component is located above the base, and the component and the base are the same size;

[0035] A refocusing module, used for refocusing based on the multiple viewing angle images to obtain a first refocusing image refocused to the plane where the top surface of the component is located and a second refocusing image refocused to the plane where the top surface of the base is located;

[0036] A first calculation module is used to perform edge detection on the first refocused image to obtain a first edge image, and calculate a first rotation angle, a first center point coordinate, a height and a width of the element based on the first edge image; the first edge image includes an edge of the element;

[0037] A second calculation module is used to perform edge detection on the second refocused image to obtain a second edge image, and calculate a second rotation angle and a second center point coordinate of the base based on the second edge image and the height and width of the component; the second edge image includes a portion of the edge of the base;

[0038] a judgment module, used for calculating the difference between the first rotation angle and the second rotation angle to obtain a rotation angle difference, calculating the difference between the first center point coordinate and the second center point coordinate to obtain a center point coordinate difference, and judging whether the rotation angle difference and the center point coordinate difference are both 0, so as to judge whether the component and the base are overlapped;

[0039] The return module is used to control the component action based on the rotation angle difference and the center point coordinate difference if no, and return to the step of "obtaining multiple viewing angle images".

[0040] In a third aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-described patch alignment methods based on visual imaging.

[0041] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described visual imaging-based patch alignment methods.

[0042] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned patch alignment methods based on visual imaging.

[0043] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0044] The present application provides a patch alignment method, device, equipment, medium and product based on visual imaging. At the same height, the component and the base are photographed from different perspectives to obtain multiple perspective images. The multiple perspective images are obtained by visual imaging, which can improve the imaging effect. Refocusing is performed based on multiple perspective images to obtain a first refocused image refocused to the plane where the top surface of the component is located and a second refocused image refocused to the plane where the top surface of the base is located. The first refocused image and the second refocused image are edge detected respectively, and the rotation angle difference and the center point coordinate difference of the component and the base are further calculated to determine whether the component and the base are overlapped. When they are not overlapped, the component action is controlled based on the rotation angle difference and the center point coordinate difference, and the next overlap judgment is performed until the component and the base overlap, thereby using the better imaging effect of multiple perspective images, combined with refocusing to obtain a clear image of the component and the base, further improve the accuracy of overlap judgment, and give corresponding alignment guidance to improve the alignment accuracy of the component and the base. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 A flowchart of a patch alignment method based on visual imaging provided in Example 1 of the present application.

[0047] Figure 2 A schematic diagram of the system framework of a patch alignment method based on visual imaging provided in Example 1 of the present application.

[0048] Figure 3 A workflow diagram of a patch alignment method based on visual imaging provided in Example 1 of the present application.

[0049] Figure 4 Schematic diagram of the principles of refocusing and edge detection provided in Example 1 of the present application.

[0050] Figure 5 A schematic diagram of the functional modules of a patch alignment device based on visual imaging provided in Example 2 of the present application.

[0051] Figure 6 A schematic diagram of the structure of a computer device provided in Example 3 of the present application.

[0052] Explanation of symbols:

[0053] 1- soldering pad; 2- base; 3- component; 4- camera; 5- slide rail. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] Example 1

[0056] like Figure 1 As shown, this embodiment provides a patch alignment method based on visual imaging, and the patch alignment method based on visual imaging includes:

[0057] S1: Acquire multiple viewpoint images; the multiple viewpoint images are obtained by photographing the component and the base at the same height from different viewpoints, the component is located above the base, and the component and the base are the same size.

[0058] S2: Refocusing is performed based on the multiple viewing angle images to obtain a first refocused image refocused to the plane where the top surface of the component is located and a second refocused image refocused to the plane where the top surface of the base is located.

[0059] S3: Perform edge detection on the first refocused image to obtain a first edge image, and calculate a first rotation angle, first center point coordinates, height and width of the element based on the first edge image; the first edge image includes the edge of the element.

[0060] S4: Perform edge detection on the second refocused image to obtain a second edge image, and calculate a second rotation angle and a second center point coordinate of the base based on the second edge image and the height and width of the component; the second edge image includes a partial edge of the base.

[0061] S5: Calculate the difference between the first rotation angle and the second rotation angle to obtain the rotation angle difference, calculate the difference between the first center point coordinates and the second center point coordinates to obtain the center point coordinate difference, and determine whether the rotation angle difference and the center point coordinate difference are both 0 to determine whether the component and the base overlap.

[0062] S6: If not, the component action is controlled based on the rotation angle difference and the center point coordinate difference, and the process returns to the step of "obtaining multiple viewing angle images".

[0063] By implementing the above-mentioned steps S1 to S6, this embodiment can capture multiple viewing angle images, and subsequently process the multiple viewing angle images to determine whether the component and the base overlap, and adjust the component position until the component and the base overlap, thereby completing the patch alignment based on visual imaging. By using visual imaging, there is no diffraction interference, and a better imaging effect is achieved, thereby achieving low-cost and high-precision patch alignment.

[0064] This embodiment uses a camera to obtain multiple viewing angle images. Figure 2 As shown, it is a schematic diagram of the system framework of the patch alignment method based on visual imaging, the base 2 is fixed on the pad 1, the component 3 is located above the base 2, the patch alignment refers to making the component 3 and the base 2 overlap, the displacement control system clamps and moves the component 3, so that the component 3 is aligned with the base 2, a high-resolution camera 4 is fixed on the strip slide 5 located above the pad 1, the base 2 and the component 3, the slide 5 can be an electric slide, the camera 4 shoots downward at equal time intervals to obtain multiple perspective images (also called multi-angle images).

[0065] Based on the above system framework, such as Figure 3 As shown, the workflow of the patch alignment method based on visual imaging includes:

[0066] (1) Turn on the camera, move it at a constant speed on the slide rail, take pictures downward at equal time intervals to obtain images from multiple perspectives, and then turn off the camera.

[0067] (2) Refocusing the multiple perspective images captured to obtain a clear image of the plane where the top surface of the component under the slide rail is located (i.e., the first refocused image) and a clear image of the plane where the top surface of the base is located (i.e., the second refocused image).

[0068] (3) Identify the edge of the component in the clear image of the plane where the top surface of the component is located after refocusing to obtain a first edge image, and identify the edge of the base in the clear image of the plane where the top surface of the base is located after refocusing to obtain a second edge image.

[0069] (4) Calculate the coordinates of the center points C1 and C2 of the component and the base and the rotation angles θ1 and θ2 respectively, and obtain the center point coordinate difference and the rotation angle difference Δθ.

[0070] (5) Determine whether the component and the base overlap. If not, transmit the coordinates of C1 and C2 and Δθ to the displacement control system. Use the displacement control system to rotate and move the component to adjust the position of the component so that the component overlaps with the base, and return to (1); if they overlap, end.

[0071] This embodiment provides a patch alignment method based on visual imaging, which uses a single camera to shoot at multiple angles to obtain multiple viewing angles, obtains clear images of the component and the base after refocusing, identifies the edges in the clear images, further calculates the center point coordinates and rotation angles of the component and the base, accurately determines whether the component and the base are aligned, and provides alignment guidance, ultimately achieving the purpose of aligning the component and the base, and improving the accuracy of patch alignment with the help of the high accuracy of visual imaging. At the same time, by using refocusing to obtain clear images of the component and the base, further identifying the edges of the two clear images obtained after refocusing, further calculating the center point coordinates and rotation angles of the component and the base, and determining whether the two are aligned, the accuracy of alignment can be improved.

[0072] In S1, multiple perspective images are obtained, specifically including: controlling the camera to move at a uniform speed along the slide rail, and during the uniform motion, controlling the camera to simultaneously shoot the component and the base once at a predetermined time interval, obtaining a perspective image each time, and multiple perspective images can be obtained by multiple shootings, so as to obtain multiple perspective images, and the camera is installed on the slide rail, and the slide rail is located above the component. This embodiment uses camera imaging without diffraction interference, has better imaging effects, and can achieve low-cost, high-precision optical alignment. At the same time, using a single camera to shoot at equal time intervals on the electric slide rail instead of the camera array can reduce imaging costs and simplify operating steps.

[0073] In this embodiment, the camera enters a dormant state after acquiring multiple viewing angle images, so that the camera works intermittently to control the camera temperature, avoid excessive camera temperature, and prevent the working environment from being affected, that is, to avoid interference with the working environment. By adopting an intermittent working mode instead of the traditional continuous working mode, the temperature of the camera can be effectively controlled to avoid abnormal operation of the camera due to temperature increase, thereby improving the reliability of the camera during the alignment process.

[0074] The camera slides at a constant speed on the electric slide rail, taking pictures at equal time intervals to obtain multiple perspective images. Light field information is extracted from the multiple perspective images taken from multiple angles. The light field information contains the propagation direction and intensity distribution of light in space, which is crucial for the subsequent refocusing process. That is, multiple perspective images lay the foundation for digital refocusing. Digital refocusing uses the camera to capture scene information at multiple angles and intensities, uses image processing methods to synthesize multiple perspective images, readjusts the focus plane of the image, and adjusts the focus position of the image while keeping the perspective unchanged, thereby achieving clear display of different depth layers in the scene.

[0075] like Figure 4 As shown, in S2, refocusing is performed based on multiple perspective images to obtain a first refocused image refocused to the plane where the top surface of the component is located and a second refocused image refocused to the plane where the top surface of the base is located, specifically including:

[0076] (1) Randomly select a perspective image from multiple perspective images and record it as the first perspective image.

[0077] Since the camera slides at a constant speed on the slide rail and takes images at equal time intervals, the shooting position is fixed. In this embodiment, the camera at each shooting position is calibrated respectively to obtain the intrinsic parameter matrix and distortion parameters of the camera at the corresponding shooting position.

[0078] (2) For each second perspective image except the first perspective image among the multiple perspective images, extract the first feature points in the first perspective image and the second feature points in the second perspective image respectively, match all the first feature points and all the second feature points to obtain a plurality of matching feature point pairs, where the matching feature point pairs include a first feature point and a second feature point that match each other; for each matching feature point pair, calculate the disparity between the first feature point and the second feature point in the matching feature point pair, and calculate the depth of each pixel in the first perspective image based on the disparity of all the matching feature point pairs.

[0079] This embodiment detects feature points in the view image by using the SIFT algorithm, and matches feature points in images of different view angles to obtain matching feature point pairs, which facilitates subsequent calculation of disparity.

[0080] When calculating the disparity, the disparity is generated by comparing the position difference between the first feature point and the second feature point in images of different viewing angles. The disparity is the difference between the horizontal coordinates (ie, the x-coordinates) of the first feature point and the second feature point in the matching feature point pair.

[0081] Based on the disparity of all matching feature point pairs, the disparity of each pixel in the first-view image can be calculated using existing methods. According to the geometric relationship between the disparity and the camera, the initial depth of each pixel in the first-view image is calculated by the following formula (1):

[0082]

[0083] In formula (1), Z is the initial depth; B is the camera baseline distance, which refers to the horizontal distance between the optical centers of the two cameras, that is, the difference between the x-coordinate of the optical center of the camera corresponding to the first-view image and the x-coordinate of the optical center of the camera corresponding to the second-view image. The position coordinates of the optical center can be obtained by calibrating the camera; f is the focal length of the camera corresponding to the first-view image, which is the internal parameter of the camera; d is the parallax.

[0084] For each second-view image, the initial depth of each pixel in the first-view image can be calculated, and the average of multiple values ​​of the initial depth of each pixel in the first-view image is calculated to obtain the depth of each pixel in the first-view image. In particular, the number of initial depths is the same as the number of second-view images.

[0085] The depth of each pixel in the first-person image constitutes a depth map of the scene, which is the light field information.

[0086] The alignment process aims to compare the edges of the component and the base at different depth positions to determine whether the two overlap. Therefore, the first field of view image needs to be focused on the plane where the component and the top surface of the base are located. For each pixel point in the first field of view image, the re-projected coordinates of the pixel point on the plane where the component and the top surface of the base are located are calculated based on the depth map and the selected focus plane (i.e., the plane where the component and the top surface of the base are located).

[0087] (3) For each pixel point in the first-perspective image, based on the coordinates and depth of the pixel point and the depth of the plane where the top surface of the component is located, the reprojected coordinates of the pixel point on the plane where the top surface of the component is located are calculated, and the reprojected coordinates of all the pixel points on the plane where the top surface of the component is located constitute a first refocused image refocused to the plane where the top surface of the component is located.

[0088] The calculation formula for the reprojected coordinates of the pixel point on the plane where the top surface of the component is located is:

[0089]

[0090] In formula (2), (x1', y1') is the reprojected coordinate of the pixel point on the plane where the top surface of the component is located, x1' is the x coordinate of the pixel point on the plane where the top surface of the component is located, y1' is the y coordinate of the pixel point on the plane where the top surface of the component is located; x is the x coordinate of the pixel point; Z is the depth of the pixel point; Z f1 is the depth of the plane where the top surface of the component is located; x0 is the x coordinate of the optical center of the camera corresponding to the first-view image; y is the y coordinate of the pixel point; y0 is the y coordinate of the optical center of the camera corresponding to the first-view image. The position coordinates of the optical center (x0, y0) are obtained by calibrating the camera corresponding to the first-view image.

[0091] (4) For each pixel point in the first-perspective image, based on the coordinates and depth of the pixel point and the depth of the plane where the top surface of the base is located, the reprojected coordinates of the pixel point on the plane where the top surface of the base is located are calculated, and the reprojected coordinates of all the pixel points on the plane where the top surface of the base is located constitute a second refocused image refocused to the plane where the top surface of the base is located.

[0092] The calculation formula for the reprojected coordinates of the pixel point on the plane where the top surface of the base is located is:

[0093]

[0094] In formula (3), (x2', y2') is the reprojected coordinate of the pixel point on the plane where the top surface of the base is located, x2' is the x coordinate of the pixel point on the plane where the top surface of the base is located, y2' is the y coordinate of the pixel point on the plane where the top surface of the base is located; x is the x coordinate of the pixel point; Z is the depth of the pixel point; Z f2is the depth of the plane where the top surface of the base is located; x0 is the x coordinate of the optical center of the camera corresponding to the first-view image; y is the y coordinate of the pixel point; y0 is the y coordinate of the optical center of the camera corresponding to the first-view image.

[0095] Using multiple viewpoints can provide more redundant information, thereby improving the accuracy of depth estimation. This is because if there is occlusion or noise in the images of certain viewpoints, the images of other viewpoints can provide complementary information. Accurate depth information is required in the refocusing process to redistribute pixel positions so that the image of the target focal plane is clear. Multi-view disparity estimation can provide more accurate depth maps, thereby generating higher quality refocused images.

[0096] In this embodiment, edge detection is performed on the first refocusing image and the second refocusing image respectively to obtain a first edge image of the component and a second edge image of the base.

[0097] During the shooting process, part of the base is blocked by the component, and the base is consistent with the component in size. Therefore, this embodiment calculates the height and width of the component, and the height and width of the component are the height and width of the base. The height and width of the base are then used to calculate the center point coordinates and rotation angle of the base.

[0098] In S3, edge detection of the first refocused image may be performed using any existing edge detection algorithm. After obtaining the first edge image, the first rotation angle, first center point coordinates, height and width of the element are calculated based on the first edge image, specifically including:

[0099] (1) The coordinates of the four vertices of the element are determined based on the first edge image.

[0100] like Figure 4 As shown, for the first refocused image of the component, after edge detection, the coordinates of the four vertices of the component are obtained based on the first edge image of the component: the first vertex A1 (x A1 ,y A1 ), the second vertex A2(x A2 ,y A2 )、The third vertex A3(x A3 ,y A3 )、The fourth vertex A4(x A4 ,y A4 ), where x A1 ,y A1 are the x and y coordinates of the first vertex A1, respectively. A2 ,y A2 are the x and y coordinates of the second vertex A2, respectively. A3 ,y A3 are the x and y coordinates of the third vertex A3, respectively. A4 ,yA4 are the x-coordinate and y-coordinate of the fourth vertex A4 respectively.

[0101] (2) Randomly select two adjacent vertices from the four vertices, and calculate the first rotation angle of the element based on the coordinates of the two adjacent vertices.

[0102] The two adjacent vertices can be the first vertex and the second vertex, the second vertex and the third vertex, the third vertex and the fourth vertex, or the fourth vertex and the first vertex. For example, the two adjacent vertices are the first vertex and the second vertex, that is, select A1(x A1 ,y A1 )、A2(x A2 ,y A2 ) This pair of adjacent vertices, the first rotation angle θ1 is calculated by the following formula (4):

[0103] θ1=arctan2(y A2 -y A1 ,x A2 -x A1 ); (4)

[0104] (3) Calculate the average value of the coordinates of the four vertices to obtain the coordinates of the first center point of the element.

[0105] The first center point coordinate C1 (C 1x ,C 1y ):

[0106]

[0107] In formula (5), C 1x ,C 1y are the x-coordinate and y-coordinate of the center point C1 of the component respectively.

[0108] (4) Randomly select two adjacent vertices along the height direction from the four vertices, calculate the distance between the two adjacent vertices along the height direction based on the coordinates of the two adjacent vertices along the height direction, and obtain the height of the component.

[0109] The two adjacent vertices along the height direction may be the first vertex and the fourth vertex, or the second vertex and the third vertex. Taking the case where the two adjacent vertices along the height direction are the first vertex and the fourth vertex as an example, the height h of the element is calculated by the following formula (6):

[0110]

[0111] (5) Randomly select two adjacent vertices along the width direction from the four vertices, calculate the distance between the two adjacent vertices along the width direction based on the coordinates of the two adjacent vertices along the width direction, and obtain the width of the component.

[0112] The two adjacent vertices along the width direction may be the first vertex and the second vertex, or the third vertex and the fourth vertex. Taking the two adjacent vertices along the width direction as the first vertex and the second vertex as an example, the width w of the element is calculated by the following formula (7):

[0113]

[0114] In S4, edge detection of the second refocused image may be performed using any existing edge detection algorithm. After obtaining the second edge image, the second rotation angle and the second center point coordinates of the base are calculated based on the second edge image and the height and width of the component, specifically including:

[0115] (1) Determine the coordinates of two adjacent vertices of the base based on the second edge image.

[0116] For the second refocused image of the base, after edge detection, the coordinates of two adjacent vertices in the second edge image can be obtained: the fifth vertex B1 (x B1 ,y B1 ) and the sixth vertex B2(x B2 ,y B2 ), where x B1 ,y B1 are the x-coordinate and y-coordinate of the fifth vertex B1, respectively. B2 ,y B2 are the x-coordinate and y-coordinate of the sixth vertex B2 respectively.

[0117] (2) Calculate the second rotation angle of the base based on the coordinates of two adjacent vertices.

[0118] The second rotation angle θ2 of the base is calculated by the following formula (8):

[0119] θ2=arctan2(y B2 -y B1 ,x B2 -x B1 ); (8)

[0120] (3) A vertex is randomly selected from two adjacent vertices as a selected vertex, and the coordinates of the second center point of the base are calculated based on the coordinates of the selected vertex, the second rotation angle, and the height and width of the component.

[0121] Taking the fifth vertex as an example, based on the coordinates, height h and width w of the fifth vertex, the coordinates of the second center point C2 (C 2x ,C 2y ):

[0122]

[0123] In formula (9), C 2x ,C 2y are the x-coordinate and y-coordinate of the center point C2 of the base respectively.

[0124] In S5, the rotation angle difference Δθ between the element and the base is:

[0125] Δθ=θ1-θ2; (10)

[0126] By comparing whether the center points coincide and whether the rotation angle difference is zero, it can be determined whether the component and the base coincide. That is, if the center point coordinate difference and the rotation angle difference are both zero, the component and the base coincide, otherwise, the component and the base do not coincide.

[0127] In S6, if the component and the base do not overlap, the coordinates of C1, C2 and Δθ are transmitted to the displacement control system, and the component is rotated and moved by the displacement control system to make the component overlap with the base. The camera is turned on, and the photo is taken and refocused again. The alignment judgment steps are repeated until the component and the base overlap, and the operation is terminated.

[0128] This embodiment also provides an application scenario, which applies the above-mentioned patch alignment method based on visual imaging. Specifically, the patch alignment method based on visual imaging provided in this embodiment can be applied in the patch alignment scenario, which includes an image acquisition link and an alignment link. Multiple perspective images are acquired through the image acquisition link, and the multiple perspective images are processed through the alignment link to determine whether the component and the base overlap, and the component position is adjusted until the patch alignment is completed. The patch alignment method based on visual imaging provided in this embodiment belongs to the alignment link.

[0129] Example 2

[0130] Based on the same inventive concept, this embodiment also provides a patch alignment device based on visual imaging for implementing the patch alignment method based on visual imaging involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the patch alignment device based on visual imaging provided below can refer to the limitations of the patch alignment method based on visual imaging above, and will not be repeated here.

[0131] like Figure 5 As shown, this embodiment provides a patch alignment device based on visual imaging, and the patch alignment device based on visual imaging includes:

[0132] The image acquisition module M1 is used to acquire multiple viewing angle images; the multiple viewing angle images are obtained by photographing the component and the base at the same height from different viewing angles. The component is located above the base, and the component and the base are the same size.

[0133] The refocusing module M2 is used to perform refocusing based on multiple viewing angle images to obtain a first refocusing image refocused to the plane where the top surface of the component is located and a second refocusing image refocused to the plane where the top surface of the base is located.

[0134] The first calculation module M3 is used to perform edge detection on the first refocused image to obtain a first edge image, and calculate the first rotation angle, first center point coordinates, height and width of the element based on the first edge image; the first edge image includes the edge of the element.

[0135] The second calculation module M4 is used to perform edge detection on the second refocused image to obtain a second edge image, and calculate the second rotation angle and the second center point coordinates of the base based on the second edge image and the height and width of the component; the second edge image includes a partial edge of the base.

[0136] The judgment module M5 is used to calculate the difference between the first rotation angle and the second rotation angle to obtain the rotation angle difference, calculate the difference between the first center point coordinates and the second center point coordinates to obtain the center point coordinate difference, and judge whether the rotation angle difference and the center point coordinate difference are both 0, so as to judge whether the component and the base overlap.

[0137] Return module M6 is used to control the component action based on the rotation angle difference and the center point coordinate difference if no, and return to the step of "obtaining multiple viewing angle images".

[0138] Example 3

[0139] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 6As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store multiple perspective images. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a patch alignment method based on visual imaging is implemented.

[0140] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0141] In an exemplary embodiment, a computer device is also provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the patch alignment method based on visual imaging described in Example 1.

[0142] Example 4

[0143] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the patch alignment method based on visual imaging described in Embodiment 1 is implemented.

[0144] Example 5

[0145] This embodiment provides a computer program product, including a computer program, which, when executed by a processor, implements the patch alignment method based on visual imaging described in Embodiment 1.

[0146] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0147] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A patch alignment method based on visual imaging, characterized in that: The patch alignment method based on visual imaging comprises: Acquire multiple viewing angle images; the multiple viewing angle images are obtained by photographing the component and the base at the same height from different viewing angles, the component is located above the base, and the component and the base are the same size; Refocusing is performed based on the multiple viewing angle images to obtain a first refocused image refocused to the plane where the top surface of the component is located and a second refocused image refocused to the plane where the top surface of the base is located; Performing edge detection on the first refocused image to obtain a first edge image, and calculating a first rotation angle, first center point coordinates, height and width of the element based on the first edge image; the first edge image includes an edge of the element; Performing edge detection on the second refocused image to obtain a second edge image, and calculating a second rotation angle and a second center point coordinate of the base based on the second edge image and the height and width of the component; the second edge image includes a portion of the edge of the base; Calculating the difference between the first rotation angle and the second rotation angle to obtain a rotation angle difference, calculating the difference between the first center point coordinate and the second center point coordinate to obtain a center point coordinate difference, and determining whether the rotation angle difference and the center point coordinate difference are both 0, so as to determine whether the component and the base overlap; If not, the component action is controlled based on the rotation angle difference and the center point coordinate difference, and the process returns to the step of "obtaining multiple viewing angle images".

2. The patch alignment method based on visual imaging according to claim 1, characterized in that: Acquire multiple perspective images, including: The camera is controlled to move at a constant speed along the slide rail, and during the constant speed movement, the camera is controlled to simultaneously shoot the component and the base at predetermined time intervals, and each shooting obtains a perspective image to obtain multiple perspective images; the camera is installed on the slide rail, and the slide rail is located above the component.

3. The patch alignment method based on visual imaging according to claim 1, characterized in that: Refocusing is performed based on multiple viewing angle images to obtain a first refocusing image refocused to a plane where the top surface of the component is located and a second refocusing image refocused to a plane where the top surface of the base is located, specifically including: A perspective image is randomly selected from multiple perspective images and recorded as the first perspective image; For each second perspective image except the first perspective image among the multiple perspective images, respectively extract the first feature points in the first perspective image and the second feature points in the second perspective image, match all the first feature points and all the second feature points to obtain a plurality of matching feature point pairs, wherein the matching feature point pairs include a first feature point and a second feature point that match each other; for each of the matching feature point pairs, calculate the disparity between the first feature point and the second feature point in the matching feature point pair, and calculate the depth of each pixel in the first perspective image based on the disparity of all the matching feature point pairs; For each pixel point in the first perspective image, based on the coordinates and depth of the pixel point and the depth of the plane where the top surface of the component is located, calculate the reprojection coordinates of the pixel point on the plane where the top surface of the component is located; the reprojection coordinates of all the pixel points on the plane where the top surface of the component is located constitute a first refocused image refocused to the plane where the top surface of the component is located; For each pixel point in the first perspective image, the reprojected coordinates of the pixel point on the plane where the top surface of the base is located are calculated based on the coordinates and depth of the pixel point and the depth of the plane where the top surface of the base is located; the reprojected coordinates of all the pixel points on the plane where the top surface of the base is located constitute a second refocused image refocused to the plane where the top surface of the base is located.

4. The patch alignment method based on visual imaging according to claim 2, characterized in that: After acquiring multiple viewing angle images, the patch alignment method based on visual imaging further includes: controlling the camera to enter a dormant state.

5. The patch alignment method based on visual imaging according to claim 1, characterized in that: Calculating the first rotation angle, the first center point coordinates, the height and the width of the component based on the first edge image specifically includes: determining coordinates of four vertices of the element based on the first edge image; Randomly select two adjacent vertices from the four vertices, and calculate a first rotation angle of the component based on the coordinates of the two adjacent vertices; Calculate the average value of the coordinates of the four vertices to obtain the coordinates of the first center point of the component; Randomly select two adjacent vertices along the height direction from the four vertices, calculate the distance between the two adjacent vertices along the height direction based on the coordinates of the two adjacent vertices along the height direction, and obtain the height of the component; Two adjacent vertices along the width direction are randomly selected from the four vertices, and based on the coordinates of the two adjacent vertices along the width direction, the distance between the two adjacent vertices along the width direction is calculated to obtain the width of the component.

6. The patch alignment method based on visual imaging according to claim 1, characterized in that: Calculating a second rotation angle and a second center point coordinate of the base based on the second edge image and the height and width of the component specifically includes: determining coordinates of two adjacent vertices of the base based on the second edge image; Calculate a second rotation angle of the base based on the coordinates of two adjacent vertices; A vertex is randomly selected from two adjacent vertices as a selected vertex, and the coordinates of the second center point of the base are calculated based on the coordinates of the selected vertex, the second rotation angle, and the height and width of the component.

7. A patch alignment device based on visual imaging, characterized in that: The patch alignment device based on visual imaging comprises: An image acquisition module is used to acquire multiple-view images; the multiple-view images are obtained by photographing the component and the base from different viewpoints at the same height, the component is located above the base, and the component and the base are the same size; A refocusing module, used for refocusing based on multiple viewing angle images to obtain a first refocusing image refocused to the plane where the top surface of the component is located and a second refocusing image refocused to the plane where the top surface of the base is located; A first calculation module is used to perform edge detection on the first refocused image to obtain a first edge image, and calculate a first rotation angle, a first center point coordinate, a height and a width of the element based on the first edge image; the first edge image includes an edge of the element; A second calculation module is used to perform edge detection on the second refocused image to obtain a second edge image, and calculate a second rotation angle and a second center point coordinate of the base based on the second edge image and the height and width of the component; the second edge image includes a portion of the edge of the base; a judgment module, used for calculating the difference between the first rotation angle and the second rotation angle to obtain a rotation angle difference, calculating the difference between the first center point coordinate and the second center point coordinate to obtain a center point coordinate difference, and judging whether the rotation angle difference and the center point coordinate difference are both 0, so as to judge whether the component and the base are overlapped; The return module is used to control the component action based on the rotation angle difference and the center point coordinate difference if no, and return to the step of "obtaining multiple perspective images".

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the patch alignment method based on visual imaging as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the patch alignment method based on visual imaging described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the patch alignment method based on visual imaging described in any one of claims 1 to 6 is implemented.

Citation Information

Patent Citations

  • Vision-based wafer pre-alignment platform and alignment method

    CN116072583A

  • Method for wafer bonding alignment through edge detection

    CN116313971A

  • Alignment method and device for wafer bonding

    CN116798926A

  • Control system for image refocusing

    CN209710210U

  • Alignment method and alignment device

    TW201925070A