Patch alignment method, device, equipment, medium and product based on visual imaging

Abstract

The present application discloses a patch alignment method, device, equipment, medium and product based on visual imaging, which relates to the field of patch alignment technology. At the same height, the component and the base are photographed from different perspectives to obtain multiple perspective images. The multiple perspective images are obtained by visual imaging, which can improve the imaging effect. Based on the multiple perspective images, refocusing and edge detection are performed, and the rotation angle difference and center point coordinate difference of the component and the base are calculated to determine whether the component and the base overlap. If they do not overlap, the component action is controlled based on the rotation angle difference and the center point coordinate difference, and the next overlap judgment is performed until the component and the base overlap. Thus, with the help of the better imaging effect of multiple perspective images, combined with refocusing, a clear image of the component and the base is obtained, the overlap judgment accuracy is further improved, and corresponding alignment guidance is given to improve the alignment accuracy of the component and the base.

Description

Technical Field

[0001] The present application relates to the field of patch alignment technology, and in particular to a patch alignment method, device, equipment, medium and product based on visual imaging. Background Art

[0002] With the rapid development of micro-electromechanical systems (MEMS) and 3D integrated packaging, multifunctional, high-performance devices, small-volume, low-power devices, high-temperature and high-voltage resistant devices, and miniaturized, highly integrated devices are currently the focus of research in the packaging field. Within the field of micro-packaging, the patch process is a key step. This involves accurately placing tiny components in predetermined locations, placing them on a base, ensuring that the components and base overlap, and securely securing them. Any slight deviation in this process can lead to device performance degradation or even failure. To achieve high-precision, efficient, and high-quality patching, continuous improvement in patch alignment accuracy is essential.

[0003] Yang Zhen et al. proposed a simple mechanical alignment device. This device rotates a driven cam by controlling a rotary handle, pushing four slides outward to move the four support posts a certain distance. The base to be bonded is then placed on the support step, and four separation pads are controlled to slide inward a certain distance. The component to be bonded is then placed on the four separation pads. The four slides are then offset by controlling a rotary handle, and returned inward by a return spring, completing the alignment of the base and component. This mechanical alignment device aligns the base and component solely through the rotation of the driven cam, the return of the return spring, and the sliding of the isolation pads. It offers the advantages of low equipment cost and high efficiency. However, the accuracy of mechanical alignment is far lower than that of optical alignment.

[0004] FAN et al. employed moiré-based grating marks and digital gratings to significantly improve the alignment accuracy of the upper and lower circles during the alignment process. This alignment device can control the error to within 10 nm while also meeting all the requirements for high-precision bonding. Its computational efficiency and speed surpass those of most commercially available image processing software, making it more conducive to subsequent bonding processes. Furthermore, the designed grating marks are sufficiently small to allow for the subsequent etching of circuits on the circles. However, moiré-based grating marks and digital gratings require reasonable control of the distance between the upper and lower circles, and the resulting diffraction effect can blur the image, reducing the imaging effect and thus affecting alignment accuracy.

[0005] Based on this, there is an urgent need for a patch alignment technology that can improve imaging effects. Summary of the Invention

[0006] The purpose of this application is to provide a patch alignment method, device, equipment, medium and product based on visual imaging, which can improve the imaging effect of multiple perspective images and further combine refocusing to improve the accuracy of patch alignment.

[0007] To achieve the above objectives, this application provides the following solutions:

[0008] In a first aspect, the present application provides a patch alignment method based on visual imaging, the patch alignment method based on visual imaging comprising:

[0009] Acquire multiple-view images; the multiple-view images are obtained by photographing the component and the base from different perspectives at the same height, with the component located above the base and having the same size as the base;

[0010] Refocusing is performed based on the multiple perspective images to obtain a first refocused image refocused to the plane where the top surface of the component is located and a second refocused image refocused to the plane where the top surface of the base is located;

[0011] Performing edge detection on the first refocused image to obtain a first edge image, and calculating a first rotation angle, first center point coordinates, height, and width of the component based on the first edge image; the first edge image includes an edge of the component;

[0012] performing edge detection on the second refocused image to obtain a second edge image, and calculating a second rotation angle and a second center point coordinate of the base based on the second edge image and the height and width of the component; the second edge image includes a portion of the edge of the base;

[0013] Calculating a difference between the first rotation angle and the second rotation angle to obtain a rotation angle difference, calculating a difference between the first center point coordinate and the second center point coordinate to obtain a center point coordinate difference, and determining whether the rotation angle difference and the center point coordinate difference are both 0 to determine whether the component and the base are aligned;

[0014] If not, the component action is controlled based on the rotation angle difference and the center point coordinate difference, and the process returns to the step of "acquiring multiple perspective images".

[0015] Optionally, acquiring multiple perspective images specifically includes:

[0016] The camera is controlled to move at a constant speed along the slide rail, and during the constant speed movement, the camera is controlled to simultaneously shoot the component and the base at predetermined time intervals, and each shooting obtains a perspective image to obtain multiple perspective images; the camera is installed on the slide rail, and the slide rail is located above the component.

[0017] Optionally, refocusing based on the multiple perspective images to obtain a first refocused image refocused to the plane where the top surface of the component is located and a second refocused image refocused to the plane where the top surface of the base is located specifically includes:

[0018] Randomly select a perspective image from multiple perspective images and record it as the first perspective image;

[0019] For each second-view image other than the first-view image from the plurality of view images, extract first feature points from the first-view image and second feature points from the second-view image, match all of the first feature points with all of the second feature points to obtain a plurality of matched feature point pairs, each of the matched feature point pairs including a first feature point and a second feature point that match each other; for each of the matched feature point pairs, calculate a disparity between the first feature point and the second feature point in the matched feature point pair, and calculate a depth of each pixel in the first-view image based on the disparities of all of the matched feature point pairs;

[0020] For each pixel in the first-view image, calculating the reprojected coordinates of the pixel on the plane where the top surface of the component is located based on the coordinates and depth of the pixel and the depth of the plane where the top surface of the component is located; the reprojected coordinates of all the pixels on the plane where the top surface of the component is located constitute a first refocused image refocused to the plane where the top surface of the component is located;

[0021] For each pixel point in the first perspective image, the reprojected coordinates of the pixel point on the plane where the top surface of the base is located are calculated based on the coordinates and depth of the pixel point and the depth of the plane where the top surface of the base is located; the reprojected coordinates of all the pixel points on the plane where the top surface of the base is located constitute a second refocused image refocused to the plane where the top surface of the base is located.

[0022] Optionally, after acquiring multiple perspective images, the patch alignment method based on visual imaging further includes: controlling the camera to enter a dormant state.

[0023] Optionally, calculating the first rotation angle, first center point coordinates, height, and width of the component based on the first edge image specifically includes:

[0024] determining coordinates of four vertices of the element based on the first edge image;

[0025] Randomly selecting two adjacent vertices from the four vertices, and calculating a first rotation angle of the component based on coordinates of the two adjacent vertices;

[0026] Calculate the average value of the coordinates of the four vertices to obtain the coordinates of the first center point of the component;

[0027] Randomly selecting two adjacent vertices along the height direction from the four vertices, and calculating the distance between the two adjacent vertices along the height direction based on the coordinates of the two adjacent vertices along the height direction to obtain the height of the component;

[0028] Two adjacent vertices along the width direction are randomly selected from the four vertices, and a distance between the two adjacent vertices along the width direction is calculated based on the coordinates of the two adjacent vertices along the width direction to obtain the width of the component.

[0029] Optionally, calculating a second rotation angle and a second center point coordinate of the base based on the second edge image and the height and width of the component specifically includes:

[0030] determining coordinates of two adjacent vertices of the base based on the second edge image;

[0031] Calculating a second rotation angle of the base based on the coordinates of two adjacent vertices;

[0032] A vertex is randomly selected from two adjacent vertices as a selected vertex, and the coordinates of the second center point of the base are calculated based on the coordinates of the selected vertex, the second rotation angle, and the height and width of the component.

[0033] In a second aspect, the present application provides a patch alignment device based on visual imaging, the patch alignment device based on visual imaging comprising:

[0034] An image acquisition module is used to acquire multiple-view images; the multiple-view images are obtained by photographing the component and the base from different perspectives at the same height, with the component located above the base and having the same size as the base;

[0035] a refocusing module, configured to perform refocusing based on the multiple perspective images to obtain a first refocused image refocused to the plane where the top surface of the component is located and a second refocused image refocused to the plane where the top surface of the base is located;

[0036] a first calculation module, configured to perform edge detection on the first refocused image to obtain a first edge image, and calculate a first rotation angle, first center point coordinates, height, and width of the component based on the first edge image; the first edge image includes an edge of the component;

[0037] a second calculation module, configured to perform edge detection on the second refocused image to obtain a second edge image, and calculate a second rotation angle and a second center point coordinate of the base based on the second edge image and the height and width of the component; the second edge image includes a portion of the edge of the base;

[0038] a determination module, configured to calculate a difference between the first rotation angle and the second rotation angle to obtain a rotation angle difference, calculate a difference between the first center point coordinate and the second center point coordinate to obtain a center point coordinate difference, and determine whether the rotation angle difference and the center point coordinate difference are both 0, so as to determine whether the component and the base are aligned;

[0039] The return module is used to control the component action based on the rotation angle difference and the center point coordinate difference if no, and return to the step of "obtaining multiple perspective images".

[0040] In a third aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any one of the above-described patch alignment methods based on visual imaging.

[0041] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any one of the above-mentioned patch alignment methods based on visual imaging.

[0042] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements any one of the above-mentioned patch alignment methods based on visual imaging.

[0043] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0044] The present application provides a patch alignment method, device, equipment, medium and product based on visual imaging, which shoots the component and the base from different perspectives at the same height to obtain multiple perspective images. The multiple perspective images are obtained by visual imaging, which can improve the imaging effect. Refocusing is performed based on the multiple perspective images to obtain a first refocused image refocused to the plane where the top surface of the component is located and a second refocused image refocused to the plane where the top surface of the base is located. Edge detection is performed on the first refocused image and the second refocused image respectively, and the rotation angle difference and center point coordinate difference of the component and the base are further calculated to determine whether the component and the base overlap. If they do not overlap, the component action is controlled based on the rotation angle difference and the center point coordinate difference, and the next overlap judgment is performed until the component and the base overlap, thereby using the better imaging effect of multiple perspective images and combining refocusing to obtain a clear image of the component and the base, further improving the accuracy of overlap judgment, and giving corresponding alignment guidance to improve the alignment accuracy of the component and the base. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 A flowchart of a patch alignment method based on visual imaging provided in Example 1 of the present application.

[0047] Figure 2 A schematic diagram of the system framework of a patch alignment method based on visual imaging provided in Example 1 of the present application.

[0048] Figure 3 This is a workflow diagram of a patch alignment method based on visual imaging provided in Example 1 of the present application.

[0049] Figure 4 Schematic diagram of the principles of refocusing and edge detection provided in Example 1 of the present application.

[0050] Figure 5 A schematic diagram of the functional modules of a patch alignment device based on visual imaging provided in Example 2 of the present application.

[0051] Figure 6 A schematic diagram of the structure of a computer device provided in Example 3 of the present application.

[0052] Explanation of symbols:

[0053] 1-soldering pad; 2-base; 3-component; 4-camera; 5-slide rail. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0055] Example 1

[0056] like Figure 1 As shown, this embodiment provides a patch alignment method based on visual imaging, and the patch alignment method based on visual imaging includes:

[0057] S1: Acquire multiple-view images; the multiple-view images are obtained by photographing the component and the base at the same height from different viewpoints, where the component is located above the base and the component and the base are of the same size.

[0058] S2: Refocusing is performed based on the multiple perspective images to obtain a first refocused image refocused to the plane where the top surface of the component is located and a second refocused image refocused to the plane where the top surface of the base is located.

[0059] S3: Perform edge detection on the first refocused image to obtain a first edge image, and calculate a first rotation angle, first center point coordinates, height, and width of the component based on the first edge image; the first edge image includes the edge of the component.

[0060] S4: Perform edge detection on the second refocused image to obtain a second edge image, and calculate the second rotation angle and the second center point coordinates of the base based on the second edge image and the height and width of the component; the second edge image includes a partial edge of the base.

[0061] S5: Calculate the difference between the first rotation angle and the second rotation angle to obtain the rotation angle difference, calculate the difference between the first center point coordinates and the second center point coordinates to obtain the center point coordinate difference, and determine whether the rotation angle difference and the center point coordinate difference are both 0 to determine whether the component and the base overlap.

[0062] S6: If not, the component action is controlled based on the rotation angle difference and the center point coordinate difference, and the process returns to the step of "acquiring multiple perspective images".

[0063] By implementing the above-mentioned steps S1 to S6, this embodiment can capture multiple perspective images, and subsequently process the multiple perspective images to determine whether the component and the base overlap, and adjust the position of the component until the component and the base overlap, thereby completing the patch alignment based on visual imaging. By using visual imaging, there is no diffraction interference, and it has a better imaging effect, thereby achieving low-cost, high-precision patch alignment.

[0064] This embodiment uses a camera to obtain images from multiple perspectives. Figure 2 As shown, it is a schematic diagram of the system framework of the patch alignment method based on visual imaging. The base 2 is fixed on the pad 1, and the component 3 is located above the base 2. Patch alignment means that the component 3 and the base 2 are overlapped. The displacement control system clamps and moves the component 3 so that the component 3 is aligned with the base 2. A high-resolution camera 4 is fixed on a strip slide 5 located above the pad 1, the base 2 and the component 3. The slide 5 can be an electric slide. The camera 4 shoots downward at equal time intervals to obtain multiple perspective images (also called multi-angle images).

[0065] Based on the above system framework, such as Figure 3 As shown in Figure 2, the workflow of the patch alignment method based on visual imaging includes:

[0066] (1) Turn on the camera, move it at a constant speed on the slide rail, take pictures downward at equal time intervals, obtain images from multiple perspectives, and then turn off the camera.

[0067] (2) Refocusing the multiple perspective images captured to obtain a clear image of the plane where the top surface of the component under the slide rail is located (i.e., the first refocused image) and a clear image of the plane where the top surface of the base is located (i.e., the second refocused image).

[0068] (3) Identify the edge of the component in the clear image of the plane where the top surface of the component is located after refocusing to obtain a first edge image, and identify the edge of the base in the clear image of the plane where the top surface of the base is located after refocusing to obtain a second edge image.

[0069] (4) Calculate the coordinates of the center points C1 and C2 of the component and the base and the rotation angles θ1 and θ2 respectively, and obtain the center point coordinate difference and the rotation angle difference Δθ.

[0070] (5) Determine whether the component and the base are aligned. If not, transmit the coordinates of C1 and C2 and Δθ to the displacement control system. Use the displacement control system to rotate and move the component to adjust the component position so that the component and the base are aligned. Return to (1). If they are aligned, end.

[0071] This embodiment provides a patch alignment method based on visual imaging. It uses a single camera to capture multiple perspective images from multiple angles. After refocusing, a clear image of each component and base is obtained. The edges in the clear images are identified, and the center point coordinates and rotation angles of each component and base are further calculated. This accurately determines whether the component and base are aligned, and provides alignment guidance, ultimately achieving the goal of aligning the component and base. Leveraging the high precision of visual imaging, the accuracy of patch alignment is improved. Furthermore, by using refocusing to obtain a clear image of each component and base, the edges of the two clear images obtained after refocusing are further identified, and the center point coordinates and rotation angles of each component and base are further calculated to determine whether the two are aligned, thereby improving the accuracy of alignment judgment.

[0072] In S1, multiple perspective images are acquired, specifically including: controlling the camera to move at a constant speed along the slide rail, and during this constant speed movement, controlling the camera to simultaneously capture the component and the base at predetermined time intervals, each capture generating a perspective image. Multiple captures can generate multiple perspective images, thereby acquiring multiple perspective images. The camera is mounted on the slide rail, which is located above the component. This embodiment utilizes camera imaging, which eliminates diffraction interference, provides better imaging results, and enables low-cost, high-precision optical alignment. Furthermore, using a single camera on the motorized slide rail to capture images at equal intervals, rather than a camera array, can reduce imaging costs and simplify operational steps.

[0073] In this embodiment, after capturing images from multiple viewpoints, the camera enters a dormant state, causing it to operate intermittently to control its temperature and prevent it from overheating, which could affect the operating environment. By employing intermittent operation instead of the traditional continuous operation mode, the camera's temperature can be effectively controlled, preventing abnormal operation caused by elevated temperatures, thereby improving the reliability of the camera during the alignment process.

[0074] The camera slides at a constant speed on a motorized slide, taking photos at equal intervals to capture multiple-viewpoint images. Light field information is extracted from these multi-viewpoint images. This information encompasses the propagation direction and intensity distribution of light in space and is crucial for the subsequent refocusing process. This multi-viewpoint imagery lays the foundation for digital refocusing. Digital refocusing utilizes the camera to capture scene information from multiple angles and intensities, synthesizes these images using image processing methods, and readjusts the image's focal plane. This adjustment maintains the same viewing angle, allowing for clear display of different depth layers within the scene.

[0075] like Figure 4 As shown, in S2, refocusing is performed based on the multiple perspective images to obtain a first refocused image refocused to the plane where the top surface of the component is located and a second refocused image refocused to the plane where the top surface of the base is located, specifically including:

[0076] (1) Randomly select a perspective image from multiple perspective images and record it as the first perspective image.

[0077] Since the camera slides at a constant speed on the slide rail and captures images at equal time intervals, the shooting position is fixed. In this embodiment, the camera at each shooting position is calibrated to obtain the intrinsic parameter matrix and distortion parameters of the camera at the corresponding shooting position.

[0078] (2) For each second perspective image other than the first perspective image in the multiple perspective images, extract the first feature points in the first perspective image and the second feature points in the second perspective image respectively, match all the first feature points and all the second feature points to obtain a number of matching feature point pairs, where the matching feature point pairs include a first feature point and a second feature point that match; for each matching feature point pair, calculate the disparity between the first feature point and the second feature point in the matching feature point pair, and calculate the depth of each pixel point in the first perspective image based on the disparity of all the matching feature point pairs.

[0079] This embodiment detects feature points in the perspective image using the SIFT algorithm, and matches feature points in images of different perspectives to obtain matching feature point pairs, which facilitates subsequent calculation of disparity.

[0080] When calculating disparity, the disparity is generated by comparing the position difference of the first feature point and the second feature point in images of different viewing angles. The disparity is the difference between the horizontal coordinates (ie, x-coordinates) of the first feature point and the second feature point in the matching feature point pair.

[0081] Based on the disparity of all matching feature point pairs, the disparity of each pixel in the first-view image can be calculated using existing methods. According to the geometric relationship between the disparity and the camera, the initial depth of each pixel in the first-view image is calculated using the following formula (1):

[0082]

[0083] In formula (1), Z is the initial depth; B is the camera baseline distance, which refers to the horizontal distance between the optical centers of the two cameras, that is, the difference between the x-coordinate of the optical center of the camera corresponding to the first-view image and the x-coordinate of the optical center of the camera corresponding to the second-view image. The position coordinates of the optical center can be obtained by calibrating the camera; f is the focal length of the camera corresponding to the first-view image, which is the internal parameter of the camera; d is the parallax.

[0084] For each second-view image, the initial depth of each pixel in the first-view image can be calculated. The depth of each pixel in the first-view image is obtained by averaging multiple initial depth values. Specifically, the number of initial depths is the same as the number of second-view images.

[0085] The depth of each pixel in the first-person perspective image constitutes a depth map of the scene, which is the light field information.

[0086] The alignment process aims to compare the edges of the component and base at different depths to determine whether they overlap. Therefore, the first field of view image must be focused on the planes containing the component and base top surfaces, respectively. For each pixel in the first field of view image, the reprojected coordinates of the pixel on the planes containing the component and base top surfaces are calculated based on the depth map and the selected focus plane (i.e., the plane containing the component and base top surfaces).

[0087] (3) For each pixel point in the first-perspective image, based on the coordinates and depth of the pixel point and the depth of the plane where the top surface of the component is located, the reprojected coordinates of the pixel point on the plane where the top surface of the component is located are calculated, and the reprojected coordinates of all the pixel points on the plane where the top surface of the component is located constitute a first refocused image refocused to the plane where the top surface of the component is located.

[0088] The calculation formula for the re-projected coordinates of a pixel point on the plane where the top surface of the component is located is:

[0089]

[0090] In formula (2), (x1', y1') is the re-projected coordinate of the pixel point on the plane where the top surface of the component is located, x1' is the x coordinate of the pixel point on the plane where the top surface of the component is located, y1' is the y coordinate of the pixel point on the plane where the top surface of the component is located; x is the x coordinate of the pixel point; Z is the depth of the pixel point; Z f1 is the depth of the plane containing the top surface of the component; x0 is the x-coordinate of the optical center of the camera corresponding to the first-view image; y is the y-coordinate of the pixel; and y0 is the y-coordinate of the optical center of the camera corresponding to the first-view image. The optical center position (x0, y0) is obtained by calibrating the camera corresponding to the first-view image.

[0091] (4) For each pixel point in the first-perspective image, based on the coordinates and depth of the pixel point and the depth of the plane where the top surface of the base is located, the reprojected coordinates of the pixel point on the plane where the top surface of the base is located are calculated, and the reprojected coordinates of all the pixel points on the plane where the top surface of the base is located constitute a second refocused image refocused to the plane where the top surface of the base is located.

[0092] The calculation formula for the reprojected coordinates of the pixel point on the plane where the top surface of the base is located is:

[0093]

[0094] In formula (3), (x2', y2') is the re-projected coordinate of the pixel point on the plane where the top surface of the base is located, x2' is the x coordinate of the pixel point on the plane where the top surface of the base is located, y2' is the y coordinate of the pixel point on the plane where the top surface of the base is located; x is the x coordinate of the pixel point; Z is the depth of the pixel point; Z f2is the depth of the plane where the top surface of the base is located; x0 is the x-coordinate of the optical center of the camera corresponding to the first-view image; y is the y-coordinate of the pixel point; y0 is the y-coordinate of the optical center of the camera corresponding to the first-view image.

[0095] Using images from multiple perspectives provides more redundant information, thereby improving the accuracy of depth estimation. This is because if images from certain perspectives are obscured or noisy, images from other perspectives can provide complementary information. The refocusing process requires accurate depth information to redistribute pixel positions to achieve a clear image of the target focal plane. Multi-perspective disparity estimation provides a more accurate depth map, resulting in a higher-quality refocused image.

[0096] In this embodiment, edge detection is performed on the first refocused image and the second refocused image respectively to obtain a first edge image of the component and a second edge image of the base.

[0097] During the shooting process, part of the base is blocked by the component, and the base is the same size as the component. Therefore, this embodiment calculates the height and width of the component, and the height and width of the component are the height and width of the base. Subsequently, the center point coordinates and rotation angle of the base are calculated with the help of the height and width of the base.

[0098] In S3, edge detection of the first refocused image may be performed using any existing edge detection algorithm. After obtaining the first edge image, the first rotation angle, first center point coordinates, height, and width of the element are calculated based on the first edge image, specifically including:

[0099] (1) The coordinates of the four vertices of the element are determined based on the first edge image.

[0100] like Figure 4 As shown, for the first refocused image of the component, after edge detection, the coordinates of the four vertices of the component are obtained based on the first edge image of the component: the first vertex A1 (x A1 ,y A1 ), the second vertex A2(x A2 ,y A2 )、the third vertex A3(x A3 ,y A3 )、the fourth vertex A4(x A4 ,y A4 ), where x A1 ,y A1 are the x and y coordinates of the first vertex A1, respectively. A2 ,y A2 are the x and y coordinates of the second vertex A2, respectively. A3 ,y A3 are the x and y coordinates of the third vertex A3, respectively. A4 ,yA4 are the x-coordinate and y-coordinate of the fourth vertex A4 respectively.

[0101] (2) Randomly select two adjacent vertices from the four vertices, and calculate the first rotation angle of the element based on the coordinates of the two adjacent vertices.

[0102] The two adjacent vertices can be the first vertex and the second vertex, the second vertex and the third vertex, the third vertex and the fourth vertex, or the fourth vertex and the first vertex. For example, the two adjacent vertices are the first vertex and the second vertex, that is, select A1(x A1 ,y A1 )、A2(x A2 ,y A2 ) For this pair of adjacent vertices, the first rotation angle θ1 is calculated using the following formula (4):

[0103] θ1=arctan2(y A2 -y A1 ,x A2 -x A1 ); (4)

[0104] (3) Calculate the average value of the coordinates of the four vertices to obtain the coordinates of the first center point of the element.

[0105] The first center point coordinate C1 (C 1x ,C 1y ):

[0106]

[0107] In formula (5), C 1x ,C 1y are the x-coordinate and y-coordinate of the center point C1 of the component respectively.

[0108] (4) Randomly select two adjacent vertices along the height direction from the four vertices, calculate the distance between the two adjacent vertices along the height direction based on the coordinates of the two adjacent vertices along the height direction, and obtain the height of the component.

[0109] The two vertices adjacent in the height direction may be the first vertex and the fourth vertex, or the second vertex and the third vertex. Taking the case where the two vertices adjacent in the height direction are the first vertex and the fourth vertex as an example, the height h of the element is calculated using the following formula (6):

[0110]

[0111] (5) Randomly select two adjacent vertices along the width direction from the four vertices, calculate the distance between the two adjacent vertices along the width direction based on the coordinates of the two adjacent vertices along the width direction, and obtain the width of the element.

[0112] The two adjacent vertices along the width direction may be the first vertex and the second vertex, or the third vertex and the fourth vertex. Taking the two adjacent vertices along the width direction as the first vertex and the second vertex as an example, the width w of the element is calculated by the following formula (7):

[0113]

[0114] In S4, edge detection of the second refocused image can be performed using any existing edge detection algorithm. After obtaining the second edge image, the second rotation angle and the second center point coordinates of the base are calculated based on the second edge image and the height and width of the component, specifically including:

[0115] (1) Determine the coordinates of two adjacent vertices of the base based on the second edge image.

[0116] For the second refocused image of the base, after edge detection, the coordinates of two adjacent vertices in the second edge image can be obtained: the fifth vertex B1 (x B1 ,y B1 ) and the sixth vertex B2(x B2 ,y B2 ), where x B1 ,y B1 are the x and y coordinates of the fifth vertex B1, respectively. B2 ,y B2 are the x-coordinate and y-coordinate of the sixth vertex B2 respectively.

[0117] (2) Calculate the second rotation angle of the base based on the coordinates of two adjacent vertices.

[0118] The second rotation angle θ2 of the base is calculated by the following formula (8):

[0119] θ2=arctan2(y B2 -y B1 ,x B2 -x B1 ); (8)

[0120] (3) Randomly select a vertex from two adjacent vertices as a selected vertex, and calculate the coordinates of the second center point of the base based on the coordinates of the selected vertex, the second rotation angle, and the height and width of the component.

[0121] Taking the fifth vertex as an example, based on the coordinates, height h and width w of the fifth vertex, the coordinates of the second center point C2 (C 2x ,C 2y ):

[0122]

[0123] In formula (9), C 2x ,C 2y are the x-coordinate and y-coordinate of the center point C2 of the base respectively.

[0124] In S5, the rotation angle difference Δθ between the component and the base is:

[0125] Δθ=θ1-θ2; (10)

[0126] By comparing whether the center points coincide and whether the rotation angle difference is zero, it can be determined whether the component and the base coincide. That is, if the center point coordinate difference and the rotation angle difference are both zero, the component and the base coincide; otherwise, the component and the base do not coincide.

[0127] In S6, if the component and the base do not overlap, the coordinates of C1 and C2 and Δθ are transmitted to the displacement control system. The component is rotated and moved by the displacement control system so that the component and the base overlap. The camera is turned on, and the photo and focus are retaken. The alignment judgment steps are repeated until the component and the base overlap, and the operation is terminated.

[0128] This embodiment also provides an application scenario that utilizes the aforementioned visual imaging-based patch alignment method. Specifically, the visual imaging-based patch alignment method provided in this embodiment can be applied in a patch alignment scenario, which includes an image acquisition step and an alignment step. The image acquisition step acquires multiple perspective images, and the alignment step processes these multiple perspective images to determine whether the component and base are aligned, and adjusts the component position until the patch alignment is complete. The visual imaging-based patch alignment method provided in this embodiment belongs to the alignment step.

[0129] Example 2

[0130] Based on the same inventive concept, this embodiment also provides a visual imaging-based patch alignment device for implementing the aforementioned visual imaging-based patch alignment method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of the visual imaging-based patch alignment device embodiment provided below can be found in the limitations of the visual imaging-based patch alignment method described above and will not be repeated here.

[0131] like Figure 5 As shown, this embodiment provides a patch alignment device based on visual imaging, and the patch alignment device based on visual imaging includes:

[0132] The image acquisition module M1 is used to acquire multiple perspective images; the multiple perspective images are obtained by photographing the component and the base from different perspectives at the same height. The component is located above the base and the component and the base are the same size.

[0133] The refocusing module M2 is used to refocus based on multiple perspective images to obtain a first refocused image refocused to the plane where the top surface of the component is located and a second refocused image refocused to the plane where the top surface of the base is located.

[0134] The first calculation module M3 is used to perform edge detection on the first refocused image to obtain a first edge image, and calculate the first rotation angle, first center point coordinates, height and width of the component based on the first edge image; the first edge image includes the edge of the component.

[0135] The second calculation module M4 is used to perform edge detection on the second refocused image to obtain a second edge image, and calculate the second rotation angle and the second center point coordinates of the base based on the second edge image and the height and width of the component; the second edge image includes a partial edge of the base.

[0136] The judgment module M5 is used to calculate the difference between the first rotation angle and the second rotation angle to obtain the rotation angle difference, calculate the difference between the first center point coordinates and the second center point coordinates to obtain the center point coordinate difference, and judge whether the rotation angle difference and the center point coordinate difference are both 0, so as to judge whether the component and the base coincide with each other.

[0137] Return module M6 is used to control the component action based on the rotation angle difference and the center point coordinate difference if no, and return to the step of "obtaining multiple perspective images".

[0138] Example 3

[0139] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 6As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store multiple perspective images. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a patch alignment method based on visual imaging is implemented.

[0140] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0141] In an exemplary embodiment, a computer device is also provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the patch alignment method based on visual imaging described in Example 1.

[0142] Example 4

[0143] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the patch alignment method based on visual imaging described in Example 1 is implemented.

[0144] Example 5

[0145] This embodiment provides a computer program product, including a computer program. When the computer program is executed by a processor, the visual imaging-based patch alignment method described in Example 1 is implemented.

[0146] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0147] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A patch alignment method based on visual imaging, characterized in that: The patch alignment method based on visual imaging includes: Acquire multiple-view images; the multiple-view images are obtained by photographing the component and the base from different perspectives at the same height, with the component located above the base and having the same size as the base; Refocusing is performed based on the multiple perspective images to obtain a first refocused image refocused to the plane where the top surface of the component is located and a second refocused image refocused to the plane where the top surface of the base is located; Performing edge detection on the first refocused image to obtain a first edge image, and calculating a first rotation angle, first center point coordinates, height, and width of the component based on the first edge image; the first edge image includes an edge of the component; performing edge detection on the second refocused image to obtain a second edge image, and calculating a second rotation angle and a second center point coordinate of the base based on the second edge image and the height and width of the component; the second edge image includes a portion of the edge of the base; Calculating a difference between the first rotation angle and the second rotation angle to obtain a rotation angle difference, calculating a difference between the first center point coordinate and the second center point coordinate to obtain a center point coordinate difference, and determining whether the rotation angle difference and the center point coordinate difference are both 0 to determine whether the component and the base are aligned; If not, the component action is controlled based on the rotation angle difference and the center point coordinate difference, and the process returns to the step of "obtaining multiple perspective images".

2. The patch alignment method based on visual imaging according to claim 1, characterized in that: Acquire images from multiple perspectives, including: The camera is controlled to move at a constant speed along the slide rail, and during the constant speed movement, the camera is controlled to simultaneously shoot the component and the base at predetermined time intervals, and each shooting obtains a perspective image to obtain multiple perspective images; the camera is installed on the slide rail, and the slide rail is located above the component.

3. The patch alignment method based on visual imaging according to claim 1, characterized in that: Refocusing is performed based on the multiple perspective images to obtain a first refocused image refocused to the plane where the top surface of the component is located and a second refocused image refocused to the plane where the top surface of the base is located, specifically including: Randomly select a perspective image from multiple perspective images and record it as the first perspective image; For each second-view image other than the first-view image from the plurality of view images, extract first feature points from the first-view image and second feature points from the second-view image, match all of the first feature points with all of the second feature points to obtain a plurality of matched feature point pairs, each of the matched feature point pairs including a first feature point and a second feature point that match each other; for each of the matched feature point pairs, calculate a disparity between the first feature point and the second feature point in the matched feature point pair, and calculate a depth of each pixel in the first-view image based on the disparities of all of the matched feature point pairs; For each pixel in the first-view image, calculating the reprojected coordinates of the pixel on the plane where the top surface of the component is located based on the coordinates and depth of the pixel and the depth of the plane where the top surface of the component is located; the reprojected coordinates of all the pixels on the plane where the top surface of the component is located constitute a first refocused image refocused to the plane where the top surface of the component is located; For each pixel point in the first perspective image, the reprojected coordinates of the pixel point on the plane where the top surface of the base is located are calculated based on the coordinates and depth of the pixel point and the depth of the plane where the top surface of the base is located; the reprojected coordinates of all the pixel points on the plane where the top surface of the base is located constitute a second refocused image refocused to the plane where the top surface of the base is located.

4. The patch alignment method based on visual imaging according to claim 2, characterized in that: After acquiring multiple perspective images, the patch alignment method based on visual imaging further includes: controlling the camera to enter a dormant state.

5. The patch alignment method based on visual imaging according to claim 1, characterized in that: Calculating the first rotation angle, the first center point coordinates, the height, and the width of the component based on the first edge image specifically includes: determining coordinates of four vertices of the element based on the first edge image; Randomly selecting two adjacent vertices from the four vertices, and calculating a first rotation angle of the component based on coordinates of the two adjacent vertices; Calculate the average value of the coordinates of the four vertices to obtain the coordinates of the first center point of the component; Randomly selecting two adjacent vertices along the height direction from the four vertices, and calculating the distance between the two adjacent vertices along the height direction based on the coordinates of the two adjacent vertices along the height direction to obtain the height of the component; Two adjacent vertices along the width direction are randomly selected from the four vertices, and a distance between the two adjacent vertices along the width direction is calculated based on the coordinates of the two adjacent vertices along the width direction to obtain the width of the component.

6. The patch alignment method based on visual imaging according to claim 1, characterized in that: Calculating a second rotation angle and a second center point coordinate of the base based on the second edge image and the height and width of the component, specifically comprising: determining coordinates of two adjacent vertices of the base based on the second edge image; Calculating a second rotation angle of the base based on the coordinates of two adjacent vertices; A vertex is randomly selected from two adjacent vertices as a selected vertex, and the coordinates of the second center point of the base are calculated based on the coordinates of the selected vertex, the second rotation angle, and the height and width of the component.

7. A patch alignment device based on visual imaging, characterized in that: The patch alignment device based on visual imaging includes: An image acquisition module is used to acquire multiple-view images; the multiple-view images are obtained by photographing the component and the base from different perspectives at the same height, with the component located above the base and having the same size as the base; a refocusing module, configured to perform refocusing based on the multiple perspective images to obtain a first refocused image refocused to the plane where the top surface of the component is located and a second refocused image refocused to the plane where the top surface of the base is located; a first calculation module, configured to perform edge detection on the first refocused image to obtain a first edge image, and calculate a first rotation angle, first center point coordinates, height, and width of the component based on the first edge image; the first edge image includes an edge of the component; a second calculation module, configured to perform edge detection on the second refocused image to obtain a second edge image, and calculate a second rotation angle and a second center point coordinate of the base based on the second edge image and the height and width of the component; the second edge image includes a portion of the edge of the base; a determination module, configured to calculate a difference between the first rotation angle and the second rotation angle to obtain a rotation angle difference, calculate a difference between the first center point coordinate and the second center point coordinate to obtain a center point coordinate difference, and determine whether the rotation angle difference and the center point coordinate difference are both 0, so as to determine whether the component and the base are aligned; The return module is used to control the component action based on the rotation angle difference and the center point coordinate difference if no, and return to the step of "obtaining multiple perspective images".

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the patch alignment method based on visual imaging according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the patch alignment method based on visual imaging according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the patch alignment method based on visual imaging according to any one of claims 1 to 6 is implemented.

Images