Two-dimensional motion axis visual positioning method and device, electronic equipment and storage medium

Abstract

The invention provides a two-dimensional motion axis visual positioning method and device, electronic equipment and a storage medium. The two-dimensional motion axis visual positioning method comprises the following steps: shooting reference images of different sub-regions of a target calibration plate in a target view field range on a two-dimensional shooting platform by using a camera; for each reference image, determining a reference positioning offset, corresponding to the reference image, of the camera according to the motion shafting coordinates corresponding to the target feature points in the reference image; shooting a target image of the target object on the two-dimensional shooting platform in the target view field range by using the camera; and according to the reference positioning offset, the position of the target object in the target image and the target motion shafting coordinate when the camera shoots the target image, determining the real motion shafting coordinate of the target object. According to the scheme, the target object can be positioned more accurately according to the shot image.

Description

Technical Field

[0001] This invention relates to the field of visual positioning technology, specifically to a two-dimensional motion axis visual positioning method, a two-dimensional motion axis visual positioning device, an electronic device, a storage medium, and a computer program product. Background Technology

[0002] Visual positioning technology refers to the use of images or video sequences captured by a camera, along with computer vision and digital image processing techniques, to calculate the position and orientation of the camera itself or the observed target within a predefined coordinate system. Visual positioning technology can be used to locate target objects within images captured by a camera, and due to its high positioning accuracy, it is applicable to various scenarios requiring high-precision positioning.

[0003] Taking automated optical inspection of wafers as an example, in an automated optical inspection system for wafers, a camera performs strip imaging on the wafer to acquire images of different sub-regions. Then, based on the acquired images, the actual position of each chip is analyzed to determine whether the wafer is qualified. To acquire images of different sub-regions of the wafer, it is usually necessary to move the wafer tray or the camera so that the camera can capture images of different sub-regions. However, in this process, due to the limitations of the motion detection platform of the automated optical inspection system, errors inevitably exist in the relative motion between the wafer and the camera, resulting in a deviation between the wafer position calculated based on the captured images and the actual wafer position.

[0004] In existing technologies, a two-dimensional laser interferometer or external measuring equipment can be used to calibrate and correct the axis system of the motion detection platform to reduce errors caused by the relative motion of the wafer and the camera. However, this method relies on external equipment, and the calibration accuracy is limited by the accuracy of the external equipment, placing high demands on the hardware. Summary of the Invention

[0005] The present invention was proposed in view of the above-mentioned problems.

[0006] According to a first aspect of the present invention, a two-dimensional motion axis visual positioning method is provided. The method includes: for each reference image, determining a reference positioning offset of the camera corresponding to the reference image based on the motion axis coordinates of each target feature point in the reference image, wherein the positioning offset is the offset between the actual motion axis coordinates and the theoretical motion axis coordinates of the image captured by the camera, and the reference positioning offset includes a first direction positioning offset and a second direction positioning offset, wherein the first direction is perpendicular to the second direction.

[0007] For example, determining the true motion axis coordinates of the target object based on the reference positioning offset, the position of the target object in the target image, and the target motion axis coordinates when the camera captures the target image includes: determining the correspondence between the actual motion axis coordinates and the predicted positioning offset when the camera captures within the target field of view on the two-dimensional shooting platform, based on the reference positioning offset corresponding to each reference image; determining the target positioning offset corresponding to the target motion axis coordinates based on the correspondence; determining the predicted motion axis coordinates of the target object based on the position of the target object in the target image; and correcting the predicted motion axis coordinates of the target object based on the target positioning offset to obtain the true motion axis coordinates of the target object.

[0008] For example, determining the correspondence between the actual motion axis coordinates and the predicted positioning offset of the camera when shooting within the target field of view on the two-dimensional shooting platform, based on the reference positioning offset corresponding to each reference image, includes: performing surface fitting based on the actual motion axis coordinates of the camera shooting each reference image on the two-dimensional shooting platform and the reference positioning offset corresponding to each reference image, to generate a first surface and a second surface respectively, wherein the first surface represents the correspondence between the actual motion axis coordinates and the positioning offset in the first direction, and the second surface represents the correspondence between the actual motion axis coordinates and the positioning offset in the second direction.

[0009] For example, determining the reference positioning offset of the camera relative to the reference image based on the motion axis coordinates corresponding to the target feature points in the reference image includes: determining the predicted field-of-view center coordinates of the reference image based on the motion axis coordinates corresponding to the target feature points in the reference image, as the theoretical motion axis coordinates for the camera to capture the reference image; calculating the deviation between the predicted field-of-view center coordinates of the reference image and the actual motion axis coordinates for the camera to capture the reference image, to obtain the reference positioning offset of the camera relative to the reference image.

[0010] For example, determining the predicted field of view center coordinates of the reference image based on the motion axis coordinates of each target feature point in the reference image includes: calculating the centroid coordinates of the feature points in the reference image based on the motion axis coordinates of each target feature point in the reference image, and using these as the predicted field of view center coordinates of the reference image.

[0011] For example, the step of extracting feature points in each reference image includes: determining the position and confidence level of the feature points in each reference image, wherein the position of the feature points is used to determine their corresponding motion axis coordinates;

[0012] The step of calculating the centroid coordinates of the feature points in the reference image based on their respective motion axis coordinates includes: calculating the centroid coordinates based on the motion axis coordinates and weight coefficients of the target feature points in the reference image, wherein the weight coefficients of the feature points are proportional to the confidence level of the feature points.

[0013] For example, the step of calculating the centroid coordinates of the feature points in the reference image based on the motion axis coordinates of the target feature points in the reference image includes: obtaining a first template image corresponding to the reference image, wherein the feature points in the first template image correspond one-to-one with the target feature points in the reference image; and determining the centroid coordinates of the feature points in the reference image based on a feature matching center estimation method, according to the difference between the target feature points in the reference image and the feature points in the first template image corresponding to the reference image.

[0014] For example, the feature points in the target calibration plate are evenly distributed, and for each reference image, the target distance corresponding to each edge of the reference image is the same, wherein the target distance corresponding to the edge is the distance between the feature point closest to the edge and the edge.

[0015] For example, determining the reference positioning offset of the camera relative to the reference image based on the motion axis coordinates corresponding to the target feature points in the reference image includes: obtaining a second template image corresponding to the reference image, wherein the feature points in the second template image correspond one-to-one with the target feature points in the reference image; and calculating the reference positioning offset of the camera relative to the reference image based on the difference in motion axis coordinates between the target feature points in the reference image and the feature points in the second template image corresponding to the reference image.

[0016] For example, before determining the true motion axis coordinates of the target object, the method further includes: determining a target reference value for the target reference positioning offset, wherein the target reference value is one of the arithmetic mean, median, weighted mean, and robust mean; for each reference positioning offset, calculating the difference between the target reference value and the reference positioning offset to replace the reference positioning offset.

[0017] For example, the pattern type of the target calibration board includes checkerboard, dot array, random dot array, striped grid and / or QR code encoding, and the number of feature points in each reference image is greater than a preset number threshold. For a target calibration board of dot array type, the feature point in the reference image is the center of the dot. For a target calibration board of checkerboard type, the feature point in the reference image is the corner point.

[0018] For example, the step of using a camera to take pictures of different sub-regions of the target calibration board on a two-dimensional imaging platform to obtain a reference image corresponding to each sub-region includes: using the camera to take pictures of different sub-regions of the target calibration board sequentially on the two-dimensional imaging platform to obtain a reference image corresponding to each sub-region, wherein the overlap rate between adjacent sub-regions is greater than a preset overlap rate threshold.

[0019] According to a second aspect of the present invention, a two-dimensional motion axis visual positioning device is also provided, comprising:

[0020] The first shooting module is used to use a camera to take pictures of different sub-regions of the target calibration board on a two-dimensional shooting platform to obtain a reference image of each sub-region, wherein the target field of view is the same when the camera takes each reference image.

[0021] The feature point extraction module is used to extract feature points from each reference image;

[0022] The offset determination module is used to determine the reference positioning offset of the camera corresponding to each reference image based on the motion axis coordinates of the target feature points in the reference image. The positioning offset is the offset between the actual motion axis coordinates and the theoretical motion axis coordinates of the image captured by the camera. The reference positioning offset includes a first direction positioning offset and a second direction positioning offset, wherein the first direction is perpendicular to the second direction.

[0023] The second shooting module is used to use the camera to shoot the target object on the two-dimensional shooting platform within the target field of view to obtain a target image;

[0024] The positioning module is used to determine the true motion axis coordinates of the target object based on the reference positioning offset, the position of the target object in the target image, and the target motion axis coordinates when the camera captures the target image.

[0025] According to a third aspect of the present invention, an electronic device is also provided, comprising: a processor and a memory, wherein the memory stores computer program instructions, which, when executed by the processor, are used to perform the above-described two-dimensional motion axis visual positioning method.

[0026] According to a fourth aspect of the present invention, a storage medium is also provided, on which program instructions are stored, which, when executed, are used to perform the above-described two-dimensional motion axis visual positioning method.

[0027] According to a fifth aspect of the present invention, a computer program product is also provided, comprising computer program instructions, which, when executed, are used to perform the above-described two-dimensional motion axis visual positioning method.

[0028] In the above technical solution, a camera is used to capture images of different sub-regions of the target calibration plate on a two-dimensional imaging platform to obtain reference images of each sub-region. The target field of view is the same when the camera captures each reference image. Then, feature points in each reference image are extracted. For each reference image, the reference positioning offset of the camera is determined based on the motion axis coordinates corresponding to all feature points of each target in the reference image. The positioning offset is the offset between the actual motion axis coordinates and the theoretical motion axis coordinates of the image captured by the camera. The reference positioning offset includes a first-direction positioning offset and a second-direction positioning offset, with the first direction perpendicular to the second direction. Subsequently, the camera is used to capture images of the target object on the two-dimensional imaging platform within the target field of view to obtain target images. Finally, the actual motion axis coordinates of the target object are determined based on the reference positioning offset, the position of the target object in the target image, and the target motion axis coordinates when the camera captures the target image. By using the feature points in the reference image obtained by the calibration plate, the reference positioning offset of the camera when capturing each reference image in the two-dimensional imaging platform can be accurately determined. Combined with this reference positioning offset, the position information of the target object can be corrected when the camera captures the target object. In this way, the position information of the target object can be obtained without too many external devices, so as to accurately locate the target object based on the captured image.

[0029] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0030] The above and other objects, features, and advantages of the present invention will become more apparent from the more detailed description of the embodiments of the invention in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same parts or steps.

[0031] Figure 1 A schematic flowchart of a two-dimensional motion axis visual positioning method according to an embodiment of the present invention is shown;

[0032] Figure 2 A schematic diagram illustrating the acquisition of a reference image by photographing a target calibration board of the dot array type according to an embodiment of the present invention;

[0033] Figure 3 A schematic flowchart illustrating the determination of the true motion axis coordinates of a target object according to an embodiment of the present invention is shown.

[0034] Figure 4 A schematic diagram showing a two-dimensional view of a first surface and a two-dimensional view of a second surface according to an embodiment of the present invention is provided.

[0035] Figure 5 A schematic flowchart illustrating the determination of a reference positioning offset of a camera corresponding to a reference image according to an embodiment of the present invention is shown.

[0036] Figure 6 A schematic flowchart illustrating the calculation of the centroid coordinates corresponding to target feature points in the reference image according to an embodiment of the present invention is shown.

[0037] Figure 7 A schematic flowchart illustrating the determination of a reference positioning offset of a camera corresponding to a reference image, according to yet another embodiment of the present invention, is shown.

[0038] Figure 8 A schematic flowchart illustrating the determination of a reference positioning offset of a camera corresponding to a reference image, according to yet another embodiment of the present invention, is shown.

[0039] Figure 9 A schematic block diagram of a two-dimensional motion axis visual positioning device according to an embodiment of the present invention is shown;

[0040] Figure 10 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the present invention more apparent, exemplary embodiments according to the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely a part of the embodiments of the present invention, and not all of the embodiments of the present invention. It should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described herein, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of the present invention.

[0042] To at least partially address the aforementioned problems, a two-dimensional motion axis visual positioning method is proposed. This method utilizes a camera to capture images of different sub-regions of a target calibration board on a two-dimensional imaging platform, obtaining reference images for each sub-region. The target field of view is the same for each reference image captured by the camera. Feature points are then extracted from each reference image. For each reference image, based on the motion axis coordinates corresponding to all feature points of the target in that reference image, a reference positioning offset is determined. This offset is the difference between the actual and theoretical motion axis coordinates of the captured image. The reference positioning offset includes a first-direction positioning offset and a second-direction positioning offset, with the first and second directions perpendicular. Subsequently, the camera is used to capture images of the target object within the target field of view on the two-dimensional imaging platform, obtaining a target image. Finally, based on the reference positioning offset, the target object's position in the target image, and the target motion axis coordinates captured by the camera, the true motion axis coordinates of the target object are determined. This scheme can more accurately locate the target object based on the captured images.

[0043] Figure 1 A schematic flowchart of a two-dimensional motion axis visual positioning method according to an embodiment of the present invention is shown. Figure 1 As shown, the two-dimensional motion axis visual positioning method may include steps S110 to S150.

[0044] In step S110, a camera is used to take pictures of different sub-regions of the target calibration plate on a two-dimensional imaging platform to obtain a reference image of each sub-region. The target field of view is the same when the camera takes each reference image.

[0045] The camera can be any type of camera, such as a monocular camera, a stereo / binocular camera, or an RGB-D camera.

[0046] In a 2D imaging platform, a target calibration plate can be placed on a stage, while a camera can be mounted on a support with its optical axis perpendicular to the plane of the stage. During the imaging process, the stage or the support holding the camera can move along an XY motion mechanism, allowing relative movement between the stage and the camera along the X / Y axes of the 2D motion axis. This enables the camera to capture images of different sub-regions of the target calibration plate separately. In commonly used 2D imaging platforms, the camera can be mounted on a support and positioned above the stage, with its optical axis pointing vertically downwards. Preferably, a 2D imaging platform where the camera support can move along an XY motion mechanism during imaging is preferred to avoid changes in the position of the target calibration plate within the stage during movement, which could affect the final positioning accuracy.

[0047] During the process of capturing images of different sub-regions of the target calibration board using a 2D imaging platform, the camera's field of view can be kept constant. Since the 2D imaging platform only supports relative movement between the stage and the camera along the 2D motion axis (i.e., relative movement between the target calibration board and the camera along the 2D motion axis), the object distance between the camera and the target calibration board is also the same. This ensures that the target field of view is the same for each reference image captured by the camera. Understandably, because the target field of view is the same for each reference image captured by the camera, the size of the sub-region targeted by each reference image is the same.

[0048] Optionally, there may be no overlap between different sub-regions captured by the camera.

[0049] For example, a camera can be used to sequentially capture images of different sub-regions of the target calibration board on a two-dimensional imaging platform to obtain reference images corresponding to each sub-region, wherein the overlap rate between adjacent captured sub-regions is greater than a preset overlap rate threshold.

[0050] Each reference image has a corresponding acquisition time, and different sub-regions captured at adjacent acquisition times are the sub-regions captured at those adjacent acquisition times. The overlap rate between different sub-regions captured by adjacent images is greater than a preset overlap rate threshold, ensuring that the same feature point can exist in different reference images. This allows for high-density sampling of feature points across the entire target calibration board, providing a solid data foundation for subsequently determining the true location of the target object. Furthermore, this approach allows for the acquisition of more reference images across the entire target calibration board, ensuring sufficient data density when determining the true location of the target object and avoiding insufficient accuracy due to sparse sampling.

[0051] For example, the number of reference images can be greater than a preset threshold. This allows for the acquisition of more reference images across the entire target calibration board area, ensuring sufficient data density when determining the true location of the target object.

[0052] For example, the pattern type of the target calibration board includes checkerboard, dot array, random dot array, striped grid and / or QR code encoding, and the number of feature points in each reference image is greater than a preset number threshold. For the dot array type target calibration board, the feature point in the reference image is the center of the dot, and for the checkerboard type target calibration board, the feature point in the reference image is the corner point.

[0053] When using a checkerboard-type target calibration board, its feature points can be corner points, i.e., the intersections of adjacent black and white square boundaries. Corner points, as the points of intersection with the most dramatic gray-scale changes in the geometric structure, possess stable spatial positions and high repeatability accuracy, serving as a fundamental reference for the offset between the actual and theoretical motion axis coordinates of the camera-captured image. When using a dot array-type target calibration board, its feature points can be dot centers, i.e., the geometric center coordinates of each circular pattern. Dot centers can be obtained through binarization and connected component analysis, centroid calculation, or ellipse fitting, and are spatially equivalent to the corner points of a checkerboard, similarly serving as feature points for determining the offset between the actual and theoretical motion axis coordinates of the camera-captured image.

[0054] By selecting different types of feature points for different types of target calibration plates, more reasonable feature points can be determined. Furthermore, for each reference image, when the number of feature points in that reference image exceeds a preset threshold, the basis for determining the reference positioning offset corresponding to that reference image is more sufficient, making the subsequently determined reference positioning offset more reasonable and accurate.

[0055] Figure 2 A schematic diagram of taking a picture of a target calibration board of the dot array type according to an embodiment of the present invention to obtain a reference image is shown.

[0056] like Figure 2 As shown, in a two-dimensional imaging platform, a camera can perform strip scanning on the target calibration board along the Y-axis of the motion axis coordinate system. That is, it sequentially captures different sub-regions of the target calibration board from one side to the other along the Y-axis (the dashed box in the figure represents one sub-region of the target calibration board in the example). After each strip scan is completed, the camera moves one strip width along the X-axis, and then continues strip scanning in the direction shown in the figure. This cycle repeats to form a "Z"-shaped scanning trajectory until the scanning area covers the entire target calibration board. Thus, a reference image can be obtained for each sub-region, and because the target field of view is the same when the camera captures each reference image, each reference image has the same size.

[0057] In step S120, feature points are extracted from each reference image.

[0058] Feature points in each reference image can be extracted using feature point detection algorithms or deep learning models.

[0059] For example, feature point detection algorithms may include Harris feature point detection, FAST algorithm, SIFT algorithm, etc.

[0060] For example, sample images with predetermined feature points can be pre-acquired as training samples. These sample images are then input into the deep learning model to output predicted feature points for the sample images. A loss value is then calculated based on the predicted feature points and the predetermined feature points in the sample images, and the deep learning model is optimized using this loss value. After multiple optimizations in this manner, a deep learning model for extracting feature points from a reference image can be obtained.

[0061] For example, the location and confidence level of feature points in each reference image can be determined, where the location of the feature point is used to determine its corresponding motion axis coordinates. When determining feature points in each reference image using a feature point detection algorithm or deep learning model, the confidence level of that feature point can be obtained. A higher confidence level indicates that the feature point is more reliable. The confidence level of the feature points can also be considered when subsequently determining the reference offset.

[0062] For example, before extracting feature points from each reference image, image distortion correction can be performed on the reference images to remove the effects of image distortion.

[0063] In step S130, for each reference image, the reference positioning offset of the camera corresponding to the reference image is determined according to the motion axis coordinates of the target feature points in the reference image. The positioning offset is the offset between the actual motion axis coordinates and the theoretical motion axis coordinates of the image captured by the camera. The reference positioning offset includes a first direction positioning offset and a second direction positioning offset, wherein the first direction is perpendicular to the second direction.

[0064] The target feature points can be all or a subset of the feature points in the reference image. For example, target feature points can be those extracted from feature points that are relatively evenly distributed. Alternatively, they can be those extracted from feature points that are relatively close to the image center. Or, they can be those extracted from feature points that meet a preset density condition. Feature points in the reference image can be filtered according to preset conditions to determine the target feature points, thereby improving the reliability of the target feature points.

[0065] In the aforementioned 2D imaging platform, the camera's optical axis is perpendicular to the plane of the stage, i.e., perpendicular to the target calibration plate plane. However, because the camera needs to capture images of different sub-regions of the target calibration plate, the stage and camera will move relative to each other along the 2D motion axis. During this process, the camera's optical axis may not be perpendicular to the plane of the stage. In the motion axis coordinate system of the aforementioned 2D imaging platform, when the camera's optical axis is perpendicular to the plane of the stage, the theoretical motion axis coordinates of the camera can be deduced from the motion axis coordinates of the feature points in the captured reference image, and these theoretical motion axis coordinates are the same as the camera's actual motion axis coordinates in this case. When the camera's optical axis is not perpendicular to the plane of the stage, the theoretical position and the camera's actual position may differ. In the motion axis coordinate system of the 2D imaging platform, this manifests as an offset between the camera's actual motion axis coordinates and the corresponding theoretical motion axis coordinates, i.e., a reference offset. This reference offset is usually due to errors caused by the hardware of the 2D imaging platform. For each reference image in the aforementioned two-dimensional imaging platform, the reference offset corresponding to the actual position of the camera when capturing that reference image is theoretically the same. That is, the error present when capturing the target calibration board at the same position is the same. Therefore, the reference offset corresponding to the actual position of the camera when capturing each reference image can be determined, and the reference offset can be corrected accordingly when capturing subsequent images, thereby reducing the positioning error corresponding to the reference offset.

[0066] The target calibration plate is fixed to the stage of the aforementioned two-dimensional imaging platform, and the position of the stage in the motion axis coordinate system of the two-dimensional imaging platform can be measured. Therefore, based on the measured position of the stage, the position of the target calibration plate in the motion axis coordinate system of the aforementioned two-dimensional imaging platform is determined. Furthermore, based on the positions of different sub-regions of the target calibration plate within the target calibration plate, the position of each sub-region of the target calibration plate in the motion axis coordinate system of the aforementioned two-dimensional imaging platform, i.e., the motion axis coordinates of the sub-region, can be determined. The arrangement and distribution of feature points in the target calibration plate are fixed; therefore, for each reference image, the motion axis coordinates corresponding to each target feature point in the reference image can be determined in the motion axis coordinate system of the aforementioned two-dimensional imaging platform, based on the motion axis coordinates of the sub-region and the position of the target feature points in the reference image.

[0067] For example, after determining the position of the target feature points in the reference image, the motion axis coordinates of each target feature point in the reference image can be calculated according to the following formulas 1 and 2:

[0068] X = u*Δx + [centre] x -0.5*FOVwidth*Δx] Formula 1

[0069] Y = v * Δy - [centre] y -0.5*FOVhight*Δy] Formula 2

[0070] Where X represents the X-coordinate of the motion axis system corresponding to the target feature point in the reference image, Y represents the Y-coordinate of the motion axis system corresponding to the target feature point in the reference image, FOVwidth represents the pixel width corresponding to the target feature point in the reference image, FOVhight represents the pixel height corresponding to the target feature point in the reference image, Δx represents the pixel equivalent in the X-axis direction of the reference image, Δy represents the pixel equivalent in the Y-axis direction of the reference image, u represents the X-coordinate of the target feature point in the reference image, v represents the Y-coordinate of the target feature point in the reference image, and centre x The X-coordinate, representing the actual motion axis coordinates of the camera, is located at [center]. y The Y-coordinate represents the actual motion axis coordinates of the camera.

[0071] For example, when the target calibration board remains in the motion axis coordinate system of the aforementioned two-dimensional imaging platform, each sub-region of the target calibration board also remains in the same position. Therefore, a template image of each sub-region can be obtained in advance, and then, based on the difference between the corresponding motion axis coordinates of the feature points in the template image and the target feature points in the reference image, the reference offset corresponding to the reference image when the camera captures the reference image can be comprehensively evaluated.

[0072] For example, the theoretical motion axis coordinates of the camera when capturing the reference image can be determined based on the motion axis coordinates corresponding to the target feature points in the reference image. These theoretical motion axis coordinates are the motion axis coordinates that the camera should be in when capturing the sub-region targeted by the reference image with its optical axis perpendicular to it. Then, for each reference image, the reference offset corresponding to that reference image can be determined based on the theoretical and actual motion axis coordinates of the camera when capturing the reference image.

[0073] For example, the predicted field-of-view center coordinates when the camera captures the reference image can be determined based on the motion axis coordinates of the target feature points in the reference image. When the camera's optical axis is perpendicular to a sub-region, the theoretical motion axis coordinates and the corresponding field-of-view center coordinates of the captured reference image should be consistent. However, because the camera's optical axis may not be perpendicular to the captured sub-region after relative movement between the camera and the target calibration plate, there is an offset between the theoretical field-of-view center coordinates of the actual field of view when the camera captures the reference image and the predicted field-of-view center coordinates. This offset reflects the difference between the camera's actual motion axis coordinates and the corresponding theoretical motion axis coordinates, i.e., the reference offset.

[0074] For example, when the camera moves in the aforementioned two-dimensional imaging platform to capture a reference image, the actual motion axis coordinates of the camera in the aforementioned two-dimensional imaging platform can be determined using a grating ruler or other type of positioning sensor set together with the camera.

[0075] For example, when the stage moves in the aforementioned two-dimensional imaging platform to capture a reference image, the actual motion axis coordinates of the camera relative to the target calibration plate in the motion axis coordinate system of the aforementioned two-dimensional imaging platform can be derived using a grating ruler or other type of positioning sensor set together with the stage.

[0076] Understandably, the motion axis coordinate system of the aforementioned two-dimensional shooting platform can be a motion axis coordinate system determined based on the position of the stage. That is, whether the stage moves or the camera moves, as long as the stage and the camera move, it can be regarded as a change in the motion axis coordinate system of the camera under the current motion axis coordinate system.

[0077] Understandably, after the camera moves relative to the target calibration plate, the shooting angle may be tilted in different directions of the motion axis coordinate system of the aforementioned two-dimensional shooting platform. Therefore, the aforementioned reference offset can include positioning offsets in mutually perpendicular directions, namely, the first direction positioning offset and the second direction positioning offset. Combining the first direction positioning offset and the second direction positioning offset, positioning offsets in any direction can be represented. The first direction positioning offset can be an X-direction positioning offset or a Y-direction positioning offset, and the second direction positioning offset can be a Y-direction positioning offset or an X-direction positioning offset. This allows for subsequent correction of the true position of the object in the captured image in the X and Y directions, respectively.

[0078] Optionally, when the first and second directions are perpendicular, the first direction can also be any direction other than the X and Y directions of the motion axis coordinate system of the aforementioned two-dimensional imaging platform, and the second direction can also be any direction other than the X and Y directions of the motion axis coordinate system of the aforementioned two-dimensional imaging platform. It is understood that even if the first direction is not the X or Y direction of the motion axis coordinate system, similar to the X and Y directions, the actual position of the object in the captured image can be subsequently corrected in the first and second directions respectively. For example, the positioning offset in the first and second directions can be represented by the X-direction positioning offset and the Y-direction positioning offset.

[0079] The first and second directions can be chosen based on actual needs.

[0080] In step S140, a camera is used to capture the target object on a two-dimensional imaging platform within the target field of view to obtain a target image.

[0081] When using a camera to capture an image of a target, the field of view used is the same as when capturing a reference image. This avoids introducing additional influencing factors due to differences in the field of view.

[0082] In step S150, the actual motion axis coordinates of the target object are determined based on the reference positioning offset, the position of the target object in the target image, and the target motion axis coordinates when the camera captures the target image.

[0083] Because the field of view used by the camera when capturing the target image is the same as the field of view used when capturing the reference image, the theoretical motion axis coordinates of the target image can be determined based on the target's motion axis coordinates and the target's field of view. Then, the theoretical motion axis coordinates of the target object can be determined based on the target object's position in the target image.

[0084] For example, the reference positioning offset corresponding to the target motion axis coordinates can be retrieved from the actual motion axis coordinates of the camera corresponding to the reference positioning offset. When the reference positioning offset corresponding to the target motion axis coordinates is retrieved, the theoretical motion axis coordinates of the target object can be corrected using the reference positioning offset to obtain the true motion axis coordinates of the target object.

[0085] When the reference positioning offset does not have a corresponding reference positioning offset for the target motion axis coordinates in the actual motion axis coordinates of the camera, interpolation can be performed based on each reference positioning offset and its corresponding actual motion axis coordinates of the camera. This interpolation calculates the reference positioning offset corresponding to the target motion axis coordinates based on the reference positioning offsets corresponding to the actual motion axis coordinates surrounding the target motion axis coordinates. This reference positioning offset can then be used to correct the X and Y coordinates of the theoretical motion axis coordinates of the target object, thus obtaining the true motion axis coordinates of the target object. Interpolation methods can include bilinear interpolation, spline interpolation, polynomial fitting, radial basis function (RBF) interpolation, kriging interpolation, or regression models based on neural networks.

[0086] For example, interpolation can be performed based on each reference positioning offset and its corresponding actual motion axis coordinates of the camera. For each motion axis coordinate in the motion axis coordinate system of the 2D imaging platform, the reference positioning offset corresponding to that motion axis coordinate is calculated based on the reference positioning offsets corresponding to the actual motion axis coordinates around it. Then, the reference positioning offset corresponding to the target motion axis coordinates can be directly retrieved from the determined reference positioning offsets. This reference positioning offset can then be used to correct the X and Y coordinates of the theoretical motion axis coordinates of the target object to obtain the target object's true motion axis coordinates. This reduces the additional computational cost required to determine the reference positioning offset corresponding to the target motion axis coordinates, thus improving the computational efficiency of the target object's true motion axis coordinates.

[0087] Furthermore, the true position of the target object in the world coordinate system can be determined based on the position of the motion axis coordinate system of the 2D shooting platform in the world coordinate system and the true motion axis coordinate system of the target object.

[0088] Taking a two-dimensional imaging platform used for automated optical inspection of wafers as an example, when the target object is a chip within the wafer, the aforementioned two-dimensional motion axis visual positioning method can accurately determine the motion axis coordinates of the wafer chip within the two-dimensional imaging platform. This allows for accurate determination of the relative positions between each wafer chip, thereby determining whether the wafer is qualified. Besides wafer chips, the target object can also be a portion of the wafer, such as the area containing the chip. The target object can also be a region within the wafer whose motion axis coordinates are the same as those of a sub-region on the calibration board.

[0089] In the above technical solution, a camera is used to capture images of different sub-regions of the target calibration plate on a two-dimensional imaging platform to obtain reference images of each sub-region. The target field of view is the same when the camera captures each reference image. Then, feature points in each reference image are extracted. For each reference image, the reference positioning offset of the camera is determined based on the motion axis coordinates corresponding to the target feature points in the reference image. The positioning offset is the offset between the actual motion axis coordinates and the theoretical motion axis coordinates of the image captured by the camera. The reference positioning offset includes a first-direction positioning offset and a second-direction positioning offset, with the first direction perpendicular to the second direction. Subsequently, the camera is used to capture images of the target object on the two-dimensional imaging platform within the target field of view to obtain target images. Finally, the actual motion axis coordinates of the target object are determined based on the reference positioning offset, the position of the target object in the target image, and the target motion axis coordinates when the camera captures the target image. By using the feature points in the reference image obtained by the calibration plate, the reference positioning offset of the camera when capturing each reference image in the two-dimensional imaging platform can be accurately determined. Combined with this reference positioning offset, the position information of the target object can be corrected when the camera captures the target object. In this way, the position information of the target object can be obtained without too many external devices, so as to accurately locate the target object based on the captured image.

[0090] Figure 3 A schematic flowchart illustrating the determination of the true motion axis coordinates of a target object according to an embodiment of the present invention is shown. Figure 3 As shown, step S150 may include steps S210 to S240.

[0091] In step S210, based on the reference positioning offset of the camera corresponding to each reference image, the correspondence between the actual motion axis coordinates and the predicted positioning offset when the camera is shooting within the target field of view on the two-dimensional shooting platform is determined.

[0092] For each reference positioning offset corresponding to a camera and each reference image, the corresponding actual motion axis coordinates can be determined. Then, based on each determined reference positioning offset and its corresponding actual motion axis coordinates, interpolation calculations can be performed to predict the predicted positioning offset for each motion axis coordinate within the motion axis coordinate system of the 2D imaging platform, based on the reference positioning offsets corresponding to the surrounding actual motion axis coordinates. A correspondence between motion axis coordinates and predicted positioning offsets can then be established. This correspondence represents the continuous change of the predicted positioning offset with the actual motion axis coordinates, and can take the form of a function, model, etc. This correspondence can smoothly calculate the reference positioning offset at any axis coordinate within the motion axis coordinate system of the 2D imaging platform, even when the target calibration plate has no corner points or the actual motion axis coordinates of the camera-captured reference images are uneven, ensuring the availability of the target positioning offset corresponding to the subsequent target motion axis coordinates.

[0093] For example, based on the actual motion axis coordinates of each reference image captured by the camera within the two-dimensional imaging platform and the corresponding reference positioning offset of each reference image, surface fitting is performed to generate a first surface and a second surface, respectively. The first surface represents the correspondence between the actual motion axis coordinates and the first direction positioning offset, and the second surface represents the correspondence between the actual motion axis coordinates and the first direction positioning offset.

[0094] The X-axis coordinates of the first and second surfaces represent the coordinates of the camera's actual motion axis system along the first direction when shooting within the target field of view on the 2D imaging platform. The Y-axis coordinates of the first and second surfaces represent the coordinates of the camera's actual motion axis system along the second direction when shooting within the target field of view on the 2D imaging platform. The Z-axis coordinate of the first surface represents the positioning offset in the first direction corresponding to the actual motion axis system coordinates of the first surface, and the Z-axis coordinate of the second surface represents the positioning offset in the second direction corresponding to the actual motion axis system coordinates of the second surface. Subsequently, the positioning offsets in the first and second directions corresponding to the target motion axis system coordinates can be determined based on the first and second surfaces respectively, to obtain the target positioning offset corresponding to the target motion axis system coordinates.

[0095] Compared to functions, the models of the first and second surfaces, in addition to representing the correspondence between the coordinates of the motion axis system and the positioning offset, allow users to more intuitively understand the positioning offset corresponding to the position of the camera when shooting within the two-dimensional shooting platform.

[0096] Figure 4A schematic diagram showing a two-dimensional view of a first surface and a two-dimensional view of a second surface according to an embodiment of the present invention is provided.

[0097] like Figure 4 As shown, the left and lower coordinate axes of the two-dimensional diagrams of the first and second surfaces represent the row and column numbers of the sub-regions of the target calibration plate. Each number corresponds to the actual motion axis coordinates of the camera when capturing images within the two-dimensional imaging platform. The upper and right coordinate axes of the two-dimensional diagrams of the first and second surfaces represent the actual motion axis coordinates of the camera when capturing images within the two-dimensional imaging platform, and the unit can be micrometers (µm). Taking the first direction as the X-direction of the motion axis coordinate system and the second direction as the Y-direction of the motion axis coordinate system as an example, the color value in the two-dimensional diagram of the first surface can reflect the positioning offset in the X-direction corresponding to the actual motion axis coordinate system, and the color value in the two-dimensional diagram of the second surface can represent the positioning offset in the Y-direction corresponding to the actual motion axis coordinate system.

[0098] In step S220, the target positioning offset corresponding to the target motion axis coordinates is determined according to the correspondence.

[0099] The target positioning offset corresponding to the target motion axis coordinates can be retrieved from the correspondence.

[0100] In step S230, the predicted motion axis coordinates of the target object are determined based on the position of the target object in the target image.

[0101] The theoretical motion axis coordinates of the target image are determined based on the target's motion axis coordinates and the target's field of view. Then, based on the target object's position in the target image, the theoretical motion axis coordinates of the target object are determined as the predicted motion axis coordinates.

[0102] In step S240, the predicted motion axis coordinates of the target object are corrected according to the target positioning offset to obtain the true motion axis coordinates of the target object.

[0103] The new coordinates can be obtained by combining the predicted motion axis coordinates, the first direction positioning offset of the target positioning offset, and the sum of the first direction positioning offsets. For example, when the first direction is the X direction of the motion axis coordinates and the second direction is the Y direction of the motion axis coordinates, the new X coordinate can be obtained by combining the predicted X coordinates of the motion axis coordinates with the X direction positioning offset of the target positioning offset, and the new Y coordinate can be obtained by combining the predicted Y coordinates of the motion axis coordinates with the Y direction positioning offset of the target positioning offset. This updated predicted motion axis coordinates can then be used as the true motion axis coordinates.

[0104] In the above technical solution, based on the reference positioning offset corresponding to each reference image, the correspondence between the actual motion axis coordinates and the predicted positioning offset when the camera is shooting within the target field of view on the 2D shooting platform is determined. Then, based on the correspondence, the target positioning offset corresponding to the target motion axis coordinates is determined. Next, based on the position of the target object in the target image, the predicted motion axis coordinates of the target object are determined. Finally, based on the target positioning offset, the predicted motion axis coordinates of the target object are corrected to obtain the true motion axis coordinates of the target object. This method can more accurately and efficiently correct the predicted motion axis coordinates of the target object by establishing the correspondence between the actual motion axis coordinates and the predicted positioning offset when the camera is shooting within the target field of view on the 2D shooting platform, thus obtaining the accurate true motion axis coordinates of the target object.

[0105] Figure 5 A schematic flowchart illustrating the determination of a reference positioning offset of a camera corresponding to a reference image, according to an embodiment of the present invention, is shown. Figure 5 As shown, step S130 may include steps S310 to S320.

[0106] In step S310, the predicted field of view center coordinates of the reference image are determined based on the motion axis coordinates of the target feature points in the reference image, so as to serve as the theoretical motion axis coordinates for the camera to capture the reference image.

[0107] Understandably, because the camera's optical axis is not perpendicular to the target calibration plate when capturing the reference image, the coordinates of the feature points in the reference image will be affected by motion axis positioning errors and straightness errors, resulting in an overall offset. Therefore, it is not advisable to directly determine the theoretical motion axis coordinates of the camera capturing the reference image based on the center position of the reference image. Specifically, the predicted field-of-view center coordinates of the reference image can be determined by comprehensively considering the motion axis coordinates of each target feature point in the reference image, and used as the theoretical motion axis coordinates of the camera capturing the reference image. For example, the arithmetic mean of the motion axis coordinates of each target feature point in the reference image can be used as the predicted field-of-view center coordinates of the reference image. Alternatively, the weighted arithmetic mean of the motion axis coordinates of each target feature point in the reference image can be used as the predicted field-of-view center coordinates of the reference image. Another example is calculating the reference field-of-view center coordinates of the reference image based on the motion axis coordinates of feature points in regions of different sizes centered on the image center, and then using the average of these reference field-of-view center coordinates as the predicted field-of-view center coordinates of the reference image.

[0108] For example, step S310 may include step S311: calculating the centroid coordinates of the feature points in the reference image based on the motion axis coordinates of each target feature point in the reference image, so as to use the predicted field of view center coordinates of the reference image.

[0109] The centroid coordinates of the feature points in the reference image are the centroids of the reference image. The centroid coordinates of the feature points in the reference image can be obtained by calculating the arithmetic mean of the coordinates of the motion axes corresponding to each target feature point in the reference image.

[0110] For example, the centroid coordinates of feature points in the reference image can be calculated using the following formulas 3 and 4:

[0111]

[0112] in, The X-coordinate represents the centroid coordinates of the feature points in the reference image. The Y-coordinate and X-coordinate represent the barycenter coordinates of the feature points in the reference image. i The X and Y coordinates of the motion axis system corresponding to the i-th feature point represent the coordinates of the i-th feature point. i The Y-coordinate represents the motion axis coordinates of the i-th feature point, and N represents the total number of feature points in the reference image.

[0113] For example, the feature points in the target calibration plate are evenly distributed, and for each reference image, the target distance corresponding to each edge of the reference image is the same, wherein the target distance corresponding to an edge is the distance between the nearest target feature point to that edge and the edge itself. This allows the target feature points in the reference image to be as close as possible to the center of the reference image and to be evenly distributed, making the determined centroid coordinates more reasonable.

[0114] In step S320, the deviation between the predicted field-of-view center coordinates corresponding to the reference image and the actual motion axis coordinates of the camera capturing the reference image is calculated to obtain the reference positioning offset between the camera and the reference image.

[0115] The deviation between the predicted field-of-view center coordinates corresponding to the reference image and the actual motion axis coordinates of the camera capturing the reference image is the offset between the actual motion axis coordinates of the camera capturing the reference image and the theoretical motion axis coordinates. Therefore, it can be used as the reference positioning offset between the camera and the reference image.

[0116] Theoretically, the centroid coordinates determined by formulas 3 and 4 above... It should be consistent with the actual motion axis coordinates of the reference image (centre). x , centre yWhile the two images are identical, there will be a deviation between them due to errors in the two-dimensional motion axes. This deviation is the reference positioning offset between the camera and the reference image.

[0117] For example, the first direction can be the X direction of the motion axis coordinate system, and the second direction can be the Y direction of the motion axis coordinate system. The reference positioning offset corresponding to the reference image can be determined according to the following formulas 5 and 6:

[0118]

[0119] Where, ΔX j This represents the X-direction positioning offset corresponding to the reference image, ΔY. j This represents the Y-direction positioning offset corresponding to the reference image.

[0120] In the above technical solution, based on the motion axis coordinates corresponding to the target feature points in the reference image, the predicted field-of-view center coordinates of the reference image are determined, serving as the theoretical motion axis coordinates for the camera capturing the reference image. Then, the deviation between the predicted field-of-view center coordinates and the actual motion axis coordinates of the camera capturing the reference image is calculated to obtain the reference positioning offset between the camera and the reference image. By combining the motion axis coordinates of the target feature points in the reference image to infer the theoretical motion axis coordinates of the camera capturing the reference image, and then combining this with the actual motion axis coordinates of the camera capturing the reference image, the reference positioning offset between the camera and the reference image can be accurately calculated.

[0121] For example, the centroid coordinates can be calculated based on the motion axis coordinates and weight coefficients corresponding to the target feature points in the reference image, wherein the weight coefficients corresponding to the feature points are proportional to the confidence level of the feature points.

[0122] Given the motion axis coordinates and weight coefficients of each target feature point in the reference image, the weighted motion axis coordinates of each target feature point can be calculated separately. Then, the arithmetic mean of the weighted motion axis coordinates can be calculated as the centroid coordinates.

[0123] The higher the confidence level of a feature point, the more reliable the feature point is. The calculated centroid coordinates are the weighted arithmetic mean of the motion axis coordinates of the target feature points in the reference image. By combining the reliability of the feature points in the reference image, a more reasonable centroid coordinate of the reference image can be determined.

[0124] Figure 6 A schematic flowchart illustrating the calculation of the centroid coordinates corresponding to target feature points in a reference image according to an embodiment of the present invention is shown. Figure 6As shown, step S311 may include steps S410 to S420.

[0125] In step S410, a first template image corresponding to the reference image is obtained, wherein the feature points in the first template image correspond one-to-one with the target feature points of the reference image.

[0126] The field of view used when capturing the first template image can be the same as that used when capturing the reference image, and the optical axis of the camera is perpendicular to the target calibration plate when capturing the first template image. A two-dimensional laser interferometer or external measuring equipment can be used to make the camera's optical axis perpendicular to the target calibration plate, so as to acquire the first template image for different sub-regions of the target calibration plate. The sub-region targeted by the reference image is the same as the sub-region targeted by the corresponding first template image.

[0127] In step S420, based on the feature matching center estimation method, the centroid coordinates corresponding to the feature points in the reference image are determined according to the difference between the feature points in the reference image and the feature points in the first template image corresponding to the reference image.

[0128] Based on the descriptors (such as Euclidean distance or Hamming distance) corresponding to feature points in the reference image and the first template image, feature points in the reference image and the first template image can be matched to obtain matching point pairs. Then, a transformation matrix from the reference image coordinate system to the template image coordinate system can be estimated using these matching point pairs. For each feature point in the reference image, its corresponding position in the template image coordinate system can be calculated using the estimated transformation matrix. Finally, the average position of the corresponding positions of the target feature points in the reference image and the corresponding positions in the template image coordinate system can be calculated as the barycentric coordinates of the target feature points in the reference image.

[0129] In the above technical solution, a first template image corresponding to the reference image is obtained, wherein the feature points in the first template image correspond one-to-one with the feature points in the reference image. Then, based on a feature matching center estimation method, the centroid coordinates corresponding to the target feature points in the reference image are determined according to the differences between the target feature points in the reference image and the feature points in the corresponding first template image. In this way, a more reasonable centroid coordinate corresponding to the feature points in the reference image can be determined using the first template image as a reference.

[0130] Figure 7 A schematic flowchart illustrating the determination of the reference positioning offset of the camera corresponding to the reference image is shown according to yet another embodiment of the present invention. Figure 7 As shown, step S130 may include steps S510 to S520.

[0131] In step S510, a second template image corresponding to the reference image is obtained, wherein the feature points in the second template image correspond one-to-one with the target feature points of the reference image.

[0132] Step S510 is similar to step S410 above, and will not be described in detail here.

[0133] In step S520, the reference positioning offset of the camera relative to the reference image is calculated based on the difference in motion axis coordinates between the target feature points in the reference image and the feature points in the second template image corresponding to the reference image.

[0134] The target feature points in the reference image and the feature points in the first template image can be matched to obtain matching point pairs.

[0135] For example, for the matching point pair where the target feature point of the reference image is located, the average value can be calculated based on the difference in motion axis coordinates between each matching point pair, and used as the reference positioning offset of the camera corresponding to the reference image.

[0136] For example, the least squares method can be used to determine the reference positioning offset of the camera corresponding to the reference image, so that the positions of the feature points in each matching point pair are as close as possible.

[0137] In the above technical solution, a second template image corresponding to the reference image is obtained. The feature points in the second template image correspond one-to-one with the target feature points in the reference image. Then, based on the difference in motion axis coordinates between the target feature points in the reference image and the feature points in the corresponding second template image, the reference positioning offset of the camera relative to the reference image is calculated. This allows for accurate determination of the camera's reference positioning offset by combining the differences in motion axis coordinates between the feature points in the reference image and the feature points in the corresponding second template image, further enabling more accurate correction of the target object's position.

[0138] Figure 8 A schematic flowchart illustrating the determination of the reference positioning offset of the camera corresponding to the reference image is shown according to yet another embodiment of the present invention. Figure 8 As shown, before step S150, the above two-dimensional motion axis visual positioning method may also include steps S131 to S132.

[0139] In step S131, a target reference value for the reference positioning offset is determined, wherein the target reference value is one of the arithmetic mean, median, weighted mean, and robust mean.

[0140] Ideally, the target reference value for the reference positioning offset should be close to zero. If a significant overall offset occurs, it indicates that the error is not inherent to the motion axis, but rather stems from inaccurate placement of the calibration plate. In practice, placing the calibration plate precisely in its correct reference position is nearly impossible. Even with fine adjustments, a global translational deviation of several to tens of micrometers may remain. For example, for each sub-region, due to the positioning error of the calibration plate, there is an error between the actual motion axis coordinates of the sub-region and the calculated theoretical motion axis coordinates. Furthermore, the motion axis coordinates corresponding to the feature points in the further determined reference object will also deviate. Therefore, the target reference values ​​of all reference positioning offsets can be used to statistically analyze the global translational deviation. This error can then be removed to offset the overall deviation of the reference positioning offset caused by it.

[0141] For example, a robust mean can be a truncated mean or a type of robust mean.

[0142] In step S132, for each reference positioning offset, the difference between the target reference value and the reference positioning offset is calculated to replace the reference positioning offset.

[0143] For each reference positioning offset, subtracting the calculated target reference value from the reference positioning offset can offset the overall deviation of the reference positioning offset caused by the placement position of the caliper, resulting in an updated and more accurate reference positioning offset.

[0144] For example, the first direction can be the X direction of the motion axis coordinate system, and the second direction can be the Y direction of the motion axis coordinate system. The updated reference positioning offset can be determined according to the following formulas 7, 8, 9 and 10:

[0145]

[0146] Where α represents the target reference value of the X-direction positioning offset, β represents the target reference value of the X-direction positioning offset, and ΔX j ΔY represents the X-direction positioning offset corresponding to the j-th reference image. j ΔX′ represents the Y-direction positioning offset corresponding to the j-th reference image. J ΔY′ represents the updated X-direction positioning offset corresponding to the j-th reference image. J This represents the updated Y-direction positioning offset corresponding to the j-th reference image.

[0147] In the above technical solution, a target reference value is determined for all reference positioning offsets. This target reference value is one of the arithmetic mean, median, weighted mean, or robust mean. Then, for each reference positioning offset, the difference between the target reference value and the offset is calculated, and this difference is used to replace the offset. Using the target reference values ​​for all reference positioning offsets, the global translation deviation is statistically analyzed. By removing this deviation, the updated reference positioning offsets can more accurately reflect the errors present when the camera captured the reference image.

[0148] Figure 9 A schematic block diagram of a two-dimensional motion axis visual positioning device according to an embodiment of the present invention is shown. Figure 9 As shown, the two-dimensional motion axis visual positioning device includes a first shooting module 910, a feature point extraction module 920, an offset determination module 930, a second shooting module 940, and a positioning module 950.

[0149] The first imaging module 910 is used to take pictures of different sub-regions of the target calibration plate on a two-dimensional imaging platform using a camera, so as to obtain a reference image of each sub-region, wherein the target field of view is the same when the camera takes each reference image.

[0150] The feature point extraction module 920 is used to extract feature points from each reference image.

[0151] The offset determination module 930 is used to determine the reference positioning offset of the camera relative to the reference image for each reference image based on the motion axis coordinates of the target feature points in the reference image. The positioning offset is the offset between the actual motion axis coordinates and the theoretical motion axis coordinates of the image captured by the camera. The reference positioning offset includes a first direction positioning offset and a second direction positioning offset, with the first direction being perpendicular to the second direction.

[0152] The second shooting module 940 is used to capture images of the target object on a two-dimensional shooting platform within the target field of view using a camera, so as to obtain a target image.

[0153] The positioning module 950 is used to determine the true motion axis coordinates of the target object based on the reference positioning offset, the position of the target object in the target image, and the target motion axis coordinates when the camera captures the target image.

[0154] For example, the positioning module 950 may include a relationship determination submodule, a target positioning offset determination submodule, a predicted motion axis coordinate determination submodule, and a true motion axis coordinate calculation submodule. The relationship determination submodule is used to determine the correspondence between the actual motion axis coordinates and the predicted positioning offset when the camera is shooting within the target field of view on the two-dimensional imaging platform, based on the reference positioning offsets corresponding to each reference image. The target positioning offset determination submodule is used to determine the target positioning offset corresponding to the target motion axis coordinates based on the correspondence. The predicted motion axis coordinate determination submodule is used to determine the predicted motion axis coordinates of the target object based on the position of the target object in the target image. The true motion axis coordinate calculation submodule is used to correct the predicted motion axis coordinates of the target object based on the target positioning offset to obtain the true motion axis coordinates of the target object.

[0155] For example, the relationship determination submodule may include a relationship calculation submodule. The relationship calculation submodule is used to perform surface fitting based on the actual motion axis coordinates of each reference image captured by the camera within the 2D imaging platform and the corresponding reference positioning offset of each reference image, to generate a first surface and a second surface, respectively. The first surface represents the correspondence between the actual motion axis coordinates and the positioning offset in a first direction, and the second surface represents the correspondence between the actual motion axis coordinates and the positioning offset in a second direction.

[0156] For example, the offset determination module 930 may include a predicted field-of-view center coordinate determination submodule and a reference positioning offset determination submodule. The predicted field-of-view center coordinate determination submodule is used to determine the predicted field-of-view center coordinates corresponding to the reference image based on the motion axis coordinates corresponding to the target feature points in the reference image, so as to serve as the theoretical motion axis coordinates for the camera to capture the reference image. The reference positioning offset determination submodule is used to calculate the deviation between the predicted field-of-view center coordinates corresponding to the reference image and the actual motion axis coordinates for the camera to capture the reference image, so as to obtain the reference positioning offset between the camera and the reference image.

[0157] For example, the predicted field of view center coordinate determination submodule may include a barycentric coordinate calculation submodule. The barycentric coordinate calculation submodule is used to calculate the barycentric coordinates corresponding to the feature points in the reference image based on the motion axis coordinates corresponding to the target feature points in the reference image, so as to serve as the predicted field of view center coordinates corresponding to the reference image.

[0158] For example, the feature point extraction module 920 may include a first extraction submodule, and the centroid coordinate calculation submodule may include a first calculation submodule. The first extraction submodule is used to determine the position and confidence level of feature points in each reference image, wherein the position of the feature point is used to determine its corresponding motion axis coordinates. The first calculation submodule is used to calculate the centroid coordinates based on the motion axis coordinates and weight coefficients corresponding to the target feature points in the reference image, wherein the weight coefficient corresponding to the feature point is proportional to the confidence level of the feature point.

[0159] For example, the barycenter coordinate calculation submodule may include a first template acquisition submodule and a matching submodule. The first template acquisition submodule is used to acquire a first template image corresponding to the reference image, wherein the feature points in the first template image correspond one-to-one with the target feature points in the reference image. The matching submodule is used to determine the barycenter coordinates corresponding to the feature points in the reference image based on a feature matching center estimation method, according to the difference between the target feature points in the reference image and the feature points in the first template image corresponding to the reference image.

[0160] For example, the feature points in the target calibration plate are uniformly distributed, and for each reference image, the target distance corresponding to each edge of the reference image is the same, wherein the target distance corresponding to the edge is the distance between the nearest target feature point to the edge and the edge.

[0161] For example, the offset determination module 930 may include a second template acquisition submodule and an offset determination submodule. The second template acquisition submodule is used to acquire a second template image corresponding to the reference image, wherein the feature points in the second template image correspond one-to-one with the target feature points in the reference image. The offset determination submodule is used to calculate the reference positioning offset of the camera relative to the reference image based on the difference in motion axis coordinates between the target feature points in the reference image and the feature points in the corresponding second template image.

[0162] For example, the above-described two-dimensional motion axis visual positioning device may further include a target reference value calculation module and an offset update module. Before determining the actual motion axis coordinates of the target object, the target reference value calculation module is used to determine a target reference value for the reference positioning offset, wherein the target reference value is one of the arithmetic mean, median, weighted mean, and robust mean. The offset update module is used to calculate the difference between the target reference value and the reference positioning offset for each reference positioning offset, and to replace the reference positioning offset.

[0163] For example, the pattern type of the target calibration board includes checkerboard, dot array, random dot array, striped grid and / or QR code encoding, and the number of feature points in each reference image is greater than a preset number threshold. For the dot array type target calibration board, the feature point in the reference image is the center of the dot, and for the checkerboard type target calibration board, the feature point in the reference image is the corner point.

[0164] For example, the first imaging module 910 may include an imaging control submodule. The imaging control submodule is used to use a camera to sequentially capture images of different sub-regions of the target calibration board on a two-dimensional imaging platform to obtain a reference image corresponding to each sub-region, wherein the overlap rate between adjacent captured sub-regions is greater than a preset overlap rate threshold.

[0165] According to another aspect of the present invention, an electronic device is also provided. Figure 10 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. Figure 10 As shown, the electronic device includes a processor and a memory, wherein the memory stores computer program instructions, which are executed by the processor to perform the two-dimensional motion axis visual positioning method as described above.

[0166] Furthermore, according to another aspect of the present invention, a storage medium is provided, on which program instructions are stored. When the program instructions are executed by a computer or processor, the computer or processor performs corresponding steps of the two-dimensional motion axis visual positioning method described above in the embodiments of the present invention, and is used to implement corresponding modules in the two-dimensional motion axis visual positioning device described above in the embodiments of the present invention. The storage medium may, for example, include a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0167] According to another aspect of the present invention, a computer program product is also provided, including computer program instructions, which, when executed, are used to perform the above-described two-dimensional motion axis visual positioning method.

[0168] Those skilled in the art can understand the specific implementation and beneficial effects of the above-described two-dimensional motion axis visual positioning device, electronic device, storage medium, and computer program product by reading the detailed description of the two-dimensional motion axis visual positioning method. For the sake of brevity, they will not be described in detail here.

[0169] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of the invention. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of the invention. All such changes and modifications are intended to be included within the scope of the invention as claimed in the appended claims.

[0170] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0171] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0172] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0173] Similarly, it should be understood that, in order to streamline the invention and aid in understanding one or more of the various aspects of the invention, features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of the invention. However, this approach should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the corresponding claims, its inventive point lies in solving the corresponding technical problem with fewer features than all of those in a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the invention.

[0174] Those skilled in the art will understand that, apart from the mutual exclusion of features, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus so disclosed can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0175] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments can be used in any combination.

[0176] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules in the two-dimensional motion axis visual positioning device according to embodiments of the present invention. The present invention can also be implemented as a device program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0177] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0178] The above description is merely a specific embodiment of the present invention or an explanation of that embodiment. The scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. The scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A two-dimensional motion axis visual positioning method, characterized in that, The method includes: Using a camera, different sub-regions of the target calibration board are photographed on a two-dimensional imaging platform to obtain a reference image for each sub-region. The target field of view is the same when the camera captures each reference image. Extract feature points from each reference image; For each reference image, the reference positioning offset of the camera corresponding to the reference image is determined based on the motion axis coordinates of the target feature points in the reference image. The positioning offset is the offset between the actual motion axis coordinates and the theoretical motion axis coordinates of the image captured by the camera. The reference positioning offset includes a first direction positioning offset and a second direction positioning offset, with the first direction perpendicular to the second direction. Using the camera, the target object is photographed on the two-dimensional imaging platform within the target field of view to obtain a target image; The true motion axis coordinates of the target object are determined based on the reference positioning offset, the position of the target object in the target image, and the target motion axis coordinates when the camera captures the target image.

2. The method according to claim 1, characterized in that, Determining the true motion axis coordinates of the target object based on the reference positioning offset, the position of the target object in the target image, and the target motion axis coordinates when the camera captured the target image includes: Based on the reference positioning offset of the camera corresponding to each reference image, the correspondence between the actual motion axis coordinates and the predicted positioning offset when the camera takes pictures within the target field of view on the two-dimensional shooting platform is determined. Based on the correspondence, determine the target positioning offset corresponding to the target motion axis coordinates; Based on the position of the target object in the target image, determine the predicted motion axis coordinates of the target object; Based on the target positioning offset, the predicted motion axis coordinates of the target object are corrected to obtain the true motion axis coordinates of the target object.

3. The method according to claim 2, characterized in that, The step of determining the correspondence between the actual motion axis coordinates of the camera when shooting within the target field of view on the two-dimensional shooting platform and the predicted positioning offset, based on the reference positioning offset corresponding to each reference image, includes: Based on the actual motion axis coordinates of each reference image captured by the camera within the two-dimensional imaging platform and the corresponding reference positioning offset of each reference image, surface fitting is performed to generate a first surface and a second surface, respectively. The first surface represents the correspondence between the actual motion axis coordinates and the positioning offset in the first direction, and the second surface represents the correspondence between the actual motion axis coordinates and the positioning offset in the second direction.

4. The method according to claim 1, characterized in that, The step of determining the reference positioning offset of the camera corresponding to the reference image based on the motion axis coordinates of each target feature point in the reference image includes: Based on the motion axis coordinates of the target feature points in the reference image, the predicted field of view center coordinates of the reference image are determined, which are then used as the theoretical motion axis coordinates of the camera capturing the reference image. The deviation between the predicted field-of-view center coordinates corresponding to the reference image and the actual motion axis coordinates of the camera capturing the reference image is calculated to obtain the reference positioning offset between the camera and the reference image.

5. The method according to claim 4, characterized in that, The step of determining the coordinates of the predicted field of view center corresponding to the reference image based on the motion axis coordinates of each target feature point in the reference image includes: Based on the motion axis coordinates of the target feature points in the reference image, the centroid coordinates of the feature points in the reference image are calculated and used as the predicted field-of-view center coordinates of the reference image.

6. The method according to claim 5, characterized in that, The extraction of feature points from each reference image includes: Determine the position and confidence level of feature points in each reference image, wherein the position of the feature points is used to determine their corresponding motion axis coordinates; The step of calculating the centroid coordinates of the feature points in the reference image based on their respective motion axis coordinates includes: The centroid coordinates are calculated based on the motion axis coordinates and weight coefficients of the target feature points in the reference image, wherein the weight coefficients of the feature points are proportional to the confidence level of the feature points.

7. The method according to claim 5, characterized in that, The step of calculating the centroid coordinates of the feature points in the reference image based on their respective motion axis coordinates includes: Obtain the first template image corresponding to the reference image, wherein the feature points in the first template image correspond one-to-one with the target feature points of the reference image; The centroid estimation method based on feature matching determines the centroid coordinates of the feature points in the reference image based on the difference between the target feature points in the reference image and the feature points in the first template image corresponding to the reference image.

8. The method according to claim 5, characterized in that, The feature points in the target calibration plate are evenly distributed, and for each reference image, the target distance corresponding to each edge of the reference image is the same, wherein the target distance corresponding to the edge is the distance between the nearest target feature point to the edge and the edge.

9. The method according to claim 1, characterized in that, The step of determining the reference positioning offset of the camera corresponding to the reference image based on the motion axis coordinates of each target feature point in the reference image includes: Obtain the second template image corresponding to the reference image, wherein the feature points in the second template image correspond one-to-one with the target feature points of the reference image; Based on the difference in motion axis coordinates between the target feature points in the reference image and the feature points in the corresponding second template image, the reference positioning offset of the camera relative to the reference image is calculated.

10. The method according to claim 1, characterized in that, Before determining the true motion axis coordinates of the target object, the method further includes: Determine a target reference value for the reference positioning offset, wherein the target reference value is one of the arithmetic mean, median, weighted mean, and robust mean; For each reference positioning offset, the difference between the target reference value and the reference positioning offset is calculated to replace the reference positioning offset.

11. The method according to claim 1, characterized in that, The target calibration board has a pattern type including checkerboard, dot array, random dot array, striped grid and / or QR code encoding, and the number of feature points in each reference image is greater than a preset number threshold. For the dot array type target calibration board, the feature point in the reference image is the center of the dot, and for the checkerboard type target calibration board, the feature point in the reference image is the corner point.

12. The method according to claim 1, characterized in that, The step of using a camera to capture images of different sub-regions of the target calibration board on a two-dimensional imaging platform to obtain a reference image corresponding to each sub-region includes: Using the camera, the two-dimensional imaging platform sequentially captures images of different sub-regions of the target calibration board to obtain reference images corresponding to each sub-region, wherein the overlap rate between adjacent captured sub-regions is greater than a preset overlap rate threshold.

13. A two-dimensional motion axis visual positioning device, characterized in that, include: The first shooting module is used to use a camera to take pictures of different sub-regions of the target calibration board on a two-dimensional shooting platform to obtain a reference image of each sub-region, wherein the target field of view is the same when the camera takes each reference image. The feature point extraction module is used to extract feature points from each reference image; The offset determination module is used to determine the reference positioning offset of the camera corresponding to each reference image based on the motion axis coordinates of the target feature points in the reference image. The positioning offset is the offset between the actual motion axis coordinates and the theoretical motion axis coordinates of the image captured by the camera. The reference positioning offset includes a first direction positioning offset and a second direction positioning offset, wherein the first direction is perpendicular to the second direction. The second shooting module is used to use the camera to shoot the target object on the two-dimensional shooting platform within the target field of view to obtain a target image; The positioning module is used to determine the true motion axis coordinates of the target object based on the reference positioning offset, the position of the target object in the target image, and the target motion axis coordinates when the camera captures the target image.

14. An electronic device comprising a processor and a memory, characterized in that, The memory stores computer program instructions, which, when executed by the processor, are used to perform the two-dimensional motion axis visual positioning method as described in any one of claims 1 to 12.

15. A storage medium on which program instructions are stored, characterized in that, The program instructions, when executed, are used to perform the two-dimensional motion axis visual positioning method as described in any one of claims 1 to 12.

16. A computer program product comprising computer program instructions, characterized in that, The computer program instructions, when executed, are used to perform the two-dimensional motion axis visual positioning method as described in any one of claims 1 to 12.

Images