A visual localization method and system in complex environments

Abstract

This invention relates to a visual positioning method and system in complex environments, comprising the following steps: establishing a coordinate system based on at least four X-corner markers pre-arranged on the target object; acquiring an image of the target object containing the X-corner markers and performing template matching to determine the coordinates of the template matching center point; determining a region of interest based on the template matching center point, and performing line detection within the region of interest to obtain k lines with the minimum distance to the template matching center point and t line intersections; determining the geometric center M0 of the t line intersections, extracting the X-corner point of the i-th marker, and solving the PnP problem using the pixel coordinates and world coordinates of the four detected X-corner points to achieve visual positioning. Through line detection, point-to-line distance, and point-to-point distance, the position of the marker's feature points is determined, other irrelevant corner points are filtered out, and sub-pixel level X-corner points are extracted through sub-pixel level corner point detection to determine the feature point position for visual positioning.

Description

Technical Field

[0001] This invention relates to the field of machine vision technology, specifically to a visual positioning method and system for complex environments. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Visual localization plays a crucial role in fields such as automated robot assembly, robot motion modeling, and autonomous driving. In the visual localization process, using a camera to detect landmark features and solving the Perspective-n-Point (Pn P) problem by establishing the correspondence between at least four sets of feature points in the pixel coordinate system and the world coordinate system is a common and effective method for pose determination.

[0004] The selection of markers requires them to be simple to manufacture, easy to place, have clear features, and be easily distinguishable from the background environment. Generally, LED light panels, different colored light sources, contour models, or rings can be used as markers for detection. However, in complex, harsh, and dangerous environments such as large mills, wells, or disaster sites, stable light sources cannot be guaranteed, significantly increasing the difficulty of marker detection. Due to the presence of background noise such as bright spots, the detection of markers based on light reflection has significant instability and limitations. In addition, the presence of eccentricity errors will also reduce the detection accuracy of contours and the center of rings, making it impossible to guarantee high positioning accuracy.

[0005] X-corner markers use only black and white pixels, making them less sensitive to lighting conditions. They are suitable for single-channel grayscale cameras, which offer superior luminous flux, detail, application range, and stability compared to RGB cameras. Furthermore, since the X-corner point is the intersection of two straight lines, this intersection still represents the real-world position of the intersection point within the camera, eliminating any eccentricity error. Therefore, X-corner markers are ideal visual positioning markers.

[0006] However, due to the presence of Gaussian noise, salt-and-pepper noise, directly performing corner detection on landmarks in complex environments based on grayscale images, binary images, or contour curves will result in redundant noise corners in the environment, making it impossible to match the feature point pixel coordinates with the world coordinates, and thus impossible to solve the PnP problem to complete the localization. Summary of the Invention

[0007] To address the technical problems existing in the background art, the present invention provides a visual positioning method and system for complex environments, which accurately detects the X corner point of an X marker in a complex environment image. This is key to achieving visual positioning in complex environments by leveraging the advantages of X corner point being insensitive to illumination, free from eccentricity error, and highly versatile.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] The first aspect of this invention provides a visual positioning method in a complex environment, comprising the following steps:

[0010] Establish a coordinate system based on at least four X-corner markers pre-arranged on the target object;

[0011] Acquire an image of the target object containing the X-corner marker and perform template matching to determine the coordinates of the template matching center point;

[0012] The region of interest is determined based on the template matching center point, and line detection is performed within the region of interest to obtain the k lines with the smallest distance from the template matching center point and t line intersection points.

[0013] Determine the geometric center M0 of the intersection of t straight lines, extract the X-corner point of the i-th marker, and solve the PnP problem using the pixel coordinates and world coordinates of the four detected X-corner points to achieve visual localization.

[0014] Establish a coordinate system by: placing at least four X-corner markers with fixed relative positions on the target object, or obtaining their relative positions through modeling after placing the X-corner markers; and establishing a world coordinate system based on the positions of the four X-corner markers.

[0015] Template matching, as shown in the following formula:

[0016]

[0017] Where x'∈[1,m] and y'∈[1,n] are the horizontal and vertical coordinates of the pixel in the template image, respectively, T(x',y') is the gray value of the point (x',y') in the template image, and I(x+x',y+y') is the gray value of the point (x+x',y+y') in the original image.

[0018] After coarse localization using template matching, the coordinates P of four template matching center points were determined in the original image. i (m i ,n i ), i∈[1,4.

[0019] The region of interest is determined based on the template matching center point, and line detection is performed within the region of interest to obtain the k lines with the smallest distance to the template matching center point and t line intersection points; including:

[0020] Point P i Four regions of interest (ROIs) are identified centered on the central point. i By setting a mask, the original image is separated from the ROI. i Set the pixels in the outer region to 0 to filter out the ROI.i Corner points outside the area;

[0021] In ROI i Line detection is performed within the range, and the i-th detected line is denoted as L. i L i The coordinates of the two pixels passed through and

[0022] Point P i to line L i The distance is:

[0023] Arrange the points and lines in ascending order of distance to obtain the distance to point P. i The k lines with the smallest distance are denoted as L1-L. k ;

[0024] according to Find the t intersection points of the lines that are closest to point P, denoted as .

[0025] Take M1-M t The geometric center M0 is the point of interest. Within a certain pixel range around M0, the X-corner point of the i-th marker is extracted using the cornerSubPix function, denoted as S. i .

[0026] A second aspect of the present invention provides a system for implementing the above-described method, comprising:

[0027] The coordinate system construction module is configured to: establish a coordinate system based on at least four X-corner markers pre-arranged on the target object;

[0028] The template matching module is configured to: acquire an image of a target object containing an X-corner marker and perform template matching to determine the coordinates of the template matching center point;

[0029] The line detection module is configured to: determine the region of interest based on the template matching center point, and perform line detection within the region of interest to obtain the k lines with the smallest distance from the template matching center point and t line intersection points;

[0030] The localization module is configured to: determine the geometric center M0 of the intersection of t straight lines, extract the X-corner point of the i-th marker, and solve the PnP problem using the pixel coordinates and world coordinates of the four detected X-corner points to achieve visual localization.

[0031] A third aspect of the present invention provides a computer-readable storage medium.

[0032] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a visual positioning method in a complex environment as described above.

[0033] A fourth aspect of the present invention provides a computer device.

[0034] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a visual positioning method in a complex environment as described above.

[0035] Compared with existing technologies, one or more of the above technical solutions have the following beneficial effects:

[0036] 1. By detecting straight lines, point-to-line distances, and point-to-point distances, the location of the landmark feature points is determined, other irrelevant corner points are filtered out, and sub-pixel level X corner points are extracted through sub-pixel level corner point detection to determine the location of feature points for visual positioning.

[0037] 2. To achieve visual positioning in scenarios with complex lighting and backgrounds, reliable markers are needed for feature point detection and solving the PnP problem. X-corner markers are designed. These markers are insensitive to lighting, have no eccentricity error, are highly versatile, and are suitable for complex environments and scenarios with high detection difficulty.

[0038] 3. For the designed landmark, existing corner detection technologies will detect redundant corners in complex environments and cannot accurately capture only the feature points of the landmark. By designing an algorithm and using template matching technology, the location of the landmark can be roughly located in the target image, and a mask can be set to filter out most of the redundant corners.

[0039] 4. To address the drawback of frequent template matching reducing processing speed, an algorithm is designed to conditionally perform line detection directly near the coordinates of the feature points detected in the previous corner detection, thereby improving both processing speed and algorithm robustness. Attached Figure Description

[0040] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0041] Figure 1 This is a frontal view of a visual positioning process in a complex environment provided by one or more embodiments of the present invention.

[0042] Figure 2 This is a side view schematic diagram of visual positioning in a complex environment provided by one or more embodiments of the present invention;

[0043] Figure 3 This is a schematic diagram of a visual positioning process in a complex environment provided by one or more embodiments of the present invention;

[0044] Figure 4 This is a template diagram of the visual positioning process in a complex environment provided by one or more embodiments of the present invention. Detailed Implementation

[0045] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0046] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0047] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0048] Perspective-n-Point (PnP) is a method for solving the motion of 3D to 2D point pairs, aiming to determine the pose of the camera coordinate system relative to the world coordinate system. It describes how to estimate the camera pose (i.e., solve for the rotation matrix R and translation vector t from the world coordinate system to the camera coordinate system) when the coordinates (relative to the world coordinate system) of n 3D points and their pixel coordinates are known.

[0049] X-corner point: An X-marked point consisting of four adjacent black and white regions. The X-corner point is the common point of the region block (e.g., ...). Figure 4 (As shown).

[0050] As described in the background section, due to the presence of Gaussian noise, salt-and-pepper noise, etc., directly performing corner detection on landmarks in complex environments based on grayscale images, binary images, or contour curves will result in redundant noise corners in the environment, making it impossible to match the feature point pixel coordinates with the world coordinates, and thus impossible to solve the PnP problem to complete the localization.

[0051] Therefore, the following embodiments provide a visual positioning method and system in complex environments. By designing an algorithm, the X-corner point of an X-marker can be accurately detected in complex environment images. This is the key to achieving visual positioning in complex environments by leveraging the advantages of X-corner point being insensitive to illumination, free from eccentricity error, and highly versatile.

[0052] Example 1:

[0053] like Figure 1-4 As shown, a visual localization method in a complex environment includes the following steps:

[0054] Establish a coordinate system based on at least four X-corner markers pre-arranged on the target object;

[0055] Acquire an image of the target object containing the X-corner marker and perform template matching to determine the coordinates of the template matching center point;

[0056] The region of interest is determined based on the template matching center point, and line detection is performed within the region of interest to obtain the k lines with the smallest distance from the template matching center point and t line intersection points.

[0057] Determine the geometric center M0 of the intersection of t straight lines, extract the X-corner point of the i-th marker, and solve the PnP problem using the pixel coordinates and world coordinates of the four detected X-corner points to achieve visual localization.

[0058] Specifically:

[0059] 1. Place four X-corner markers with fixed relative positions on the target object, or place these markers and obtain their relative positions through 3D modeling. Based on the world positions of the four X-corner markers, establish a world coordinate system. The world coordinates of the X-corner points are: Where i∈[1,4] represents the i-th X-corner point.

[0060] 2. A template is created by cropping the X-marker from the first frame image, or by pre-creating X-marker images of different sizes. The template size is m×n. Template matching is performed in the original first frame image I. Here, a relevant matching method is selected:

[0061]

[0062] Where x'∈[1,m] and y'∈[1,n] are the horizontal and vertical coordinates of the pixel in the template image, respectively, T(x',y') is the gray value of the point (x',y') in the template image, and I(x+x',y+y') is the gray value of the point (x+x',y+y') in the original image.

[0063] 3. After coarse localization using template matching, the coordinates P of the four template matching center points in the original image are determined. i (m i ,n i ), i∈[1,4.

[0064] 4. Point P i Using w and h as the center point, define four regions of interest (ROIs). i Its top left and bottom right pixel coordinates are respectively By setting a mask, the original image is separated from the ROI. i Setting the outer region pixels to 0 can filter out the ROI. i Corner points outside the area.

[0065] 5. In ROI i Line detection is performed within the range, and the i-th detected line is denoted as L. i L can be obtained i The coordinates of the two pixels passed through and according to:

[0066] ① Two non-coincident points have three-dimensional homogeneous forms M(x1,y1,1) and N(x2,y2,1), M×N=(y1-y2,x2-x1,x1y2-y1x2).

[0067] ② The straight line passing through points M and N can be expressed as y1 = kx1 + b or y2 = kx2 + b, which simplifies to...

[0068] ③ Transform the line y = kx + b into the form ax + by + c = 0, resulting in (a, b, c) = (-k, 1, -b), which simplifies to (y1 - y2, x2 - x1, x1y2 - y1x2). i It can be represented as:

[0069]

[0070] 6 points P i to line L i The distance can be expressed as:

[0071]

[0072] Arrange the points and lines in ascending order of distance to obtain the distance to point P. i The k lines with the smallest distance are denoted as L1-L. k .

[0073] 7. Based on:

[0074] ① Two straight lines F1(x): a1x+b1y+c1=0, F2(x): a2x+b2y+c2=0, take three-dimensional points XM(a1,b1,c1) and N(a2,b2,c2).

[0075] ② The intersection of the two lines should satisfy a1x + b1y + c1 = a2x + b2y + c2, which gives us...

[0076] ③M×N=(b1c2-b2c1,a2c1-a1c2,a1b2-a2b1) is denoted as (A,B,C), and the intersection point of the two lines in the two-dimensional plane is... homogeneous form is

[0077] Through the straight line L1-L k The expression yields the k lines. The intersection points are denoted as _____. according to We can find the t intersection points of the lines that are closest to point P, denoted as .

[0078] 8 take M1-M t Taking the geometric center M0 as the point of interest, within a certain pixel range around M0, the X-corner point of the i-th marker is accurately extracted using the cornerSubPix function of the OpenCV open-source library, denoted as S. i .

[0079] 9. Through the 4 detected X corner points S i The pixel coordinates, world coordinates, and camera intrinsic and extrinsic parameters are used to solve the PnP problem and achieve visual localization.

[0080] 10. For the second frame and subsequent frames, if the template matching condition is met, the current frame P is determined through template matching. i Otherwise, template matching is not performed, and the previous frame image S is used. i As the current frame P i This step improves speed while maintaining robustness. Repeat steps 4-9 to achieve real-time visual positioning.

[0081] 11. Matching conditions are: ① During the first frame detection. ② After the first frame detection, if fewer or more than 4 S's are detected. i ③ If the number of lines detected in step 6 is less than 2, the intersection point of the lines cannot be obtained. ④ For each v pose calculation completed, v can be determined based on the target's movement speed within the camera's field of view.

[0082] The above process requires reliable markers for feature point detection and PnP problem solving in scenarios with complex lighting and backgrounds for visual positioning. An X-corner marker is designed that is insensitive to lighting, has no eccentricity error, is highly versatile, and is suitable for scenarios with complex environments and high detection difficulty.

[0083] For the designed landmark, existing corner detection technologies will detect redundant corners in complex environments, but cannot accurately capture only the feature points of the landmark. By designing an algorithm and using template matching technology, the location of the landmark can be roughly located in the target image, and a mask can be set to filter out most of the redundant corners.

[0084] By detecting straight lines, point-to-line distances, and point-to-point distances, the positions of the landmark feature points are further determined, other irrelevant corner points are filtered out, and sub-pixel level X corner points are extracted through sub-pixel level corner point detection to determine the position of the feature points for visual localization.

[0085] To address the drawback of frequent template matching reducing processing speed, this invention designs an algorithm that conditionally performs line detection directly near the coordinates of the feature points detected in the previous corner detection, thereby improving both processing speed and algorithm robustness.

[0086] Example 2:

[0087] A system for implementing the above method includes:

[0088] The coordinate system construction module is configured to: establish a coordinate system based on at least four X-corner markers pre-arranged on the target object;

[0089] The template matching module is configured to: acquire an image of a target object containing an X-corner marker and perform template matching to determine the coordinates of the template matching center point;

[0090] The line detection module is configured to: determine the region of interest based on the template matching center point, and perform line detection within the region of interest to obtain the k lines with the smallest distance from the template matching center point and t line intersection points;

[0091] The localization module is configured to: determine the geometric center M0 of the intersection of t straight lines, extract the X-corner point of the i-th marker, and solve the PnP problem using the pixel coordinates and world coordinates of the four detected X-corner points to achieve visual localization.

[0092] By detecting straight lines, point-to-line distances, and point-to-point distances, the location of landmark feature points is determined, and other irrelevant corner points are filtered out. Subpixel-level corner point detection is used to extract subpixel-level X-corner points, and the feature point locations are then determined for visual localization. This approach enables visual localization in scenes with complex lighting and backgrounds, improving both processing speed and algorithm robustness.

[0093] Example 3:

[0094] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a visual positioning method in a complex environment as described in Embodiment 1 above.

[0095] By detecting straight lines, point-to-line distances, and point-to-point distances, the location of landmark feature points is determined, and other irrelevant corner points are filtered out. Subpixel-level corner point detection is used to extract subpixel-level X-corner points, and the feature point locations are then determined for visual localization. This approach enables visual localization in scenes with complex lighting and backgrounds, improving both processing speed and algorithm robustness.

[0096] Example 4:

[0097] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the visual positioning method in a complex environment as described in Embodiment 1 above.

[0098] By detecting straight lines, point-to-line distances, and point-to-point distances, the location of landmark feature points is determined, and other irrelevant corner points are filtered out. Subpixel-level corner point detection is used to extract subpixel-level X-corner points, and the feature point locations are then determined for visual localization. This approach enables visual localization in scenes with complex lighting and backgrounds, improving both processing speed and algorithm robustness.

[0099] The steps or modules involved in Embodiments 2 to 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0100] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A visual positioning method in complex environments, characterized in that: The method comprises the following steps: establishing a coordinate system according to at least four X corner point markers arranged in advance on the target object to be measured; acquiring an image of the target object containing the X corner point markers and performing template matching to determine the template matching center point coordinates; According to the template matching center point, a region of interest is determined, and straight line detection is performed in the region of interest to obtain a straight line with the smallest distance from the template matching center point and a straight line intersection point ​ determining geometric center of intersection of the straight lines , extract the x corner points of the first , solve the PnP problem through the pixel coordinates and world coordinates of the four detected x corner points to realize visual positioning.

2. The visual positioning method in a complex environment according to claim 1, wherein: establishing a coordinate system, specifically: arranging at least four X corner point markers with fixed relative positions on the target object to be measured, or obtaining the relative positions of the X corner point markers after arranging the X corner point markers through modeling; and establishing a world coordinate system according to the positions of the four X corner point markers.

3. The visual positioning method in a complex environment of claim 1, wherein: The template matching is as follows: ； wherein respectively the horizontal and vertical coordinates of a pixel in the template image, the grey value of a pixel in the template image the grey value of a pixel in the template image the grey value of a pixel in the original image the grey value of a pixel in the original image 4. The visual positioning method in a complex environment of claim 1, wherein: After rough positioning by template matching, 4 template matching center point coordinates are determined in the original image .

5. The visual positioning method in a complex environment of claim 1, wherein: According to the template matching center point, a region of interest is determined, and straight line detection is performed in the region of interest to obtain a straight line with the smallest distance from the template matching center point and a straight line intersection point; comprising: Point Determine 4 regions of interest for the center point Divide the original image by setting a mask to 0 for pixels outside Filter out corners outside the region regions In straight line detection is performed in the image, and the straight line detected in the step of the first aspect is a straight line , passing through two pixel point coordinates and .

6. The visual positioning method in a complex environment according to claim 4, characterized in that: According to the template matching center point, a region of interest is determined, and straight line detection is performed in the region of interest to obtain a straight line with the smallest distance from the template matching center point and a straight line intersection point . Further comprising: Point to a straight line is: , according to the point line distance from small to large, get with the point distance minimum straight line, recorded as ; according to , obtain and point Minimum distance The intersection of the three lines is denoted as . .

7. The visual positioning method in a complex environment according to claim 6, characterized in that: Take the geometric center of the point of interest, in a certain pixel range around it, use the cornerSubPix function to extract the X corner point of the first marker, recorded as .

8. A visual positioning system in complex environments, characterized by: comprising: a coordinate system construction module configured to establish a coordinate system according to at least four X corner point markers arranged in advance on the target object to be measured; a template matching module configured to acquire an image of the target object containing the X corner point markers and perform template matching to determine the template matching center point coordinates; The straight line detection module is configured to determine a region of interest according to the template matching center point, and perform straight line detection in the region of interest to obtain a straight line with a minimum distance from the template matching center point and a straight line intersection point . The positioning module is configured to determine a geometric center of the intersection of the two straight lines , extract the X corner points of the first marker, and solve a PnP problem through pixel coordinates and world coordinates of the four detected X corner points to realize visual positioning. 9.A computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the steps of the visual positioning method in a complex environment according to any one of claims 1-7. 10.A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the visual positioning method in a complex environment according to any one of claims 1-7 when executing the program.

Images