Spatial circular ring detection and pose estimation method based on monocular vision

Through the spatial ring detection and position estimation method based on monocular vision, using monocular camera and iterative optimization technology, the problems of traditional methods in environmental robustness and position estimation accuracy are solved, and high-precision circles are realized Detection and position estimation are suitable for autonomous intelligence of drones.

CN118918176AActive Publication Date: 2024-11-08BEIJING INST OF TECH

Patent Information

Application Number
CN202410901757.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-05
Publication Date
2024-11-08
Estimated Expiration
2044-07-05

AI Technical Summary

Technical Problem

The traditional ring detection method is poorly robust under environmental factors such as lighting and occlusion, and there are large systematic errors in pose estimation based on the monocular positioning model, making it difficult to achieve accurate spatial ring detection and pose estimation.

Method used

Using spatial ring detection and position estimation methods based on monocular vision, images are obtained through monocular cameras, the minimum external rectangle is calculated, and the initial position solution of the monocular positioning model is performed. Through iterative optimization, the least squares optimization problem and Gaussian are used. Newton's method solves the reprojection error, optimizes the pose of the ring, and takes into account the elliptical characteristics of the ring for detection and pose estimation.

Benefits of technology

It realizes robust spatial ring recognition and detection and accurate positioning estimation, improves positioning accuracy, reduces system errors, and is suitable for autonomous intelligence of drone systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918176A_ABST
    Figure CN118918176A_ABST
Patent Text Reader

Abstract

The invention discloses a space circular ring detection and pose estimation method based on monocular vision. According to the method, only the monocular vision image is adopted as system input, the three-dimensional position and the spatial attitude of the spatial circular ring are optimized at the same time through the method of carrying out nonlinear optimization on the circular ring attitude, and the accurate circular ring attitude can be output. Compared with a conventional binocular or radar target positioning algorithm, the method is simple in input information, facilitates the testing and deployment of the algorithm, and is more convenient in overall use. Meanwhile, a set of ellipse detection-based method is constructed by considering the characteristic that most of images imaged by the space circular ring in the camera are elliptical graphs, the ellipse mapped by the space circular ring in the image can be extracted, and the robust pose estimation of the space circular ring is realized. The method does not need to label data and training time, saves computing resources, and is simple in input and robust and accurate in output.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of robot perception and computer vision technology, and in particular to a method for detecting and estimating a space ring based on monocular vision. Background Art

[0002] In autonomous drone competitions at home and abroad, ring drilling is a very spectacular and challenging subject. The drone needs to rely only on its own sensors and onboard computing equipment to pass through a series of suspended rings at the fastest speed and with the highest accuracy within the specified time. In autonomous drone competitions, in order to achieve the need for drones to successfully pass through the rings, its basic technology is to be able to use its own sensors to accurately perceive the space rings. Due to the small size and load capacity of drones, the sensors they carry are generally image sensors. Therefore, the images obtained by the drone are used to estimate the position and posture of the rings through some image processing algorithms. This is the basic work of drones passing through the rings in autonomous drone competitions, which directly determines the autonomous capabilities of the drones.

[0003] Traditional methods for detecting circular rings usually use methods based on template matching or Hough transform. The template matching method compares the pre-prepared circular ring template image with the image to be identified to find the most similar area to achieve the recognition and positioning of the target circular ring, but its working environment requirements are high and its robustness to environmental factors such as lighting and occlusion is poor. The Hough transform-based method uses Hough transform to detect the circular contour of the circular ring. It extracts the circular features in the image by extracting the contour line in the image to achieve the detection of the circular ring. However, in actual work, the projection of the circular ring in the image is most likely not a standard circle, and the accuracy of this method will be seriously reduced.

[0004] Traditionally, the spatial pose estimation of a circular ring usually adopts the traditional monocular positioning model. Based on the principle of similar triangles, the traditional monocular positioning model believes that for a lens with a fixed focal length, when the target projection size remains unchanged, the target depth is proportional to the target size. Therefore, the circular ring detection frame size extracted from the detection algorithm is used as the circular ring projection size and substituted into the monocular positioning model to solve the target position information. However, in reality, the circular ring has a certain degree of angular deflection, so directly using the detection frame size for pose estimation will bring about a large system error.

[0005] Therefore, designing a robust spatial circle perception and detection algorithm and realizing accurate pose estimation of the spatial circle according to the circle detection results in the image, and realizing spatial circle detection and pose estimation based only on monocular vision are crucial for further autonomous and intelligent UAV systems and for promoting the development of autonomous UAV competitions. Summary of the invention

[0006] In view of this, the present invention provides a spatial ring detection and pose estimation method based on monocular vision, which can utilize simple monocular vision image input to achieve robust spatial ring recognition detection and accurate pose estimation output.

[0007] The monocular vision-based spatial ring detection and pose estimation method of the present invention comprises:

[0008] S1, detect the spatial ring in the image acquired by the monocular camera and calculate its minimum circumscribed rectangle;

[0009] S2, uses a monocular positioning model to perform a preliminary position solution for the spatial ring detected by S1;

[0010] S3, iteratively optimize the preliminary position of the space ring obtained in S2, specifically:

[0011] S31, taking the center of the space ring as the origin, establish the local coordinate system of the object; establish the space ring three-dimensional point model according to the known space ring size a priori, and take a space three-dimensional point at regular intervals on the edges of the inner and outer rings of the space ring, denoted as {P i |1≤i≤N}, N is the total number of sampling points;

[0012] S32, using the spatial projection relationship, based on the preliminary position t of the spatial ring obtained in S2, the spatial three-dimensional point is projected onto the image to obtain a projected two-dimensional pixel point p i ;

[0013] S33, construct the least squares optimization problem:

[0014] By finding the nearest observed pixel of each projected two-dimensional pixel point in the aerial ring contour detected by S1, the pixel error between each pair of points, i.e., the reprojection error, is established; let each projection point p i The closest point matched on the contour of the detected air ring is u i , then the objective function is:

[0015]

[0016] S34, by solving the least squares optimization problem constructed in S33, the optimal solution of the spatial ring posture T is obtained.

[0017] Preferably, in S1, the detection of the space ring in the image is completed based on the ellipse template, specifically:

[0018] First, extract the straight line segments in the image;

[0019] Next, the extracted straight line segments are used as arc support segments to construct arc support groups. Specifically, if any two connected arc support segments have similar geometric properties at the head and tail connection points, they are connected together, and multiple connected arc support segments together constitute an arc support group.

[0020] Then, the internal span angle of the arc support group is calculated, and the arc support group whose internal span angle exceeds the set threshold is taken as the valid arc support group;

[0021] Finally, sampling is performed on each valid arc support group, and the sampling points are fitted to an elliptic curve by the least squares method to obtain a preliminary ellipse parametric equation; mean shift clustering is performed on all ellipse parametric equations to obtain the final ellipse parametric equation, and the ellipse with the most significant integrity and geometric properties is selected as the detected spatial ring output.

[0022] Preferably, the LSD algorithm, Hough line detection, EDLines line detection and other algorithms are used to extract straight line segments in the image.

[0023] Preferably, the LSD algorithm is used to extract straight line segments in the image, specifically:

[0024] First, the image is Gaussian filtered to remove noise in the image; then, the gradient strength and direction of each pixel in the image are calculated; then, non-maximum suppression is used to remove pixels with weak gradients, and only the local maximum gradient strength is retained; then, according to the gradient direction, the edge points are voted to the subdivision units in the five-dimensional Hough space, and the maximum points are found in the Hough space to determine the description of the line segment; finally, based on the description, short line segments (within 5 pixels in length) and overlapping line segments are eliminated to obtain the final straight line segment detection result.

[0025] Preferably, the Sobel operator, Canny operator, Laplace operator, etc. are used to calculate the gradient strength and direction of each pixel in the image.

[0026] Preferably, in S1, after completing the detection of the spatial rings in the image, the spatial rings with too small an area (within 100 pixels) are filtered out.

[0027] Preferably, in S1, the RGB image acquired by the monocular camera is first preprocessed:

[0028] Convert the RGB image acquired by the monocular camera to the HSV color space; set a threshold based on the color of the space ring; in the HSV color space, extract the image pixel color based on the threshold to obtain a binary grayscale image that only retains the space ring; if the space ring has multiple colors, set multiple thresholds accordingly, extract the image pixel colors respectively and then merge them to obtain a binary grayscale image that only retains the space ring;

[0029] Perform spatial ring detection on the binary grayscale image obtained by preprocessing.

[0030] Preferably, in S1, before performing spatial ring detection, median filtering is performed on the image to remove noise in the image.

[0031] Preferably, in S2, the initial position of the space ring is t = [X, Y, Z] T ,in,

[0032]

[0033]

[0034]

[0035] Among them, X, Y, and Z are the three-dimensional positions of the center of the space ring in the camera coordinate system; H is the actual height prior value of the space ring; x o ,y o is the center pixel coordinate of the minimum bounding rectangle, c x 、c y 、f y is the internal parameter of the monocular camera, and h is the pixel height of the minimum circumscribed rectangle.

[0036] Better,

[0037]

[0038] Among them, K is the camera intrinsic parameter matrix obtained by camera calibration of the monocular camera, Z is the depth value of the three-dimensional point in space, R and t are the rotation matrix and translation vector from the world coordinate system to the ring coordinate system respectively;

[0039] The Gauss-Newton method is used to solve the least squares optimization problem constructed by S33, where the Jacobian matrix of the Lie algebra of the reprojection error relative to the camera pose is:

[0040]

[0041] Through the constructed Jacobian matrix, the three-dimensional points in the space are traversed in turn, and the posture optimization direction ΔT is obtained according to the Gauss-Newton optimization method:

[0042] ΔT=(J T J) -1 (-J T e)

[0043] Where e is the reprojection error term. The action of ΔT is accumulated, and the pose matrix T of the ring is continuously optimized and updated iteratively to finally optimize the optimal ring pose.

[0044] Beneficial effects:

[0045] (1) The present invention uses only monocular vision images as sensor input to achieve spatial ring detection and pose estimation. The input information is simple, which is convenient for algorithm testing and deployment. Compared with traditional binocular or radar target positioning algorithms, the present invention starts from the imaging process of the target in the image and performs nonlinear optimization based on spatial projection, which is more convenient to use overall.

[0046] (2) The method of the present invention performs nonlinear optimization positioning on the circular ring posture, and optimizes the three-dimensional position and spatial posture of the spatial circular ring at the same time. Compared with the traditional monocular positioning model, it is more reasonable and has higher positioning accuracy. Compared with other optimization-based positioning methods, it also optimizes the target posture and can estimate the circular ring posture more accurately.

[0047] (3) The present invention adopts a nonlinear optimization method based on the Gauss-Newton method, constructs the Jacobian matrix of the Lie algebra of the reprojection error relative to the camera posture for the Gauss-Newton method solution, and the error gradient direction is more accurate; a rough solution of the ring posture based on the monocular vision model is adopted, and the rough value is used as the initial value of the optimization for iteration. The algorithm is optimized in two steps from coarse to fine, and the solution is faster and the optimization convergence result is more accurate.

[0048] (4) Taking into account the fact that the two-dimensional projection of a ring is mostly an ellipse, the present invention detects the ellipse information in the image to extract the spatial ring information in the image and obtain a more accurate minimum circumscribed rectangle of the spatial ring. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a framework for circle detection and position estimation.

[0050] Figure 2 This is the output effect diagram of the algorithm of the present invention.

[0051] Figure 3 This is an intermediate process diagram of the algorithm of the present invention. DETAILED DESCRIPTION

[0052] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0053] The present invention provides a method for detecting and estimating a space ring based on monocular vision.

[0054] The present invention is mainly designed based on the scheme of shape detection, but abandons the traditional scheme of identifying circles. The present invention takes into account the characteristics that most of the images of space rings in the camera are elliptical figures, and constructs a set of ring recognition methods based on ellipse detection, which can extract ellipses in the image to correspond to the spatial ring mapping, and introduces an ellipse pose estimation module with back-end optimization to achieve robust pose estimation of space rings. The scheme does not require labeled data and training time, and has a certain robustness, and can stably achieve ring detection and pose estimation.

[0055] The process of the present invention is as follows Figure 1 As shown, the specific steps include:

[0056] Step 1: First, preprocess the RGB image acquired by the monocular camera to extract the color features of the ring and simplify the subsequent steps.

[0057] The space ring generally has obvious color features, so HSV color space threshold segmentation can be constructed to extract the image pixel color. Specifically, the space ring is generally bright and obvious red and yellow, so the image pixel color is effectively extracted in the HSV space, and two mask images after threshold extraction are obtained. Then, the merged image after segmentation is obtained by image superposition, and a binary grayscale image that only retains the space ring is obtained.

[0058] Since color extraction inevitably produces a small number of noise points, a median filter is performed on the visual image after color extraction to remove salt and pepper noise in the image, so that the circular feature information in the output image is more obvious and the noise is smaller. This embodiment uses a 3*3 median filter.

[0059] Step 2: Extract the spatial ring based on the preprocessed RGB image, and obtain the minimum circumscribed rectangle of the spatial ring in the two-dimensional image through template matching and other methods.

[0060] The present invention considers that the general imaging effect of a spatial ring in an image is an ellipse, detects the ellipse information in the preprocessed RGB image, extracts the ring parameter information in the image according to the ellipse detection method, and obtains a more accurate minimum circumscribed rectangle of the spatial ring.

[0061] Ellipse detection is as follows:

[0062] S21, extracting straight line segments in the image to determine basic components of the ring to be estimated;

[0063] The straight line segments in the image can be extracted using algorithms such as LSD (Line Segment Detector), Hough line detection, and EDLines line detection.

[0064] This embodiment adopts the LSD algorithm, which is as follows:

[0065] First, the image is Gaussian filtered to remove noise in the image; then, the gradient strength and direction of each pixel in the image are calculated, wherein the Sobel operator, Canny operator, Laplace operator, etc. can be used to calculate the image gradient; then, the non-maximum suppression method is used to remove the pixels with weak gradients, and only the local maximum gradient strength is retained; then, according to the gradient direction, the edge points are voted to the subdivision units in the five-dimensional Hough space, and the maximum points are found in the Hough space to determine the description of the line segment, and based on the description, short line segments and overlapping line segments are eliminated to obtain the final line segment detection result.

[0066] S22, using the detected straight lines as arc support line segments to construct an arc support group to estimate possible ellipses.

[0067] Arc support segments are robustly connected through a certain discrimination method, that is, if any two connected arc support segments have similar geometric properties at the head and tail connection points, they are connected together, and multiple connected arc support segments together constitute an arc support group.

[0068] S23, perform ellipse extraction:

[0069] It is easier to form an ellipse when the span angle within the arc support group is approximately 360 degrees. The span angle score is calculated by voting, and the group with a score exceeding the threshold is regarded as the valid arc support group approximating the corresponding ellipse; finally, sampling is performed on each valid group to obtain a certain number of two-dimensional sampling points, and the points are fitted to an elliptical curve through the least squares method to obtain a preliminary ellipse equation. The ellipse parameter equation is clustered by mean shift among all the generated ellipses to reduce repeated ellipse predictions and obtain a more representative ellipse equation; finally, the integrity and geometric properties of the generated ellipse are considered, and the most significant ellipse parameter results are selected for output.

[0070] Step 3: According to the obtained plane ellipse equation, the initial value of the spatial ring position is calculated according to the monocular vision positioning principle. The size threshold of the ellipse extracted in step 2 is checked to filter out ellipses with too small an area. The minimum circumscribed rectangle is calculated according to the obtained ellipse equation, and the boundary of the spatial ring is obtained, and the pixel size and coordinates of the spatial ring in the image projection can be obtained.

[0071] According to the pixel coordinates of the ring and its pre-defined spatial geometric constraints, the position can be solved through a simple monocular positioning model, which can be expressed as:

[0072]

[0073]

[0074]

[0075] Among them, X, Y, and Z represent the three-dimensional position of the center of the space ring in the camera coordinate system, H is the actual prior value of the outer diameter of the space ring, and x o ,y o is the pixel coordinate of the center of the ellipse, c x 、c y 、f y is the internal parameter of the monocular camera, and h is the pixel height of the minimum circumscribed rectangle of the ellipse. The initial positioning result of the target relative to the camera is:

[0076] t=[X,Y,Z] T

[0077] Step 4: According to the previous step, the initial value of the three-dimensional coordinate of the center of the space ring can be obtained. However, since the monocular vision positioning model assumes that the ring is in the same plane for calculation, there is a certain error in the result obtained, especially in long-distance measurement. The error is unacceptable. Therefore, the present invention constructs a state estimation scheme based on back-end optimization, and iteratively optimizes the position and posture of the space ring in a nonlinear optimization manner combined with the image sensor acquisition results.

[0078] First, the local coordinate system of the object is established with the center of the space ring as the origin. The 3D point model of the ring is established based on the known size of the space ring. A 3D point is taken every 10 degrees on the edge of the inner and outer rings of the ring, for a total of 72 3D points. It is recorded as:

[0079] {P i |1≤i≤72}

[0080] Using the spatial projection relationship, according to the known preliminary positioning result T, the spatial point is projected onto the image to obtain the projected two-dimensional pixel point. The process can be expressed as:

[0081]

[0082] Where K is the camera intrinsic parameter matrix obtained by camera calibration of the monocular camera, Z is the depth value of a three-dimensional point in space, R and t are the rotation matrix and translation vector from the world coordinate system to the circular coordinate system, respectively, and the two together constitute the pose T of the circular coordinate system.

[0083] 72 projected pixel points can be obtained on the pixel plane. When the pose estimation is completely accurate, all projected pixel points should be on the two-dimensional elliptical ring contour obtained by direct imaging. Therefore, the least squares problem can be established by using the distance between the two-dimensional elliptical ring contour and the projected pixel points. By finding the nearest observed pixel of each projected pixel in the image contour, the pixel error between each pair of points, i.e., the reprojection error, can be established. Suppose that each projection point p i The closest point matched on the contour in the figure is ui , then the objective function is:

[0084]

[0085] The optimal solution of the ring pose T is the value of T that minimizes L(T). The Gauss-Newton method is used to solve this least squares optimization problem, where the Jacobian matrix of the Lie algebra of the reprojection error relative to the camera pose is:

[0086]

[0087] By constructing the Jacobian matrix, we traverse the 72 groups of points in turn to accumulate the effects on ΔT, and continuously iterate and optimize the ring's posture matrix T to ultimately optimize the optimal ring posture.

[0088] Example: Figure 2 , 3 The figure shows the effect of the implementation example of the algorithm. The red space ring is detected and spatially positioned in the airsim simulator. It mainly shows the original RGB monocular image, image preprocessing, LSD line detection, line segment arc search, preliminary ellipse fitting, mean shift filtering, comparison of the space ring point cloud projection effect before and after back-end optimization, and the real-time recognition and detection effect of the ring. The final result shows that this method has high positioning accuracy and can achieve accurate positioning of the space ring based only on monocular vision.

[0089] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting and estimating a spatial ring based on monocular vision, characterized in that: include: S1, detect the spatial ring in the image acquired by the monocular camera and calculate its minimum circumscribed rectangle; S2, uses a monocular positioning model to perform a preliminary position solution for the spatial ring detected by S1; S3, iteratively optimize the preliminary position of the space ring obtained in S2, specifically: S31, taking the center of the space ring as the origin, establish the local coordinate system of the object; establish the space ring three-dimensional point model according to the known space ring size a priori, and take a space three-dimensional point at regular intervals on the edges of the inner and outer rings of the space ring, denoted as {P i |1≤i≤N}, N is the total number of sampling points; S32, using the spatial projection relationship, based on the preliminary position t of the spatial ring obtained in S2, the spatial three-dimensional point is projected onto the image to obtain a projected two-dimensional pixel point p i ; S33, construct the least squares optimization problem: By finding the nearest observed pixel of each projected two-dimensional pixel point in the aerial ring contour detected by S1, the pixel error between each pair of points, i.e., the reprojection error, is established; let each projection point p i The closest point matched on the contour of the detected air ring is u i , then the objective function is: S34, by solving the least squares optimization problem constructed in S33, the optimal solution of the spatial ring posture T is obtained.

2. The method according to claim 1, characterized in that In S1, the detection of the space ring in the image is completed based on the ellipse template, specifically: First, extract the straight line segments in the image; Next, the extracted straight line segments are used as arc support segments to construct arc support groups. Specifically, if any two connected arc support segments have similar geometric properties at the head and tail connection points, they are connected together, and multiple connected arc support segments together constitute an arc support group. Then, the internal span angle of the arc support group is calculated, and the arc support group whose internal span angle exceeds the set threshold is taken as the valid arc support group; Finally, sampling is performed on each valid arc support group, and the sampling points are fitted to an elliptic curve by the least squares method to obtain a preliminary elliptic parameter equation; All ellipse parametric equations are clustered by mean shift to obtain the final ellipse parametric equation, and the ellipse with the most significant integrity and geometric properties is selected as the detected spatial ring output.

3. The method according to claim 2, characterized in that Use LSD algorithm, Hough line detection or EDLines line detection to extract straight line segments in the image.

4. The method according to claim 3, characterized in that The LSD algorithm is used to extract straight line segments in the image, specifically: First, the image is Gaussian filtered to remove noise in the image. Then, the gradient strength and direction of each pixel in the image are calculated. Next, non-maximum suppression is used to remove pixels with weak gradients, and only the local maximum value of the gradient strength is retained. Then, according to the gradient direction, the edge points are voted to the subdivision units in the five-dimensional Hough space, and the maximum points are found in the Hough space to determine the description of the line segment; finally, based on the description, short line segments and overlapping line segments are eliminated to obtain the final straight line segment detection result.

5. The method according to claim 4, characterized in that The Sobel operator, Canny operator or Laplace operator is used to calculate the gradient strength and direction of each pixel in the image.

6. The method according to claim 2, characterized in that In S1, after completing the detection of the spatial rings in the image, the spatial rings with too small an area are filtered out.

7. The method according to any one of claims 1 to 6, characterized in that: In S1, the RGB image acquired by the monocular camera is first preprocessed: Convert the RGB image acquired by the monocular camera to the HSV color space; set a threshold based on the color of the space ring; in the HSV color space, extract the image pixel color based on the threshold to obtain a binary grayscale image that only retains the space ring; if the space ring has multiple colors, set multiple thresholds accordingly, extract the image pixel colors respectively and then merge them to obtain a binary grayscale image that only retains the space ring; Perform spatial ring detection on the binary grayscale image obtained by preprocessing.

8. The method according to claim 7, characterized in that In S1, before performing spatial ring detection, median filtering is performed on the image to remove noise in the image.

9. The method according to claim 1, characterized in that In S2, the initial position of the space ring is t = [X, Y, Z] T ,in, Among them, X, Y, and Z are the three-dimensional positions of the center of the space ring in the camera coordinate system; H is the actual height prior value of the space ring; x o ,y o is the center pixel coordinate of the minimum bounding rectangle, c x 、c y 、f y is the internal parameter of the monocular camera, and h is the pixel height of the minimum circumscribed rectangle.

10. The method according to claim 9, characterized in that Among them, K is the camera intrinsic parameter matrix obtained by camera calibration of the monocular camera, Z is the depth value of the three-dimensional point in space, R and t are the rotation matrix and translation vector from the world coordinate system to the ring coordinate system respectively; The Gauss-Newton method is used to solve the least squares optimization problem constructed by S33, where the Jacobian matrix of the Lie algebra of the reprojection error relative to the camera pose is: Through the constructed Jacobian matrix, the three-dimensional points in the space are traversed in turn, and the posture optimization direction ΔT is obtained according to the Gauss-Newton optimization method: AT=(J T J) -1 (-J T e) Where e is the reprojection error term. The action of ΔT is accumulated, and the pose matrix T of the ring is continuously optimized and updated iteratively to finally optimize the optimal ring pose.

Citation Information

Patent Citations

  • A circle identification method, apparatus and electronic device

    CN107578438A

  • Monocular vision pose measuring and adjusting method and system

    CN113516716A

  • Spatial positioning method for multiple circular targets based on binocular vision

    CN118052883A

Cited By

  • Attitude estimation method for tea buds and leaves in tea garden environment

    CN120635708A