A monocular vision-based space ring detection and pose estimation method

Abstract

The application discloses a kind of space annular detection and pose estimation method based on monocular vision.The application only uses monocular vision image as system input, through the method of nonlinear optimization of annular pose, the three-dimensional position of space annular and space attitude are optimized simultaneously, and accurate annular pose can be output.Compared with traditional binocular or radar target positioning algorithm, input information is simple, convenient for algorithm test and deployment, overall use is more convenient.At the same time, considering the characteristics that most of the space annular is imaged as an elliptical pattern in the camera, a set of ellipse detection based method is constructed, which can extract the ellipse mapped by the space annular in the image, and realize the robust pose estimation of the space annular.The application does not need to label data and training time, saves computing resources, and the input is simple, the output is robust and accurate.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robot perception and computer vision, and in particular to a space ring detection and pose estimation method based on monocular vision. BACKGROUND

[0002] In domestic and foreign autonomous unmanned aircraft competitions, drilling rings is a very attractive and challenging subject. The unmanned aerial vehicle needs to rely only on its own sensors and on-board computing devices to achieve the fastest speed and highest accuracy in crossing a series of suspended rings within a specified time. In autonomous unmanned aircraft competitions, in order to achieve the need for unmanned aerial vehicles to successfully cross the rings, the basic technology is to be able to use its own sensors to achieve accurate perception of space rings. Unmanned aerial vehicles are usually small in size and weight, and the sensors carried are generally image sensors. Therefore, through some image processing algorithms, the pose of the ring is estimated based on the images obtained by the unmanned aerial vehicle, which is the basic work for the unmanned aerial vehicle to cross the ring in the autonomous unmanned aircraft competition, and directly determines the autonomous ability of the unmanned aerial vehicle.

[0003] Traditional ring detection methods usually use template matching-based methods or Hough transform-based methods. The template matching-based method compares the pre-prepared ring template image with the image to be recognized, so as to find the most similar area, so as to realize the recognition and positioning of the target ring, but its working environment requirement is high, and its robustness to environmental factors such as light and shielding is poor. The Hough transform-based method uses Hough transform to detect the circular contour of the ring, which extracts the circular feature in the image by extracting the contour line in the image, and realizes the detection of the ring, but in actual work, the projection of the ring in the image is most likely not a standard circle, at which time the accuracy of this method will be seriously reduced.

[0004] Traditional ring space pose estimation usually uses a traditional monocular positioning model. According to the principle of similar triangles, the traditional monocular positioning model considers that for a lens with a fixed focal length, when the target projection size is constant, the target depth is proportional to the target size. Therefore, the size of the ring detection box extracted by the detection algorithm is used as the projection size of the ring to solve the target position information by the monocular positioning model. However, due to the fact that the ring has a certain degree of angular deflection in reality, directly using the detection box size for pose estimation will bring a large system error.

[0005] Therefore, it is of great importance to design a robust space ring perception detection algorithm, to achieve accurate pose estimation of space rings based on the ring detection results in the image, to realize space ring detection and pose estimation based only on monocular vision, to further the autonomous and intelligent unmanned aerial vehicle system, and to promote the development of autonomous unmanned aircraft competitions. SUMMARY

[0006] Therefore, the application provides a monocular vision-based space ring detection and pose estimation method.

[0007] The monocular vision-based space ring detection and pose estimation method of the application comprises:

[0008] S1, detecting a space ring in an image obtained by a monocular camera and calculating a minimum circumscribed rectangle thereof;

[0009] S2, solving a preliminary position of the space ring detected in S1 by using a monocular positioning model;

[0010] S3, iteratively optimizing the preliminary position of the space ring obtained in S2, specifically:

[0011] S31, establishing a local coordinate system of an object with the center of the space ring as the origin; establishing a three-dimensional point model of the space ring according to a known size of the space ring; taking a space three-dimensional point on the edge of the inner ring and the outer ring of the space ring at a certain interval, respectively, and recording as {P i |1≤i≤N} and N is the total number of sampling points;

[0012] S32, projecting the space three-dimensional points to an image based on the preliminary position t of the space ring obtained in S2 by using a space projection relationship to obtain projected two-dimensional pixel points p i ;

[0013] S33, constructing a least squares optimization problem:

[0014] By finding the nearest observation pixel of each projected two-dimensional pixel point in the contour of the space ring detected in S1, a pixel error, i.e., a re-projection error, between each pair of points is established; let each projected point p i The nearest point matched on the detected contour of the space ring is u i , and the objective function is:

[0015]

[0016] S34, obtaining an optimal solution of the pose T of the space ring by solving the least squares optimization problem constructed in S33.

[0017] Preferably, in S1, the detection of the space ring in the image is completed based on an ellipse template, specifically:

[0018] First, a straight line segment in the image is extracted;

[0019] Next, the extracted straight line segments are taken as arc support line segments to construct an arc support group, specifically, if any two connected arc support line segments have similar geometric properties at the connection, they are connected together, and multiple connected arc support line segments together constitute an arc support group;

[0020] Then, the internal span angle of the arc support group is calculated, and the arc support group with an internal span angle exceeding a set threshold is taken as an effective arc support group.

[0021] Finally, sampling is performed on each effective arc support group, and the sampling points are fitted to an elliptical curve by least squares to obtain a preliminary elliptical parameter equation; mean shift clustering is performed on all elliptical parameter equations to obtain a final elliptical parameter equation, and the ellipse with the most significant completeness and geometric properties is selected as the output of the detected spatial ring.

[0022] Preferably, the LSD algorithm, Hough line detection, EDLines line detection, or the like is used to extract the straight line segments in the image.

[0023] Preferably, the LSD algorithm is used to extract the straight line segments in the image, specifically as follows:

[0024] First, the image is subjected to Gaussian filtering to remove noise in the image; then, the gradient intensity and direction of each pixel point in the image are calculated; next, non-maximum suppression is used to remove pixel points with weak gradient intensity, and only the locally maximum value of the gradient intensity is retained; then, the edge points are voted to the subdivided unit in the five-dimensional Hough space according to the gradient direction, and the maximum value point in the Hough space is found to determine the description of the line segment; finally, short line segments (within 5 pixels in length) and coincident line segments are eliminated based on the description to obtain the final straight line segment detection result.

[0025] Preferably, the Sobel operator, Canny operator, Laplace operator, or the like is used to calculate the gradient intensity and direction of each pixel point in the image.

[0026] Preferably, in S1, after the detection of the spatial ring in the image, spatial rings with a small area (within 100 pixels in area) are filtered out.

[0027] Preferably, in S1, first, the RGB image obtained by the monocular camera is preprocessed as follows:

[0028] The RGB image obtained by the monocular camera is converted to the HSV color space; based on the color of the spatial ring, a threshold value is set; in the HSV color space, the image pixel color is extracted based on the threshold value to obtain a binary grayscale image that retains only the spatial ring; if the spatial ring has multiple colors, multiple threshold values are set correspondingly, and the image pixel color is extracted and combined after being extracted respectively to obtain a binary grayscale image that retains only the spatial ring;

[0029] The binary gray image obtained by preprocessing is subjected to spatial annulus detection.

[0030] Preferably, in S1, the image is subjected to median filtering to remove noise in the image before spatial annulus detection.

[0031] Preferably, in S2, the preliminary position of the spatial annulus is t = [X, Y, Z] T , wherein

[0032]

[0033]

[0034]

[0035] wherein X, Y and Z are three-dimensional positions of the center of the spatial annulus in the camera coordinate system; H is an actual height prior value of the spatial annulus; x o , y o are the center pixel coordinates of the minimum circumscribed rectangle, c x , c y , f y is the intrinsic parameter of the monocular camera, and h is the pixel height of the minimum circumscribed rectangle.

[0036] Preferably,

[0037]

[0038] wherein K is the camera intrinsic matrix of the monocular camera obtained through camera calibration, Z is the depth value of the spatial three-dimensional point, and R and t are respectively the rotation matrix and the translation vector from the world coordinate system to the annulus coordinate system;

[0039] The Gauss-Newton method is used to solve the least square optimization problem constructed in S33, wherein the Jacobian matrix of the re-projection error with respect to the Lie algebra of the camera pose is:

[0040]

[0041] By constructing the Jacobian matrix, the spatial three-dimensional points are sequentially traversed, and the pose optimization direction ΔT is obtained according to the Gauss-Newton optimization method.

[0042] ΔT = (J T J) -1 (-J T e)

[0043] wherein e is the re-projection error term, the effect amount of ΔT is accumulated, the pose matrix T of the annulus is constantly iteratively optimized and updated, and the optimal annulus pose can be finally optimized.

[0044] Advantages:

[0045] (1) The application only uses monocular vision images as sensor inputs to realize the detection and pose estimation of a space ring, the input information is simple, and the algorithm is convenient for testing and deployment. Compared with traditional binocular or radar target positioning algorithms, the application starts from the imaging process of the target in the image, performs nonlinear optimization based on space projection, and is more convenient as a whole.

[0046] (2) The method for nonlinear optimization positioning of the ring pose simultaneously optimizes the three-dimensional position and the space attitude of the space ring, is more reasonable than the traditional monocular positioning model, has higher positioning accuracy, and simultaneously optimizes the target attitude compared with other optimization-based positioning methods, so that the ring attitude can be more accurately estimated.

[0047] (3) The application adopts a nonlinear optimization method based on the Gauss-Newton method, constructs a Jacobian matrix of the Lie algebra of the re-projection error with respect to the camera pose for Gauss-Newton method solving, the error gradient direction is more accurate, and a ring pose coarse solving based on a monocular vision model is adopted, the coarse value is used as an optimization initial value for iteration, the algorithm is optimized in two steps from coarse to fine, the solving is more rapid, and the optimization convergence result is more accurate.

[0048] (4) The application considers that the two-dimensional projection of the ring is mostly an ellipse, detects the ellipse information in the image to extract the space ring information in the image, and obtains a more accurate minimum circumscribed rectangle of the space ring. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 The figure is a ring detection and position estimation framework.

[0050] Figure 2 The figure is an algorithm output effect diagram of the application.

[0051] Figure 3 The figure is an intermediate process diagram of the algorithm of the application. DETAILED DESCRIPTION

[0052] The application will be described in detail below with reference to the accompanying drawings and examples.

[0053] The application provides a space ring detection and pose estimation method based on monocular vision.

[0054] The application is mainly designed based on a shape detection scheme, but discards the traditional circle recognition scheme, considers the feature that most of the space annular rings are imaged as elliptical patterns in the camera, constructs an annular ring recognition method based on elliptical detection, can extract the ellipse in the image to correspond to the space annular ring mapping, and introduces an elliptical pose estimation module with a rear-end optimization, so as to realize the robust pose estimation of the space annular ring. The scheme does not need to be labeled and trained, and has a certain robustness, and can stably realize the annular ring detection and pose estimation.

[0055] The flow of the application is shown in Figure 1 The specific steps include the following steps:

[0056] Step 1: First, the RGB image obtained by the monocular camera is preprocessed, and the color feature of the annular ring is extracted, and the subsequent steps are simplified.

[0057] The space annular ring generally has obvious color features, so the HSV color space threshold segmentation can be constructed to extract the image pixel color. Specifically, the space annular ring is generally bright and obvious red and yellow, so the image pixel color is effectively extracted in the HSV space, two mask images extracted after thresholding are obtained, and then a combined image after segmentation is obtained by image superposition, and a binary grayscale image retaining only the space annular ring is obtained.

[0058] Since color extraction inevitably appears a small part of noise points, the median filtering is performed on the extracted color vision image, the salt and pepper noise in the image is removed, the annular ring feature information in the output image is more obvious, and the noise is smaller. In this embodiment, 3*3 median filtering is adopted.

[0059] Step 2: According to the preprocessed RGB image, the space annular ring is extracted, and the minimum circumscribed rectangle of the space annular ring in the two-dimensional image is obtained by template matching and other methods.

[0060] The application considers that the general imaging effect of the space annular ring in the image is an ellipse, the ellipse information in the preprocessed RGB image is detected, the annular ring parameter information in the image is extracted according to the ellipse detection method, and the minimum circumscribed rectangle of the space annular ring is obtained.

[0061] The ellipse detection is specifically:

[0062] S21, the straight line segment in the image is extracted to determine the basic constituent element in the estimated annular ring;

[0063] The LSD (Line Segment Detector) algorithm, Hough line detection, EDLines line detection and other algorithms can be used to extract the straight line segment in the image.

[0064] In this embodiment, the LSD algorithm is adopted, and the specific process is as follows:

[0065] First, Gaussian filtering is applied to the image to remove noise. Then, the gradient intensity and direction of each pixel in the image are calculated, using operators such as Sobel, Canny, and Laplace. Next, non-maximum suppression is used to remove pixels with weak gradients, retaining only the local maximum gradient values. Then, based on the gradient direction, edge points are voted onto subdivision units in the five-dimensional Hough space. Maximum points are found in the Hough space to determine the description of line segments. Based on the description, short and overlapping line segments are eliminated to obtain the final line segment detection result.

[0066] S22, the detected straight lines are used as arc support segments to construct an arc support group to estimate possible ellipses.

[0067] The arc support segments are robustly connected by a certain discrimination method. That is, if any two connected arc support segments have similar geometric properties at the beginning and end of the connection, they are connected together. Multiple connected arc support segments together form an arc support group.

[0068] S23, perform ellipse extraction:

[0069] When the span angle within an arc support group is approximately 360 degrees, it is easier to form an ellipse. The span angle score is calculated by voting, and the groups with scores exceeding a threshold are taken as valid arc support groups and approximate the corresponding ellipse. Finally, sampling is performed on each valid group to obtain a certain number of two-dimensional sampling points. The points are fitted to an elliptic curve using the least squares method to obtain a preliminary ellipse equation. Mean drift clustering is performed on the ellipse parameter equations among all generated ellipses to reduce duplicate ellipse predictions and obtain more representative ellipse equations. Finally, the completeness and geometric properties of the generated ellipse are considered, and the most significant ellipse parameter result is selected for output.

[0070] Step 3: Based on the obtained planar ellipse equation, calculate the initial position of the spatial annulus according to the principle of monocular vision positioning. Perform size threshold verification on the ellipse extracted in Step 2, filtering out ellipses with excessively small areas. Calculate its minimum bounding rectangle based on the obtained ellipse equation to obtain the boundary of the spatial annulus, thus obtaining the pixel size and coordinates of the spatial annulus in the image projection.

[0071] Based on the pixel coordinates of the ring and its pre-defined spatial geometric constraints, the position can be solved using a simple monocular localization model, which can be expressed as:

[0072]

[0073]

[0074]

[0075] wherein X, Y, Z represent the three-dimensional position of the center of the spatial ring in the camera coordinate system, H is the actual outer diameter prior value of the spatial ring, x o , y o is the center pixel coordinate of the ellipse, c x , c y , f y is the internal parameter of the monocular camera, and h is the pixel height of the minimum circumscribed rectangle of the ellipse. The preliminary positioning result of the target relative to the camera is:

[0076] t = [X, Y, Z] T

[0077] Step 4: According to the previous step, the three-dimensional coordinate initial value of the center of the spatial ring can be obtained. However, since the monocular vision positioning model assumes that the ring is in the same plane for calculation, the result obtained has certain error, especially when measuring at a long distance, the error is unacceptable. Therefore, the application constructs a state estimation scheme based on back-end optimization, which iteratively optimizes the pose of the spatial ring in combination with the image sensor acquisition result in a nonlinear optimization manner.

[0078] First, the local coordinate system of the object is established with the center of the spatial ring as the origin. According to the known spatial ring size prior, a three-dimensional point model of the ring is established. A total of 72 three-dimensional space points are taken every 10 degrees on the inner and outer edges of the ring. Denoted as:

[0079] {P i |1≤i≤72}

[0080] Using the spatial projection relationship, according to the known preliminary positioning result T, the spatial points are projected onto the image to obtain the projected two-dimensional pixel points. The process can be represented as:

[0081]

[0082] wherein K is the camera internal parameter matrix of the monocular camera obtained through camera calibration, Z is the depth value of the three-dimensional space point, R and t are respectively the rotation matrix and translation vector of the world coordinate system to the ring coordinate system, which together constitute the pose T of the ring coordinate system.

[0083] In the pixel plane, 72 projected pixel points can be obtained. In the case of completely accurate pose estimation, all projected pixel points should be on the directly imaged two-dimensional elliptical ring contour. Therefore, the distance between the two-dimensional elliptical ring contour and the projected pixel points can be used to establish a least squares problem. By finding the nearest observed pixel of each projected pixel in the image contour, the pixel error, i.e. the re-projection error, between each pair of points can be established. Let each projected point p i The nearest point matched on the contour in the image is ui The target function obtained is:

[0084]

[0085] The optimal solution of the circular ring pose T is the value of T that makes L(T) minimum. The Gauss-Newton method is used to solve this least squares optimization problem, and the Jacobian matrix of the re-projection error with respect to the Lie algebra of the camera pose is:

[0086]

[0087] By constructing the Jacobian matrix, the action amount of the 72 groups of points on ΔT is added in turn, and the pose matrix T of the circular ring is constantly iterated and optimized and updated, so that the optimal circular ring pose can be finally optimized.

[0088] Embodiment: as Figure 2 , 3 The embodiment effect of the algorithm is shown in FIGS. 1 to 8, and the red space circular ring is detected and positioned in the airsim simulator. The main display includes the original RGB monocular picture, image preprocessing, LSD line detection, line segment arc search, preliminary ellipse fitting, mean shift filtering, space circular ring point cloud projection effect comparison before and after the back-end optimization, and circular ring real-time recognition detection effect. The final result shows that the method has high positioning accuracy and can realize accurate positioning of the space circular ring based on monocular vision.

[0089] To sum up, the above is only a preferred embodiment of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A monocular vision based spatial circle detection and pose estimation method, characterized in that, Comprise: S1, based on the elliptical template, the space ring in the image obtained by monocular camera is detected, specifically: First, extract the straight line segment in the image; Next, take the extracted straight line segment as the arc support line segment, and construct the arc support group, specifically: if any two connected arc support line segments have similar geometric properties at the connection, they are connected together, and multiple connected arc support line segments together constitute an arc support group; Then, calculate the internal span angle of the arc support group, and take the arc support group with internal span angle exceeding the set threshold as the effective arc support group; Finally, sample on each effective arc support group, and fit the sampling points to an elliptical curve by least square method to obtain the preliminary elliptical parameter equation; All the elliptical parameter equations are subjected to mean shift clustering to obtain the final elliptical parameter equation, and the ellipse with the most significant completeness and geometric property is selected as the output of the detected space ring; Calculate the minimum circumscribed rectangle of the space ring; S2, a monocular positioning model is used to solve the preliminary position of the space ring detected by S1; S3, the preliminary position of the space ring obtained by S2 is iteratively optimized, specifically: S31, taking the center of the space ring as the origin, establishing a local coordinate system of the object; according to the known size of the space ring, a three-dimensional point model of the space ring is established in advance, and a space three-dimensional point is taken on the edge of the inner ring and the outer ring of the space ring at a certain interval, respectively, and is recorded as , N is the total number of sampling points; S32, using the spatial projection relationship, based on the preliminary position t of the spatial ring obtained in S2, projecting the three-dimensional spatial point onto the image to obtain a projected two-dimensional pixel point ; S33, a least square optimization problem is constructed: The pixel error between each pair of points, i.e. the re-projection error, is established by finding the nearest observation pixel of each projected two-dimensional pixel point in the air circle profile detected by S1; let each projected point The nearest point matched on the detected air circle profile is The objective function is: S34, the optimal solution of the space ring pose T is obtained by solving the least square optimization problem constructed in S33.

2. The method of claim 1, wherein, The LSD algorithm, Hough line detection or EDLines line detection is used to extract the straight line segment in the image.

3. The method of claim 2, wherein, The LSD algorithm is used to extract the straight line segment in the image, specifically: First, the image is subjected to Gaussian filtering to remove noise in the image; then, the gradient intensity and direction of each pixel point in the image are calculated; then, the non-maximum suppression method is used to remove the pixel points with weak gradient, and only the locally maximum gradient intensity is retained; Then, according to the gradient direction, the edge points are voted to the subdivided unit in the five-dimensional Hough space, and the maximum value point in the Hough space is found to determine the description of the line segment; finally, the short line segments and overlapping line segments are eliminated based on the description to obtain the final straight line segment detection result.

4. The method of claim 3, wherein, The Sobel operator, Canny operator or Laplace operator is used to calculate the gradient intensity and direction of each pixel point in the image.

5. The method of claim 1, wherein, In the S1, after the detection of the space ring in the image is completed, the space ring with too small area is filtered out.

6. The method according to any one of claims 1 to 5, wherein In the S1, first, the RGB image obtained by the monocular camera is preprocessed: The RGB image obtained by the monocular camera is converted to the HSV color space; based on the color of the space ring, the threshold value is set; in the HSV color space, the image pixel color is extracted based on the threshold value to obtain a binary grayscale image that only retains the space ring; if the space ring has multiple colors, multiple threshold values are set correspondingly, and the image pixel color is extracted and combined after being extracted respectively to obtain a binary grayscale image that only retains the space ring; The binary grayscale image obtained by preprocessing is subjected to space ring detection.

7. The method of claim 6, wherein, In the S1, before the space ring detection is performed, the image is subjected to median filtering to remove noise in the image.

8. The method of claim 1, wherein, In S2, the preliminary position of the spatial torus is wherein, Wherein, X, Y, Z are three-dimensional positions of the spatial circular ring center in the camera coordinate system; H is the actual height prior value of the spatial circular ring; is the center pixel coordinate of the minimum circumscribed rectangle, is the intrinsic parameter of the monocular camera, h is the pixel height of the minimum circumscribed rectangle.

9. The method of claim 8, wherein, Wherein, K is the camera intrinsic matrix obtained by camera calibration, Z is the depth value of the spatial three-dimensional point, and R and t are the rotation matrix and translation vector from the world coordinate system to the ring coordinate system, respectively; The Gauss-Newton method is used to solve the least squares optimization problem constructed by S33, and the Jacobian matrix of the re-projection error with respect to the Lie algebra of the camera pose is: By constructing the Jacobian matrix, the three-dimensional points in the space are traversed in sequence, and a pose optimization direction is obtained according to a Gauss-Newton optimization method : where e is the re-projection error term, and The action of the optimization is to accumulate the T matrix of the ring, and constantly iterate and update the pose matrix T of the ring, that is, the optimal ring pose can be finally optimized.

Images