A method of close-range relative positioning between two cameras based on feature points

Abstract

The present invention discloses a feature point-based close-range visual relative positioning method for two aircraft. The method comprises performing color space conversion on acquired close-range images of the two aircraft to extract a mask image of the engine nozzle; performing edge detection on the mask image of the engine nozzle to determine the center position of the engine nozzle; determining the connecting line between the leading and trailing edges of the wing and the side edges based on the center position of the engine nozzle, and tracking the feature points at the intersection of the leading and trailing edges of the wing and the side edges; and performing close-range visual relative positioning of the two aircraft based on the feature point of the center of the engine nozzle and the feature points of the connecting line between the leading and trailing edges of the wing and the side edges. The present invention can improve the adaptability, intelligence, and stealth of dual-aircraft pose estimation.

Description

Technical Field

[0001] The present invention relates to the technical field of machine vision, and in particular to a method for close-range visual relative positioning of two machines based on feature points. Background Art

[0002] Currently, the sensors used for relative positioning and navigation primarily include inertial navigation technology, the Global Positioning System (GPS), differential GPS (DGPS), and active / passive vision sensors. Each of these sensor-based navigation methods has distinct characteristics and, therefore, varying scopes of application. While DGPS navigation technology is relatively mature, it is susceptible to electromagnetic interference, and aircraft wings and tails can affect GPS satellite signal reception. Furthermore, its update frequency is low, so DGPS navigation is generally used for long-distance measurement.

[0003] Due to limited prior information, relative positioning between two aircraft in the air in an environment without GPS information is particularly difficult, significantly impacting the efficiency of the two aircraft in executing their missions. Navigation methods based on active and passive vision sensors offer advantages such as high accuracy, rapid update frequency, compact size, low cost, and good real-time performance. Therefore, their application in autonomous aerial refueling scenarios has been widely studied. In recent years, vision-based autonomous navigation technology has also been widely studied and achieved certain success in fields such as autonomous driving, robotics, and drones. Summary of the Invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a feature point-based dual-machine close-range visual relative positioning method.

[0005] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0006] A method for close-range visual relative positioning of two machines based on feature points includes the following steps:

[0007] S1. Perform color space conversion on the acquired close-up images of the two aircraft to extract the mask image of the engine nozzle;

[0008] S2. Perform edge detection on the mask image of the engine nozzle to determine the center position of the engine nozzle;

[0009] S3. Determine the connecting lines of the front and rear edges of the wing and the side edges according to the center position of the engine nozzle, and track the feature points of the intersections of the front and rear edges of the wing and the side edges;

[0010] S4. Perform close-range visual relative positioning of the two aircraft based on the characteristic points of the center of the engine nozzle and the characteristic points of the connecting lines of the front and rear edges and sides of the wings.

[0011] Furthermore, step S1 specifically includes the following steps:

[0012] S11, obtaining a close-up image of the current aircraft and the aircraft ahead;

[0013] S12, converting the close-range images of the two cameras from the RGB color space to the HSV color space;

[0014] S13, performing color segmentation on the three HSV channels of the close-range image of the two aircraft to obtain an initial mask image of the engine nozzle;

[0015] S14: performing corrosion and expansion processing on the initial mask image of the engine nozzle to obtain a mask image of the engine nozzle.

[0016] Furthermore, step S2 specifically includes the following steps:

[0017] S21. Perform Hough circle detection on the mask image of the engine nozzle to obtain a circle detection result of the engine nozzle;

[0018] S22, performing Canny edge detection on the mask image of the engine nozzle to obtain an edge extraction result of the engine nozzle;

[0019] S23, segmenting the edge extraction result of the engine nozzle using the circle detection result of the engine nozzle to obtain a binary edge image of the engine nozzle;

[0020] S24 . Determine the center position of the engine nozzle according to the edge points in the binary edge image of the engine nozzle.

[0021] Furthermore, step S24 specifically includes:

[0022] The edge point positions in the binary edge image of the engine nozzle are averaged to obtain the center position of the engine nozzle.

[0023] Furthermore, step S3 specifically includes the following steps:

[0024] S31. Crop the mask image of the engine nozzle to obtain an image of the connection between the wing and the fuselage;

[0025] S32, performing Hough transform line detection on the image of the connection between the wing and the fuselage to obtain the front and rear edges of the wing and the connection line of the connection between the wing and the fuselage;

[0026] S33, performing slope filtering on the connecting lines of the front and rear edges of the wing and the connection between the wing and the fuselage to obtain connecting lines of the front and rear edges of the wing and the side edges;

[0027] S34, determining characteristic points of the line connecting the front and rear edges of the wing and the side edges according to the line connecting the front and rear edges of the wing and the side edges;

[0028] S35, using a sparse optical flow method to track feature points of the front and rear edges and the side connection line of the wing;

[0029] S36: Update the tracked feature points of the front and rear edges of the wing and the side connection line to obtain the tracking results of the feature points in each frame of image.

[0030] Furthermore, step S35 specifically includes the following steps:

[0031] S351. Construct a pixel movement speed matrix based on the movement distance of pixels in adjacent images, expressed as:

[0032]

[0033] Among them, I x =θI / θx,I y =θI / θy,I t =θI / θt, where I is the grayscale value of the pixel, u is the moving speed of the pixel in the X direction, v is the moving speed of the pixel in the Y direction, x is the coordinate of the pixel in the X direction, y is the coordinate of the pixel in the Y direction, and t is the moving time of the pixel;

[0034] S352: Construct a neighborhood pixel movement speed matrix based on the movement distances of all pixels in the neighborhood of the pixel, expressed as:

[0035]

[0036] Among them, I xk =θI k / θx k , I yk =θI k / θy k , I tk =θI k / θt k , I k is the gray value of the neighboring pixel, u is the moving speed of the neighboring pixel in the X direction, v is the moving speed of the neighboring pixel in the Y direction, x k is the coordinate of the neighborhood pixel in the X direction, y k is the coordinate of the pixel in the Y direction, t k is the movement time of the neighborhood pixels;

[0037] S353. Solve the above equation using the least squares principle to obtain the moving speed of the pixel in the X direction and the Y direction, which is expressed as:

[0038]

[0039] in, T is the transpose symbol.

[0040] Furthermore, step S36 specifically includes the following steps:

[0041] S361. Calculate the distance between the front and rear edges of the wing and the line connecting the side edges.

[0042] S362: Determine whether the distance between the front and rear edges of the wing and the connecting line between the side edges is less than a set distance threshold;

[0043] If so, the tracked feature point positions are updated using the feature point detection results of the current frame;

[0044] Otherwise, the feature point detection results of the previous frame are used to continue tracking the feature points of the current frame.

[0045] Furthermore, step S4 specifically includes the following steps:

[0046] S41. Determine the rotation vector and translation vector between the body coordinate system of the preceding aircraft and the camera coordinate system based on the coordinates of the center feature point of the engine nozzle and the feature points of the connecting line between the front and rear edges and the side edges of the wing in the world coordinate system;

[0047] S42. Perform coordinate system conversion on the rotation vector and translation vector between the body coordinate system of the front aircraft and the camera coordinate system to obtain a close-range visual relative positioning result of the two aircraft.

[0048] The present invention has the following beneficial effects:

[0049] The present invention adopts a positioning method based on visual feature points to achieve the relative positioning of two aircraft flying in front and behind each other. It has the advantages of low cost, simple equipment, large amount of information, and strong autonomy. It can well meet the requirements of relative posture estimation between two aircraft, so as to improve the adaptability, intelligence and concealment of the posture estimation of two aircraft. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 Schematic diagram of the process of the dual-machine close-range visual relative positioning method based on feature points in the present invention;

[0051] Figure 2 Schematic diagram of the center of the engine nozzle, the leading and trailing edges of the wing, the side lines, and the intersection points in the present invention;

[0052] Figure 3 Schematic diagram of slope filtering results in the present invention;

[0053] Figure 4 Schematic diagram of optical flow tracking in the present invention. DETAILED DESCRIPTION

[0054] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0055] like Figure 1 As shown, the embodiment of the present invention provides a method for close-range visual relative positioning of two machines based on feature points, comprising the following steps S1 to S4:

[0056] S1. Perform color space conversion on the acquired close-up images of the two aircraft to extract the mask image of the engine nozzle;

[0057] In an optional embodiment of the present invention, step S1 of this embodiment specifically includes the following steps:

[0058] S11, obtaining a close-up image of the current aircraft and the aircraft ahead;

[0059] Specifically, this embodiment obtains image information of the aircraft in front through a forward-looking RGB monocular camera installed in the positive direction of the aircraft's x-axis. The original image information is an RGB color image with a resolution of 800*450 pixels and an image frame rate of approximately 30 frames per second.

[0060] S12, converting the close-range images of the two cameras from the RGB color space to the HSV color space;

[0061] Specifically, in the sky, the aircraft and the background clouds and sky have obvious differences in hue, color saturation, and brightness. To better distinguish the aircraft from the background clouds in subsequent processing, this embodiment converts the original RGB color space to the HSV color space. The color parameters in the HSV model are: hue (H), saturation (S), and value (V).

[0062] First, the RGB channels are normalized to obtain the maximum value and range of the three channels.

[0063] R′=R / 255

[0064] G′=G / 255

[0065] B′=B / 255

[0066] C max =max(R′,G′,B′)

[0067] C min =min(R′,G′,B′)

[0068] Δ=C max -C min

[0069] Then calculate H, S, V respectively

[0070] H channel:

[0071]

[0072] S channel:

[0073]

[0074] V channel:

[0075] V=C max

[0076] S13, performing color segmentation on the three HSV channels of the close-range image of the two aircraft to obtain an initial mask image of the engine nozzle;

[0077] Specifically, since the engine nozzles differ in color from the aircraft fuselage, this embodiment utilizes this characteristic to perform color segmentation on the engine nozzles. Color segmentation is performed on the V channel, or the brightness channel. Because the engine nozzles are relatively dark, the V channel clearly distinguishes the engine from the background and fuselage.

[0078] Color segmentation uses the pixel value of the image and sets a suitable threshold to binarize the image as follows:

[0079]

[0080] Where x represents the pixel value of each pixel in the V channel. When x is greater than the high threshold or less than the low threshold, the pixel value is set to 0, turning it black. If it is within the threshold range, it is set to 255, turning it white.

[0081] After binarizing the image, we can obtain a rough mask image of the engine nozzle. However, since the pixel values ​​of the engine nozzle in the image are not exactly the same, a complete circular mask of the engine nozzle cannot be obtained after color segmentation, so morphological processing of dilation and erosion is required.

[0082] Dilation is an operation to find the local maximum. From a mathematical point of view, dilation and erosion are convolution operations between the original image and the convolution kernel. The convolution kernel can be of any shape and size, and it has a single defined reference point, called an anchor point. In the dilation operation, the convolution kernel is convolved with the original image, that is, the maximum value of the pixels in the area covered by the convolution kernel is calculated and assigned to the pixel specified by the reference point (anchor point). This will gradually increase the highlight area (foreground color) in the image. The mathematical expression for dilation is:

[0083]

[0084] After expansion erosion, the black spots in the white engine nozzle mask will be removed, and the nozzle edge will become smoother, which will help in the subsequent extraction of the center position of the nozzle.

[0085] S14: performing corrosion and expansion processing on the initial mask image of the engine nozzle to obtain a mask image of the engine nozzle.

[0086] Specifically, the Hough transform is a feature extraction technique used in image processing. This process calculates the local maximum of the cumulative results in a parameter space to obtain a set of features that conform to a specific shape as the result of the Hough transform. In the present invention, the image input to the Hough circle detection algorithm is a binary image after color segmentation.

[0087] The basic principle of the Hough circle transform is similar to that of the Hough line transform. The two-dimensional polar radius and polar angle space corresponding to the midpoint in the Hough line transform is replaced by the three-dimensional center and radius space in the Hough circle transform. In the standard Hough circle transform, for any point on the edge image, all possible circles passing through this point are represented in three-dimensional space by three parameters: the center position and the radius, which corresponds to a curve in three-dimensional space. For multiple edge points, the more points there are, the more three-dimensional space curves corresponding to these points intersect at one point, and the more points on the common circle they pass through. Similarly, we can use the same threshold method to determine whether a circle is detected. This is the principle of the standard Hough circle transform. The equation of the circle in the Cartesian coordinate system is:

[0088] (xa) 2 +(yb) 2 =r 2

[0089] Where (a, b) is the center of the circle and r is the radius, which can also be expressed as:

[0090] x=a+rcosθ

[0091] y=b+rsinθ

[0092] Right now

[0093] a=x-rcosθ

[0094] b=y-rsinθ

[0095] Therefore, in the three-dimensional coordinate system composed of abr, a point can uniquely determine a circle.

[0096] In the Cartesian xy coordinate system, all circles passing through a certain point are mapped to the abr coordinate system as a three-dimensional curve: all circles passing through all non-zero pixels in the xy coordinate system constitute many three-dimensional curves in the abr coordinate system.

[0097] All points on the same circle in the xy coordinate system have the same equation, and they map to the same point in the abr coordinate system. Therefore, in the abr coordinate system, that point should intersect with N0 curves, the total number of pixels in the circle. By determining the cumulative number of intersections for each point in the abr system, points with a value greater than a certain threshold are considered circles.

[0098] S2. Perform edge detection on the mask image of the engine nozzle to determine the center position of the engine nozzle;

[0099] In an optional embodiment of the present invention, step S2 of this embodiment specifically includes the following steps:

[0100] S21. Perform Hough circle detection on the mask image of the engine nozzle to obtain a circle detection result of the engine nozzle;

[0101] Specifically, edge detection based on the Canny operator mainly consists of five steps, namely Gaussian filtering, pixel gradient calculation, non-maximum suppression, hysteresis threshold processing and isolated weak edge suppression.

[0102] Gaussian filtering is a low-pass filter used to smooth and blur images. The basic idea behind Gaussian filtering is that the value of each pixel in an image is a weighted average of its own value and the values ​​of other pixels in its neighborhood. This involves scanning each pixel in the image with a convolution kernel, multiplying the pixel values ​​by the weights at the corresponding locations, and then taking the sum.

[0103] The Gaussian convolution kernel comes from a two-dimensional Gaussian function, which is a two-dimensional normal distribution density function. The mean is 0 and the variance σ 2 The two-dimensional Gaussian function is as follows.

[0104]

[0105] The Gaussian convolution kernel is a discretized representation of a continuous two-dimensional Gaussian function. Therefore, a Gaussian template of any size can be obtained by establishing a (2k+1)×(2k+1) matrix M. The element value at the (i, j) position can be determined as follows:

[0106]

[0107] Here is an example of a Gaussian convolution kernel with σ = 1.4 and size 3×3 (note the normalization):

[0108]

[0109] If a 3×3 window in the image is A and the pixel to be filtered is e, then after Gaussian filtering, the brightness value of pixel e is:

[0110]

[0111] Where * is the convolution symbol, and sum represents the sum of all elements in the matrix. The choice of Gaussian convolution kernel size affects detector performance. A larger kernel size reduces the detector's sensitivity to noise, but also slightly increases edge detection localization errors.

[0112] The Sobel operator used in pixel gradient calculation is a discrete differential operator used for edge detection. It is used to calculate the approximate gradient of the image grayscale function. Using the Sobel operator at any point in the image will generate the corresponding gradient vector or its normal vector. In the Canny edge detection algorithm, the Sobel operator is two 3×3 matrices, S x and S y The former is used to calculate the image G x Directional pixel gradient matrix, which is used to calculate the image G y Directional pixel gradient matrix. The specific form is:

[0113]

[0114]

[0115] Where I is the grayscale image matrix, and * here represents the cross-correlation operation (the convolution operation can be regarded as the cross-correlation operation after the convolution kernel is rotated 180°). It should be noted that the origin of the image matrix coordinate system is in the upper left corner, and the positive x direction is from left to right, and the positive y direction is from top to bottom. Then

[0116]

[0117] The gradient strength matrix G can be calculated xy .

[0118] The purpose of non-maximum pixel gradient suppression is to eliminate spurious responses caused by edge detection and thus filter out edges. The basic approach is to compare the gradient strength of the current pixel with the gradient strength of adjacent pixels along the positive and negative gradient directions. If the gradient strength is the maximum (i.e., the extreme value), the pixel is retained as an edge point. If it is not the maximum, it is suppressed and not considered an edge point. For more accurate calculations, linear interpolation is usually used between two adjacent pixels that straddle the gradient direction to obtain the pixel gradient to be compared.

[0119] Even after non-maximum suppression, the image still contains many noise points. The Canny algorithm uses a double-thresholding technique. This involves setting an upper and lower threshold (usually manually specified in OpenCV). Pixels above the upper threshold are considered strong edges, while those below the lower threshold are considered non-edges. Pixels between these two thresholds are considered weak edge candidates and require further processing.

[0120] Finally, there is isolated weak edge suppression. This method considers weak edges connected to strong edges as edges, and other weak edges are suppressed. Generally speaking, weak edge pixels caused by real edges will be connected to strong edge pixels, while noise responses are not connected. By looking at the weak edge pixel and its eight neighboring pixels, it can be judged based on its connection with the strong edge. Generally, it can be defined that as long as one of the neighboring pixels is a strong edge pixel, the weak edge can be retained as a strong edge, that is, a true edge point.

[0121] S22, performing Canny edge detection on the mask image of the engine nozzle to obtain an edge extraction result of the engine nozzle;

[0122] Specifically, this embodiment uses Hough transform circle detection to detect the number of nozzles in the image, outputting the circle center position and radius. Canny edge detection is used to identify the circular boundaries. Due to the limited accuracy of Hough transform circle detection and the significant jumps in detection results between image frames, this invention only uses the Hough transform to locate the circular shape of the engine nozzles. The circular shape of the engine is extracted using the circular edge detected by Canny edge detection after color segmentation.

[0123] S23, segmenting the edge extraction result of the engine nozzle using the circle detection result of the engine nozzle to obtain a binary edge image of the engine nozzle;

[0124] Specifically, this embodiment first uses the results of Hough circle detection to obtain the approximate position of each circle, and then uses the approximate position of the circle to segment the edge image of each circle on the edge detection image.

[0125] S24 . Determine the center position of the engine nozzle according to the edge points in the binary edge image of the engine nozzle.

[0126] Specifically, after obtaining a segmented image of each nozzle, this embodiment averages the positions of these points, as they represent the nozzle's edge points. Because this method relies on edge detection rather than Hough circle detection, sudden changes in the Hough circle detection results do not affect the center position extraction.

[0127] S3. Determine the connecting lines of the front and rear edges of the wing and the side edges according to the center position of the engine nozzle, and track the feature points of the intersections of the front and rear edges of the wing and the side edges;

[0128] In an optional embodiment of the present invention, step S3 of this embodiment specifically includes the following steps:

[0129] S31. Crop the mask image of the engine nozzle to obtain an image of the connection between the wing and the fuselage;

[0130] Specifically, after obtaining the engine nozzle position of a frame of image, this embodiment uses the engine nozzle position to crop the original image. The cropped image is significantly smaller than the original image. This can not only speed up the calculation time, but also eliminate interference caused by other parts of the aircraft for subsequent line detection.

[0131] S32, performing Hough transform line detection on the image of the connection between the wing and the fuselage to obtain the front and rear edges of the wing and the connection line of the connection between the wing and the fuselage;

[0132] Specifically, a straight line can be represented by two variables in the two-dimensional space of the image. There are two cases: in the rectangular coordinate system, it can be represented by the parameters slope and intercept (k, b); in the polar coordinate system, it can be represented by the parameters polar diameter and polar angle (r θ ,θ) represents.

[0133] For Hough transform, we use the second method to represent the straight line, so the expression of the straight line is:

[0134]

[0135] Simplifying, we can get:

[0136] r=x cosθ+y sinθ

[0137] Generally speaking, for a point (x0, y0), a cluster of straight lines passing through this point can be uniformly defined as:

[0138] r θ=x0cosθ+y0sinθ

[0139] This means that for every pair (r θ ,θ) represents a straight line passing through the point (x0,y0). For a given point (x0,y0), we draw all the straight lines passing through it in the polar coordinate polar plane, and we will get a sine curve.

[0140] We can perform the above operation on all points in the image. If the curves obtained by applying the above operation to two different points intersect in the plane θ-r, it means that they pass through the same line. Therefore, a line can be detected by finding the number of curves that intersect at a point in the plane θ-r. The more lines intersect at a point, the more points the line represented by this intersection is composed of. Therefore, we can set a threshold for the number of points on the line to define how many lines must intersect at a point before a line is considered detected.

[0141] In the specific calculation, the parameter space can be regarded as a discrete space, and a two-dimensional cumulative array A(θ,r) is established. The range of the first dimension is the possible range of ρ in the image coordinate space, and the range of the second dimension is the possible range of r in the image coordinate space. Initially, A(θ,r) is initialized to 0, and then for each foreground point (x i ,y i ), bring each discrete value θ in the parameter space into r θ =x0cosθ+y0sinθ, so the corresponding r value can be calculated. Every time a pair of (θ, r) is calculated, the corresponding array element A(θ, r) is increased by 1. After all the calculations are completed, the maximum peak of A(θ, r) is found in the parameter space voting results. The corresponding θ0, r0 are the parameters of the equation of the straight line with the largest number of collinear points in the original image. Next, we can continue to look for the parameter values ​​of the second peak and the third peak, which correspond to the straight lines with slightly fewer collinear points in the original image.

[0142] S33, performing slope filtering on the connecting lines of the front and rear edges of the wing and the connection between the wing and the fuselage to obtain connecting lines of the front and rear edges of the wing and the side edges;

[0143] Specifically, because multiple line segments may be detected for the same edge, it is necessary to filter out the lines that are detected repeatedly. By taking the average of the straight line segments with similar slopes and intercepts, it is possible to ensure that there is only one detected line segment for each of the front and rear edges of the wing and the connecting line. For example, Figure 3 shown.

[0144] S34, determining characteristic points of the line connecting the front and rear edges of the wing and the side edges according to the line connecting the front and rear edges of the wing and the side edges;

[0145] Specifically, the intersection of the front and rear edges of the wing and the side connection lines can be calculated based on the front and rear edges of the wing and the side connection lines, and the intersection of the front and rear edges of the wing and the side connection lines is used as a feature point, such as Figure 2 shown.

[0146] S35, using a sparse optical flow method to track feature points of the front and rear edges and the side connection line of the wing;

[0147] Step S35 specifically includes the following steps:

[0148] S351. Construct a pixel movement speed matrix based on the movement distance of pixels in adjacent images, expressed as:

[0149]

[0150] Among them, I x =θI / θx,I y =θI / θy,I t =θI / θt, where I is the grayscale value of the pixel, u is the moving speed of the pixel in the X direction, v is the moving speed of the pixel in the Y direction, x is the coordinate of the pixel in the X direction, y is the coordinate of the pixel in the Y direction, and t is the moving time of the pixel;

[0151] S352: Construct a neighborhood pixel movement speed matrix based on the movement distances of all pixels in the neighborhood of the pixel, expressed as:

[0152]

[0153] Among them, I xk =θI k / θx k , I yk =θI k / θy k , I tk =θI k / θt k , I k is the gray value of the neighboring pixel, u is the moving speed of the neighboring pixel in the X direction, v is the moving speed of the neighboring pixel in the Y direction, x k is the coordinate of the neighborhood pixel in the X direction, y k is the coordinate of the pixel in the Y direction, t k is the movement time of the neighborhood pixels;

[0154] S353. Solve the above equation using the least squares principle to obtain the moving speed of the pixel in the X direction and the Y direction, which is expressed as:

[0155]

[0156] in, T is the transpose symbol.

[0157] Specifically, the optical flow method uses the changes in pixels in an image sequence to find the correspondence between the previous image and the current image, and then obtains the motion state of the object between the two frames. The optical flow method has two strong assumptions:

[0158] 1) The brightness of the pixels corresponding to the same object in the image remains unchanged. Since the optical flow method finds the motion relationship of the target in two frames of images based on pixel brightness, if the pixel brightness changes, it will be impossible to match the same object or pixel in the two frames.

[0159] 2) The two frames of images must have a small movement. The optical flow method only searches for corresponding pixels near the original pixels. Therefore, the pixel positions in the two frames of images cannot move a large distance.

[0160] The schematic diagram of the optical flow method is as follows Figure 4 The three images in the figure are three consecutive frames taken over time, with the time interval between the two frames being dt. The boxes in the images represent pixels, and the grayscale value of each pixel is represented by . From the first frame to the second, the pixel moves by (dx, dy). Since the grayscale value of the pixel remains unchanged, the following relationship exists before and after the pixel movement.

[0161] I(x,y,t)=I(x+dx,y+dy,t+dt)

[0162] Taylor expansion of the above formula gives the following results:

[0163]

[0164] Further simplifying, dividing both sides of the equation by dt, the result is as follows:

[0165]

[0166] Among them, dx / dt and dy / dt represent the movement speed of the pixel in the X direction and Y direction respectively, which can be expressed in matrix form as follows:

[0167]

[0168] Among them, I x =θI / θx, I x =θI / θy, I x =θI / θt. These quantities can be calculated based on the image information, and then the speed at which the pixel brings you in the X and Y directions can be obtained. The speed in the two directions is represented by u=dx / dt and u=dy / dt.

[0169] Since there are two unknowns in the above equation, it is impossible to directly solve the movement speed in both directions. Therefore, the movement speed of the entire neighborhood in the X and Y directions can be obtained by combining the information of all pixels in the neighborhood. However, this method requires the assumption that all pixels in the neighborhood have the same motion state. Assuming a neighborhood of size w×w, the motion state of each pixel in it can be expressed as follows:

[0170]

[0171] Combining the motion state of each pixel, we get:

[0172]

[0173] in:

[0174]

[0175]

[0176] Using the least squares principle to solve the above equation, we can get:

[0177]

[0178] The above formula can be used to calculate the velocity magnitude in the X and Y directions of the neighborhood.

[0179] S36: Update the tracked feature points of the front and rear edges of the wing and the side connection line to obtain the tracking results of the feature points in each frame of image.

[0180] Step S36 specifically includes the following steps:

[0181] S361. Calculate the distance between the front and rear edges of the wing and the line connecting the side edges.

[0182] S362: Determine whether the distance between the front and rear edges of the wing and the connecting line between the side edges is less than a set distance threshold;

[0183] If so, the tracked feature point positions are updated using the feature point detection results of the current frame;

[0184] Otherwise, the feature point detection results of the previous frame are used to continue tracking the feature points of the current frame.

[0185] Specifically, the tracking algorithm tracks two points: the intersection of the leading and trailing edges of the wing and the fuselage. We calculate the position of these feature points for each frame and use the distance between the calculated feature points and the oblique line to determine the quality of the detection results. Only when the distance between the leading and trailing edge endpoints and the oblique line is less than a certain threshold is the detection result considered good for that frame, and the tracking result is then updated based on this frame's detection results. Otherwise, feature point tracking continues for this frame using the results from the previous frame.

[0186] S4. Perform close-range visual relative positioning of the two aircraft based on the characteristic points of the center of the engine nozzle and the characteristic points of the connecting lines of the front and rear edges and sides of the wings.

[0187] In an optional embodiment of the present invention, step S4 of this embodiment specifically includes the following steps:

[0188] S41. Determine the rotation vector and translation vector between the body coordinate system of the preceding aircraft and the camera coordinate system based on the coordinates of the center feature point of the engine nozzle and the feature points of the connecting line between the front and rear edges and the side edges of the wing in the world coordinate system;

[0189] S42. Perform coordinate system conversion on the rotation vector and translation vector between the body coordinate system of the front aircraft and the camera coordinate system to obtain a close-range visual relative positioning result of the two aircraft.

[0190] Specifically, in this invention, we use the EPNP solution to solve the PNP problem. The EPnP algorithm uses the known world coordinates of n 3D points, selects four control points (with known world coordinates) through PCA, and establishes a new local coordinate system, thereby expressing the world coordinates of the 3D points using the new control points (world coordinates). Then, using the camera projection model and the 2D point pixel coordinates, a linear system of equations is established regarding the coordinates of the control points in the camera coordinate system. The coordinates of the four control points in the camera coordinate system are solved, and then the coordinates of the n 3D points in the camera coordinate system are solved. At this point, we know the coordinates of the n 3D points in the world coordinate system and the camera coordinate system, and the problem is converted to a 3D-3D problem, which is then solved using ICP.

[0191] Because the PNP outputs the rotation and translation vectors between the forward aircraft's body coordinate system and the camera's coordinate system, while the present invention requires the relative pose between the two aircraft's body coordinate systems, a coordinate transformation is required on the PNP output. Considering that the camera is rigidly mounted directly in front of the aircraft's flight direction, the x, y, and z axes of the camera's coordinate system correspond to the y, z, and x axes of the aircraft's body coordinate system. Therefore, only a coordinate transformation matrix is ​​required on the output.

[0192] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0193] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0194] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0195] Specific embodiments are used in the present invention to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

[0196] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

Claims

1. A dual-machine close-range visual relative positioning method based on feature points, characterized in that: The following steps are involved: S1. Perform color space conversion on the acquired close-up images of the two aircraft to extract the mask image of the engine nozzle; S2. Perform edge detection on the mask image of the engine nozzle to determine the center position of the engine nozzle; The specific steps include: S21. Perform Hough circle detection on the mask image of the engine nozzle to obtain a circle detection result of the engine nozzle; S22, performing Canny edge detection on the mask image of the engine nozzle to obtain an edge extraction result of the engine nozzle; S23, segmenting the edge extraction result of the engine nozzle using the circle detection result of the engine nozzle to obtain a binary edge image of the engine nozzle; S24, determining the center position of the engine nozzle according to the edge points in the binary edge image of the engine nozzle; specifically, averaging the edge point positions in the binary edge image of the engine nozzle to obtain the center position of the engine nozzle; S3, determining the connecting lines of the front and rear edges of the wing and the side edges according to the center position of the engine nozzle, and tracking the feature points of the intersections of the front and rear edges of the wing and the side edges; specifically comprising the following steps: S31. Crop the mask image of the engine nozzle to obtain an image of the connection between the wing and the fuselage; S32, performing Hough transform line detection on the image of the connection between the wing and the fuselage to obtain the front and rear edges of the wing and the connection line of the connection between the wing and the fuselage; S33, performing slope filtering on the connecting lines of the front and rear edges of the wing and the connection between the wing and the fuselage to obtain connecting lines of the front and rear edges of the wing and the side edges; S34, determining characteristic points of the line connecting the front and rear edges of the wing and the side edges according to the line connecting the front and rear edges of the wing and the side edges; S35, using a sparse optical flow method to track feature points of the front and rear edges and the side connection line of the wing; S36, updating the tracked feature points of the front and rear edges of the wing and the side connection line to obtain the tracking results of the feature points in each frame of the image; S4. Perform close-range visual relative positioning of the two aircraft based on the characteristic points of the center of the engine nozzle and the characteristic points of the connecting lines of the front and rear edges and sides of the wings.

2. The method for close-range visual relative positioning of two machines based on feature points according to claim 1, characterized in that: Step S1 specifically includes the following steps: S11, obtaining a close-up image of the current aircraft and the aircraft ahead; S12, converting the close-range images of the two cameras from the RGB color space to the HSV color space; S13, performing color segmentation on the three HSV channels of the close-range image of the two aircraft to obtain an initial mask image of the engine nozzle; S14: performing corrosion and expansion processing on the initial mask image of the engine nozzle to obtain a mask image of the engine nozzle.

3. The method for close-range visual relative positioning of two machines based on feature points according to claim 1, characterized in that: Step S35 specifically includes the following steps: S351. Construct a pixel movement speed matrix based on the movement distance of pixels in adjacent images, expressed as: in, , , , I is the grayscale value of the pixel, u is the moving speed of the pixel in the X direction, v is the moving speed of the pixel in the Y direction, x is the coordinate of the pixel in the X direction, y is the pixel's coordinate in the Y direction, t is the pixel movement time; S352: Construct a neighborhood pixel movement speed matrix based on the movement distances of all pixels in the neighborhood of the pixel, expressed as: in, , , , I k is the grayscale value of the neighborhood pixel, u is the moving speed of the pixel in the X direction, v is the moving speed of the pixel in the Y direction, x k is the coordinate of the neighborhood pixel in the X direction, y k is the pixel's coordinate in the Y direction, t k is the movement time of the neighborhood pixels; S353. Solve the above equation using the least squares principle to obtain the moving speed of the pixel in the X direction and the Y direction, which is expressed as: in, , , T is the transpose symbol.

4. The method for close-range visual relative positioning of two machines based on feature points according to claim 1, characterized in that: Step S36 specifically includes the following steps: S361. Calculate the distance between the front and rear edges of the wing and the line connecting the side edges. S362: Determine whether the distance between the front and rear edges of the wing and the connecting line between the side edges is less than a set distance threshold; If so, the tracked feature point positions are updated using the feature point detection results of the current frame; Otherwise, the feature point detection results of the previous frame are used to continue tracking the feature points of the current frame.

5. The method for close-range visual relative positioning of two machines based on feature points according to claim 1, characterized in that: Step S4 specifically includes the following steps: S41. Determine the rotation vector and translation vector between the body coordinate system of the preceding aircraft and the camera coordinate system based on the coordinates of the center feature point of the engine nozzle and the feature points of the connecting line between the front and rear edges and the side edges of the wing in the world coordinate system; S42. Perform coordinate system conversion on the rotation vector and translation vector between the body coordinate system of the front aircraft and the camera coordinate system to obtain a close-range visual relative positioning result of the two aircraft.

Images