Relative pose measurement method based on sequence frame image

Abstract

The invention provides a relative pose measurement method based on a sequence frame image, and the method comprises the steps: collecting an infrared image of a target in real time through an infrared camera, and collecting the inertial pose information of an aircraft in real time through an inertial navigation system in the process of measuring the relative position pose of the aircraft; grouping the acquired images in the form of a sliding window; obtaining a first homography matrix and a second homography matrix; determining a final homography matrix; obtaining a final point line feature on the current frame image; acquiring vanishing points of the target left line and the target right line intersected on the image coordinate system and vanishing points of the target starting line on the image coordinate system; establishing a pose resolving equation; and performing pose calculation on the pose calculation equation to obtain a pitch angle, a course angle and a roll angle of the aircraft and lateral displacement, forward displacement and vertical displacement of the aircraft relative to the target.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method for relative pose measurement based on a sequence of frame images. Background Technology

[0002] In the process of relative position and attitude measurement of aircraft, the existing method of relative pose measurement based on point and line features extracted from single-frame images is greatly affected by factors such as environmental changes and ground object interference, which can significantly affect the image quality and the reliable extraction of geometric features. This results in problems such as measurement jumps and discontinuities in relative pose measurement based on single-frame images, which seriously affects the reliability of relative pose measurement. Summary of the Invention

[0003] This invention provides a relative pose measurement method based on sequential frame images. By utilizing the correlation between features between sequential frames, the sequential images can be processed to improve the robustness and accuracy of the relative pose measurement algorithm.

[0004] This invention provides a relative pose measurement method based on a sequence of frame images, the method comprising:

[0005] During the relative position and attitude measurement of the aircraft, infrared images of the target are acquired in real time using an infrared camera, and inertial navigation system is used to acquire the inertial attitude information of the aircraft in real time.

[0006] The acquired images are grouped in the form of a sliding window; each group includes n consecutive frames, the previous n-1 frames are historical frames, and the nth frame is the current frame; a feature recognition network is used to recognize each frame to obtain the target point and line feature coordinates; the target point and line features include the target's edge line, center line, and corner points;

[0007] A super-point image feature extraction network is used to extract image features from each group of n frames, resulting in n extracted image features. A super glue feature registration network is used to register the extracted image features from the previous n-1 frames with the extracted image features from the nth frame, obtaining the registration relationship between the historical frame images and the current frame images. A random sample consensus algorithm is used to obtain the first homography matrix based on the registration relationship between the historical frame images and the current frame images. The second homography matrix is ​​obtained based on the inertial pose information of the aircraft.

[0008] Determine whether the pixel coordinate distance between the target point line feature projection results of the first homography matrix and the second homography matrix is ​​less than a first preset value. If so, use the first homography matrix as the final homography matrix; otherwise, use the second homography matrix as the final homography matrix.

[0009] Based on the target point and line feature coordinates and the final homography matrix of each frame image, the target point and line features on the previous n-1 frames are projected onto the current frame image to obtain the target point and line feature extraction results on the current frame image. The target point and line feature extraction results on the current frame image are then comprehensively optimized to obtain the final point and line features on the current frame image.

[0010] Based on the final point and line features on the current frame image, establish the straight line equations of the three lines on the image: the left line of the target, the starting line of the target, and the right line of the target. Based on the straight line equations, obtain the disappearance points of the left line of the target and the right line of the target on the image coordinate system, and the disappearance points of the starting line of the target on the image coordinate system.

[0011] Based on the vanishing points of the left and right lines of the target in the image coordinate system, the vanishing points of the target's starting line in the image coordinate system, the camera intrinsic parameter matrix, the rotation matrix from the world coordinate system to the camera coordinate system, the unit direction vector of the target's edge line, and the unit direction vector of the target's starting line, a pose calculation equation is established. The pose calculation equation is then used to calculate the pitch angle, heading angle, roll angle, and the lateral, forward, and vertical displacements of the aircraft relative to the target.

[0012] Preferably, the target point and line feature extraction results on the current frame image are comprehensively optimized to obtain the final point and line features on the current frame image, including:

[0013] Obtain the preliminary center point corresponding to n sets of target point and line features in the target point and line feature extraction results of the current frame image; wherein, the target point and line feature extraction results of the current frame image include n sets of target point and line features, each frame image corresponds to a set of target point and line features, and each set of target point and line features includes several target point and line features; the preliminary center point is the mean point of the coordinates of the n sets of target point and line features.

[0014] The Z-score of each target point line feature in each group of target point line features is obtained based on the coordinates of each group of target point line features and the coordinates of the preliminary center point.

[0015] Select the target point and line features corresponding to Z scores whose absolute values ​​are less than or equal to the second preset value, and use the preliminary center points of all selected target point and line features as the final point and line features on the current frame image.

[0016] Preferably, the Z-score of each target point / line feature in each group of target point / line features is obtained by the following formula:

[0017]

[0018] In the formula, (x α ,y α (α=1,2,…,n) represents the coordinates of a target point / line feature in the α-th group of n target point / line features, (xc ,y c The coordinates of the initial center point are obtained by calculating the mean of the characteristic coordinates of n sets of target point lines, μ. d σ d Z represents the mean and standard deviation of the distances from the coordinates of the n sets of target point line features to the initial center point, respectively. α Z-score for each target point line feature.

[0019] Preferably, the pose calculation equation is established by the following formula:

[0020]

[0021] In the formula, K is the camera intrinsic parameter matrix. Let be the rotation matrix from the world coordinate system to the camera coordinate system. Let the point be the point where the target edges intersect in the image coordinate system and disappear. Let d1 and d2 be the vanishing points of the target's starting line in the image coordinate system, respectively, and let a1 and a2 be the vanishing points. The proportionality factor.

[0022] Preferably, the pitch angle, yaw angle, and roll angle of the aircraft are obtained by the following formula:

[0023]

[0024] In the formula, θ, ψ, and γ represent the pitch angle, yaw angle, and roll angle of the aircraft, respectively. for The element in the i-th row and j-th column of the matrix, where i = 1, 2, 3 and j = 1, 2, 3.

[0025] Preferably, the lateral displacement, forward displacement, and vertical displacement of the aircraft relative to the target are obtained by the following formulas:

[0026]

[0027] In the formula, These represent the lateral displacement, forward displacement, and vertical displacement of the aircraft relative to the target, α. left The scaling factor for the left side of the target, w is the target width, and l is the target length. lx l lz These are the x and y coordinates of the left edge of the target in the image coordinate system, respectively. sx l sy These are the x and y coordinates of the target's starting line in the image coordinate system, respectively. ij for The corresponding element in the i-th row and j-th column of the matrix, i = 1, 2, 3, j = 1, 2, 3, K* It is the cofactor matrix of the camera intrinsic parameter matrix.

[0028] By applying the technical solution of this invention, images inferred by a target feature recognition network are grouped in the form of a sliding window. A deep learning network is then used to extract and register features between the grouped historical frame images and the current frame, calculating the corresponding homography matrix. Point and line features from the historical frame images are projected onto the current frame image, removing those with significant deviations. Finally, the optimized point and line features are used as the result for relative pose measurement. This method provides more accurate and robust relative position and pose information. Attached Figure Description

[0029] The accompanying drawings, which form part of this specification, are provided to further illustrate embodiments of the invention and, together with the textual description, explain the principles of the invention. It is obvious that the drawings described below are merely some embodiments of the invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0030] Figure 1 A flowchart of a relative pose measurement method based on a sequence of frame images according to an embodiment of the present invention is shown;

[0031] Figure 2 A schematic diagram of a sequence frame image grouping provided according to an embodiment of the present invention is shown. Detailed Implementation

[0032] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0034] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0035] like Figure 1 As shown, the present invention provides a relative pose measurement method based on a sequence of frame images, the method comprising:

[0036] During the relative position and attitude measurement of the aircraft, infrared images of the target are acquired in real time using an infrared camera, and inertial navigation system is used to acquire the inertial attitude information of the aircraft in real time.

[0037] The acquired images are grouped in the form of sliding windows, such as Figure 2 As shown; each group includes n consecutive frames of images, the previous n-1 frames are historical frames, and the nth frame is the current frame; a feature recognition network is used to recognize each frame of images to obtain the target point and line feature coordinates; the target point and line features include the target's edge line, center line, and corner points;

[0038] A super-point image feature extraction network is used to extract image features from each group of n frames, resulting in n extracted image features. A super glue feature registration network is used to register the extracted image features from the previous n-1 frames with the extracted image features from the nth frame, obtaining the registration relationship between the historical frame images and the current frame images. A random sample consensus algorithm is used to obtain the first homography matrix based on the registration relationship between the historical frame images and the current frame images. The second homography matrix is ​​obtained based on the inertial pose information of the aircraft.

[0039] Determine whether the pixel coordinate distance between the target point line feature projection results of the first homography matrix and the second homography matrix is ​​less than a first preset value. If so, use the first homography matrix as the final homography matrix; otherwise, use the second homography matrix as the final homography matrix.

[0040] Based on the target point and line feature coordinates and the final homography matrix of each frame image, the target point and line features on the previous n-1 frames are projected onto the current frame image to obtain the target point and line feature extraction results on the current frame image. The target point and line feature extraction results on the current frame image are then comprehensively optimized to obtain the final point and line features on the current frame image.

[0041] Based on the final point and line features on the current frame image, establish the straight line equations of the three lines on the image: the left line of the target, the starting line of the target, and the right line of the target. Based on the straight line equations, obtain the disappearance points of the left line of the target and the right line of the target on the image coordinate system, and the disappearance points of the starting line of the target on the image coordinate system.

[0042] Based on the vanishing points of the left and right lines of the target in the image coordinate system, the vanishing points of the target's starting line in the image coordinate system, the camera intrinsic parameter matrix, the rotation matrix from the world coordinate system to the camera coordinate system, the unit direction vector of the target's edge line, and the unit direction vector of the target's starting line, a pose calculation equation is established. The pose calculation equation is then used to calculate the pitch angle, heading angle, roll angle, and the lateral, forward, and vertical displacements of the aircraft relative to the target.

[0043] To gain a further understanding of the present invention, the following description is provided in conjunction with... Figure 1 and Figure 2 The relative pose measurement method based on sequential frame images of the present invention will be described in detail.

[0044] In this embodiment, the relative pose measurement method based on a sequence of frame images is implemented through the following four processes, such as... Figure 1 As shown:

[0045] (1) Image preprocessing:

[0046] Flight data is collected via an aircraft. The aircraft is equipped with a strapdown inertial navigation system (SINS), an infrared camera, and a data processing unit. The SINS is used to acquire inertial attitude information; the forward-looking infrared (FLIR) camera's optical axis is parallel to the longitudinal axis of the carrier and is used to acquire infrared images; the data processing unit is installed inside the infrared camera and is used to store and process the data collected by the various sensors, performing image recognition, feature extraction and registration, and visual relative pose calculation.

[0047] The image sequence is grouped using a sliding window approach, with each group consisting of n consecutive frames. The first n-1 frames of each group represent the sequence's history, and the nth frame is the current frame. Taking n=6 as an example, the images are recursively grouped into groups of 6 frames, as illustrated in the diagram below. Figure 2 As shown, each frame of the image is inferred by a feature recognition network to obtain the coordinates of the target features on each frame. The target features include edges, center lines, and corner points.

[0048] (2) Homography matrix calculation:

[0049] To establish the homography matrix between sequential frames, a super-point image feature extraction network is used to extract image features from the sequential frame images, and a super-glue feature registration network is used to establish the registration relationship between image features. Feature extraction and registration are performed on the current frame and the previous n-1 historical frames during the acquisition process to obtain their registration relationships. After establishing the registration relationship between the sequential historical frame images and the current frame image, the random sample consensus algorithm is used to solve for the homography matrix. The random sample consensus algorithm uses all the feature point pairs of the registered image output by the registration network to calculate the homography matrix between the sequential historical frames and the current frame.

[0050] Inertial information is introduced because the time interval between sequential frames is short, and the accuracy of inertial information is high. The image registration results can be evaluated based on the continuity of inertial information. For ease of distinction, the homography matrix calculated from the image is called the first homography matrix, and the homography matrix calculated using inertial information is called the second homography matrix. When the pixel coordinate distance of the projection results of point and line features in the first and second homography matrices is less than a preset value, the calculation result of the first homography matrix is ​​considered correct, and it is used for subsequent calculations; otherwise, the calculation result of the first homography matrix is ​​considered incorrect, and the second homography matrix is ​​selected as the homography matrix for subsequent calculations. Inertial incremental information is used to eliminate point and line features with large deviations caused by recognition errors or image registration errors.

[0051] (3) Optimization of target key features based on sequence frames:

[0052] Using the pixel coordinates of the point and line features obtained in each frame of the image preprocessing section, and the homography matrix calculated in the second part, the point and line features from the previous n-1 frames are projected onto the current frame image. Finally, the target features are comprehensively optimized and used as the feature extraction result for the current image. The comprehensive optimization mainly adopts the Z-score method, and the specific process is as follows: First, the preliminary center point of the point and line feature coordinates is calculated. The preliminary center point is the mean point of the point and line feature coordinates. Then, the Z-score of each point and line feature coordinate relative to the preliminary center point is calculated.

[0053] Since each frame of the image corresponds to a set of target point and line features, and a set of target point and line features includes features such as target edges, center lines, and corners, the formula uses one point and line feature from a set of target point and line features as an example, and the calculation method is as follows:

[0054]

[0055] In the formula, (x α ,y α (α=1,2,…,n) represents the coordinates of a target point / line feature in the α-th group of n target point / line features, (x c,y c The coordinates of the initial center point are obtained by calculating the mean of the characteristic coordinates of n sets of target point lines, μ. d σ d Z represents the mean and standard deviation of the distances from the coordinates of the n sets of target point line features to the initial center point, respectively. α Z-score for each target point line feature.

[0056] Finally, set the threshold k. Remove |Z α For points |> k, retain the remaining target point-line features. Then calculate the center coordinates of the remaining feature points as the final point-line features on the current frame image for subsequent calculations.

[0057] (4) Relative pose calculation:

[0058] Relative pose measurement is performed using the optimized features of the final target's left and right edges and the starting line. Based on the coordinates of the final point-line features obtained in step 3 on the image, and considering the relationship that two points lie on the same straight line on the target imaging plane, the linear equations of the target's left edge, starting line, and right edge on the image are established. 0i l 1i l 2i The corresponding coordinate in the world coordinate system is L. 0w L 1w L 2w The unit direction vectors of the target edge line and the target starting line are d1 and d2, respectively.

[0059] The left and right edges of the target intersect at the vanishing point in the image coordinate system. and Target starting line L 1w The disappearance point on the image coordinate system is Establish the pose calculation equation:

[0060]

[0061] In the formula, K is the camera intrinsic parameter matrix. Let be the rotation matrix from the world coordinate system to the camera coordinate system. Let the point be the point where the target edges intersect in the image coordinate system and disappear. Let d1 and d2 be the vanishing points of the target's starting line in the image coordinate system, respectively, and let a1 and a2 be the vanishing points. The proportionality factor.

[0062] The pitch angle, yaw angle, and roll angle of the aircraft can be obtained from the following formulas:

[0063]

[0064] In the formula, θ, ψ, and γ represent the pitch angle, yaw angle, and roll angle of the aircraft, respectively. for The element in the i-th row and j-th column of the matrix, where i = 1, 2, 3 and j = 1, 2, 3.

[0065] Based on the Plücker coordinates of the target's left and right edges and the starting line (with the direction vector d and moment m known), the lateral displacement of the aircraft relative to the target can be calculated after processing. Forward displacement With vertical displacement

[0066]

[0067] In the formula, These represent the lateral displacement, forward displacement, and vertical displacement of the aircraft relative to the target, α. left The scaling factor for the left side of the target, w is the target width, and l is the target length. lx l lz These are the x and y coordinates of the left edge of the target in the image coordinate system, respectively. sx l sy These are the x and y coordinates of the target's starting line in the image coordinate system, respectively. ij for The corresponding element in the i-th row and j-th column of the matrix, i = 1, 2, 3, j = 1, 2, 3, K * It is the cofactor matrix of the camera intrinsic parameter matrix.

[0068] In summary, this invention provides a relative pose measurement method based on a sequence of frame images. By calculating the homography matrix between the sequence of frame images, the projection of the target key point and line features identified on the sequence of frame images onto the current image is obtained. Through feature filtering, point and line features with higher accuracy and robustness are obtained, providing more accurate and robust relative position and pose information. This method can suppress misidentification and measurement result jumps caused by visual information, improve the stability of point and line feature extraction, and provide reliable information support.

[0069] For ease of description, spatial relative terms such as "above," "on top of," "on the upper surface of," "above," etc., are used herein to describe the spatial positional relationship of a device or feature as shown in the figures to other devices or features. It should be understood that spatial relative terms are intended to encompass different orientations in use or operation beyond the orientation of the device as described in the figures. For example, if the device in the figures were inverted, a device described as "above" or "on top of" other devices or structures would subsequently be positioned as "below" or "under" other devices or structures. Thus, the exemplary term "above" can include both "above" and "below." The device may also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatial relative descriptions used herein will be interpreted accordingly.

[0070] Furthermore, it should be noted that the use of terms such as "first" and "second" to define components is merely for the purpose of distinguishing the corresponding components. Unless otherwise stated, the above terms have no special meaning and therefore should not be construed as limiting the scope of protection of this invention.

[0071] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for relative pose measurement based on a sequence of frame images, characterized in that, The method includes: During the relative position and attitude measurement of the aircraft, infrared images of the target are acquired in real time using an infrared camera, and inertial navigation system is used to acquire the inertial attitude information of the aircraft in real time. The acquired images are grouped in the form of a sliding window; each group includes n consecutive frames, the previous n-1 frames are historical frames, and the nth frame is the current frame; a feature recognition network is used to recognize each frame to obtain the target point and line feature coordinates; the target point and line features include the target's edge line, center line, and corner points; A super-point image feature extraction network is used to extract image features from each group of n frames, resulting in n extracted image features. A super glue feature registration network is used to register the extracted image features from the previous n-1 frames with the extracted image features from the nth frame, obtaining the registration relationship between the historical frame images and the current frame images. A random sample consensus algorithm is used to obtain the first homography matrix based on the registration relationship between the historical frame images and the current frame images. The second homography matrix is ​​obtained based on the inertial pose information of the aircraft. Determine whether the pixel coordinate distance between the target point line feature projection results of the first homography matrix and the second homography matrix is ​​less than a first preset value. If so, use the first homography matrix as the final homography matrix; otherwise, use the second homography matrix as the final homography matrix. Based on the target point and line feature coordinates and the final homography matrix of each frame image, the target point and line features on the previous n-1 frames are projected onto the current frame image to obtain the target point and line feature extraction results on the current frame image. The target point and line feature extraction results on the current frame image are then comprehensively optimized to obtain the final point and line features on the current frame image. Based on the final point and line features on the current frame image, establish the straight line equations of the three lines on the image: the left line of the target, the starting line of the target, and the right line of the target. Based on the straight line equations, obtain the disappearance points of the left line of the target and the right line of the target on the image coordinate system, and the disappearance points of the starting line of the target on the image coordinate system. Based on the vanishing points of the left and right lines of the target in the image coordinate system, the vanishing points of the target's starting line in the image coordinate system, the camera intrinsic parameter matrix, the rotation matrix from the world coordinate system to the camera coordinate system, the unit direction vector of the target's edge line, and the unit direction vector of the target's starting line, a pose calculation equation is established. The pose calculation equation is then used to calculate the pitch angle, heading angle, roll angle, and the lateral, forward, and vertical displacements of the aircraft relative to the target.

2. The method according to claim 1, characterized in that, The target point and line feature extraction results on the current frame image are comprehensively optimized to obtain the final point and line features on the current frame image, including: Obtain the preliminary center point corresponding to n sets of target point and line features in the target point and line feature extraction results on the current frame image; wherein, the target point and line feature extraction results on the current frame image include n sets of target point and line features, each frame image corresponds to a set of target point and line features, and each set of target point and line features includes several target point and line features; The initial center point is the mean point of the characteristic coordinates of the n sets of target point lines; The Z-score of each target point line feature in each group of target point line features is obtained based on the coordinates of each group of target point line features and the coordinates of the preliminary center point. Select the target point and line features corresponding to Z scores whose absolute values ​​are less than or equal to the second preset value, and use the preliminary center points of all selected target point and line features as the final point and line features on the current frame image.

3. The method according to claim 1 or 2, characterized in that, The Z-score of each target point line feature in each group of target point line features is obtained using the following formula: In the formula, (x α ,y α (α=1,2,…,n) represents the coordinates of a target point / line feature in the α-th group of n target point / line features, (x c ,y c The coordinates of the initial center point are obtained by calculating the mean of the characteristic coordinates of n sets of target point lines, μ. d σ d Z represents the mean and standard deviation of the distances from the coordinates of the n sets of target point line features to the initial center point, respectively. α Z-score for each target point line feature.

4. The method according to any one of claims 1-3, characterized in that, The pose calculation equation is established using the following formula: In the formula, K is the camera intrinsic parameter matrix. Let be the rotation matrix from the world coordinate system to the camera coordinate system. Let the point be the point where the target edges intersect in the image coordinate system and disappear. Let d1 and d2 be the vanishing points of the target's starting line in the image coordinate system, respectively, and let a1 and a2 be the vanishing points. The proportionality factor.

5. The method according to claim 1, characterized in that, The pitch angle, yaw angle, and roll angle of the aircraft can be obtained from the following formulas: In the formula, θ, ψ, and γ represent the pitch angle, yaw angle, and roll angle of the aircraft, respectively. for The element in the i-th row and j-th column of the matrix, where i = 1, 2, 3 and j = 1, 2, 3.

6. The method according to claim 1, characterized in that, The lateral, forward, and vertical displacements of the aircraft relative to the target are obtained using the following formulas: In the formula, These represent the lateral displacement, forward displacement, and vertical displacement of the aircraft relative to the target, α. left The scaling factor for the left side of the target, w is the target width, and l is the target length. lx l lz These are the x and y coordinates of the left edge of the target in the image coordinate system, respectively. sx l sy These are the x and y coordinates of the target's starting line in the image coordinate system, respectively. ij for The corresponding element in the i-th row and j-th column of the matrix, i = 1, 2, 3, j = 1, 2, 3, K * It is the cofactor matrix of the camera intrinsic parameter matrix.

Images