A Virtual Camera Image Reprojection Method Based on Aircraft Attitude

Abstract

This application provides a virtual camera image reprojection method based on aircraft attitude. The method includes: Step 1: Establishing a camera coordinate system, a pixel coordinate system, and a navigation coordinate system; projecting the image from the navigation coordinate system to the camera coordinate system, and then from the camera coordinate system to the pixel coordinate system; establishing the transformation relationship of the actual acquired image from the navigation coordinate system to the pixel coordinate system; and obtaining the coordinates u and v of the target point in the pixel coordinate system and the three-axis coordinates x in the navigation coordinate system. w y w z w The transformation relationship. Step 2: Based on the target point's coordinates u, v in the pixel coordinate system and the three-axis coordinates x in the navigation coordinate system. w y w z w The transformation relationship is used to obtain the mathematical formula for the pixel projection coordinates W of the image under the actual attitude of the aircraft; based on W, the mathematical formula for the pixel projection W' of the virtual camera view image under the ideal attitude of the aircraft is obtained. Step 3: Calculate W' based on the mathematical formula for the pixel projection W' of the virtual camera view image.

Description

Technical Field

[0001] This application relates to the field of flight control, specifically to a virtual camera image reprojection method based on aircraft attitude. Background Technology

[0002] Airborne forward-looking cameras are essential payloads for aircraft ground station command and control and airborne visual navigation. They are capable of image acquisition and transmission, supporting autonomous takeoff, landing, and flight under satellite-denied conditions. Due to factors such as the perspective requirements of ground pilots and cost and reliability considerations, aircraft forward-looking cameras are usually directly mounted to the fuselage. The camera's field of view changes with the aircraft's attitude, making it difficult for the aircraft to directly control its autonomous flight trajectory based on image changes under visually guided flight. Summary of the Invention

[0003] The technical problem to be solved by this invention is to propose a virtual ideal camera image reprojection method based on aircraft attitude. This method completes the reprojection of virtual camera view image under the ideal attitude of the aircraft by using the real-time attitude of the aircraft and actual image information, thereby decoupling attitude and trajectory information during image changes and eliminating the influence of dynamic changes in aircraft attitude on the image.

[0004] Invention technical solution: A virtual camera image reprojection method based on aircraft attitude, the method comprising:

[0005] Step 1: Establish the camera coordinate system, pixel coordinate system, and navigation coordinate system. Project the image from the navigation coordinate system onto the camera coordinate system, and then project it from the camera coordinate system back onto the pixel coordinate system. Establish the transformation relationship between the actual acquired image and the navigation coordinate system and the pixel coordinate system. Obtain the target point's coordinates u and v in the pixel coordinate system and its three-axis coordinates x in the navigation coordinate system. w y w z w The transformation relationship;

[0006] Step 2: Based on the target point's coordinates u and v in the pixel coordinate system and its three-axis coordinates x in the navigation coordinate system w y w z w The transformation relationship is used to obtain the mathematical formula for the pixel projection coordinates W of the image under the actual attitude of the aircraft; based on W, the mathematical formula for the pixel projection W' of the virtual camera view image under the ideal attitude of the aircraft is obtained.

[0007] Step 3: Calculate W' based on the mathematical formula of the pixel projection W' of the virtual camera view image.

[0008] Preferably, step 1 specifically includes:

[0009] Step 11: Establish the camera coordinate system, pixel coordinate system, and navigation coordinate system;

[0010] Step 12: Collect the aircraft's actual pitch angle θ, actual roll angle φ, actual yaw angle γ, and camera intrinsic parameter matrix K;

[0011] Step 13: Calculate the actual attitude rotation matrix R of the aircraft based on the actual pitch angle θ, the actual roll angle φ, and the actual yaw angle γ of the aircraft.

[0012] Step 14: Preset the three-axis coordinates (x, y) of the target point in the navigation coordinate system. w y w z w Preset camera's three-axis coordinates (x, y) in the navigation coordinate system f y f z f ;

[0013] Step 15: Based on the three-axis coordinates x w y w z w and three-axis coordinates x f y f z f Obtain the target point's coordinates u and v in the pixel coordinate system and its three-axis coordinates x in the navigation coordinate system. w y w z w The transformation relationship.

[0014] Preferably, step 13 specifically includes:

[0015] Calculate the actual attitude rotation matrix of the aircraft using the following formula.

[0016]

[0017] Preferably, step 15 specifically includes:

[0018] Based on the three-axis coordinates x w y w z w and three-axis coordinates x f y f z f Using the formula Obtain the target point's coordinates u and v in the pixel coordinate system and its three-axis coordinates x in the navigation coordinate system. w y w z w The transformation relationship.

[0019] Preferably, step 2 includes:

[0020] Based on the target point's coordinates u and v in the pixel coordinate system and its three-axis coordinates x in the navigation coordinate system w y wz w The transformation relationship is used to obtain the pixel projection coordinates W = [uv1] of the image under the actual aircraft attitude. T The mathematical expression;

[0021] According to W, using the formula

[0022] The mathematical formula for obtaining the pixel projection W' of the virtual camera view image under the ideal attitude of the aircraft is obtained, where the virtual ideal attitude of the aircraft image navigation is [θ'φ'γ'], θ' is the ideal pitch angle of the aircraft, φ' is the ideal roll angle of the aircraft, γ' is the ideal yaw angle of the aircraft, and the corresponding ideal attitude rotation matrix is ​​R'.

[0023] Preferably, step 3 includes:

[0024] Step 31: Let A = R T K -1 W determines A(1). If A(1) ≠ 0, proceed to step 32; if A(1) = 0, proceed to step 33.

[0025] Step 32: According to the formula Calculate α1; according to the formula Calculate β1; based on α1 and β1, calculate W';

[0026] Step 33: According to the formula Calculate α²; according to the formula Calculate β2; based on α2 and β2, calculate W'.

[0027] Preferably, step 32 specifically includes:

[0028] Based on α1 and β1, using the formula Calculate W'.

[0029] Preferably, step 33 specifically includes:

[0030] Based on α² and β², using the formula Calculate W'.

[0031] In summary, this application provides a virtual camera image reprojection method based on aircraft attitude, which realizes virtual camera image reprojection based on aircraft attitude, eliminates the influence of dynamic changes in aircraft attitude on the image, and the virtual camera image under the ideal aircraft attitude after reprojection is no longer affected by aircraft attitude maneuvering, and can be directly used for aircraft flight trajectory control.

[0032] Explanation of the attached diagram

[0033] Figure 1 A schematic diagram of virtual camera reprojection under the ideal attitude of the aircraft. Detailed Implementation

[0034] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0035] like Figure 1 As shown, this invention is a virtual camera image reprojection method based on aircraft attitude, comprising the following steps:

[0036] Step 1: Establish the camera coordinate system, pixel coordinate system, and navigation coordinate system. Project the image from the navigation coordinate system onto the camera coordinate system, and then project it from the camera coordinate system back onto the pixel coordinate system. Establish the transformation relationship between the actual acquired image and the navigation coordinate system and the pixel coordinate system. Obtain the target point's coordinates u and v in the pixel coordinate system and its three-axis coordinates x in the navigation coordinate system. w y w z w The transformation relationship.

[0037] Step 1 specifically includes:

[0038] Step 11: Establish the camera coordinate system, pixel coordinate system, and navigation coordinate system;

[0039] In the camera coordinate system: the origin is the camera, the x-axis is the horizontal axis of the camera (positive to the right), the y-axis is the vertical axis of the camera (positive downwards), and the z-axis is the optical axis of the camera (positive forwards).

[0040] Pixel coordinate system: The origin is the top left corner of the image, the x-axis is the horizontal axis of the image (positive to the right), and the y-axis is the vertical axis of the image (positive downwards).

[0041] Navigation coordinate system: The origin is the center of the runway starting line, the x-axis points east, the y-axis points north, and the z-axis points upward.

[0042] Step 12: Collect the aircraft's actual pitch angle θ, actual roll angle φ, actual yaw angle γ, and camera intrinsic parameter matrix K;

[0043] Step 13: Calculate the actual attitude rotation matrix R of the aircraft based on the actual pitch angle θ, the actual roll angle φ, and the actual yaw angle γ of the aircraft.

[0044] Specifically, the actual attitude rotation matrix of the aircraft is calculated using the following formula.

[0045]

[0046] Step 14: Preset the three-axis coordinates (x, y) of the target point in the navigation coordinate system. w y w z w Preset camera's three-axis coordinates (x, y) in the navigation coordinate system f y f z f ;

[0047] Step 15: Based on the three-axis coordinates x w y w z w and three-axis coordinates x f y f z f Using the formula Calculate the coordinates u and v of the target point in the pixel coordinate system.

[0048] It should be noted that, since the camera coordinate system is fixed to the aircraft, the coordinate projection from the navigation coordinate system to the camera coordinate system can be obtained from the aircraft's attitude.

[0049]

[0050]

[0051] Where, x c y c z c Let x be the three-axis coordinates of the target point in the camera coordinate system. w y w z w Let R be the three-axis coordinates of the target point in the navigation coordinate system, θ be the actual aircraft attitude rotation matrix, φ be the actual aircraft pitch angle, φ be the actual aircraft roll angle, γ be the actual aircraft yaw angle, and T be the coordinates of the origin of the navigation coordinate system in the camera coordinate system. Considering that the camera is the origin of the camera coordinate system, therefore...

[0052]

[0053] Where x f y f z f The coordinates of the camera are the three axes in the navigation coordinate system.

[0054] The projection of the camera coordinate system onto the pixel coordinate system can be calculated using the camera intrinsic parameter matrix, as shown in the following formula:

[0055]

[0056] Where u and v are the coordinates of the target point in the pixel coordinate system, and K is the camera intrinsic parameter matrix, which is obtained by calibrating the camera.

[0057] Substituting equations (1), (2), and (3) into equation (4), we obtain the projection of the target point from the navigation coordinate system to the pixel coordinate system as follows:

[0058]

[0059] Step 2: Based on the target point's coordinates u and v in the pixel coordinate system and its three-axis coordinates x in the navigation coordinate system w y w z w The transformation relationship is used to obtain the mathematical formula for the pixel projection coordinates W of the image under the actual attitude of the aircraft; based on W, the mathematical formula for the pixel projection W' of the virtual camera view image under the ideal attitude of the aircraft is obtained.

[0060] Specifically, step 2 includes:

[0061] Based on the target point's coordinates u and v in the pixel coordinate system and its three-axis coordinates x in the navigation coordinate system w y w z w The transformation relationship is used to obtain the pixel projection coordinates W = [uv1] of the image under the actual aircraft attitude. T The mathematical expression;

[0062] According to W, using the formula

[0063] The mathematical formula for obtaining the pixel projection W' of the virtual camera view image under the ideal attitude of the aircraft is obtained, where the virtual ideal attitude of the aircraft image navigation is [θ'φ'γ'], θ' is the ideal pitch angle of the aircraft, φ' is the ideal roll angle of the aircraft, γ' is the ideal yaw angle of the aircraft, and the corresponding ideal attitude rotation matrix is ​​R'.

[0064] Understandably, by setting the ideal attitude of the aircraft under image navigation, and based on the invariance of the image target point in the navigation coordinate system, a pixel projection mapping relationship is established between the virtual camera view image and the actual image under the ideal attitude of the aircraft.

[0065] At the same moment, the coordinates of the target point and the camera in the world coordinate system are unique and invariant. Therefore, the pixel projection mapping relationship between the virtual camera view image and the actual image can be obtained from formula (5):

[0066]

[0067]

[0068] Where W' is the pixel projection of the virtual camera view image under the ideal attitude of the aircraft.

[0069] Step 3: Calculate W' based on the mathematical formula of the pixel projection W' of the virtual camera view image.

[0070] Specifically, step 3 includes:

[0071] Step 31: Let A = R T K -1W determines A(1). If A(1) ≠ 0, proceed to step 32; if A(1) = 0, proceed to step 33.

[0072] Step 32: According to the formula Calculate α1; according to the formula Calculate β1; based on α1 and β1, use the formula Calculate W';

[0073] Step 33: According to the formula Calculate α²; according to the formula Calculate β²; based on α² and β², use the formula Calculate W'.

[0074] It should be noted that A(1) is the first element of vector A, A(2) is the second element of vector A, and A(3) is the third element of vector A.

[0075] Understandably, the relative position of the target point and the camera in the navigation coordinate system is obtained based on the actual image pixels. Then, the reprojection of the virtual camera view image is calculated by combining the actual view of the aircraft's real camera, the ideal view of the virtual camera, and the intrinsic parameter matrix of the real camera.

[0076] In formula (7), KR'R T K -1 W is calculated directly based on known variables, K is obtained through camera calibration, R' is calculated by setting the ideal aircraft attitude, R is calculated by acquiring real aircraft attitude data, and W is obtained by acquiring real images from the camera. Further calculation of the latter half is required. ratio.

[0077] Because the z-axis coordinate of the target point in the camera coordinate system is z c It will not be 0. Since it is a non-zero constant term, it can be seen from formula (6) that... With R T K -1 W represents the proportional relationship.

[0078] Let A = R T K -1 W, when A(1)≠0, let

[0079]

[0080] at this time

[0081]

[0082] When A(1) = 0, that is, the eastward coordinates of the camera and the target point are the same, but the northward coordinates will not be the same. Let

[0083]

[0084] at this time

[0085]

[0086] Substituting formula (9) or formula (11) into formula (7) yields the pixel projection W' of the virtual camera view image under the ideal aircraft attitude.

Claims

1. A method for virtual camera image re-projection based on aircraft attitude, characterized in that, The method comprises: Step 1: Establish the camera coordinate system, pixel coordinate system and navigation coordinate system, project to the camera coordinate system through the navigation coordinate system, and then project to the pixel coordinate system from the camera coordinate system, establish the conversion relationship of the actual collected image from the navigation coordinate system to the pixel coordinate system, and obtain the coordinates of the target point in the pixel coordinate system 、 and the conversion relationship of the three-axis coordinates in the navigation coordinate system 、 、 ​ Step 2: Based on the target point's coordinates in the pixel coordinate system , and three-axis coordinates in the navigation coordinate system , , The transformation relationship is used to obtain the pixel projection coordinates of the image under the actual attitude of the aircraft. The mathematical formula; according to the above Obtain virtual camera viewpoint image pixel projection under ideal aircraft attitude The mathematical formulas; specifically including: Based on the coordinates of the target point in the pixel coordinate system , and three-axis coordinates in the navigation coordinate system , , The transformation relationship is used to obtain the pixel projection coordinates W=[u,v,1] of the image under the actual aircraft attitude. T The mathematical expression; according to Using formula Obtain the pixel projection of the virtual camera view image under the ideal attitude of the aircraft. The mathematical formula, where the virtual ideal attitude of the aircraft image navigation is... , For the ideal pitch angle of the aircraft, For the ideal roll angle of the aircraft, The ideal yaw angle of the aircraft corresponds to the ideal attitude rotation matrix as follows: ; Step 3: Projecting the virtual camera perspective image pixels according to the mathematical formula ;​ Step 1 specifically comprises: Step 11: Establishing a camera coordinate system, a pixel coordinate system, and a navigation coordinate system; Step 12: Collect the actual pitch angle of the airplane , the actual roll angle of the airplane , the actual yaw angle of the airplane , the camera intrinsic matrix ; Step 13: calculating the actual pitch angle of the aircraft based on the actual pitch angle of the aircraft Step 14: calculating the actual roll angle of the aircraft based on the actual roll angle of the aircraft Step 15: calculating the actual yaw angle of the aircraft based on the actual yaw angle of the aircraft Step 16: calculating the actual attitude rotation matrix of the aircraft based on the actual pitch angle, the actual roll angle and the actual yaw angle of the aircraft ; Step 14: preset the three-axis coordinates of the target point in the navigation coordinate system , , ; preset the three-axis coordinates of the camera in the navigation coordinate system , , ; Step 15: Based on the three-axis coordinates , , and three-axis coordinates , , Obtain the coordinates of the target point in the pixel coordinate system. , and three-axis coordinates in the navigation coordinate system , , The transformation relationship.

2. The method of claim 1, wherein, Step 13 specifically comprises: The actual attitude rotation matrix of the aircraft is calculated by using the following formula 。 3. The method of claim 1, wherein, Step 15 specifically comprises: According to the three-axis coordinates , , and the three-axis coordinates , , , using the formula , obtain the coordinate of the target point in the pixel coordinate system 、 and the three-axis coordinate in the navigation coordinate system 、 、 the conversion relationship.

4. The method of claim 1, wherein, Step 3 comprises: Step 31: Let ,right Make a judgment when When, proceed to step 32; when Then, proceed to step 33; Step 32: Calculate according to the formula ; calculate according to the formula ; calculate and according to ; Step 33: Calculate ; according to the formula ; calculate ; according to the formula ; calculate and ; calculate .

5. The method of claim 4, wherein, Step 32 specifically comprises: According to and , the calculation is made using the formula .

6. The method of claim 4, wherein, Step 33 specifically comprises: According to and , with the formula , compute .

Images