Custom view angle image generation method based on single real shot image and pose information

Abstract

The invention provides a self-defined visual angle image generation method based on a single real shot image and pose information. The method comprises the following steps: shooting an original image at the current moment by using a camera installed on an aircraft; obtaining the pose information of the aircraft at the current moment by using an inertial navigation system; obtaining the vertical distance from the center of the camera to the ground at the current moment based on the height of the aircraft at the current moment and the altitude of the aircraft on the ground; acquiring a rotation matrix from a carrier coordinate system to a navigation coordinate system at the current moment based on a roll angle, a pitch angle and a yaw angle of the aircraft at the current moment; converting the ground normal vector twice to obtain a unit normal vector of the imaging plane in a camera coordinate system; obtaining a homography matrix between the target view angle image and the original image; and performing homography transformation on the original image based on the homography matrix between the target view angle image and the original image, and generating an analog image under the user-defined view angle.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and image processing technology, and in particular to a method for generating custom viewpoint images based on a single real-world image and pose information. Background Technology

[0002] Aircraft vision systems play a central role in remote sensing and mapping, target detection and tracking, and autonomous navigation. However, developing high-performance intelligent algorithms (such as deep learning-based detection models) heavily relies on massive amounts of image samples covering multiple viewpoints and poses. In actual flight, acquiring sufficiently diverse airborne image data for the same scene is costly, time-consuming, and limited by various factors such as airspace and weather, making it impractical.

[0003] To address the problem of data scarcity, academia and industry have developed several virtual image generation technologies.

[0004] Firstly, there are view synthesis techniques based on 3D scene reconstruction. For example, classic techniques like structure-of-motion motion reconstruction and multi-view stereo capture reconstruct a dense 3D point cloud or mesh model of the scene by acquiring multiple images of the same scene, and then using this model to render images from any new perspective. However, the core drawback of such methods is that they rely on acquiring images of the same scene from multiple angles, which contradicts the initial goal of "expanding the dataset from a single image" and fails to fundamentally solve the cost problem of data acquisition.

[0005] Secondly, there are two-dimensional transformation techniques based on model or image registration. Simple affine or perspective transformations of images can quickly generate new perspectives, but they usually ignore the real geometric constraints between two images caused by camera motion, resulting in geometrically inaccurate images with obvious distortions, making them unusable as high-quality training samples.

[0006] Therefore, how to generate geometrically accurate and realistic multi-view simulation images using only a single airborne image and its easily obtainable aircraft pose information without relying on multi-image acquisition and complex 3D reconstruction is a technical challenge that urgently needs to be solved in the field of aircraft vision technology. Summary of the Invention

[0007] This invention provides a method for generating custom viewpoint images based on a single real-world image and pose information. It can quickly and accurately generate simulated images of the same scene after a specified viewpoint transformation using only a single aircraft image and its initial pose and altitude information without constructing a 3D model. This expands the diversity and quantity of remote sensing image samples in a low-cost and high-efficiency manner.

[0008] According to one aspect of the present invention, a method for generating a custom viewpoint image based on a single real-world image and pose information is provided, the method comprising:

[0009] The system uses a camera mounted on the aircraft to capture the raw image of the current moment; and uses an inertial navigation system to obtain the aircraft's current attitude information, which includes latitude, longitude, altitude, roll angle, pitch angle, and yaw angle.

[0010] The vertical distance from the camera center to the ground at the current moment is obtained based on the aircraft's current altitude and its altitude when it is on the ground.

[0011] The rotation matrix from the carrier coordinate system to the navigation coordinate system at the current moment is obtained based on the roll angle, pitch angle, and yaw angle of the aircraft at the current moment;

[0012] Based on the rotation matrix from the current vehicle coordinate system to the navigation coordinate system and the transformation matrix from the vehicle coordinate system to the camera coordinate system, the ground normal vector is transformed twice to obtain the unit normal vector of the imaging plane in the camera coordinate system.

[0013] Based on the vertical distance from the camera center to the ground at the current moment, the unit normal vector of the imaging plane in the camera coordinate system, the camera intrinsic parameter matrix, the expected rotation matrix and the expected translation vector required to transform to the desired target pose, the homography matrix between the target view image and the original image is obtained;

[0014] Based on the homography matrix between the target view image and the original image, a homography transformation is performed on the original image to generate a simulated image under a custom viewpoint.

[0015] Preferably, the navigation coordinate system adopts the "North-Sky-East" coordinate system, with the origin at the center of the inertial navigation system mounting plane of the aircraft, X... n The axis is positive when it points north in the horizontal plane; Z n The axis is positive when it points east in the horizontal plane; Y n The axis is perpendicular to both, with the sky as the positive direction; the carrier coordinate system adopts a "front-up-right" coordinate system, with the origin at the center of the inertial navigation system mounting plane. b The axis is along the transverse axis of the carrier and forward is positive, Z b The axis is along the longitudinal axis of the carrier and is positive to the right. X b Y b Z b The axes are set to a right-handed coordinate system; the camera coordinate system is a front-view mounted coordinate system, using a "right-lower-front" coordinate system, Z... c The axis is along the optical axis and forward is positive, X c Y c Z c The axes form a right-handed coordinate system.

[0016] Preferably, the rotation matrix from the navigation coordinate system to the vehicle coordinate system at the current moment is obtained by the following formula:

[0017] R bn =R x (γ1)·R y (θ1)·R z (ψ1)

[0018] In the formula, R bn R is the rotation matrix from the navigation coordinate system to the vehicle coordinate system. x (γ1), R y (θ1), R z (ψ1) represents the roll angle rotation matrix, pitch angle rotation matrix, and yaw angle rotation matrix, respectively, and γ1, θ1, and ψ1 represent the roll angle, pitch angle, and yaw angle of the aircraft, respectively.

[0019] Preferably, the unit normal vector of the imaging plane in the camera coordinate system is obtained by the following formula:

[0020] N = R bc ·R bn ·n b

[0021] In the formula, N is the unit normal vector of the imaging plane in the camera coordinate system, and R bc Let n be the transformation matrix from the carrier coordinate system to the camera coordinate system. b This is the ground normal vector.

[0022] Preferably, the homography matrix between the target view image and the original image is obtained by the following formula:

[0023]

[0024] In the formula, H is the homography matrix between the target view image and the original image, K is the camera intrinsic parameter matrix, d is the vertical distance from the camera center to the ground, and ΔR and ΔT are the desired rotation matrix and desired translation vector required to transform to the desired target pose, respectively.

[0025] Preferably, the process of performing homography transformation on the original image based on the homography matrix between the target viewpoint image and the original image to generate a simulated image under a custom viewpoint includes:

[0026] Each pixel in the original image is mapped to its corresponding position in the target image using the homography matrix between the target view image and the original image;

[0027] The mapped image is filled and interpolated to obtain a simulated image from a custom viewpoint.

[0028] According to another aspect of the present invention, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the methods described above.

[0029] By applying the technical solution of this invention, the ground scene captured by the aircraft is idealized as a plane, and the inertial navigation data of the aircraft is cleverly combined with the imaging model of the camera. Through a rigorous coordinate transformation chain, the image homography matrix corresponding to the custom pose transformation is accurately calculated. Finally, the homography matrix is ​​used to complete a high-quality image viewpoint mapping transformation.

[0030] Compared with the prior art, the present invention has the following significant advantages:

[0031] 1. Low data generation cost and high efficiency: Only a single airborne image and its corresponding pose data are needed to generate multi-view images, completely eliminating the dependence on multi-angle flight acquisition or complex 3D reconstruction, and greatly reducing the cost and time of sample expansion.

[0032] 2. High geometric accuracy of generated images: This invention strictly follows the geometric model of camera imaging, and the calculated homography matrix accurately reflects the image mapping relationship brought about by the real pose transformation; compared with simple two-dimensional image transformation, the generated simulated image has a qualitative improvement in geometric consistency and is more suitable as training data for high-precision algorithms.

[0033] 3. The solution is robust and easy to implement: The computational chain of this invention is clear, and all input parameters (pose, altitude, camera intrinsic parameters) are standard data that can be directly obtained or calibrated by the aircraft and camera system. The method has good numerical stability and engineering feasibility, and is easy to integrate into the existing aircraft data processing chain. Attached Figure Description

[0034] The accompanying drawings, which form part of this specification, are provided to further illustrate embodiments of the invention and, together with the textual description, explain the principles of the invention. It is obvious that the drawings described below are merely some embodiments of the invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0035] Figure 1 A flowchart of a custom viewpoint image generation method based on a single real-shot image and pose information according to an embodiment of the present invention is shown;

[0036] Figure 2 A schematic diagram of the original image is shown;

[0037] Figure 3 A schematic diagram of the simulated image generated after the viewpoint change is shown. Detailed Implementation

[0038] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0040] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0041] like Figure 1 As shown, this invention provides a method for generating custom viewpoint images based on a single real-world image and pose information, specifically including the following steps:

[0042] Step S1: Coordinate system definition and parameter initialization

[0043] First, define the navigation coordinate system, the vehicle coordinate system, and the camera coordinate system.

[0044] Navigation coordinate system (n-frame, OX)n Y n Z n The coordinate system adopted is "North-Sky-East". The origin of the coordinate system is selected as the center of the inertial navigation system base plate mounting plane, X... n The axis is positive when it points north in the horizontal plane; Z n The axis is positive when it points east in the horizontal plane; Y n The axis is perpendicular to both, pointing to the sky as the center.

[0045] Carrier coordinate system (b system, OX) b Y b Z b ): A "front-up-right" coordinate system is adopted. The origin of the coordinate system is the center of the mounting plane of the aircraft's inertial navigation system base plate, X b The axis is along the transverse axis of the carrier and forward is positive, Z b The axis is along the longitudinal axis of the carrier and is positive to the right. X b Y b Z b The axes form a right-handed coordinate system.

[0046] Camera coordinate system (c-frame, OX-frame) c Y c Z c (): The camera is mounted in a front-view orientation and uses a "right-bottom-front" coordinate system. Z c The axis is along the optical axis and forward is positive, X c Y c Z c The axes form a right-handed coordinate system.

[0047] Subsequently, the following key parameters are read from or preset from the aircraft's flight control system:

[0048] • Camera intrinsic parameter matrix K;

[0049] • The camera's mounting matrix relative to the body, i.e., the transformation matrix R from the carrier coordinate system to the camera coordinate system. bc ;

[0050] • The altitude h of the aircraft at the takeoff point (ground) g ;

[0051] When capturing the current live image, the aircraft's real-time attitude information includes altitude h1 and attitude angles (roll angle γ1, pitch angle θ1, yaw angle ψ1).

[0052] Step S2: Calculate key geometric parameters

[0053] Based on the parameters obtained in step S1, calculate the following key geometric parameters:

[0054] Camera imaging plane distance: Calculate the vertical distance d = h1 - h from the camera center to the ground (i.e., the imaging plane).g The key here lies in cleverly obtaining the true "camera-to-ground distance" by using the ground altitude of the takeoff point as a reference, a value crucial for homography matrix calculation.

[0055] Normal vector of the imaging plane:

[0056] Under ideal ground conditions, the unit normal vector of the imaging plane (ground) in the carrier coordinate system is n. b =[0,-1,0] T .

[0057] Calculate the rotation matrix R from the vehicle coordinate system to the navigation coordinate system when the aircraft is capturing the current image. bn This matrix is ​​constructed from the real-time attitude angles (γ1, θ1, ψ1) of the aircraft according to the rotational sequence "ZYX", namely "yaw-pitch-roll". Its calculation formula is as follows:

[0058] R bn =R x (γ1)·R y (θ1)·R z (ψ1)

[0059] The components are:

[0060]

[0061] In the formula, R bn R is the rotation matrix from the navigation coordinate system to the vehicle coordinate system. x (γ1), R y (θ1), R z (ψ1) represents the roll angle rotation matrix, pitch angle rotation matrix, and yaw angle rotation matrix, respectively, and γ1, θ1, and ψ1 represent the roll angle, pitch angle, and yaw angle of the aircraft, respectively.

[0062] The ground normal vector n b Through R bn and R bc After two transformations, the final unit normal vector N of the imaging plane in the camera coordinate system is obtained. The calculation formula is as follows:

[0063] N = R bc ·R bn ·n b

[0064] In the formula, N is the unit normal vector of the imaging plane in the camera coordinate system, and R bc Let n be the transformation matrix from the carrier coordinate system to the camera coordinate system. b This is the ground normal vector.

[0065] Step S3: Calculate the homography matrix of the custom viewpoint transformation.

[0066] Define a custom, desired transformation parameter of the target pose relative to the current pose, which is described by a desired rotation matrix ΔR and a desired translation vector ΔT.

[0067] Using the parameters obtained in step S2 and the camera intrinsic parameter K from step S1, the homography matrix H between the target view image and the original image is calculated according to the following formula:

[0068]

[0069] in:

[0070] K is a known camera intrinsic parameter matrix;

[0071] (ΔR, ΔT) are user-defined or predefined viewpoint transformation parameters that define the relative motion of the "second camera" (virtual) relative to the "first camera" (real). ΔR and ΔT are the desired rotation matrix and desired translation vector required to transform to the desired target pose, respectively.

[0072] N is the unit normal vector of the imaging plane in the first camera coordinate system, calculated in step S2.

[0073] d is the vertical distance from the center of the first camera to the imaging plane, calculated in step S2.

[0074] The core advantage of this invention is that both N and d are precisely calculated using single-flight data and camera parameters, avoiding the cumbersome and error-prone nature of traditional methods that rely on multi-view or multi-stage estimation.

[0075] Step S4: Generate a simulated image

[0076] Based on the homography matrix H calculated in step S3, a homography transformation is performed on the original real-shot image. Each pixel in the original image is mapped to its corresponding position in the target image through H. After filling and interpolation, a simulated image under a custom viewpoint can be generated.

[0077] To gain a further understanding of the present invention, the following description is provided in conjunction with... Figures 1-3 The present invention provides a detailed description of the custom viewpoint image generation method based on a single real-shot image and pose information.

[0078] This embodiment uses a certain type of aircraft equipped with a forward-looking camera as an example, as shown in the attached diagram. Figure 1 The process shown is used to execute the method disclosed in this invention.

[0079] Step S1: Parameter Initialization

[0080] Let the altitude of the aircraft's takeoff point be h. g=344.98m. Photos attached. Figure 2 When the image shown appears, the real-time data recorded by the aircraft's flight control system is: altitude h1 = 654.63 m, roll angle γ1 = 0.010986. ° The pitch angle θ1 = 4.729614° and the yaw angle ψ1 = -25.136719°.

[0081] The camera intrinsic parameter matrix K is obtained after calibration:

[0082]

[0083] (Note: f) x =f y =19mm / 12μm≈1583.3 pixels, c x =1280 / 2=640, c y =1024 / 2=512).

[0084] The camera is mounted forward, and its mounting relationship with the aircraft (assuming the camera has no pitch or roll, only 90-degree yaw) results in a transformation matrix R from the carrier coordinate system to the camera coordinate system. bc for:

[0085]

[0086] (Note: This matrix is ​​for example only and should be calculated based on the actual installation angle.)

[0087] Step S2: Calculate key geometric parameters

[0088] Camera imaging plane distance: d = h1 - h g =654.63-344.98=309.65m.

[0089] Normal vector of the imaging plane:

[0090] First, calculate the rotation matrix R from the carrier coordinate system to the navigation coordinate system at the shooting time. bn It is the product of rotations around the Z-axis (yaw ψ), Y-axis (pitch θ), and X-axis (roll γ). Substituting the values ​​of (γ1, θ1, ψ1) into the calculation yields R. bn The specific numerical matrix.

[0091] 2. Ground normal vector in the loading system n b =[0,-1,0] T .

[0092] 3. Switch to the first camera system: N = R bc ·R bn ·n b The final normal vector N is obtained through calculation.

[0093] Step S3: Calculate the homography matrix of the custom viewpoint transformation.

[0094] In this embodiment, the custom viewpoint transformation is set as follows: the aircraft moves along its carrier coordinate system Z. b Translation of the axis (right) T = [0, 0, 100] T Meters, no rotation, i.e., ΔR is the identity matrix I, and ΔT in the camera frame is represented as ΔT c =R bc ·[0,0,100] T .

[0095] Let K,ΔR=I,ΔT c Substituting N and d into the core formula, the specific 3×3 homography matrix H is calculated.

[0096] Step S4: Generate a simulated image

[0097] Using the warpPerspective function from computer vision libraries such as OpenCV, Figure 2 Using the input image and H as the transformation matrix, the following image can be generated. Figure 3 The image shown is a simulated view after shifting the perspective 100 meters to the right.

[0098] In summary, this invention provides a method for generating custom viewpoint images based on a single real-world image and pose information. Compared with existing technologies, this invention has the following significant advantages:

[0099] 1. Low data generation cost and high efficiency: Only a single airborne image and its corresponding pose data are needed to generate multi-view images, completely eliminating the dependence on multi-angle flight acquisition or complex 3D reconstruction, and greatly reducing the cost and time of sample expansion.

[0100] 2. High geometric accuracy of generated images: This invention strictly follows the geometric model of camera imaging, and the calculated homography matrix accurately reflects the image mapping relationship brought about by the real pose transformation; compared with simple two-dimensional image transformation, the generated simulated image has a qualitative improvement in geometric consistency and is more suitable as training data for high-precision algorithms.

[0101] 3. The solution is robust and easy to implement: The computational chain of this invention is clear, and all input parameters (pose, altitude, camera intrinsic parameters) are standard data that can be directly obtained or calibrated by the aircraft and camera system. The method has good numerical stability and engineering feasibility, and is easy to integrate into the existing aircraft data processing chain.

[0102] For ease of description, spatial relative terms such as "above," "on top of," "on the upper surface of," "above," etc., are used herein to describe the spatial positional relationship of a device or feature as shown in the figures to other devices or features. It should be understood that spatial relative terms are intended to encompass different orientations in use or operation beyond the orientation of the device as described in the figures. For example, if the device in the figures were inverted, a device described as "above" or "on top of" other devices or structures would subsequently be positioned as "below" or "under" other devices or structures. Thus, the exemplary term "above" can include both "above" and "below." The device may also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatial relative descriptions used herein will be interpreted accordingly.

[0103] Furthermore, it should be noted that the use of terms such as "first" and "second" to define components is merely for the purpose of distinguishing the corresponding components. Unless otherwise stated, the above terms have no special meaning and therefore should not be construed as limiting the scope of protection of this invention.

[0104] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating a custom viewpoint image based on a single real-shot image and pose information, characterized in that, The method includes: The system uses a camera mounted on the aircraft to capture the raw image of the current moment; and uses an inertial navigation system to obtain the aircraft's current attitude information, which includes latitude, longitude, altitude, roll angle, pitch angle, and yaw angle. The vertical distance from the camera center to the ground at the current moment is obtained based on the aircraft's current altitude and its altitude when it is on the ground. The rotation matrix from the carrier coordinate system to the navigation coordinate system at the current moment is obtained based on the roll angle, pitch angle, and yaw angle of the aircraft at the current moment; Based on the rotation matrix from the current vehicle coordinate system to the navigation coordinate system and the transformation matrix from the vehicle coordinate system to the camera coordinate system, the ground normal vector is transformed twice to obtain the unit normal vector of the imaging plane in the camera coordinate system. Based on the vertical distance from the camera center to the ground at the current moment, the unit normal vector of the imaging plane in the camera coordinate system, the camera intrinsic parameter matrix, the expected rotation matrix and the expected translation vector required to transform to the desired target pose, the homography matrix between the target view image and the original image is obtained; Based on the homography matrix between the target view image and the original image, a homography transformation is performed on the original image to generate a simulated image under a custom viewpoint.

2. The method according to claim 1, characterized in that, The navigation coordinate system adopts the "North-Sky-East" coordinate system, with the origin at the center of the inertial navigation system mounting plane. n The axis is positive when it points north in the horizontal plane; Z n The axis is positive when it points east in the horizontal plane; Y n The axis is perpendicular to both, with the sky as the positive direction; the carrier coordinate system adopts a "front-up-right" coordinate system, with the origin at the center of the inertial navigation system mounting plane. b The axis is along the transverse axis of the carrier and forward is positive, Z b The axis is along the longitudinal axis of the carrier and is positive to the right. X b Y b Z b The axes are set to a right-handed coordinate system; the camera coordinate system is a front-view mounted coordinate system, using a "right-lower-front" coordinate system, Z... c The axis is along the optical axis and forward is positive, X c Y c Z c The axes form a right-handed coordinate system.

3. The method according to claim 1, characterized in that, The rotation matrix from the navigation coordinate system to the vehicle coordinate system at the current moment can be obtained using the following formula: R bn =R x (γ1)·R y (θ1)·R z (ψ1) In the formula, R bn R is the rotation matrix from the navigation coordinate system to the vehicle coordinate system. x (γ1), R y (θ1), R z (ψ1) represents the roll angle rotation matrix, pitch angle rotation matrix, and yaw angle rotation matrix, respectively, and γ1, θ1, and ψ1 represent the roll angle, pitch angle, and yaw angle of the aircraft, respectively.

4. The method according to claim 1, characterized in that, The unit normal vector of the imaging plane in the camera coordinate system can be obtained by the following formula: N=R bc ·R bn ·n b In the formula, N is the unit normal vector of the imaging plane in the camera coordinate system, and R bc Let n be the transformation matrix from the carrier coordinate system to the camera coordinate system. b This is the ground normal vector.

5. The method according to claim 1, characterized in that, The homography matrix between the target view image and the original image is obtained using the following formula: In the formula, H is the homography matrix between the target view image and the original image, K is the camera intrinsic parameter matrix, d is the vertical distance from the camera center to the ground, and ΔR and ΔT are the desired rotation matrix and desired translation vector required to transform to the desired target pose, respectively.

6. The method according to claim 1, characterized in that, Based on the homography matrix between the target view image and the original image, a homography transformation is performed on the original image to generate a simulated image under a custom viewpoint, including: Each pixel in the original image is mapped to its corresponding position in the target image using the homography matrix between the target view image and the original image; The mapped image is filled and interpolated to obtain a simulated image from a custom viewpoint.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 6.

Images