Monocular vision-based flight target trajectory reconstruction method and system

Abstract

The application discloses a monocular vision-based flight target orbit reconstruction method and system, and mainly solves the problems of high device dependence, high sampling complexity and low orbit reconstruction authenticity in the prior art flight target orbit reconstruction. The implementation scheme comprises the following steps: three rotational degrees of freedom of a target are divided into two degrees of freedom perpendicular to the camera sight direction and one degree of freedom along the camera sight direction; the accurate values of the three degrees of freedom are respectively determined through multi-round iterative rotation sampling and comparison with the original image, and then the rotation matrix of the target is calculated; the distance from the camera to the target is calculated by using the camera imaging principle, and the spatial vector of the target in the camera coordinate system is determined in combination with the position of the target in the image, and then the spatial vector is converted into the world coordinate system; and the accurate pose of the target is determined according to the spatial vector and the rotation matrix. The application can realize the automation of flight target orbit reconstruction, effectively reduce the device dependence, reduce the sampling complexity, and has high authenticity of the obtained result, and can be used for three-dimensional engine scene simulation.

Description

Technical Field

[0001] This invention belongs to the field of aerospace technology, and further relates to a method and system for reconstructing the trajectory of a flight target. It can be used for three-dimensional engine scene simulation, and reconstructs the flight trajectory of the target in the simulation scene by using pre-experimental data. Background Technology

[0002] In the process of reconstructing the trajectory of a flight target, the core task is model matching. The aim is to calculate the target's rotation matrix and position vector in three-dimensional space by comparing the similarity between an existing 3D model of the flight target and image data acquired from a monocular camera. This process typically involves extracting feature points from the image and matching these feature points to determine the relative position and attitude between the camera and the target. Since monocular vision systems can only provide two-dimensional image information, effectively recovering the target's 3D trajectory from this two-dimensional data is a major challenge in this field.

[0003] Model matching not only has wide applications in the aerospace field but also demonstrates significant value in several other areas. In autonomous driving, using monocular vision for target tracking and trajectory prediction enables real-time perception and decision support of the surrounding environment, improving the safety and reliability of autonomous driving systems. Furthermore, in robotics, monocular vision is widely used for navigation and obstacle avoidance, aiding robots in real-time localization and path planning through accurate trajectory reconstruction. In addition, monocular vision trajectory reconstruction technology has shown great application potential in other high-precision positioning scenarios such as object detection and recognition, and drone monitoring.

[0004] Existing technologies for calculating rotation matrices employ either single-round coarse sampling with local interpolation or multi-round progressive optimization. Both methods involve extracting images of the target from multiple angles and matching them with the target image. The former uses single-round sampling with local interpolation for coarse estimation, while the latter uses multi-round sampling for progressive approximation. Existing technologies for calculating position vectors include triangulation and trilateration. Triangulation is a positioning method based on angle observation, measuring the angles between the target and two or more known observation points, and then using geometric relationships to calculate the target's position. Trilateration measures the distances between the target and multiple known reference points, combining this with geometric methods to determine the target's three-dimensional coordinates.

[0005] Patent application CN 201810911836.5 discloses a panoramic stitching method and system based on multi-camera calibration. The main steps of this method are: internal parameter calibration and distortion coefficient calculation for each camera; extrinsic parameter calibration data acquisition for each pair of adjacent cameras; corner detection after distortion correction of paired checkerboard images; RANSAC algorithm for inlier selection and estimation of the initial homography matrix between adjacent cameras, and calculation of the initial rotation matrix based on the homography matrix and internal parameters; and global optimization of the rotation matrix between all cameras using the LM algorithm to obtain the final optimized result. This system utilizes a global optimization method to optimize the extrinsic parameters of all cameras, eliminating the cumulative error of stitching parameters in traditional methods. Furthermore, this method does not rely on feature point information in the scene, but achieves stitching by calibrating the geometric relationships between cameras, avoiding the need for feature point extraction and matching for each stitch, thus improving stitching speed. Moreover, this method achieves excellent stitching results even with few feature points or low overlap. However, this method heavily relies on precise intrinsic and extrinsic parameter calibration for each camera. The camera's intrinsic parameters and distortion coefficients can change due to environmental variations such as temperature and humidity, or mechanical variations within the camera itself, such as lens loosening. Therefore, the calibration results need to be updated periodically, increasing maintenance costs. Furthermore, since this method assumes the camera array's position is fixed and the relative positions of the cameras do not change during the stitching process, recalibration is required if the cameras move or vibrate during use, thus placing high demands on the hardware. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of the prior art by proposing a method and system for reconstructing the trajectory of a flight target based on monocular vision, thereby reducing dependence on equipment, lowering sampling complexity, and improving the realism of trajectory reconstruction.

[0007] To achieve the above objectives, the technical solution of the present invention includes:

[0008] 1. A method for reconstructing the trajectory of a flight target based on monocular vision, characterized in that:

[0009] (1) Initialize the 3D engine and adjust the positional relationship between the target and the camera to ensure that the target is fully projected into the camera's field of view;

[0010] (2) The target rotation 3 degrees of freedom are split into 2+1;

[0011] (3) Read a frame of the original image in sequence, perform multiple rounds of iterative rotation sampling of the target around the first two degrees of freedom perpendicular to the camera's viewing direction in the 3D engine, and compare it with the original image to determine the precise values ​​of these two degrees of freedom.

[0012] (4) Perform multiple rounds of iterative rotation sampling of the target around the camera's viewing direction and compare it with the original image to determine the precise value of the third degree of freedom;

[0013] (5) Calculate the rotation matrix of the target based on the numerical results of the three rotational degrees of freedom:

[0014] (6) Based on the camera imaging principle, the distance from the camera to the target plane is calculated by using the proportional relationship that the distance from the camera to the target plane is proportional to the ratio of the camera focal length and the ratio of the actual size of the target to the image size; the spatial vector of the target in the camera coordinate system is determined by using the imaging position of the target in the original image and the distance from the camera to the target plane; the spatial vector of the target in the camera coordinate system is converted into a spatial vector in the world coordinate system by coordinate transformation in computer graphics.

[0015] (7) Use spatial vectors and rotation matrices to obtain the precise pose of the target's orbit in this frame;

[0016] (8) Repeat steps (3) to (7) to calculate the orbit pose of the target for all frames and complete the orbit reconstruction of the flight target.

[0017] Preferably, the step of splitting the three degrees of freedom of target rotation into 2+1 means splitting the three degrees of freedom of target rotation into two degrees of freedom perpendicular to the camera's viewing direction and one degree of freedom parallel to the target's viewing direction.

[0018] The two degrees of freedom perpendicular to the camera's viewing direction are the x-axis and y-axis of the world coordinate system;

[0019] One degree of freedom parallel to the direction the target is looking is the z-axis of the world coordinate system.

[0020] 2. A flight target trajectory reconstruction system based on monocular vision, characterized in that it comprises:

[0021] The system initialization module is used to initialize the 3D engine and provide the basic environment for subsequent sampling simulation.

[0022] The track frame extraction module is used to extract the image and camera parameters of each frame in track frame order;

[0023] The target pose matching module is used to iteratively sample and compare targets to determine their precise pose matrix.

[0024] The target position calculation module is used to calculate the actual position of the target in space and determine the target's precise position vector.

[0025] Compared with the prior art, the present invention has the following advantages:

[0026] First, since the raw data of this invention is acquired from a single camera, no complex equipment is required, thus reducing the dependence on equipment.

[0027] Secondly, this invention reduces the complexity of sampling by splitting the three degrees of freedom of target rotation into 2+1 and by using a multi-round progressive optimization method, thus ensuring the authenticity of the reconstruction results while reducing matching time.

[0028] Third, because the present invention calculates the position of the target from the perspective of camera imaging principle, it not only has the advantages of simple and efficient calculation process, but also ensures the authenticity and reliability of the target position calculation result by using this calculation based on physical laws. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating the implementation of the monocular vision-based flight target trajectory reconstruction method of the present invention.

[0030] Figure 2 This is a schematic diagram of the latitude and longitude sphere referenced for the target rotation in the method of this invention;

[0031] Figure 3 This is a schematic diagram comparing the target after it has been rotated and matched around the x-axis and y-axis in the method of this invention with the original image;

[0032] Figure 4 This is a schematic diagram comparing the target pose matching result with the original image in the method of this invention;

[0033] Figure 5 This is a schematic diagram of the camera imaging principle in the method of the present invention;

[0034] Figure 6 This is a schematic diagram comparing the final pose determined in the method of the present invention with the original image;

[0035] Figure 7 This is a block diagram of the flight target trajectory reconstruction system based on monocular vision according to the present invention. Detailed Implementation

[0036] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, other embodiments obtained by those skilled in the art without creative effort should all fall within the protection scope of the present invention.

[0037] Example 1: A method for reconstructing the trajectory of a flight target based on monocular vision.

[0038] This example adds the target model and camera to the 3D engine, adjusts the camera parameters to match those of the actual shooting camera, adjusts the target's rotation posture by rotating and sampling frame by frame, and obtains the target's rotation angle by comparing the camera screenshot with the original image. Based on the target's imaging position and size in the graphics, the target's position vector is calculated using camera imaging principles and computer graphics methods.

[0039] Reference Figure 1 The implementation steps of this example include the following:

[0040] Step 1: Initialize the 3D engine and adjust the positional relationship between the target and the camera.

[0041] This example performs target pose matching in a 3D engine. To obtain accurate target pose results, it is necessary to load the accurate model of the target and adjust the camera parameters in the 3D engine to be consistent with the parameters of the actual shooting camera. In addition, to ensure that the target is a complete image within the camera's field of view, the distance between the target and the camera needs to be adjusted appropriately.

[0042] The aforementioned 3D engine refers to abstracting real-world objects into polygons or various curves, constructing a virtual 3D world in the computer, and displaying it on the screen. The specific implementation of this step includes the following:

[0043] 1.1) Adjust the camera field of view and imaging pixel values ​​in the 3D engine to make them consistent with those of a real camera;

[0044] 1.2) Place the target at the origin, and position the camera in the positive z-axis direction, looking towards the origin, to facilitate target rotation;

[0045] 1.3) Adjust the distance between the target and the camera so that the target forms a complete image within the camera's field of view:

[0046] 1.3.1) Calculate the camera's lateral field of view (fovx) based on the longitudinal field of view (fovy) value provided by the camera:

[0047]

[0048] Where width c and height c These are the width and height of the camera's field of view, respectively;

[0049] 1.3.2) Based on the target's width in the x-axis direction t Calculate the camera distance d when the target occupies half of the camera's horizontal field of view. camx :

[0050]

[0051] 1.3.3) Based on the target's length (height) in the y-axis direction t Calculate the camera distance d when the target occupies half of the camera's longitudinal field of view. camy :

[0052]

[0053] 1.3.4) Take d camx With d camy The maximum value is used to adjust the distance d between the camera and the origin. cam To ensure that the target forms a complete image within the camera's field of view:

[0054] d cam =max(d camx ,d camy ).

[0055] Step 2: Obtain the raw frame data of the track.

[0056] Read one frame of the original image sequentially;

[0057] When the image was captured, the camera was facing f. c The upward direction u and the camera position p.

[0058] Step 3: Adjust the target to rotate around the x-axis and y-axis, so that each face of the target faces the camera. Perform multiple rounds of iterative rotation sampling and compare with the original image to determine the precise values ​​of these two degrees of freedom.

[0059] To rationally manipulate target rotation and range compression, the concept of a latitude and longitude sphere is borrowed. The intersections of latitude and longitude lines are rotated to the positive z-axis, and latitude and longitude are converted to Euler angles for rotation sampling. Image comparison uses SIFT to extract feature points, and quantity and geometric consistency are used as evaluation metrics. The latitude and longitude sphere is like... Figure 2 As shown.

[0060] The specific implementation of this step includes the following:

[0061] 3.1) Rotate the target around the x-axis and y-axis for sampling:

[0062] 3.1.1) Let the initial sampling range of latitude be [0, 180], the range of longitude be [0, 360], the step size be step, the longitude of the intersection of latitude and longitude lines be θ, the latitude be φ, and the radius be r = 1. Calculate the Cartesian coordinate system:

[0063]

[0064] Transform the Cartesian coordinate system into Euler angles:

[0065]

[0066] Where roll, pitch, and yaw are the rotation angles around the x, y, and z axes of the target coordinate system, respectively;

[0067] 3.1.2) Rotate the target according to Euler angles, and rotate all intersection points around the x-axis and y-axis of the world coordinate system to the positive z-axis direction in turn:

[0068] 3.1.3) After each rotation, use the 3D engine to capture a screenshot to obtain a sampled image;

[0069] 3.2) Use the SIFT algorithm to extract feature points from all sampled images and the original image;

[0070] 3.3) The feature points of all sampled images and the original image are matched using Euclidean distance. Cross-matching is enabled during the matching process to obtain multiple sets of matching point pairs.

[0071] 3.4) Compare the sampled image with the original image using the matching point ratio and inlier ratio, and select the optimal comparison result:

[0072] 3.4.1) Calculate the matching point ratio R match :

[0073]

[0074] 3.4.2) By filtering matching points through geometric consistency and affine transformation, matching points that do not conform to geometric constraints are removed, and the number of interior points of the matching points is obtained;

[0075] 3.4.3) Calculate the interior point ratio to measure R inlier :

[0076]

[0077] 3.4.4) According to R match With R inlier Calculate the comparison index R XY :

[0078] R XY =0.7·R match +0.3·R inlier ;

[0079] 3.4.5) Calculate R for all sampled images and the original image. XY Select R XY The latitude and longitude (lon, lat) corresponding to the largest image is used as the result of this round of matching;

[0080] 3.5) Using (lon,lat) as the center, modify the longitude range to (lon-step,lon+step) and the latitude range to (lan-step,lan+step), and reduce the step size by half: step = step / 2;

[0081] 3.6) Repeat steps 3.1)-3.5) iteratively reducing step until step is less than 1°, and then convert the final latitude and longitude result (lon) to the desired value. last ,lat last Transformed into Euler angles using Cartesian coordinates (row) last pitch last The Euler angles are the precise values ​​of the target's rotation around the x-axis and y-axis.

[0082] At this point, the rotated target appears in the camera with the same shape as the original image, but the rotation angle, size, and position are different in the two-dimensional plane of the camera's viewpoint. Figure 3 As shown, where Figure 3 (a) is the original image. Figure 3 (b) The image of the target from the camera's perspective after determining the first two degrees of freedom.

[0083] Step 4: Perform multiple rounds of iterative rotation sampling of the target around the z-axis and compare it with the original image to determine the precise value of the third degree of freedom.

[0084] Since the image of the target obtained in step 3 within the camera's field of view has the same shape as the original image, but the rotation angle is different in the two-dimensional plane of the camera's view, it is necessary to rotate the target around the z-axis of the world coordinate system to achieve rotation sampling in the two-dimensional plane of the camera's view and compare it with the original image to determine the precise value of this third degree of freedom. The implementation includes the following:

[0085] 4.1) Keep the target's rotation angle around the x-axis and y-axis constant. Let the first round sampling step size be step and the rotation range around the z-axis be [0, 360]. Within the rotation range, rotate around the z-axis of the world coordinate system according to the step size, and take a screenshot to obtain the sampled image.

[0086] 4.2) The SIFT algorithm is used to extract feature points from the sampled image and the original image, and each feature point has its unique descriptor;

[0087] 4.3) Feature points in all sampled images and the original image are matched using Euclidean distance, with cross-matching enabled during the matching process, resulting in multiple pairs of matching points:

[0088] In the process of image feature point matching, Euclidean distance is used as a metric to match feature points of all sampled images with those of the original image. That is, the Euclidean distance between each feature point descriptor in the sampled image and all feature point descriptors in the original image is calculated. The smaller the Euclidean distance, the closer the two feature points are in the feature space, and the greater the probability of matching.

[0089] During the matching process, a cross-matching mechanism is enabled, which means that matching is performed not only from the sampled image to the original image, but also from the original image to the sampled image in reverse. Only when the matching results in both directions are consistent is the feature point pair considered to be a valid match. In this way, some erroneous matches are effectively filtered out, resulting in multiple sets of matching point pairs, thereby improving the accuracy and reliability of the matching.

[0090] 4.4) Compare images using the intersections of matching points and connecting lines:

[0091] Since the imaging target in this round of sampling rotates on the camera's imaging plane, the results obtained by comparing quantity and geometric consistency are not significantly different. Therefore, this step first filters the matching points through geometric consistency and affine transformation, eliminating matching points that do not meet the geometric constraints. Then, the number of intersections between the line segments connected to the matching points is calculated, so that in ideal conditions, the matching points completely coincide or the extensions of these line segments intersect at a single point. The specific implementation includes the following:

[0092] 4.4.1) Estimate an optimal affine transformation matrix A using the keypoint coordinate pairs and an affine transformation model:

[0093]

[0094] Where a 11 and a 22 Let a represent the scaling factors for the x-axis and y-axis, respectively. 12 and a 21 Let t represent the shearing factor of the y-axis with respect to the x-axis and the shearing factor of the x-axis with respect to the y-axis, respectively. x ,t y These represent the translation amounts along the x-axis and y-axis, respectively; this matrix describes the geometric relationship between the first set of key points and the second set of key points.

[0095] 4.4.2) For each matching point pair, use the affine transformation matrix A to transform point p1 in the first image to coordinate p'2 in the coordinate system of the second image:

[0096] p'2 = A·p1;

[0097] 4.4.3) Calculate the Euclidean distance dis between the actual corresponding point and the transformed point based on the matching point p2 of p1 in the second figure and the affine transformation point p'2:

[0098] dis = ||p'2-p2||;

[0099] 4.4.4) Set a threshold T based on the calculation accuracy, compare it with the Euclidean distance dis, and determine whether the point meets the geometric consistency requirement:

[0100] If dis < T, then the point is considered to meet geometric consistency, and step 4.4.5 is executed.

[0101] Otherwise, the matching pair corresponding to that point is removed, and that point will not be considered again in the future;

[0102] This example sets, but is not limited to, T=3.

[0103] 4.4.5) Use the coordinates of the matched point pairs after filtering to form line segments. Combine all line segments pairwise and determine whether they intersect:

[0104] 4.4.5a) Define the two line segments as [p1, p2] and [q1, q2] respectively, in vector form:

[0105]

[0106] 4.4.5b) Calculate the cross product d1×d2 of the two line segment vectors and determine whether the result is zero:

[0107] If the result is zero, the line segments are either parallel or collinear.

[0108] Otherwise, calculate the relative positions t1 and t2 of the intersection point on the two line segments:

[0109]

[0110] 4.4.5c) Determine whether t1 and t2 are both within the range [0,1]:

[0111] If so, then the two line segments intersect;

[0112] Otherwise, the two line segments will not intersect;

[0113] 4.4.5d) Use the methods in steps 4.4.5a)-4.4.5c) to calculate the number of intersections between all matching point pairs in a single sampled image and the original image;

[0114] 4.5) Using the method in step 4.4), calculate the number of intersections between the lines connecting the matching point pairs of all sampled images and the original image, and select the rotation angle yaw corresponding to the sampled image with the fewest intersections as the result of this round of matching;

[0115] 4.6) Using yaw as the center, modify the rotation range to (yaw-step, yaw+step), and reduce the step size by half.

[0116] step = step / 2;

[0117] 4.7) Repeat steps 4.1)-4.6) iteratively reducing step until step is less than 1°, resulting in the final rotation yaw. last This is the precise value of rotation around the z-axis.

[0118] After rotation, the image of the target in the camera is basically the same as the shape of the target in the original image, such as... Figure 4 As shown, where Figure 4 (a) is the original image. Figure 4 (b) The image of the target in the camera's view after the three degrees of freedom are determined.

[0119] Step 5: Calculate the rotation matrix of the target based on the numerical results of the three rotational degrees of freedom.

[0120] Since steps 3 and 4 involve acquiring the target's pose with the target at the origin and the camera in the positive z-axis direction, it is necessary to rotate the camera to the pose at the time of capture in that frame, so that the target rotates synchronously with the camera, and obtain the rotation matrix corresponding to the original image. The implementation includes the following:

[0121] 5.1) Following the rotation order of first rotating around the Z-axis, then around the Y-axis, and finally around the X-axis, multiply the rotation matrices of the three axes to calculate the rotation matrix R of the target simulation result:

[0122] R = Rx(row) last )·Ry(pitch last )·Rz(yaw last )

[0123] Where Rx(row) last ), Rx(row) last ), Rx(row) last ) are rotation matrices for the x-axis, y-axis, and z-axis, respectively;

[0124] 5.2) Rotate the camera and the target:

[0125] 5.2.1) Based on the camera parameters (camera orientation f1, upward direction u1, and camera position p1) in the simulation, construct the camera transformation matrix M1 for the simulation:

[0126]

[0127] Where r1 = f1 × u1 is the right vector of the camera in the simulation state, and u1 = r1 × f1 is the upward vector of the camera after adjustment in the simulation state;

[0128] 5.2.2) Based on the initial camera parameters at the time of image capture, including camera orientation f2, upward direction u2, and camera position p2, construct the camera transformation matrix M2 for image capture:

[0129]

[0130] Where r2 = f2 × u2 is the right vector of the camera when the image is taken, and u2 = r2 × f2 is the upward vector of the camera after adjustment when the image is taken;

[0131] 5.2.3) Rotate the camera from the simulated pose to the actual shooting pose, i.e., transform from matrix M1 to matrix M2. The target rotates synchronously with the camera to obtain the rotation matrix R2 of the target in this frame:

[0132]

[0133] M2 rot M2 is a 3×3 matrix located in the upper left corner. It is the transpose of the top left 3×3 matrix M1.

[0134] Step 6: Calculate the spatial vector of the target.

[0135] 6.1) Based on the target's imaging position and size in the image, accurately calculate the spatial vector of the target's center in the camera coordinate system using the camera imaging principle:

[0136] 6.1.1) Obtain the geometric relationship between the camera and the target:

[0137] From such Figure 5 As shown in the camera lens imaging principle diagram, since an object forms an inverted, reversed, and reduced image on the imaging sensor after passing through the camera lens, and the camera processes the sensor's image into a forward-reduced image as the output image, the image seen by the eye can be equivalently understood as the equivalent imaging plane in the diagram. This leads to the following proportional relationship between the camera and the target:

[0138]

[0139] Where D is the distance between the plane where the target is located and the camera, f is the focal length of the camera, l is the length of any position of the target in the image, and L is the actual length of the target at the position corresponding to l.

[0140] 6.1.2) Using the proportional relationship between the camera and the target, obtain the proportional relationship between the target and the camera at different distances, and calculate the distance D1 from the camera to the plane where the target is located during the actual shooting.

[0141]

[0142] Where L1 is the length of the minimum bounding matrix of the target in the original image, L2 is the length of the minimum bounding matrix of the target in the final pose downsampled image, and D2 is the distance between the final camera and the plane where the target is located.

[0143] 6.1.3) Let the image center be the origin, the right side be the positive x-axis, and the upward direction be the positive y-axis. Determine the position of the imaging center of the target in the original image as (x... p ,y p Based on the target imaging position and camera focal length, determine the spatial vector ct of the target imaging center in the camera coordinate system.

[0144] ct=(x p ,-y p ,-f)

[0145] Where f is the camera focal length, and the camera uses a right-handed coordinate system;

[0146] 6.1.4) Calculate the distance d between the camera and the target imaging center. p :

[0147]

[0148] 6.1.5) Calculate the distance d between the target center and the camera:

[0149]

[0150] 6.1.6) Normalize the spatial vector ct of the target imaging center in the camera coordinate system to a unit direction vector e, and calculate the spatial vector α of the target in the camera coordinate system:

[0151] α = de;

[0152] 6.2) Calculate the target's position vector in the world coordinate system using graphical methods:

[0153] 6.2.1) Construct a homogeneous vector P representing the target spatial location based on the target spatial vector. camera :

[0154] P camera =(α x ,α y ,α z ,1)

[0155] Where, α x α y α z These represent the magnitudes of the target in the x, y, and z directions in the camera coordinate system, respectively.

[0156] 6.2.2) Based on the transformation matrix M2 and homogeneous vector P in step 5.2.2),camera The position vector of the target in the world coordinate system is calculated as P. world :

[0157] P world =M2*P camera .

[0158] Step 7: Save the rotation matrix calculated in Step 5 and the position vector calculated in Step 6 as the reconstructed pose parameters of the target in the current frame. In the 3D engine, apply these parameters as the pose parameters of the target to obtain the image from the camera's perspective.

[0159] The image from the camera's perspective is compared with the original image, and the result is as follows: Figure 6 As shown. Among them Figure 6 (a) is the original image. Figure 6 (b) is an image from the camera's perspective in the 3D engine. Figure 6 It can be seen that the image of the target in the camera's viewpoint is basically consistent with the shape and position of the target in the original image, indicating that the reconstruction result of the current frame has high fidelity.

[0160] Step 8: Repeat steps 2 to 7 to calculate the target pose for all frames of the orbit and complete the orbit reconstruction of the flight target based on monocular vision.

[0161] Example 2: Flight target trajectory reconstruction system based on monocular vision.

[0162] Reference Figure 7 This example system includes an initialization module, an orbit frame extraction module, a target attitude matching module, a target position calculation module, and a judgment module, among which:

[0163] The initialization module initializes the 3D engine, providing a basic environment for subsequent sampling.

[0164] The orbit frame extraction module extracts the image and camera parameters of one orbit frame in sequence, and transmits the original image and camera parameters of the extracted frame to the target pose matching module and the target position calculation module in sequence.

[0165] The target pose matching module first divides the three degrees of freedom of target rotation into 2+1, then iteratively samples the first two degrees of freedom and the last degree of freedom, and compares them with the original image of the frame sent by the orbit frame extraction module using different comparison indicators. Based on the comparison results, the range of iterative sampling is gradually narrowed. When the range is narrowed to the set range, the precise value of the three degrees of freedom can be determined. Finally, the precise rotation matrix of the target is determined based on the precise value of the three degrees of freedom and the camera parameters sent by the orbit frame extraction module.

[0166] The target position calculation module uses the proportional relationship that the distance from the camera to the target plane is proportional to the camera focal length and the ratio of the actual size of the target to the image size to calculate the distance from the camera to the target plane. It uses the distance from the camera to the target plane and the image position of the target in the original image of the frame sent by the orbit frame extraction module to determine the spatial vector of the target in the camera coordinate system. It uses the camera parameters sent by the orbit frame extraction module to convert the spatial vector of the target in the camera coordinate system into a spatial vector in the world coordinate system.

[0167] The judgment module determines whether the current frame is the last frame. If it is, the system operation ends. Otherwise, the system runs the orbit frame extraction module to extract the raw data of the next frame and runs the target attitude matching module and the target position calculation module in sequence to perform calculations until the last frame is calculated.

[0168] It should be noted that the step numbers in the specification and claims of this invention are only for the purpose of clearly describing the embodiments of this invention and facilitating understanding, and their order is not limited.

Claims

1. A method for reconstructing the trajectory of a flight target based on monocular vision, characterized in that, include: (1) Initialize the 3D engine and adjust the positional relationship between the target and the camera to ensure that the target is fully projected into the camera's field of view; (2) The target rotation 3 degrees of freedom are split into 2+1; (3) Read a frame of the original image in sequence, perform multiple rounds of iterative rotation sampling of the target around the first two degrees of freedom perpendicular to the camera's viewing direction in the 3D engine, and compare it with the original image to determine the precise values ​​of these two degrees of freedom. (4) Perform multiple rounds of iterative rotation sampling of the target around the camera's viewing direction and compare it with the original image to determine the precise value of the third degree of freedom; (5) Calculate the rotation matrix of the target based on the numerical results of the three rotational degrees of freedom; (6) Based on the camera imaging principle, the distance from the camera to the target plane is calculated by using the proportional relationship that the distance from the camera to the target plane is proportional to the ratio of the camera focal length and the ratio of the actual size of the target to the image size; the spatial vector of the target in the camera coordinate system is determined by using the imaging position of the target in the original image and the distance from the camera to the target plane; the spatial vector of the target in the camera coordinate system is converted into the spatial vector in the world coordinate system by using the coordinate transformation relationship in computer graphics. (7) Use spatial vectors and rotation matrices to obtain the precise pose of the target's orbit in this frame; (8) Repeat steps (3) to (7) to calculate the target pose for all frames of the orbit and complete the reconstruction of the flight target orbit.

2. The method according to claim 1, characterized in that, Step (1) initializes the 3D engine and adjusts the positional relationship between the target and the camera. This is achieved as follows: (1a) Adjust the camera field of view and imaging pixel values ​​in the engine to make them consistent with the real camera; (1b) Place the target at the origin of the coordinate system, and position the camera in the positive z-axis direction, looking towards the origin, so that the target can rotate; (1c1) Calculate the camera distance d when the target occupies half of the camera's lateral field of view. camx : Where width t Let fovx be the length of the target along the x-axis, and fovx be the lateral field of view of the camera. (1c2) Calculate the camera distance d when the target occupies half of the camera's longitudinal field of view. camy : height t Let fovy be the length of the target along the y-axis, and fovy be the longitudinal field of view of the camera. (1c3) Take d camx With d camy The maximum value is used to adjust the distance d between the camera and the origin. cam Ensure that the target occupies half of the camera's field of view: d cam =max(d camx ,d camy )。 3. The method according to claim 1, characterized in that, In step (2), the three degrees of freedom of target rotation are split into 2+1, which means splitting the three degrees of freedom of target rotation into two degrees of freedom perpendicular to the camera's viewing direction and one degree of freedom parallel to the target's viewing direction. The two degrees of freedom perpendicular to the camera's viewing direction are the x-axis and y-axis of the world coordinate system; One degree of freedom parallel to the direction the target is looking is the z-axis of the world coordinate system.

4. The method according to claim 1, characterized in that, In step (3), the target is subjected to multiple rounds of iterative rotation sampling around the first two degrees of freedom perpendicular to the camera's viewing direction in the 3D engine to determine the precise values ​​of these two degrees of freedom. The implementation includes the following: (3a) Drawing on the concept of latitude and longitude spheres, the first sampling is set with the latitude range as [0,180] and the longitude range as [0,360]. The step size is set to step. The operation target rotates the intersection point around the x-axis and y-axis of the world coordinate system to the positive z-axis in sequence, and then takes a screenshot to obtain the sampled image. (3b) Use the SIFT algorithm to extract feature points from all sampled images and the original image; (3c) Multiple sets of matching point pairs are obtained by matching feature points of all sampled images and the original image respectively using Euclidean distance; (3d) Compare the sampled image with the original image using the matching point ratio and inlier ratio, and select the most comparable results: (3d1) Calculate the matching point ratio R match : (3d2) By filtering the matching points through geometric consistency and affine transformation, the matching points that do not conform to the geometric constraints are removed, and the number of interior points of the matching points is obtained. (3d3) Calculate the interior point ratio measure R inlier : (3d4) According to R match With R inlier Calculate the comparison index R XY : R XY =0.7·R match +0.3·R inlier ; (3d5) Calculate R for all sampled images and the original image. XY Select R XY The latitude and longitude (lon, lat) corresponding to the largest image is used as the result of this round of matching; (3e) Using (lon,lat) as the center, modify the longitude range to (lon-step,lon+step) and the latitude range to (lan-step,lan+step), and reduce the step size by half: step = step / 2; (3f) Repeat (3a)-(3e) iteratively reducing step until step is less than 1°, and then calculate the final latitude and longitude result (lon). last ,lat last Transformed into Euler angles using Cartesian coordinates (row) last pitch last The Euler angles are the precise values ​​of the target's rotation around the x-axis and y-axis.

5. The method according to claim 1, characterized in that, In step (4), the target is iteratively rotated and sampled multiple times around the camera's viewing direction and compared with the original image to determine the precise value of the third degree of freedom. This process includes the following steps: (4a) Let the sampling step size be step and the rotation range be [0,360]. The target is rotated around the z-axis of the world coordinate system in sequence according to the step size within the rotation range, and a screenshot is taken to obtain the sampled image. (4b) Use the SIFT algorithm to extract feature points from the sampled image and the original image; (4c) Multiple sets of matching point pairs are obtained by matching feature points of the sampled image and the original image using Euclidean distance; (4d) The matching points are filtered by geometric consistency and affine transformation, and the matching points that do not conform to the geometric constraints are eliminated. (4e) Compare all sampled images with the original image, calculate the number of intersections between the lines connecting the matching point pairs, and select the rotation angle yaw corresponding to the sampled image with the fewest intersections as the result of this round of matching; (4f) With yaw as the center, modify the rotation range to (yaw-step, yaw+step), and reduce the step size by half: step = step / 2; (4g) Repeat (4a)-(4f) iteratively reducing step until step is less than 1°, and the final rotation result is yaw. last This is the precise value of rotation around the z-axis.

6. The method according to claim 1, characterized in that, Step (5) calculates the rotation matrix of the target based on the numerical results of the three rotational degrees of freedom. The implementation includes the following: (5a) Following the rotation order of first rotating around the Z-axis, then around the Y-axis, and finally around the X-axis, multiply the rotation matrices of the three axes to calculate the rotation matrix R of the target simulation result: R=Rx(row last )·Ry(pitch last )·Rz(yaw last ), Where Rx(row) last ), Ry (pitch) last ), Rz(yaw) last ) are rotation matrices for the x-axis, y-axis, and z-axis, respectively; (5b) Based on the camera transformation matrix M1 during simulation and the camera transformation matrix M2 during image capture, transform the camera from matrix M1 to matrix M2 to obtain the rotation matrix R2 of the target in this frame: M2 rot M2 is a 3×3 matrix located in the upper left corner. It is the transpose of the top left 3×3 matrix M1.

7. The method according to claim 1, characterized in that, Step (6) calculates the distance from the camera to the plane containing the target, which is implemented as follows: (6a) Based on the camera imaging principle, the following proportional relationship between the camera and the target is obtained: Where D is the distance between the target and the camera, f is the camera focal length, l is the length of any position of the target in the image, and L is the actual length of the target at the position corresponding to l. (6b) Using the proportional relationship between the camera and the target, the following proportional relationship is obtained at different distances between the target and the camera. The distance D1 from the camera to the plane containing the target during actual shooting is then calculated: Where L1 is the length of the minimum bounding matrix of the target in the original image, L2 is the length of the minimum bounding matrix of the target in the final pose downsampled image, and D2 is the distance between the final camera and the plane where the target is located.

8. The method according to claim 7, characterized in that, Step (6) uses the target's imaging position in the original image and the distance from the camera to the target's plane to determine the target's spatial vector in the camera coordinate system. This is achieved as follows: (6c) In the image, let the image center be the origin, the right side be the positive x-axis, and the upward direction be the positive y-axis. Determine the position of the imaging center of the target in the original image as (x... p ,y p Based on the target imaging position and camera focal length, determine the spatial vector ct of the target imaging center in the camera coordinate system. ct=(x p ,-y p ,-f); Where f is the camera focal length, and the camera uses a right-handed coordinate system; (6d) Calculate the distance d between the camera and the center of the target image. p : (6e) Calculate the distance d between the target center and the camera: (6f) Normalize the spatial vector ct of the target imaging center in the camera coordinate system to a unit direction vector e, and calculate the spatial vector α of the target in the camera coordinate system: α = de.

9. The method according to claim 1, characterized in that, In step (6), the spatial vector of the target in the camera coordinate system is converted into a spatial vector in the world coordinate system through coordinate transformation relationships in computer graphics. The implementation includes the following: (6g) Based on the camera orientation f c The transformation matrix M2 of the camera in its original state is constructed using the upward direction u and the camera position p. (6h) Construct the homogeneous vector P of the target space position based on the target space vector. camera : P camera ＝(α x ,α y ,α z ,1)； Where α is the spatial vector of the target in the camera coordinate system; (6i) Based on the transformation matrix M2 and the homogeneous vector P camera Using the coordinate transformation formula in computer graphics, the position vector of the target in the world coordinate system is calculated as P. world : P world =M2*P camera 。 10. A flight target trajectory reconstruction system based on monocular vision, the system being used to implement the method as described in any one of claims 1-9, characterized in that, include: The system initialization module is used to initialize the 3D engine and provide the basic environment for subsequent sampling simulation. The track frame extraction module is used to extract the image and camera parameters of each frame in track frame order; The target pose matching module is used to iteratively sample and compare targets to determine their precise pose matrix. The target position calculation module is used to calculate the actual position of the target in space and determine the target's precise position vector; The judgment module is used to determine whether the current frame is the last frame of the track. If it is, the system operation ends; otherwise, the track frame extraction module is run.

Images