A comprehensive visual scene generation method based on photoelectric video motion estimation

Abstract

The present application belongs to the field of airborne photoelectric reconnaissance and situation awareness, and discloses a kind of integrated visual scene generation method based on photoelectric video motion estimation, comprising the following steps: determining the visual scene geographic range according to the photoelectric starting point position;For adjacent frames of photoelectric image, feature point extraction and matching are carried out;For the starting geographic area of photoelectric image, select the integrated visual scene geographic area and the corresponding photoelectric image for matching;According to the matching feature point pair of adjacent frames, the relative motion rotation matrix and translation matrix are solved;According to the matching result, the absolute positioning result of the carrier is solved;Fusion absolute positioning and relative positioning result, form continuous positioning result output;Establish a virtual camera imaging model;Based on virtual camera, integrated visual scene is generated.The present application can meet the integrated visual scene generation ability of existing airborne equipment under certain degree of anti-interference condition, and can be applied in multiple airborne task scenes by designing special software module, to help realize high-precision matching ability between multiple modal images.

Description

Technical Field

[0001] This invention belongs to the field of airborne optoelectronic reconnaissance and situational awareness, and relates to a comprehensive visual scene generation method based on optoelectronic video motion estimation. Background Technology

[0002] Current integrated visual technology primarily relies on pose data acquired from onboard position and attitude sensors, with typical devices including inertial navigation and GPS. However, traditional methods of acquiring pose data have certain problems in some scenarios: First, the pose data itself contains noise and has a certain range of error; second, the pose data provided by navigation devices has a probability of drift, and the process of acquiring pose data based on integrated inertial navigation devices is easily interfered with, leading to service interruptions; finally, in special scenarios, the aircraft may be deceived by false data and thus be trapped.

[0003] The adjacent frame data of the photoelectric sensor actually implies the pose changes of the photoelectric system. Based on the modeling of the image data and the imaging parameters of the photoelectric system, the relative motion of the spatial pose can be extracted, and the absolute pose can be calculated by using the partial matching results with satellite imagery, thereby generating the corresponding comprehensive visual scene. Summary of the Invention

[0004] (I) Purpose of the Invention

[0005] In order to generate a corresponding comprehensive visual scene using photoelectric sensor data, this invention proposes a comprehensive visual scene generation method based on photoelectric potential pose motion estimation of adjacent image frames. This method can calculate the relative motion using photoelectric images, and obtain the initial absolute pose by combining the matching results with satellite imagery, thereby generating a comprehensive visual scene image.

[0006] (II) Technical Solution

[0007] For photoelectric video and integrated visual image, the main steps of the method of this invention include: 1. Determining the initial geographical range of the integrated visual image based on the starting position and aiming direction of the photoelectric sensor; 2. Matching the geographical area of ​​the integrated visual image with the corresponding photoelectric image for the starting geographical area of ​​the photoelectric image; 3. Solving for the corresponding geographical coordinates based on the pixel coordinates of the matching point pairs; 4. Solving for the absolute positioning result of the photoelectric sensor based on the geographical coordinate result of the matching points; 5. Performing feature detection, extraction, description, and matching based on adjacent frame images; 6. Solving for the relative motion rotation matrix and translation matrix of adjacent frames based on the matching point pairs; 7. Fusing the absolute positioning result and the relative positioning result of the carrier; 8. Establishing a virtual camera imaging model; 9. Generating the integrated visual image based on the virtual camera and the positioning result.

[0008] (III) Beneficial Effects

[0009] The method provided by this invention is of great significance in special situations. When the integrated navigation sensors in an airborne platform are interfered with or GPS is denied, this method can be used to generate a comprehensive visual scene from photoelectric sensor data. This elevates the aircraft's motion reflected in the two-dimensional photoelectric sensor images to a three-dimensional spatial presentation, providing a "God's-eye view" of the aircraft's movement and offering three-dimensional situational awareness. This invention has good potential for practical engineering applications, meeting the requirements of existing airborne equipment for generating comprehensive visual scenes under certain anti-interference conditions. Designing a dedicated software module for it can be applied in multiple airborne mission scenarios, helping to achieve high-precision matching capabilities between multimodal images. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the process composition of the method in this invention.

[0011] Figure 2 This is a schematic diagram of the coordinates of the four corner points of the photoelectric sensor for ground detection, obtained by solving the detection area of ​​the photoelectric sensor using a three-dimensional spatial geometric relationship as a model in an embodiment of the present invention.

[0012] Figure 3 This is a schematic diagram of the heterogeneous image matching block obtained in the embodiments of the present invention, with the center point as the matching point pair. Detailed Implementation

[0013] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0014] like Figure 1 As shown, the airborne multimodal image feature extraction method of this invention includes the following steps: 1. Determine the initial geographical range of the comprehensive view based on the starting position and aiming direction of the photoelectric sensor; 2. Select the geographical area of ​​the comprehensive view and match it with the corresponding photoelectric image for the initial detection geographical area of ​​the photoelectric image; 3. Solve for the corresponding geographical coordinates based on the pixel coordinates of the matching point pairs; 4. Solve for the absolute positioning result of the photoelectric sensor based on the geographical coordinate result of the matching point; 5. Perform feature detection, extraction, description, and matching based on adjacent frame images; 6. Solve for the relative motion rotation matrix and translation matrix of adjacent frames based on the matching point pairs; 7. Fuse the absolute positioning result and the relative positioning result of the aircraft; 8. Establish a virtual camera imaging model; 9. Generate a comprehensive view based on the virtual camera and the positioning result.

[0015] The following is a detailed description of each step in the above process:

[0016] S1: Determine the initial geographical range of the integrated visual field based on the starting position and aiming direction of the photoelectric sensor.

[0017] In this case, the starting position of the photoelectric sensor is known, or approximately known, denoted as P. eo (b,l,h) Typically, the starting position includes three data points: longitude, latitude, and altitude, denoted as b, l, and h respectively. The initial attitude is known and denoted as A. eo (r,p,y) Typically, the initial attitude is based on the northeast-northeast coordinate system and includes three data points: roll angle, pitch angle, and azimuth angle, represented by r, p, and y, respectively. The field of view (FOV) parameters of the photoelectric sensor are known, denoted as FOV = (α, β), where α represents the corresponding azimuth angle and β represents the corresponding pitch angle. Based on the location and attitude of the photoelectric sensor, and the FOV as input, a three-dimensional spatial geometric relationship is used as the model, such as... Figure 2 As shown, the detection area of ​​the photoelectric sensor can be solved to obtain the coordinates of the four corner points of the photoelectric sensor's ground detection, represented as C1, C2, C3, and C4.

[0018] S2: For the initial detection geographic area of ​​the photoelectric image, select the comprehensive visual geographic area and match it with the corresponding photoelectric image.

[0019] Referring to the solution results of the initial detection geographic area of ​​the photoelectric image in step S1, the comprehensive visual image is appropriately enlarged. Based on empirical values, the geographic coverage of the visual image can be enlarged to 1.2 times. Multimodal matching is then performed between the comprehensive visual image and the corresponding photoelectric image.

[0020] The matching method consists of two stages: image feature extraction and feature matching. Image feature extraction can be performed using contour extraction, phase consistency extraction, gradient-based methods, or even neural network-based methods. Feature matching can be performed using correlation-based methods or high-dimensional space vector distance metrics.

[0021] Based on the above approach, heterogeneous image matching blocks are obtained, with the center point as the matching point pair, such as... Figure 3 As shown.

[0022] S3: Calculate the corresponding geographic coordinates based on the pixel coordinates of the matched point pairs.

[0023] Taking the pixel coordinates of the matching point pairs in the composite scene image output in step S2 as input, let the pixel coordinates of the scene image in the matching point pair be (Pix x Pix y The process involves determining the corresponding geographic coordinates. First, the cascaded equivalent transformation matrix from the absolute geocentric coordinate system to the current display window coordinate system (hereinafter referred to as the viewport coordinate system) can be obtained from the view engine, denoted as M. mvpw In viewport coordinates, let the coordinates of the starting and ending points of the intersection line segment be P. intersector_start P intersector_end The calculation is as follows:

[0024] P intersector_start =(Pix x Pix y ,0)*M mvpw

[0025] P intersector_end =(Pix x Pix y ,1)*M mvpw

[0026] The first intersection point between the line segment and the ground surface is found using the intersection algorithm, and denoted as P. intersector Convert it to geographic coordinates in geocentric coordinates, denoted as P. world_coor The calculation is as follows:

[0027] P world_coor =M -1 mvpw *P intersector

[0028] Where M -1 mvpw For M mvpw The inverse matrix.

[0029] S4: Based on the geographic coordinates of the matching points, calculate the absolute positioning result of the photoelectric sensor.

[0030] Using the coordinates of the matching point pairs output in step S3 as input, the absolute positioning result of the carrier is solved. The solution process uses the EPnP algorithm as a model. By inputting the geographic coordinates and pixel coordinates of three or more matching point pairs, the positioning result of the carrier can be obtained. The position and attitude in the absolute geocentric coordinate system are denoted as P. eo A eo .

[0031] S5: Perform feature detection, description, and matching based on adjacent frames.

[0032] Using the adjacent frames of the photoelectric sensor image output described in step S1 as input, feature point detection is performed. The detection method adopts the ORB method, and the steps include: first, using the FAST algorithm to detect feature corner points; second, using the BRIEF algorithm to perform feature description and generate feature descriptors; third, using the FLANN algorithm to perform feature matching on the feature descriptors to obtain matching point pairs; it is necessary to adjust the threshold parameters for feature point detection and matching to ensure that the final number of matching point pairs is greater than 8.

[0033] S6: Solve for the relative motion rotation and translation matrices of adjacent frames based on the matching point pairs.

[0034] Taking the matching point pairs in adjacent frames of the photoelectric sensor described in step S5 as input, let one of the point pairs be m1 and m2, according to the epipolar constraint

[0035]

[0036] Where K is the intrinsic parameter matrix of the photoelectric sensor, which is known data; t^ represents the transformation of vector t into an antisymmetric matrix; R, t are the relative motion rotation and translation matrices of adjacent frames, which are the data to be solved; let E = t^R represent the essential matrix; the epipolar constraint can be written as

[0037]

[0038] By using the eight-point method and substituting the eight matching points into the above equation, the essential matrix E can be solved, and the rotation matrix R and translation matrix t can be separated from it.

[0039] S7: Fusion of absolute and relative positioning results of the carrier aircraft

[0040] Based on the aircraft positioning results output from steps S4 and S6 and the motion of adjacent frames, the transformation matrix from the photoelectric coordinate system to the absolute geocentric coordinate system can be easily solved from the photoelectric sensor pose output from step S4, denoted as T. c-w =f(P eo A eo The motion of the photoelectric sensor in adjacent frames, output from step S6 (i.e., the rotation matrix R and translation matrix t), can be used to easily solve for the local coordinate system transformation matrix of the photoelectric sensor in adjacent frames, denoted as T. r-c = f(R, t).

[0041] Based on absolute positioning, and by fusing the pose motion of photoelectric sensors between adjacent frames, the updated pose can be calculated using the following formula, denoted as T. update The calculation process is as follows:

[0042] T update =T r-c *T c-w

[0043] Where * denotes the multiplication of transformation matrices, subsequent updates can be iterated, assigning the updated transformation matrix to the current local-to-world transformation matrix.

[0044] T c-w =T update

[0045] Continue updating the motion transformation between adjacent frames, the calculation process is as follows:

[0046] T update =T r-c *T c-w

[0047] By repeating this process, the transformation matrix of the current pose in the absolute geocentric coordinate system can be obtained quickly and continuously, denoted as T. rt_update It is a rapidly updated and constantly changing matrix.

[0048] S8: Establish a virtual camera imaging model

[0049] First, a virtual camera imaging model is established, which consists of a viewpoint transformation model, a perspective projection transformation model, and a viewport transformation model. These three models are constructed according to the following process:

[0050] Using the approximate starting position of the photoelectric sensor used in step S1 as input, the viewpoint transformation model can be easily calculated, denoted as M. view Using the field of view of the photoelectric sensor used in step S1, the image aspect ratio, and the ratio of the near and far clipping planes as inputs, the perspective projection transformation model can be easily calculated, denoted as M. projection The field of view can be appropriately enlarged based on empirical values, with a reference enlargement value of 1.2. Based on the pixel resolution of the photoelectric sensor used in step S1, the viewport transformation model can be easily calculated, denoted as M. viewport The virtual camera imaging model is denoted as CAM. virtual It can be established according to the following formula:

[0051] CAM virtual =M viewport *M projection *M view

[0052] Simultaneously, a virtual digital globe is constructed based on the WGS84 coordinate system, and terrain data is added. The terrain files are organized in tile format, and the constructed terrain nodes are denoted as MP. node .

[0053] S9: Generates a comprehensive visual scene based on virtual camera and positioning results

[0054] Based on the results of step S8, at the terrain node MP node The above uses a virtual camera CAM virtual Imaging, based on the results of step S7, using T rt_update By continuously updating the pose of the virtual camera, a comprehensive view can be generated.

[0055] As can be seen from the above technical solution, this invention, after performing multiple operations such as initial position determination, heterogeneous visual image matching, keyframe absolute positioning, adjacent frame feature extraction, relative positioning, and positioning fusion on airborne optoelectronic images, can generate a comprehensive visual image. This method can generate a comprehensive visual image using only optoelectronic sensors and preset terrain data. Compared with existing comprehensive visual image generation technologies, it does not rely on navigation and positioning equipment and can assist in generating comprehensive visual images even in scenarios with electromagnetic interference. This method combines the ideas of computer vision and artificial intelligence processing methods to design a new comprehensive visual image generation method, which has strong engineering application significance for airborne avionics systems. It is a fundamental technology for visual navigation and autonomous intelligence, and its tactical significance deserves further exploration to improve the battlefield survivability of helicopters.

[0056] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A comprehensive scene generation method based on photoelectric video motion estimation, characterized in that, Includes the following steps: S1: Determine the initial geographical range of the integrated view; S2: Select the comprehensive visual geographic area and match it with the corresponding photoelectric image; S3: Calculate the corresponding geographic coordinates based on the pixel coordinates of the matching points; Taking the pixel coordinates of the matching point pairs in the composite visual image output in step S2 as input, let the pixel coordinates of the visual image in the matching point pair be ( The process of solving for the corresponding geographic coordinates involves: first, obtaining the cascaded equivalent transformation matrix from the absolute geocentric coordinate system to the current display window coordinate system from the view engine, denoted as . The coordinate system of the currently displayed window is denoted as the viewport coordinate system. In the viewport coordinate system, let the coordinates of the starting and ending points of the intersection line segment be... The calculation is as follows: The first intersection point between the line segment and the ground surface is found using the intersection algorithm, denoted as . Convert it to geographic coordinates in geocentric coordinates, denoted as... The calculation is as follows: in, for The inverse matrix; S4: Solve for the absolute positioning result of the photoelectric sensor; S5: Perform feature detection, description, and matching on adjacent frames of the photoelectric image; S6: Solve for the relative motion rotation and translation matrices between adjacent frames; S7: Fusion of absolute and relative positioning results of the carrier aircraft; Based on the aircraft positioning results output from steps S4 and S6 and the motion of adjacent frames, the transformation matrix from the photoelectric coordinate system to the absolute geocentric coordinate system is solved from the photoelectric sensor pose output from step S4, denoted as... From the motion of the photoelectric sensor in adjacent frames output in step S6, i.e., the rotation matrix R and the translation matrix t, the local coordinate system transformation matrix of the photoelectric sensor in adjacent frames is solved, denoted as... ; Based on absolute positioning, the pose motion of the photoelectric sensors between adjacent frames is fused, and the updated pose is calculated using the following formula, denoted as: The calculation process is as follows: in This represents the multiplication of transformation matrices. Subsequent update loops iterate, assigning the updated transformation matrix to the current local-to-world transformation matrix. Continue updating the motion transformation between adjacent frames, the calculation process is as follows: This process is repeated until the transformation matrix of the current pose in the absolute geocentric coordinate system is obtained, denoted as... ; S8: Establish a virtual camera imaging model; S9: Generates a comprehensive view based on virtual camera and positioning results.

2. The comprehensive scene generation method based on photoelectric video motion estimation as described in claim 1, characterized in that, In step S1, the detection area of ​​the photoelectric sensor is solved using the three-dimensional spatial geometric relationship as a model, and the coordinates of the four corner points of the photoelectric sensor's ground detection are obtained.

3. The comprehensive scene generation method based on photoelectric video motion estimation as described in claim 2, characterized in that, In step S2, for the initial detection geographic area of ​​the photoelectric image, a comprehensive visual geographic area is selected and matched with the corresponding photoelectric image; Referring to the solution results of the initial detection geographic area of ​​the photoelectric image in step S1, the comprehensive visual image is magnified, and multimodal matching is performed between the comprehensive visual image and the corresponding photoelectric image; The matching method consists of two stages: image feature extraction and feature matching. Image feature extraction employs contour extraction or phase consistency extraction, or gradient-based methods, or neural network-based methods; feature matching employs correlation-based methods, or methods using high-dimensional space vector distance metrics. Obtain heterogeneous image matching blocks, with the center point as the matching point pair.

4. The comprehensive scene generation method based on photoelectric video motion estimation as described in claim 3, characterized in that, In step S4, the coordinates of the matching point pairs output in step S3 are used as input to solve for the absolute positioning result of the aircraft. The solution process uses the EPnP algorithm as a model, inputting the geographic coordinates and pixel coordinates of three or more matching point pairs to obtain the aircraft positioning result. The position and attitude in the absolute geocentric coordinate system are denoted as... .

5. The comprehensive scene generation method based on photoelectric video motion estimation as described in claim 4, characterized in that, In step S5, the adjacent frames of the photoelectric sensor image output in step S1 are used as input to perform feature point detection. The detection method adopts the ORB method, and the steps include: first, using the FAST algorithm to detect feature corner points; second, using the BRIEF algorithm to perform feature description and generate feature descriptors; third, using the FLANN algorithm to perform feature matching on the feature descriptors to obtain matching point pairs; and adjusting the threshold parameters for feature point detection and matching to ensure that the final number of matching point pairs is greater than 8.

6. The comprehensive scene generation method based on photoelectric video motion estimation as described in claim 5, characterized in that, In step S6, the matching point pairs in adjacent frames of the photoelectric sensor described in step S5 are taken as input, and let one pair of point pairs be... According to the epipolar constraint: in, This is the intrinsic parameter matrix of the photoelectric sensor, which contains known data. This means converting vector t into an antisymmetric matrix. Let be the relative motion rotation and translation matrices of adjacent frames, and be the data to be solved. The essential matrix, with epipolar constraints, is written as: Using the eight-point method, by substituting the eight matching points into the above equation, the essential matrix E is solved, and the rotation matrix R and translation matrix t are separated from it.

7. The comprehensive scene generation method based on photoelectric video motion estimation as described in claim 6, characterized in that, In step S8, a virtual camera imaging model is established. This model consists of a viewpoint transformation model, a perspective projection transformation model, and a viewport transformation model. The three models are constructed according to the following process: Using the initial position of the photoelectric sensor used in step S1 as input, calculate the viewpoint transformation model, denoted as... Using the field of view of the photoelectric sensor used in step S1, the image aspect ratio, and the near-far cropping plane ratio as inputs, calculate the perspective projection transformation model, denoted as... Based on the pixel resolution of the photoelectric sensor used in step S1, calculate the viewport transformation model, denoted as... The virtual camera imaging model is denoted as... Established based on the following formula: Simultaneously, a virtual digital globe is constructed based on the WGS84 coordinate system, and terrain data is added. The terrain files are organized in tile format, and the constructed terrain nodes are denoted as... .

8. The comprehensive scene generation method based on photoelectric video motion estimation as described in claim 7, characterized in that, In step S9, based on the result of step S8, at the terrain node... The above uses a virtual camera Imaging, based on the results of step S7, using The virtual camera's pose is continuously updated to generate a comprehensive view.

Images