Real-time remote sensing map generation method under denial condition

Abstract

The invention provides a real-time remote sensing map generation method under a denial condition. The method comprises the following steps: acquiring multiple frames of images of a remote sensing map; based on a pre-trained SuperPoint model, feature point extraction is carried out on each frame of image in the multiple frames of images to obtain feature points of each frame of image, and the number of the feature points is multiple; based on a pre-trained LightGlue model, initial feature matching is carried out on feature points of any two continuous frame images in the multiple frame images to obtain a plurality of initial feature matching points, and the two continuous frame images are an initial first image and an initial second image respectively; performing image rotation on the initial second image according to the plurality of initial feature matching points to obtain an intermediate second image; performing secondary feature matching on the intermediate second image and the initial first image, and performing calculation to obtain a homography matrix; and based on the homography matrix, splicing two continuous frame images to complete real-time updating of the remote sensing map.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a real-time remote sensing map generation method under denial conditions. Background Art

[0002] In recent years, the importance of real-time maps has become more and more significant with the development of technology and the growth of social needs. Real-time map generation is conducive to the development of precise navigation, disaster monitoring and other aspects. Existing real-time remote sensing image generation technology usually relies on accurate GPS positioning information and high-precision map data. The positioning information is combined with high-precision map data to generate pictures containing longitude and latitude information, which are pasted on the map to achieve map updates. This may have limitations in some application scenarios, especially when the drone is in a denied environment where the GPS signal is unstable or completely ineffective. In order to achieve real-time map updates under denied conditions, computer vision navigation technology is commonly used. Computer vision navigation is a technology that uses visual information (such as images and videos) for navigation and positioning.

[0003] Based on the above, on the one hand, most of the existing real-time map update methods need to use inertial navigation information to determine the approximate area. If there are errors in the positioning equipment of the drone or in the case of poor signal (such as canyons, forests, underground facilities, etc.), the exact location may not be determined, thus affecting the accuracy of map construction.

[0004] On the other hand, existing computer vision methods usually rely on orthophoto maps and use them as a benchmark for image matching. If the orthophoto map has insufficient resolution, is not updated in a timely manner, has insufficient regional coverage, or lacks details, the matching accuracy will be reduced and the map cannot be effectively merged. In order to solve the problem of relying on orthophoto maps, the LightGlue image matching method is generally used. As an efficient image matching algorithm based on deep learning, the LightGlue image matching method is very suitable for the generation of real-time maps, but its rotation invariance is poor. Under rejection conditions, due to the lack of heading angle, angle correction cannot be performed. Therefore, when the rotation angle is large, image matching cannot be performed accurately, which in turn affects the generation of real-time remote sensing images. In addition, due to the limitations of terminal computing power, power consumption and other conditions, the use of traditional CPUs (Central Processing Units) for deep learning algorithm deployment often cannot meet real-time requirements. Summary of the invention

[0005] The present invention provides a real-time remote sensing map generation method under denial conditions, comprising: acquiring multiple frames of remote sensing maps; extracting feature points of each frame of the multiple frames based on a pre-trained SuperPoint model to obtain feature points of each frame of the multiple frames, wherein the feature points include multiple; performing initial feature matching on feature points of any two consecutive frame images in the multiple frames based on a pre-trained LightGlue model to obtain multiple initial feature matching points, wherein the two consecutive frame images are an initial first image and an initial second image respectively; performing image rotation on the initial second image according to the multiple initial feature matching points to obtain an intermediate second image; performing secondary feature matching on the intermediate second image and the initial first image, and calculating a homography matrix; and splicing two consecutive frame images based on the homography matrix to complete the real-time update of the remote sensing map.

[0006] In the above scheme, the initial second image is rotated according to multiple initial feature matching points to obtain an intermediate second image, including: calculating the main direction of feature matching according to multiple initial feature matching points; and rotating the initial second image according to the main direction of feature matching to obtain an intermediate second image.

[0007] In the above scheme, two continuous frame images are stitched based on the homography matrix, including: based on the homography matrix, the intermediate second image is subjected to image transmission transformation to obtain the target second image, and the two continuous frame images are stitched according to the overlapping area between the target second image and the initial first image to complete the real-time update of the remote sensing map.

[0008] In the above scheme, secondary feature matching is performed on the intermediate second image and the initial first image, and a homography matrix is ​​calculated, including: secondary feature matching is performed on the intermediate second image and the initial first image to obtain multiple secondary feature matching points, wherein the secondary feature matching points include at least four; based on the multiple secondary feature matching points, the homography matrix is ​​calculated.

[0009] In the above scheme, the number of secondary feature matching points is proportional to the calculation accuracy of the homography matrix.

[0010] In the above scheme, the overlapping area between the target second image and the initial first image is obtained by the following method, including: obtaining the coordinates of the four corner points of the target second image based on the homography matrix; determining the overlapping area between the target second image and the initial first image according to the coordinates of the four corner points of the target second image and the known coordinates of the four corner points of the initial first image.

[0011] In the above solution, obtaining multiple frames of images of the remote sensing map includes: obtaining multiple frames of images of the remote sensing map through an image acquisition device or a video shooting device.

[0012] In the above scheme, both the SuperPoint model and the LightGlue model are deployed on the image acquisition device by using the RKNN tool.

[0013] In the above scheme, both the SuperPoint model and the LightGlue model are deployed on the image acquisition device by using the RKNN tool, including: pre-training the SuperPoint model and the LightGlue model respectively; transferring the architecture of the pre-trained SuperPoint model and the LightGlue model to the ONNX architecture; using the RKNN tool to convert the architecture from ONNX to RKNN, and deploying it on the image acquisition device.

[0014] In the above solution, the secondary feature matching points include at least four.

[0015] The technical solution of the embodiment of the present invention has at least the following beneficial effects:

[0016] (1) By introducing image feature extraction and matching, with feature point matching and main direction calculation of adjacent frame images as the core, a real-time positioning and map updating method that does not rely on inertial navigation information is constructed. Even in an absolutely denied environment, image feature matching can still serve as an independent data source to provide reliable location information, eliminating the dependence on inertial navigation information, so that the system can still operate normally in a denied environment.

[0017] (2) By calculating the main direction of feature matching and performing phase rotation correction on the next frame of image, the image rotation is adjusted to a uniform direction, thereby reducing the inconsistency of feature point descriptors due to changes in rotation angles. Compared with the traditional LightGlue image matching method, when the image rotation changes greatly, the present invention can obtain more matching points, thereby improving the image stitching effect of continuous frames of real-time maps. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flowchart of a method for generating a real-time remote sensing map under a denial condition according to an embodiment of the present invention is schematically shown;

[0019] Figure 2A A diagram schematically shows a secondary feature matching result according to an embodiment of the present invention;

[0020] Figure 2B A diagram schematically shows a feature matching result of the LightGlue image matching method according to an embodiment of the present invention;

[0021] Figure 3A A diagram schematically shows an image stitching imaging result according to an embodiment of the present invention;

[0022] Figure 3BThe image stitching imaging result diagram of the LightGlue image matching method according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0023] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0024] First, the technical terms described in this article are explained and described as follows.

[0025] ‌SuperPoint model, a model for feature point detection and descriptor extraction based on self-supervised training;

[0026] LightGlue model, a graph neural network model designed for local feature matching, aims to improve the efficiency and accuracy of feature matching in image processing;

[0027] ‌ONNX architecture, an open source standard architecture for deep learning model interoperability;

[0028] RKNN architecture, an inference framework that supports multiple deep learning methods.

[0029] Figure 1 The flowchart of the method for generating a real-time remote sensing map under denial conditions according to an embodiment of the present invention is schematically shown.

[0030] Please refer to Figure 1 The specific process of the real-time remote sensing map generation method under the rejection condition of the embodiment of the present invention includes operations S110~S160.

[0031] In operation S110 , a plurality of frames of images of a remote sensing map are acquired.

[0032] In an embodiment of the present invention, multiple frames of images of a remote sensing map are acquired by an image acquisition device or a video shooting device.

[0033] For example, in this embodiment, the aerial remote sensing map video collected by the drone in real time is used as the research object, and the image of the continuous frame of the video is captured. For example, the image of each frame of the aerial video is represented as , T is the total number of frames, and X×Y is the size of the screenshot.

[0034] In operation S120, feature points are extracted from each frame of the multiple frames of images based on the pre-trained SuperPoint model to obtain feature points of each frame of the image, where the feature points include a plurality of feature points.

[0035] In operation S130, based on the pre-trained LightGlue model, initial feature matching is performed on feature points of any two consecutive frame images in the multiple frame images to obtain multiple initial feature matching points, wherein the two consecutive frame images are an initial first image and an initial second image.

[0036] In an embodiment of the present invention, feature points of each frame of the drone aerial video are extracted using a pre-trained SuperPoint model, and then initial feature matching is performed on the feature points of any two consecutive frame images (for example, an initial first image and an initial second image) in the multiple frames using a pre-trained LightGlue model to obtain multiple initial feature matching points.

[0037] Specifically, the pre-trained SuperPoint model is used to extract feature points from each frame of the drone aerial video. For example, the tth frame image I t (that is, the initial first image) has n feature points, expressed as , each feature point p i Two-dimensional coordinates can be used To represent; the t+1 frame image I t+1 (that is, the initial second image) has m feature points, expressed as , each feature point q i Two-dimensional coordinates can be used To express.

[0038] Furthermore, by matching the feature point sets of consecutive frames with the pre-trained LightGlue model, the initial feature matching points (p i , q i ) and the matching score W of the initial feature matching point i ,The larger matching score indicates a stronger matching relationship.

[0039] In the embodiment of the present invention, the wrong matching points are removed by a random sampling consensus algorithm to obtain a set of N correct feature matching points. .

[0040] It should be noted that, in the embodiment of the present invention, the above-mentioned SuperPoint model and LightGlue model are both deployed on the image acquisition device by using the RKNN tool.

[0041] Specifically, the SuperPoint model and the LightGlue model are deployed on the image acquisition device by using the RKNN tool, including: pre-training the SuperPoint model and the LightGlue model respectively; transferring the architecture of the pre-trained SuperPoint model and the LightGlue model to the ONNX architecture; using the RKNN tool to convert the architecture from ONNX to RKNN, and deploying it on the drone.

[0042] Through the embodiments of the present invention, NPU (Neural Processing Unit) is used to accelerate artificial intelligence computing. For ARM instruction set architecture chips, RKNN (Rockchip Neural Network) tools are used to deploy deep learning algorithms, thereby implementing the deployment of algorithms on the terminal side, while ensuring feature point extraction and matching performance, and meeting the algorithm's real-time requirements.

[0043] In addition, for feature point extraction, in addition to using the SuperPoint model, other feature point detection and description methods can also be used. For feature matching, in addition to using the LightGlue model, other feature matching algorithms can also be used, such as EfficientLoFTR, RoMa, etc.

[0044] In operation S140, the initial second image is rotated according to the plurality of initial feature matching points to obtain an intermediate second image.

[0045] In an embodiment of the present invention, an initial second image is rotated according to a plurality of initial feature matching points to obtain an intermediate second image, including: calculating a feature matching main direction according to the plurality of initial feature matching points; and rotating the initial second image according to the feature matching main direction to obtain an intermediate second image.

[0046] Specifically, according to each initial feature matching point (p i , q i ), calculate the angle direction of the matching point connection line and the y-axis in the two-dimensional coordinates ,Right now:

[0047]

[0048] For all matching points, a set of feature angles can be formed, which is . Calculate the weighted average of the angle vector to estimate the main direction of feature matching ,as follows:

[0049]

[0050] Among them, the weight can be the matching score W of the feature matching point i The matching main direction obtained in this way is Indicates the real-time heading angle of the drone.

[0051] Furthermore, according to the feature matching main direction , the image of the t+1th frame is rotated to obtain the intermediate second image. For example, the rotated image of the t+1th frame, that is, the intermediate second image is expressed as .

[0052] By calculating the main direction of feature matching and performing phase rotation correction on the next frame of image, the image rotation is adjusted to a uniform direction, thereby reducing the inconsistency of feature point descriptors due to changes in rotation angles. Compared with the traditional LightGlue image matching method, when the image rotation changes greatly, the present invention can obtain more matching points, thereby improving the image stitching effect of continuous frames of real-time maps.

[0053] In operation S150, secondary feature matching is performed on the intermediate second image and the initial first image, and a homography matrix is ​​calculated.

[0054] In an embodiment of the present invention, secondary feature matching is performed on the intermediate second image and the initial first image, and a homography matrix is ​​calculated, including: secondary feature matching is performed on the intermediate second image and the initial first image to obtain multiple secondary feature matching points, wherein the secondary feature matching points include at least four; based on the multiple secondary feature matching points, a homography matrix is ​​calculated, wherein the number of secondary feature matching points is proportional to the calculation accuracy of the homography matrix.

[0055] Specifically, in an embodiment of the present invention, t and The purpose of performing secondary feature matching on images is to make the algorithm rotationally invariant under the rejection condition that the heading angle of the UAV is unknown, thus ensuring the quality of image matching.

[0056] Furthermore, secondary feature matching is performed on the intermediate second image and the initial first image to obtain multiple secondary feature matching points, wherein the secondary feature matching points include at least four. Based on the multiple secondary feature matching points, a homography matrix is ​​calculated, wherein the number of secondary feature matching points is proportional to the calculation accuracy of the homography matrix, that is, the more matching points there are, the higher the calculation accuracy of the homography matrix.

[0057] In operation S160, two consecutive frame images are spliced ​​based on the homography matrix to complete the real-time update of the remote sensing map.

[0058] In an embodiment of the present invention, two continuous frame images are stitched based on a homography matrix, including: performing a transmission transformation on an intermediate second image based on the homography matrix to obtain a target second image; and stitching two continuous frame images based on an overlapping area between the target second image and the initial first image to complete real-time updating of a remote sensing map.

[0059] Specifically, the homography matrix A point can be mapped from one plane (image) to another plane, which can be used to quantify the image I of adjacent frames. t and The geometric relationship between the t-th frame image I t As the benchmark, for the t+1th frame image (i.e. the second image in the middle) is subjected to transmission transformation to obtain the transformed image of the t+1th frame (i.e. the second target image). A feature point For example, the coordinates of the transformed feature points can be estimated , as shown in the formula:

[0060]

[0061] The more correct matching points there are, the more accurate the calculation of the homography matrix H is. Therefore, the number of correct matching points is an important performance indicator for evaluating image matching.

[0062] Furthermore, the overlapping area between the target second image and the initial first image is obtained by the following method, including: obtaining the coordinates of the four corner points of the target second image based on the homography matrix; determining the overlapping area between the target second image and the initial first image according to the coordinates of the four corner points of the target second image and the known coordinates of the four corner points of the initial first image.

[0063] Specifically, according to the initial first image I t The coordinates of the four corner points are determined by the transformed target second image The four corner points of , thus obtaining the overlapping area between adjacent frame images. The linear weighted fusion method is used, as shown in the following formula:

[0064]

[0065] in, is the weight parameter, usually 0.5, and Represents adjacent frame images respectively and I t In this way, the images of adjacent frames are stitched in sequence to achieve the update of real-time remote sensing maps.

[0066] In the embodiment of the present invention, the t-th frame image I can also be calculated t And the transformed t+1th frame image The center distance (unit: pixel) is recorded as Under the denial condition, although the longitude and latitude, heading angle, flight speed and other information of the drone cannot be obtained, the focal length of the drone itself, the width of the camera sensor, and the flight altitude can be obtained. Therefore, the flight distance of the drone from time t to time t+1 is , as shown below:

[0067]

[0068] in, is the ground sample distance (GSD), which indicates the actual distance on the ground corresponding to a pixel in the aerial image. It can be calculated by the following formula:

[0069]

[0070] Where h is the flight altitude of the drone, b 相机 is the width of the camera sensor (in millimeters), f is the focal length of the camera (in millimeters), b 图像 Indicates the width of the aerial image (unit: pixel). In this way, the flight distance of the drone from time t to time t+1 can be calculated In addition, based on the angle of the main direction of feature matching , therefore, the relative latitude and longitude information of the drone at time t+1 can be obtained as follows:

[0071]

[0072] in, and are the relative longitude and relative latitude, is the scaling factor of latitude to longitude distance (the higher the latitude, the smaller the ground distance covered by the longitude), and R is the radius of the earth. In addition, if the longitude and latitude of the take-off position of the drone are known, the current absolute position information (longitude and latitude) can be calculated in real time based on continuous frames.

[0073] Through the embodiments of the present invention, the matching main direction and actual displacement of adjacent frame images are calculated by taking the weighted average of the matching point feature angles and the matching scores, thereby obtaining the relative longitude and latitude and heading angle information of the UAV, thereby ensuring the accuracy of remote sensing map updates under denial conditions.

[0074] Figure 2A A diagram schematically shows a secondary feature matching result according to an embodiment of the present invention. Figure 2B A diagram schematically shows a feature matching result of the LightGlue image matching method according to an embodiment of the present invention. Figure 3A The figure schematically shows the image stitching imaging result according to an embodiment of the present invention. Figure 3B The image stitching imaging result diagram of the LightGlue image matching method according to an embodiment of the present invention is schematically shown.

[0075] Based on the above-mentioned real-time remote sensing map generation method under denial conditions, real-time map updates under denial conditions are achieved through the terminal deployment of computer vision and algorithms. For the aerial video screenshots of the drone, the pre-trained SuperPoint model and LightGlue model are first used to extract and match the feature points of adjacent frame images, and the main matching direction is estimated through the relative position relationship of the matching points. Then, the next frame image is phase rotated and corrected according to the matching main direction, and the rotated image is feature matched. Finally, the image is stitched, and the results are shown in Figures 2 and 3 respectively. Figure 2A As shown, it is a secondary feature matching result diagram of an embodiment of the present invention. Figure 2B As shown, this is the feature matching result diagram of the LightGlue image matching method.

[0076] It can be seen that in the embodiment of the present invention, when there is a large rotation between adjacent frames, more matching points can still be obtained, such as Figure 2A As shown in FIG. 1 , the matching feature points are represented by connecting lines. The method of the present invention detects 20 pairs of secondary feature matching points. Figure 2B As shown, the LightGlue image matching method can only detect 5, that is, the method of the present invention obtains 15 more matching points, which is 3 times that of the LightGlue image matching method, indicating that the feature matching result of the present invention is better.

[0077] Further, such as Figure 3A As shown in FIG. 1 , it is an image stitching imaging result diagram of an embodiment of the present invention, such as Figure 3B As shown, it is the image stitching imaging result diagram of the LightGlue image matching method. It can be seen that the image stitching effect of the present invention is better, there is no image distortion, and a continuous stitching result is obtained; while only using the LightGlue image matching rule has obvious image distortion. This is because the rotation invariance of the LightGlue image matching method is poor, and under the rejection condition, due to the lack of the heading angle, the angle correction cannot be performed. Therefore, when the rotation angle is large, the image matching cannot be performed accurately, which further affects the generation of real-time remote sensing images and produces image distortion. It can be seen that the method of the present invention has a better stitching effect than directly using the LightGlue image matching method.

[0078] Through the embodiments of the present invention, a technology for performing rotation correction on adjacent frame images using the main direction of feature matching is proposed, which enhances the rotation invariance of the image matching method and solves the image matching and stitching problems when the drone rotates at a large angle and the heading angle is unknown, thereby completing the update of the remote sensing map.

[0079] The above specific embodiments further illustrate the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating a real-time remote sensing map under denial conditions, characterized in that: The method comprises: Acquire multiple frames of remote sensing maps; Based on the pre-trained SuperPoint model, feature points are extracted from each frame of the multiple images to obtain feature points of each frame, where the feature points include multiple ones; Based on the pre-trained LightGlue model, initial feature matching is performed on feature points of any two consecutive frame images in the multiple frame images to obtain multiple initial feature matching points, wherein the two consecutive frame images are an initial first image and an initial second image respectively; According to the multiple initial feature matching points, the initial second image is rotated to obtain an intermediate second image; Performing secondary feature matching on the intermediate second image and the initial first image, and calculating a homography matrix; Based on the homography matrix, two consecutive frame images are spliced ​​to complete the real-time update of the remote sensing map.

2. The real-time remote sensing map generation method according to claim 1, characterized in that: The step of rotating the initial second image according to the plurality of initial feature matching points to obtain an intermediate second image comprises: According to multiple initial feature matching points, the main direction of feature matching is calculated; The initial second image is rotated according to the feature matching main direction to obtain an intermediate second image.

3. The real-time remote sensing map generation method according to claim 1 or 2, characterized in that: The step of stitching two consecutive frame images based on the homography matrix includes: Based on the homography matrix, performing image transmission transformation on the intermediate second image to obtain a target second image, According to the overlapping area between the target second image and the initial first image, two continuous frame images are spliced ​​to complete the real-time update of the remote sensing map.

4. The real-time remote sensing map generation method according to claim 1, characterized in that: The performing secondary feature matching on the intermediate second image and the initial first image and calculating a homography matrix includes: Performing secondary feature matching on the intermediate second image and the initial first image to obtain a plurality of secondary feature matching points; Based on the multiple secondary feature matching points, a homography matrix is ​​calculated.

5. The real-time remote sensing map generation method according to claim 4, characterized in that: The number of the secondary feature matching points is proportional to the calculation accuracy of the homography matrix.

6. The real-time remote sensing map generation method according to claim 3, characterized in that: The overlapping area between the target second image and the initial first image is obtained by the following method, including: Based on the homography matrix, obtaining the coordinates of four corner points of the target second image; An overlapping area between the target second image and the initial first image is determined according to the coordinates of the four corner points of the target second image and the known coordinates of the four corner points of the initial first image.

7. The real-time remote sensing map generation method according to claim 1, characterized in that: The method of obtaining multiple frames of images of a remote sensing map comprises: The multiple frames of images of the remote sensing map are acquired through an image acquisition device or a video shooting device.

8. The real-time remote sensing map generation method according to claim 7, characterized in that: The SuperPoint model and the LightGlue model are both deployed on the image acquisition device by using the RKNN tool.

9. The real-time remote sensing map generation method according to claim 8, characterized in that: The SuperPoint model and the LightGlue model are both deployed on the image acquisition device by using the RKNN tool, including: Pre-training the SuperPoint model and the LightGlue model respectively; Transfer the architecture of the pre-trained SuperPoint model and LightGlue model to the ONNX architecture; The RKNN tool is used to convert the architecture from ONNX to RKNN and deployed on the image acquisition device.

10. The real-time remote sensing map generation method according to claim 4, characterized in that: The secondary feature matching points include at least four.

Images