Heterogeneous remote sensing image matching generalization enhancement method and device

Abstract

The invention discloses a heterogeneous remote sensing image matching generalization enhancement method and device, and relates to the technical field of remote sensing images. Generating a dense matching label between adjacent frames corresponding to the reference frame in the same video by utilizing the image matching model after the preliminary training and combining a complementary image matching method; filtering wrong matching in the dense matching labels through a robust fitting algorithm, and propagating the labels corresponding to accurate matching from adjacent frames to far frames through a matching relationship between frames in the same video to obtain far frame data; enhancing the far-frame data by using a strong data enhancement technology to obtain far-frame data with strong augmentation; and finally training the image matching model on the far-frame data with strong augmentation to obtain an image matching model with enhanced generalization ability so as to realize matching of the unmanned aerial vehicle aerial image and the satellite image. The problem that images in different scenes are difficult to match accurately in practical application in the prior art is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing images, and particularly relates to a heterogeneous remote sensing image matching generalization enhancement method and device. BACKGROUND

[0002] Image matching, as a key technology in the field of computer vision, aims to find the corresponding relationship of the same or similar objects and scenes in different images, and has a wide range of applications in aircraft autonomous navigation, three-dimensional reconstruction, target recognition and other fields. For example, in unmanned aerial vehicle aerial photography, image matching can determine the relationship between ground objects photographed by the unmanned aerial vehicle at different positions, thereby achieving precise positioning and path planning. In actual application scenarios, image matching faces many challenges, such as rotation variation, scale difference, and light difference, which significantly increase the difficulty of image matching.

[0003] Currently, some image matching models use the method of training separate models for different scene types (such as desert, field, and city). This specific scene training method results in poor generalization effect of the model on scenes that have not been previously encountered. In terms of data construction, the commonly used methods currently include methods relying on RGBD scanning, and methods from motion recovery (SfM) and multi-angle stereo vision (MVS), but the label construction of existing training data is difficult, which limits the diversity of standard image matching data sets.

[0004] Although the existing technology improves the performance of the image matching model to some extent, due to the limited nature of the training data scenes, the generalization ability of the model is still greatly limited. This lack of generalization ability directly makes it difficult to accurately match images under different scenes in actual applications, causing difficulties in the image matching task. SUMMARY

[0005] In the embodiments of the present application, by providing a heterogeneous remote sensing image matching generalization enhancement method, the problem that the existing technology is difficult to accurately match images under different scenes in actual applications, causing difficulties in the image matching task, is solved.

[0006] In a first aspect, the embodiments of the present application provide a heterogeneous remote sensing image matching generalization enhancement method, which comprises: preliminarily training an image matching model on a data set; wherein the data set comprises a standard data set for an image matching task and a data pair artificially labeled and composed of an unmanned aerial vehicle aerial photograph and a paired aerial photograph; using the preliminarily trained image matching model and combining a complementary image matching method, generating dense matching labels between nearby frames corresponding to a reference frame within the same video for satellite videos and unmanned aerial vehicle aerial videos respectively; filtering the matching errors in the dense matching labels by using a robust fitting algorithm, and propagating the labels corresponding to the accurate matching from the adjacent frames to the far frames by using the matching relationship between the frames within the same video, to obtain far frame data; using strong data enhancement technology to enhance the far frame data to obtain far frame data with strong augmentation; the strong data enhancement technology comprises extreme dark light environment data enhancement and random data enhancement; finally training the image matching model on the far frame data with strong augmentation to obtain an image matching model with enhanced generalization capability, so as to realize the matching of unmanned aerial vehicle aerial photographs and satellite photographs.

[0007] In a possible implementation manner, the step of using the preliminarily trained image matching model and combining a complementary image matching method to generate dense matching labels between nearby frames corresponding to a reference frame within the same video for satellite videos and unmanned aerial vehicle aerial videos respectively comprises: selecting a frame as a reference frame every first preset frame number for the satellite videos and the unmanned aerial vehicle aerial videos; selecting a frame as a nearby frame every second preset frame number from each determined reference frame to construct a corresponding relationship between the reference frame and the nearby frame; and using the preliminarily trained image matching model and combining a complementary image matching method to perform feature extraction and matching on the reference frame and the corresponding nearby frame, to generate a matching corresponding relationship between the reference frame and the nearby frame, so as to generate the dense matching labels between the nearby frames corresponding to the reference frame within the same video.

[0008] In a possible implementation manner, the step of filtering the matching errors in the dense matching labels by using a robust fitting algorithm and propagating the labels corresponding to the accurate matching from the adjacent frames to the far frames by using the matching relationship between the frames within the same video to obtain far frame data comprises: defining the matching relationship between the adjacent frames as a matching matrix: wherein C AB is a matching matrix of frame I A and frame I B adjacent to frame I A , r A is a pixel number in frame I B , r B is a pixel number of frame I , and matrix element C A is used to identify whether pixel i in frame I BCorresponding to pixel j in , if corresponding then otherwise For the matching matrix C AB Each matrix element with a value of 1 is determined to be frame I A Pixel i in the adjacent frame I B There is a matching relationship between pixel j in frame I; define frame I B and adjacent frame I C The matching matrix is ​​C BC , if the matching matrix C BC There are matrix elements in And frame I B The distance between pixels j and j′ in frame I is less than 1 pixel. A Pixel i in frame I C There is also a matching relationship between pixel k in frame I; where j′ is the pixel k in frame I B Middle and frame I C There is another pixel in the matching relationship between pixel k and pixel k; in the matching matrix C AC In the example, set the matrix elements To achieve accurate matching, the corresponding labels are propagated from adjacent frames to distant frames.

[0009] In a possible implementation, the method filters out erroneous matches in dense matching labels through a robust fitting algorithm, and propagates accurate matching corresponding labels from adjacent frames to distant frames through matching relationships between frames within the same video to obtain distant frame data, and further includes: setting a preset number threshold for determining whether to perform a matching relationship propagation operation; when the number of corresponding matching points between two frames is greater than the preset number threshold, performing a matching relationship propagation operation; wherein the matching relationship propagation operation is performed in an order of gradually doubling the propagation frame interval; the matching relationship propagation operation includes: performing a matching relationship propagation operation on every N frames. A first propagation operation of the matching relationship is performed to obtain a first matching relationship; after the first propagation operation is completed, a second propagation operation of the matching relationship is performed on every 2N frames to obtain a second matching relationship; the second matching relationship is merged with the first matching relationship to obtain a third matching relationship; a third propagation operation of the matching relationship is performed on every 4N frames to obtain a fourth matching relationship; the fourth matching relationship is merged with the third matching relationship; the propagation operation and the merging operation of the matching relationship are continuously performed, and the propagation frame interval is doubled each time, until the number of corresponding matching points between the two frames is less than or equal to the preset number threshold, so as to obtain far frame data.

[0010] In a possible implementation, the data enhancement of an extremely dark light environment is performed for each pair of frames having a matching relationship, and specifically includes: performing directional brightness reduction processing on the frames according to a preset brightness reduction amplitude; performing contrast disturbance on the frames after the brightness reduction is completed, according to a preset contrast range; performing blur processing on the frames, adding static blur effects, and simulating linear motion blur; adjusting the brightness of the frames through a power law transformation to perform gamma correction; reducing the saturation of the frames according to a preset saturation range to suppress color saturation.

[0011] In a possible implementation, random data enhancement is performed for each pair of frames having a matching relationship, and specifically includes: adding motion blur to the frames to perform dynamic interference; randomly adjusting the brightness, contrast, saturation, and hue of the frames to simulate changes in the frames under different lighting conditions; adding lines and water droplet occlusions to the frames to simulate occlusion in actual scenes; simulating lens backlight or strong light source interference; performing compression processing on the frames according to a preset compression quality range; and adding sensor noise to the frames to simulate noise points in a dark light environment.

[0012] In a possible implementation, the final training of the image matching model on the remote frame data with strong augmentation is performed to obtain the image matching model with enhanced generalization capability, so as to realize the matching of the unmanned aerial vehicle aerial image and the satellite image.

[0013] In a second aspect, the embodiments of the present application provide a device for enhancing generalization of heterogeneous remote sensing image matching, which comprises: a preliminary training module configured to preliminarily train an image matching model on a data set; wherein the data set comprises a standard data set for an image matching task and a data pair artificially labeled and composed of an unmanned aerial vehicle aerial image and a paired aerial image; a label generation module configured to generate dense matching labels between nearby frames corresponding to an internal reference frame of a same video by using the preliminarily trained image matching model and combining a complementary image matching method, respectively for a satellite video and an unmanned aerial vehicle aerial video; a remote frame data obtaining module configured to filter incorrect matching in the dense matching labels by using a robust fitting algorithm, and propagate labels corresponding to accurate matching from adjacent frames to remote frames by using matching relationships between internal frames and frames of a same video to obtain remote frame data; an enhancement module configured to enhance the remote frame data by using strong data enhancement technology to obtain remote frame data with strong augmentation; the strong data enhancement technology comprises data enhancement of an extremely dark light environment and random data enhancement; and a final training module configured to finally train the image matching model on the remote frame data with strong augmentation to obtain an image matching model with enhanced generalization capability, so as to realize the matching of the unmanned aerial vehicle aerial image and the satellite image.

[0014] In a third aspect, the embodiments of the present application provide a heterogeneous remote sensing image matching generalization enhancement server, comprising a memory and a processor; the memory is configured to store computer executable instructions; and the processor is configured to execute the computer executable instructions to implement the method in the first aspect or any possible implementation manner of the first aspect.

[0015] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium, which stores executable instructions, and the computer executes the executable instructions to implement the method in the first aspect or any possible implementation manner of the first aspect.

[0016] The one or more technical solutions provided in the embodiments of the present application have at least the following technical effects: the embodiments of the present application provide a heterogeneous remote sensing image matching generalization enhancement method, which preliminarily trains an image matching model on a data set, and the data set of the present application includes a standard data set for an image matching task and a data pair artificially annotated and composed of an unmanned aerial vehicle aerial photograph and a paired aerial photograph, which greatly enriches the diversity of training data. After preliminary training, in combination with a complementary image matching method, dense matching labels between nearby frames corresponding to an internal reference frame of the same video are generated for satellite videos and unmanned aerial vehicle aerial videos, and a robust fitting algorithm is used to filter false matching, and accurate matching labels are propagated from adjacent frames to distant frames, further expanding the scene coverage of the training data. The application of the robust fitting algorithm filters out the false matching in the dense matching labels, ensuring the quality of the matching labels. At the same time, the accurate matching labels are propagated from the adjacent frames to the distant frames, so that the distant frame data also has high-quality matching labels, providing a more reliable data basis for the training of the model. The strong data enhancement technology simulates various extreme and complex scene conditions, so that the model can be exposed to more challenging samples during the training process, thereby improving the matching accuracy of the model in actual application. Under different light conditions, different angles, and the presence of occlusions, the model can more accurately find the corresponding relationship between the unmanned aerial vehicle aerial photograph and the satellite photograph, providing more accurate image matching results for applications such as autonomous navigation of aircraft. The method solves the problem that the prior art cannot accurately match images in different scenes in actual application, causing difficulties in image matching tasks. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application or the prior art. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0018] Figure 1 A flowchart of a heterogeneous remote sensing image matching generalization enhancement method provided for an embodiment of the present application is shown in FIG. 1.

[0019] Figure 2 A flowchart of a heterogeneous remote sensing image matching generalization enhancement method provided for an embodiment of the present application is shown in FIG. 1.

[0020] Figure 3 A comparison chart of matching quality visualization effects of a raw image matching model and a generalization capability enhanced image matching model of the present application on a complex heterogeneous remote sensing image dataset provided for an embodiment of the present application is shown in FIG. 2.

[0021] Figure 4 A comparison chart of positioning results of the present application and GPS in actual 1000-meter high-altitude flight experiments provided for an embodiment of the present application is shown in FIG. 3.

[0022] Figure 5 A schematic diagram of a heterogeneous remote sensing image matching generalization enhancement device provided for an embodiment of the present application is shown in FIG. 4.

[0023] Figure 6 A schematic diagram of a heterogeneous remote sensing image matching generalization enhancement server provided for an embodiment of the present application is shown in FIG. 5. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the scope of protection of the present application.

[0025] The following describes some technologies related to the embodiments of the present application to help understanding, which should be considered only as exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Also, for the sake of clarity and conciseness, the following description omits the description of some well-known functions and structures.

[0026] The embodiments of the present application provide a heterogeneous remote sensing image matching generalization enhancement method, as shown in FIG. 1, which includes steps S101 to S105. Among them, Figure 1 Only one execution order is shown for the embodiments of the present application, which does not represent the only execution order of a heterogeneous remote sensing image matching generalization enhancement method. As long as the final result can be achieved, Figure 1 The steps shown can be executed in parallel or in reverse. Figure 1

[0027] ​S101: Preliminary training of the image matching model on a data set. The data set includes a standard data set for the image matching task and a manually annotated data pair composed of a UAV aerial photograph and a paired aerial photograph.

[0028] Specifically, the standard data set can be MegaDepth. In the data acquisition stage of the manually annotated data pair composed of a UAV aerial photograph and a paired aerial photograph, a high-precision camera is carried by a UAV, and a large number of UAV aerial images are obtained by multi-angle and multi-temporal aerial photography of a specific area according to a preset flight trajectory and shooting parameter. At the same time, other aerial images corresponding to the UAV aerial photography area, i.e., the UAV aerial photograph and the paired aerial photograph, are collected, which can be derived from satellite shooting, manned aerial photography, or images of the same area taken by the same UAV at different times and under different conditions. The image matching model can be GIM.

[0029] S102: Using the preliminarily trained image matching model and combining a complementary image matching method, dense matching labels between nearby frames corresponding to a reference frame in the same video are generated for satellite videos and UAV aerial videos, respectively.

[0030] Figure 2 A flowchart of a heterogeneous remote sensing image matching generalization enhancement method provided by an embodiment of the present application.

[0031] Using the preliminarily trained image matching model and combining a complementary image matching method, dense matching labels between nearby frames corresponding to a reference frame in the same video are generated for satellite videos and UAV aerial videos, respectively, including the following steps.

[0032] Every first preset frame number of the satellite videos and the UAV aerial videos is selected as a reference frame.

[0033] Starting from each determined reference frame, every second preset frame number is selected as a nearby frame thereof, and a corresponding relationship between the reference frame and the nearby frame is constructed.

[0034] Using the preliminarily trained image matching model and combining a complementary image matching method, feature extraction and matching are performed on the reference frame and the nearby frame corresponding thereto, a matching corresponding relationship between the reference frame and the nearby frame is generated, and dense matching labels between nearby frames corresponding to a reference frame in the same video are generated.

[0035] Specifically, the frame in the present application is a video frame. The first preset frame number can be 20 frames. The second preset frame number can be 20 frames. This sampling strategy effectively reduces the redundant information between frames and reduces the computational complexity. For the selected reference frame X, it is respectively combined with the subsequent 20th frame (X+20), the 40th frame (X+40) and the 80th frame (X+80) to form frame pairs, that is, {X, X+20}, {X, X+40} and {X, X+80}, thereby constructing the corresponding relationship between the reference frame and the nearby frame.

[0036] The image matching model after preliminary training is used to extract and match the features of the reference frame and its corresponding nearby frame, and the matching corresponding relationship between {X, X+20}, {X, X+40} and {X, X+80} is generated. The complementary image matching method of the present application adopts RootSIFT, SuperGlue, LoFTR and DKM as complementary models, and this multi-model integration strategy fully embodies the idea of model fusion. Through this fusion, the density of the label is significantly improved. The model of the present application can learn the unique advantages of each complementary method. (1) Inherit the stability of RootSIFT to illumination changes, scale transformations and rotation changes, and maintain the robustness of traditional SIFT. (2) It has excellent performance in complex dynamic scenes (such as motion blur, sudden changes in viewing angle). (3) It obtains the matching ability of LoFTR in low-texture areas and challenging scenes such as repetitive patterns. (4) It integrates the advantages of DKM in cross-view matching scenes, especially the strong generalization ability to perspective deformation and non-rigid deformation. This multi-model fusion strategy not only improves the quality and diversity of the training labels, but also enables the final trained model to integrate the advantages of each complementary method, and to perform excellent matching performance in various complex scenes.

[0037] Further, the matching corresponding relationship between the reference frame and the nearby frame generated by the present application can also be corrected by RANSAC robust fitting. The generation of dense matching labels between nearby frames corresponding to the same video reference frame is to establish matching corresponding relationships for numerous pixel points in the reference frame and the nearby frame, and these relationships constitute the dense matching labels.

[0038] S103: Filter the incorrect matches in the dense matching labels through a robust fitting algorithm, and through the matching relationship between the frames in the same video, propagate the accurate matching corresponding labels from the adjacent frames to the distant frames to obtain the distant frame data.

[0039] Specifically, in the field of image matching, existing image matching methods rely on strong supervision signals from small overlapping images, but multi-method matching is difficult to achieve. Because even if the most advanced robust fitting algorithm is used to filter outliers of the matching results, the corresponding relationship generated by the existing method is no longer reliable when the frame interval exceeds 80 frames. But learning based on video data has significant advantages. The dense correspondence between video frames and nearby frames is usually located at the same pixel position, which makes it possible to propagate the matching relationship to distant frames.

[0040] Filtering the wrong matches in the dense matching labels by a robust fitting algorithm, and propagating the accurate matching corresponding labels from the adjacent frames to the distant frames through the matching relationship between the frames within the same video, obtaining the distant frame data, including the following steps.

[0041] Defining the matching relationship between adjacent frames as a matching matrix: Wherein, C AB is the matching matrix of frame I A and adjacent frame I B , r A is the number of pixels in frame I A , r B is the number of pixels in frame I B , and the matrix element is used to identify whether the pixel i in frame I A corresponds to the pixel j in frame I B , if it corresponds Otherwise

[0042] For each matrix element with a value of 1 in the matching matrix C AB , it is determined that there is a matching relationship between the pixel i in frame I A and the pixel j in the adjacent frame I B .

[0043] Defining the matching matrix of frame I B and adjacent frame I C as C BC , if there is a matrix element in the matching matrix C BC and the distance between the pixels j and j' in frame I B is less than 1 pixel, then the pixel i in frame I A also has a matching relationship with the pixel k in frame I C . Wherein, j' is another pixel in frame I B which has a matching relationship with the pixel k in frame I C .

[0044] In the matching matrix C AC , the matrix element implementing a preliminary propagation of accurate matching corresponding labels from neighboring frames to distant frames.

[0045] filtering out the false matches in the dense matching labels by a robust fitting algorithm, and propagating the accurate matching corresponding labels from neighboring frames to distant frames by the matching relationship between the frames within the same video to obtain the distant frame data, further comprising: setting a preset number threshold for determining whether to perform the propagation operation of the matching relationship. When the number of corresponding matching points between two frames is greater than the preset number threshold, the propagation operation of the matching relationship is performed. The propagation operation of the matching relationship is performed in an order of gradually doubling the frame interval of propagation. The propagation operation of the matching relationship comprises: performing a first propagation operation of the matching relationship on every Nth frame to obtain a first matching relationship. After the first propagation operation is completed, a second propagation operation of the matching relationship is performed on every 2Nth frame to obtain a second matching relationship. The second matching relationship is merged with the first matching relationship to obtain a third matching relationship. A third propagation operation of the matching relationship is performed on every 4Nth frame to obtain a fourth matching relationship. The fourth matching relationship is merged with the third matching relationship. The propagation operation and the merging operation of the matching relationship are continuously performed, and the frame interval of propagation is doubled each time until the number of corresponding matching points between two frames is less than or equal to the preset number threshold, so as to obtain the distant frame data.

[0046] Specifically, in order to obtain strong supervision information, a preset number threshold (1024) is set, and when the number of corresponding matching points between two frames is greater than the preset number threshold, the corresponding matching relationship is propagated as much as possible. The specific operation is as follows: the propagation operation of the matching relationship is performed on each sampling frame. In this application, the sampling frame interval is set to every Nth frame, for example, N = 20 frames. Specifically, the basic matching relationship propagation is initially established between every 20th, 40th and 80th frame. After a round of propagation, the propagation range is expanded from every 20th matching point to every 40th, and the matching relationship obtained by propagation is merged with the basic matching relationship to enhance the density and accuracy of matching. After obtaining the matching corresponding relationship between every 40th interval frame, the same operation is performed to generate the matching relationship transmission of every 80th frame. Since there is no basic matching relationship propagation exceeding 80 frames in the initial stage, the remaining propagation rounds do not need to perform the merging operation, but continue to double the frame interval until the number of matching points does not exceed the preset number threshold of 1024. The reason for forcibly setting the 1024 matching point number threshold is to balance the difficulty of the learning problem and avoid the model from being biased towards difficult or easy samples. Although the standard method of uniformly sampling from different overlap ratios can also be adopted, simply limiting the number of corresponding and saving the farthest frame pair as the final training data can save more space and computing resources.

[0047] S104: The remote frame data is enhanced using strong data augmentation techniques to obtain remote frame data with strong augmentation. The strong data augmentation techniques include extreme low-light environment data augmentation and random data augmentation.

[0048] For each pair of frames with a matching relationship, extreme low-light environment data augmentation is performed, specifically including the following steps. The frames are subjected to directional brightness reduction processing according to a preset brightness reduction amplitude. The preset brightness reduction amplitude is 60%. After the brightness reduction is completed, the frames are subjected to contrast disturbance according to a preset contrast range. The frames are subjected to blurring processing, adding static blur effects, and simulating linear motion blur. The preset contrast range is -50% to +30%. Static blur uses a 3x3 to 9x9 Gaussian kernel to blur the image. The brightness of the frames is adjusted through a power-law transformation for gamma correction. The saturation of the frames is reduced according to a preset saturation range to suppress color saturation. The preset saturation range is 40% to 100%.

[0049] Table 1 is a description table of the extreme low-light environment data augmentation method and augmentation disturbance.

[0050] Table 1

[0051]

[0052] For each pair of frames with a matching relationship, random data augmentation is performed, specifically including the following steps. Motion blur is added to the frames for dynamic interference. The brightness, contrast, saturation, and hue of the frames are randomly adjusted to simulate changes in the frames under different lighting conditions. Lines and water droplet obstructions are added to the frames to simulate obstruction situations in actual scenes. Lens back light or strong light source interference is simulated. The frames are subjected to compression processing according to a preset compression quality range. The preset compression quality range is 50-100. Sensor noise is added to the frames to simulate noise points in a dark environment.

[0053] Table 2 is a description table of the random data augmentation method and augmentation disturbance.

[0054] Table 2

[0055]

[0056] S105: The image matching model is finally trained on the remote frame data with strong augmentation to obtain an image matching model with enhanced generalization ability to realize matching of unmanned aerial vehicle aerial images and satellite images.

[0057] The image matching model is finally trained on the remote frame data with strong augmentation to obtain an image matching model with enhanced generalization ability to realize matching of unmanned aerial vehicle aerial images and satellite images, including: alternately inputting satellite videos and unmanned aerial vehicle aerial videos in the final training process.

[0058] Specifically, in the practice link, in order to obtain an image matching model with strong generalization ability to realize the matching of unmanned aerial vehicle aerial images and satellite images, in the generation of video training data, the main calculation overhead is concentrated on the dense matching label. Moreover, the average processing time of each frame does not increase significantly with the increase of the length of the input video, which makes this method perform well in efficiency and versatility, and thus can efficiently expand the scale of video training. Specifically, the application can process 1.25 hours of video per day using 1 A100 GPU. In this training, a total of 30 hours of video was used, half of which was aerial video and half of which was satellite video, and the model was alternately fed during the training process. This alternating input training method helps the model to learn the different features of unmanned aerial vehicle aerial images and satellite images at the same time, enhancing the adaptability of the model to images of two different sources. The 30 hours of video can generate 100k pairs of training data, providing a rich sample for the training of the model.

[0059] In the experimental verification link, the proposed generalization enhancement method for heterogeneous remote sensing image matching is researched in multiple aspects, including experimental result comparison, ablation experiment analysis and application result display, which fully verifies the effectiveness and advantages of the application.

[0060] The original image matching model and the generalization ability enhanced image matching model of the application are tested and compared using the test set, and the indicators are set consistently. The test set is 500 pairs of typical scene images, covering weak texture, rotation, scale difference, etc.

[0061] Table 3 is a test and comparison table of the original image matching model and the generalization ability enhanced image matching model of the application. Among them, OURs is the generalization ability enhanced image matching model of the application. px is pixel. Model is model. Matches is the number of matches. AUC is the area under the curve, and the higher the value, the better the model performance. Rep is the degree of repetition, and the higher the value, the better the model performance. H error is the reprojection error, and the lower the value, the better. The arrow in the table indicates that the higher the corresponding indicator, the better. The arrow in the table indicates that the lower the corresponding indicator, the better. The generalization ability enhanced image matching model has a significant improvement in matching accuracy compared to the original image matching model, and the repetition degree and reprojection error are also optimized, fully proving the effectiveness of the application in improving the performance of the model.

[0062] Table 3

[0063]

[0064] Figure 3 Figure 1 is a comparison chart of the matching quality visualization effect of the original image matching model provided by the embodiment of the application and the generalization ability enhanced image matching model of the application on the complex heterogeneous remote sensing image dataset.Figure 3 (a), (c), and (e) are the matching quality visualization effects of the original image matching model. Figure 3 (b), (d), and (f) are visualization effects of the matching quality of the image matching model with enhanced generalization ability. For samples in the test set where the original image matching model has large matching errors, the image matching model with enhanced generalization ability of this application can all be correctly matched. The number of matching pairs output by the image matching model with enhanced generalization ability far exceeds that of the original image matching model, and the consistency of the matching pairs is good. The matching pairs of the original image matching model are sparse and messy, and some erroneous matching pairs result in a large difference between the projection matrix and the true value. The image matching model with enhanced generalization ability can obtain dense and consistent matching pairs in a variety of scenarios, and the projection relationship is closer to the true value.

[0065] To investigate the impact of training video data length and content on the model, we conducted ablation experiments, as shown in Table 4. Table 4 shows the results of the strategy comparison experiments. The experiment found that the more video data used for training, the stronger the model's matching ability. When training with only aerial video, the model's performance actually decreased with more training data, indicating that the model overfits to aerial scenes and cannot achieve robust feature extraction from both aerial and satellite imagery. Therefore, we ultimately chose to alternate between aerial and satellite video for model training to ensure that the model does not overfit to either scene and can extract reliable feature points for matching in both images.

[0066] Table 4

[0067]

[0068] The generalized enhancement method for heterogeneous remote sensing image matching proposed in this application provides core technical support for autonomous and precise positioning of aircraft. It can be deployed on an airborne edge terminal board as a plug-and-play autonomous navigation module to meet the needs of various scenarios. To verify the effectiveness of this application, the algorithm was carried out on a DJI M600 rotary-wing aircraft. The experimental system includes a gimbal camera, an airborne edge terminal board, etc. During the autonomous navigation test phase, the algorithm was evaluated with reference to GPS receiver information, and a wireless image transmission device was used to monitor the matching and positioning results in real time. The experiment was divided into two phases: offline and actual flight testing. The offline test directly uses aerial video input to model inference and output matching results, and confirms the longitude and latitude position based on satellite images. In the actual flight test, a high-resolution satellite image library and feature library are prepared in advance and stored in the drone terminal. During the flight, the gimbal camera shoots and transmits video in real time. After image retrieval and matching inference, the real-time position is determined in combination with satellite image information.

[0069] Figure 4A comparison chart of the positioning effect of the application and the positioning result of GPS in the actual 1000-meter high altitude flight experiment is provided for the embodiments of the application. The yellow color is the positioning result of the application, and the red color is the GPS positioning result. It can be seen that the positioning accuracy of the application is higher.

[0070] The experimental results show that the positioning error of the application can be controlled within 3% of the flight height under the flight height of 50 to 9000 meters, the flight speed of 100 meters per second, and various terrain environments and light conditions. Compared with the traditional inertial navigation system, the positioning error does not accumulate with time, and the reliability is higher. The aircraft carrying the method of the application has strong anti-interference ability, high cost-effectiveness, good portability, and other practical application values, and can be widely applied in agricultural monitoring, logistics distribution, emergency rescue and other fields, and has important theoretical value and practical application significance.

[0071] The embodiments of the application also provide a heterogeneous remote sensing image matching generalization enhancement device 500, as shown in Figure 5 The device comprises a preliminary training module 501, a label generation module 502, an obtained far frame data module 503, an enhancement module 504 and a final training module 505.

[0072] The preliminary training module 501 is used for preliminarily training the image matching model on the data set. The data set comprises a standard data set for an image matching task and a data pair artificially annotated and composed of an unmanned aerial vehicle aerial photograph and a matched aerial photograph.

[0073] The label generation module 502 is used for generating dense matching labels between nearby frames corresponding to the internal reference frames of the same video by using the preliminarily trained image matching model and combining a complementary image matching method, respectively for satellite videos and unmanned aerial vehicle aerial videos.

[0074] The obtained far frame data module 503 is used for filtering the incorrect matching in the dense matching labels by a robust fitting algorithm, and propagating the labels corresponding to the accurate matching from the adjacent frames to the far frames by the matching relationship between the internal frames and the frames of the same video, to obtain the far frame data.

[0075] The enhancement module 504 is used for enhancing the far frame data by using a strong data enhancement technology to obtain far frame data with strong augmentation. The strong data enhancement technology comprises data enhancement in an extreme dark light environment and random data enhancement.

[0076] The final training module 505 is used for finally training the image matching model on the far frame data with strong augmentation to obtain the image matching model with enhanced generalization capability, so as to realize the matching between the unmanned aerial vehicle aerial photograph and the satellite photograph.

[0077] Some of the modules in the apparatus described in the present application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, classes, etc., that perform particular tasks or implement particular abstract data types. The present application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices.

[0078] The apparatus or modules described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above apparatus is described as various modules with functions. In the implementation of the embodiments of the present application, the functions of the modules can be implemented in one or more software and / or hardware. Of course, the modules implementing certain functions can also be implemented by a combination of multiple sub-modules or sub-units.

[0079] The methods, apparatuses or modules described in the present application can be implemented in a computer-readable program code in any appropriate manner, for example, the controller can take the form of, for example, a microprocessor or a processor, and a computer-readable medium storing computer-readable program code (such as software or firmware) executable by the (micro) processor, logic gates, switches, application specific integrated circuits (Application Specific Integrated Circuit, abbreviated as: ASIC), programmable logic controllers and embedded microcontrollers. Examples of the controller include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that, in addition to implementing the controller in a pure computer-readable program code manner, the same function can also be implemented by logically programming the method steps in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers. Therefore, such a controller can be considered as a hardware component, and the means included therein for implementing various functions can also be regarded as structures within the hardware component. Alternatively, the means for implementing various functions can also be regarded as both software modules implementing the method and structures within the hardware component.

[0080] As Figure 6As shown, the embodiment of the present application further provides a heterogeneous remote sensing image matching generalization enhancement server, comprising a memory 601 and a processor 602; the memory 601 is used for storing computer executable instructions; the processor 602 is used for executing the computer executable instructions to realize the heterogeneous remote sensing image matching generalization enhancement method provided by the embodiment of the present application.

[0081] The embodiment of the present application further provides a computer readable storage medium, which stores executable instructions, and when a computer executes the executable instructions, the computer readable storage medium can realize the heterogeneous remote sensing image matching generalization enhancement method provided by the embodiment of the present application.

[0082] From the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and necessary hardware. Based on such an understanding, the technical solutions of the present application can be embodied in the form of a software product or in the form of data migration in the implementation process. The computer software product can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions for causing a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the method described in the embodiments of the present application.

[0083] The various embodiments in the specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment mainly describes the difference from other embodiments. The whole or part of the present application can be used in many general or special computer system environments or configurations.

[0084] The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the present application.

Claims

1. A generalized enhancement method for heterogeneous remote sensing image matching, characterized in that: include: Initially train the image matching model on a dataset consisting of a standard dataset for image matching tasks and manually annotated pairs of drone aerial images and paired aerial images. Using the preliminarily trained image matching model and combining it with a complementary image matching method, we generate dense matching labels between nearby frames corresponding to the reference frame within the same video for satellite videos and drone aerial videos respectively. The robust fitting algorithm is used to filter out incorrect matches in dense matching labels, and through the matching relationship between frames within the same video, the labels corresponding to accurate matches are propagated from adjacent frames to distant frames to obtain distant frame data; Using strong data enhancement technology to enhance the teleframe data to obtain teleframe data with strong augmentation; the strong data enhancement technology includes data enhancement for extreme dark light environments and random data enhancement; The image matching model is finally trained on the far-frame data with strong augmentation to obtain an image matching model with enhanced generalization ability, which can realize the matching of UAV aerial images and satellite images.

2. The generalization enhancement method for heterogeneous remote sensing image matching according to claim 1, characterized in that: The method utilizes the preliminarily trained image matching model and combines it with the complementary image matching method to generate dense matching labels between nearby frames corresponding to the reference frame within the same video for satellite videos and drone aerial videos, respectively, including: Selecting one frame from the satellite video and the drone aerial video as a reference frame every first preset number of frames; Starting from each determined reference frame, a frame is selected every second preset number of frames as its neighboring frame, and a correspondence relationship between the reference frame and the neighboring frames is established; The preliminarily trained image matching model is used in combination with the complementary image matching method to extract and match the reference frame and its corresponding nearby frames, and the matching correspondence between the reference frame and the nearby frames is generated to generate dense matching labels between the nearby frames corresponding to the reference frame in the same video.

3. The generalization enhancement method for heterogeneous remote sensing image matching according to claim 2, characterized in that: The method filters out incorrect matches in dense matching labels through a robust fitting algorithm, and propagates accurate matching labels from adjacent frames to distant frames through matching relationships between frames within the same video to obtain distant frame data, including: The matching relationship between adjacent frames is defined as a matching matrix: Among them, C AB Frame I A and adjacent frame I B The matching matrix, r A Frame I A The number of pixels in r B Frame I B The number of pixels, matrix elements Used to identify frame I A Is pixel i in frame I B Corresponding to pixel j in , if corresponding then otherwise For the matching matrix C AB Each matrix element with a value of 1 is determined to be frame I A Pixel i in the adjacent frame I B There is a matching relationship between pixel j in; Define frame I B and adjacent frame I C The matching matrix is ​​C BC , if the matching matrix C BC There are matrix elements in And frame I B The distance between pixels j and j′ in frame I is less than 1 pixel. A Pixel i in frame I C There is also a matching relationship between pixel k in frame I; where j′ is the pixel k in frame I B Middle and frame I C There is another pixel in the pixel k that matches; In the matching matrix C AC In the example, set the matrix elements To achieve accurate matching, the corresponding labels are propagated from adjacent frames to distant frames.

4. The generalization enhancement method for heterogeneous remote sensing image matching according to claim 3, characterized in that: The method further includes filtering out incorrect matches in dense matching labels by a robust fitting algorithm, and propagating accurate matching corresponding labels from adjacent frames to distant frames through matching relationships between frames within the same video to obtain distant frame data. Set a preset number threshold to determine whether to perform the propagation operation of the matching relationship; When the number of corresponding matching points between the two frames is greater than a preset number threshold, a propagation operation of the matching relationship is performed; wherein the propagation operation of the matching relationship is performed in an order of gradually doubling the propagation frame interval; The propagation operations of the matching relationship include: Performing a first propagation operation of the matching relationship on every N frames to obtain a first matching relationship; After the first propagation operation is completed, performing a second propagation operation of the matching relationship on every 2N frames to obtain a second matching relationship; Merge the second matching relationship with the first matching relationship to obtain a third matching relationship; Performing a third propagation operation of the matching relationship on every 4N frames to obtain a fourth matching relationship; Merge the fourth matching relationship with the third matching relationship; The propagation operation and the merging operation of the matching relationship are continuously performed, and the propagation frame interval is doubled each time until the number of corresponding matching points between the two frames is less than or equal to the preset number threshold, so as to obtain the far frame data.

5. The generalization enhancement method for heterogeneous remote sensing image matching according to claim 4, characterized in that: This includes performing data enhancement for extreme dark light environments for each pair of matching frames, specifically: Performing directional brightness reduction processing on the frame according to the preset brightness reduction amplitude; After the brightness reduction is completed, the contrast is disturbed according to the preset contrast range; the frame is blurred, a static blur effect is added, and linear motion blur is simulated; Adjust the brightness of the frame through power law transformation and perform gamma correction; Reduces the saturation of the frame and suppresses color saturation according to the preset saturation range.

6. The generalization enhancement method for heterogeneous remote sensing image matching according to claim 5, characterized in that: This involves performing random data augmentation on each pair of matching frames, specifically: Add motion blur to the frame for dynamic interference; Randomly adjust the brightness, contrast, saturation and hue of the frame to simulate the changes of the frame under different lighting conditions; Add lines and water splashes and raindrops to the frame to simulate the occlusion in the actual scene; Simulate lens backlight or strong light source interference; Compress the frames according to the preset compression quality range; Adds sensor noise to frames to simulate noise in low-light environments.

7. The generalization enhancement method for heterogeneous remote sensing image matching according to claim 6, characterized in that: The image matching model is finally trained on the far-frame data with strong augmentation to obtain an image matching model with enhanced generalization capability to achieve matching between drone aerial images and satellite images, including: Satellite videos and drone aerial videos are input alternately during the final training process.

8. A generalization enhancement device for heterogeneous remote sensing image matching, characterized in that: include: A preliminary training module, used to perform preliminary training on the image matching model on a dataset, including a standard dataset for image matching tasks and manually annotated pairs of drone aerial images and paired aerial images; The label generation module is used to use the preliminarily trained image matching model and the complementary image matching method to generate dense matching labels between nearby frames corresponding to the reference frame in the same video for satellite video and drone aerial video respectively; Obtaining far-frame data module is used to filter out incorrect matches in dense matching labels through a robust fitting algorithm, and propagate the labels corresponding to accurate matches from adjacent frames to far-frames through the matching relationship between frames within the same video to obtain far-frame data; An enhancement module is used to enhance the teleframe data using a strong data enhancement technique to obtain teleframe data with strong augmentation; the strong data enhancement technique includes data enhancement for extreme dark environments and random data enhancement; The final training module is used to perform final training on the image matching model on the far-frame data with strong augmentation, obtaining an image matching model with enhanced generalization ability to achieve matching between drone aerial images and satellite images.

9. A generalized enhanced server for heterogeneous remote sensing image matching, characterized in that: including memory and processor; The memory is used to store computer-executable instructions; The processor is configured to execute the computer-executable instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores executable instructions, and when a computer executes the executable instructions, the method according to any one of claims 1 to 7 can be implemented.

Images