A method and device for weakly paired image pseudo-label annotation

Abstract

This application discloses a method and apparatus for pseudo-labeling weakly paired images. The method includes: aligning heterogeneous geospatial data to construct aerial satellite image pairs, and extracting matching points to obtain a first matching point pair; quantifying the matching difficulty of the aerial satellite image pairs based on this; inputting the weakly paired images into multiple models to extract matching points to obtain a second matching point pair, removing low-value point pairs to obtain a high-quality matching point pair set, extracting two sets of second matching point pairs from these sets, and extracting the corresponding aerial images to form an aerial image pair; extracting matching points from the aerial image pairs to obtain a third matching point pair; and filtering satellite image points that simultaneously exist in the two sets of second matching point pairs. If a satellite image point simultaneously matches two aerial image points in the third matching point pair, a strongly paired pseudo-label dataset is obtained. This method reduces reliance on manual intervention and provides high-quality, standardized training samples for incremental continuous learning of cognitive models.

Description

Technical Field

[0001] This application relates to the field of remote sensing image processing technology, and in particular to a method and apparatus for pseudo-labeling weakly paired images. Background Technology

[0002] With the development and popularization of remote sensing technology, the scale of heterogeneous geospatial data, such as satellite images and aerial images, and its application scenarios continue to expand, gradually exhibiting characteristics of "massive volume, fragmentation, and spatiotemporal heterogeneity." According to relevant statistics, a single high-resolution remote sensing satellite can generate hundreds of gigabytes of data daily, and the data modalities are diverse, covering various types such as infrared and visible light, and are widely used in geographic information monitoring, resource exploration, environmental governance, and other fields. In the incremental continuous learning scenario of cognitive models, a serious "semantic gap" exists between the heterogeneity and unstructured state of the original data and the high-quality standardized datasets required for model training. How to efficiently transform these fragmented heterogeneous geospatial data into training samples that can directly drive model iteration has become a core bottleneck restricting the learning efficiency of intelligent interpretation systems.

[0003] To address the aforementioned data transformation needs, existing technologies primarily employ two approaches: the first is the traditional, manual annotation-driven model, where professionals manually perform semantic segmentation, target recognition, or category labeling to transform fragmented, heterogeneous geospatial data into a labeled training sample set; the second is a semi-automated, assisted annotation tool that uses basic image processing algorithms (such as threshold segmentation and edge detection) to perform preliminary screening or coarse annotation of the heterogeneous geospatial data, followed by manual review and correction to improve efficiency. Both approaches require human intervention to fill in missing semantic information in the data, ultimately outputting a labeled dataset suitable for model training.

[0004] However, both of these methods rely on manual processing to varying degrees, resulting in high annotation costs and long annotation cycles, failing to meet the timeliness requirements of incremental data updates. Furthermore, manual annotation is susceptible to subjective differences in experience (e.g., different annotators may have different standards for defining the same target), and the coarse annotation results from semi-automated tools may also contain biases, leading to noisy and inconsistent training datasets. In addition, in incremental learning scenarios, the distribution difference between new and historical data can easily lead to "catastrophic forgetting" of the model, and current technologies lack targeted optimizations for the characteristics of incremental data. In summary, low annotation efficiency limits the timely integration of new data, and low-quality annotation further leads to slow model convergence and decreased generalization ability. Summary of the Invention

[0005] This application provides a pseudo-labeling method and apparatus for weakly paired images, which solves the problem that existing data labeling methods are unable to overcome the efficiency and quality bottlenecks of labeling massive amounts of scattered data, and thus cannot provide reliable training data for incremental and continuous learning of cognitive data.

[0006] In a first aspect, embodiments of this application provide a pseudo-labeling method for weakly paired images, comprising: aligning aerial images extracted from live-flight aerial video with satellite images to construct aerial-satellite image pairs with consistent geographic mapping relationships; extracting matching points from the aerial-satellite image pairs using a dense matching model to obtain a first pair of matching points including confidence levels; determining the spatial divergence of the aerial-satellite image pairs based on the first pair of matching points to quantify the matching difficulty of the aerial-satellite image pairs; determining weakly paired images in the aerial-satellite image pairs according to the matching difficulty, and inputting them into multiple models for matching point extraction. The second matching point pairs, including confidence scores, are obtained from the output of each model. Based on the second matching point pairs and their confidence scores, low-value point pairs are removed to obtain a high-quality matching point pair set. Two sets of second matching point pairs are extracted from the high-quality matching point pair set, and the aerial images corresponding to the two sets of second matching point pairs are extracted to form aerial image pairs. The matching points of the aerial image pairs are extracted to obtain the third matching point pairs. Satellite image points that exist simultaneously in the two sets of second matching point pairs are selected. If the satellite image points match two aerial image points in the third matching point pairs, a strongly paired pseudo-label dataset is obtained.

[0007] In conjunction with the first aspect, in one possible implementation, aligning the aerial images extracted from the actual flight video with satellite images to construct a pair of aerial and satellite images with consistent geographic mapping includes: acquiring a flight dataset and a satellite image tile database; wherein the flight dataset includes actual flight video and flight parameter recording files; uniformly sampling the actual flight video at preset intervals to obtain an aerial image sequence; matching flight parameter records to the flight parameter recording files based on the timestamps of the aerial image sequence; wherein the flight parameter records include the spatial position and attitude angle of the UAV; determining the geographic coordinates of the four corner points of the corresponding aerial image based on the spatial position and attitude angle of the UAV to obtain the geographic coverage area corresponding to the aerial image; and matching the corresponding satellite images in the satellite image tile database based on the geographic coverage area to obtain a pair of aerial and satellite images with consistent geographic mapping.

[0008] In conjunction with the first aspect, in one possible implementation, the step of matching corresponding satellite images in the satellite image tile database according to the geographical coverage area to obtain aerial satellite image pairs with consistent geographical mapping relationships includes: extracting tile images with spatial intersection from the satellite image tile database according to the geographical coverage area, and stitching them together into a satellite image according to geospatial relationships; determining a perspective transformation matrix based on the geographic coordinate projection of the four corner points of the aerial image; and projecting the satellite image onto the aerial viewpoint through the perspective transformation matrix to form the aerial satellite image pair with the corresponding aerial image.

[0009] In conjunction with the first aspect, in one possible implementation, determining the spatial divergence of the aerial satellite image pair based on the first matching point pair includes: dividing the aerial satellite image pair into multiple grids according to a set grid size; counting the number of first matching point pairs in each grid to determine the spatial coverage of the first matching point pairs in the aerial / satellite image; determining the proportion of the first matching point pairs in each grid to the total number of matching points, and determining the grid uniformity of the aerial / satellite image based on this proportion; and performing a weighted average of the spatial coverage and the grid uniformity of the aerial / satellite image to determine the spatial divergence of the corresponding aerial satellite image pair.

[0010] In conjunction with the first aspect, in one possible implementation, the multiple models include LoFTR, ELoFTR, MatchFormer, and D2-Net.

[0011] In conjunction with the first aspect, in one possible implementation, the step of removing low-value point pairs from the second matching point pairs output by each model and their confidence levels to obtain a high-quality matching point pair set includes: identifying mismatched point pairs in the second matching point pairs output by each model using the Ransac algorithm and removing them; and performing a secondary screening based on the confidence levels of the remaining second matching point pairs using a screening threshold to retain the second matching point pairs with high confidence levels; wherein the screening threshold is higher than the confidence level within each model.

[0012] In conjunction with the first aspect, in one possible implementation, the step of removing low-value point pairs from the second matching point pairs output by each model and their confidence scores to obtain a high-quality matching point pair set includes: merging the second matching point pairs output by each model into a matching point coordinate array; converting the coordinates of each second matching point pair in the matching point coordinate array into an integer type and removing duplicates row by row; traversing each deduplicated second matching point pair and removing other second matching point pairs within its first radius to obtain the high-quality matching point pair set.

[0013] Secondly, embodiments of this application provide a pseudo-labeling device for weakly paired images, comprising: a construction module, configured to align aerial images extracted from live aerial video with satellite images to construct aerial-satellite image pairs with consistent geographic mapping relationships; a first matching module, configured to extract matching points from the aerial-satellite image pairs using a dense matching model to obtain a first matching point pair containing confidence; and to determine the spatial divergence of the aerial-satellite image pairs based on the first matching point pairs to quantify the matching difficulty of the aerial-satellite image pairs; and a second matching module, configured to determine weakly paired images in the aerial-satellite image pairs according to the matching difficulty, and input them into multiple models for matching. The algorithm involves several steps: First, a matching point extraction module is used to obtain second matching point pairs output by each model, including their confidence scores. Second, an optimization module is used to remove low-value point pairs from the second matching point pairs output by each model and their confidence scores to obtain a high-quality matching point pair set. Third, a filtering module is used to extract two sets of second matching point pairs from the high-quality matching point pair set, extract the corresponding aerial images to form aerial image pairs, and extract the matching points from the aerial image pairs to obtain third matching point pairs. Finally, satellite image points that simultaneously exist in the two sets of second matching point pairs are filtered. If a satellite image point matches two aerial image points in the third matching point pair, a strongly paired pseudo-label dataset is obtained.

[0014] Thirdly, embodiments of this application provide an apparatus comprising: a processor; a memory for storing processor-executable instructions; wherein, when the processor executes the executable instructions, it implements the method as described in the first aspect or any possible implementation of the first aspect.

[0015] Fourthly, embodiments of this application provide a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium including storage for storing a computer program or instructions that, when executed, cause the method described in the first aspect or any possible implementation of the first aspect to be implemented.

[0016] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: This application's embodiments construct aerial satellite image pairs with consistent geographical mapping relationships, providing a precise spatial benchmark for subsequent matching point extraction and matching difficulty quantification, ensuring geographical alignment between images of different modalities. For identified weakly paired images, a multi-model fusion matching strategy is adopted, effectively compensating for the matching limitations of a single model in complex scenes (such as changes in illumination and differences in viewpoint). The second matching point pair in the high-quality matching point pair set after removing low-value point pairs not only has high confidence but also a more uniform spatial distribution, effectively reducing information redundancy caused by duplicate or adjacent point pairs. It can integrate the advantages of multi-model output, filter out interference from low-value point pairs, and ensure that the final matching point pairs are both accurate and representative, thereby improving the overall quality of pseudo-label annotation and laying a reliable foundation for the construction of a strong-paired pseudo-label dataset. Finally, the strong-paired pseudo-label dataset generated through satellite point matching verification across aerial image pairs not only significantly reduces the dependence on manual annotation but also provides high-quality, standardized training samples for the incremental continuous learning of cognitive models. This effectively solves the problem that existing data annotation methods struggle to overcome the efficiency and quality bottlenecks of massive, scattered data annotation, thus making it difficult to provide reliable training data for the incremental continuous learning of cognitive data. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a pseudo-labeling method for weakly paired images provided in this application embodiment; Figure 2 A schematic diagram of the structure of a pseudo-labeling device for weakly paired images provided in an embodiment of this application; Figure 3 Example diagram of aerial images from a set of aerial satellite image pairs provided in this application embodiment; Figure 4 Example diagram of satellite images in a set of aerial satellite image pairs provided in the embodiments of this application; Figure 5 An example diagram illustrating the extraction of matching points from a set of aerial satellite image pairs provided in this application embodiment; Figure 6 An example diagram of three-party matching provided for embodiments of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] The following description of some technologies involved in the embodiments of this application is provided to aid understanding and should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, some descriptions of well-known functions and structures are omitted in the following description.

[0021] Figure 1 This is a flowchart of a pseudo-labeling method for weakly paired images provided in an embodiment of this application, including steps 101 to 105. Figure 1 This is merely one execution order shown in the embodiments of this application, and does not represent the only execution order for a pseudo-labeling method for weakly paired images. Where the final result can be achieved, Figure 1 The steps shown can be performed in parallel or in reverse order.

[0022] Step 101: Align the aerial images extracted from the live-flight aerial video with satellite images to construct aerial-satellite image pairs with consistent geographic mapping. In this embodiment, a live-flight dataset and a satellite image tile database are obtained; the live-flight dataset includes live-flight aerial video and flight parameter record files; the live-flight aerial video is uniformly sampled at preset intervals to obtain an aerial image sequence; based on the timestamp of the aerial image sequence, flight parameter records are matched to it in the flight parameter record files; the flight parameter records include the spatial position and attitude angle of the UAV; the geographic coordinates of the four corner points of the corresponding aerial image are determined according to the spatial position and attitude angle of the UAV to obtain the geographic coverage area corresponding to the aerial image; the corresponding satellite image is matched in the satellite image tile database according to the geographic coverage area to obtain an aerial-satellite image pair with consistent geographic mapping.

[0023] Based on the geographic coverage, the corresponding satellite images are matched in the satellite image tile database to obtain aerial satellite image pairs with consistent geographic mapping relationships. This includes: extracting tile images with spatial intersection from the satellite image tile database based on the geographic coverage and stitching them together into a satellite image according to geographic spatial relationships; determining the perspective transformation matrix based on the geographic coordinate projection of the four corner points of the aerial image; and projecting the satellite image onto the aerial viewpoint using the perspective transformation matrix to form an aerial satellite image pair with the corresponding aerial image.

[0024] Specifically, the actual flight dataset and corresponding flight parameter record files of the UAV during its actual flight are acquired. The actual flight dataset includes aerial video footage and flight parameter record files. The flight parameter record files contain a sequence of timestamps, spatial location (longitude, latitude, and altitude), and attitude angles (roll, pitch, and yaw) of the UAV during its actual flight. The aerial video footage is uniformly sampled at preset intervals (exemplarily set to 50 frames) to extract keyframes. The aerial images corresponding to each keyframe are stored as an aerial image sequence in the extraction order. The timestamp of each aerial image is read and matched with the closest flight parameter record in the flight parameter record file.

[0025] Furthermore, by comparing the timestamp corresponding to the aerial image with the timestamp sequence in the flight parameter record file, the flight parameter record with the closest timestamp in time is taken as the flight parameter record corresponding to the aerial image.

[0026] Based on this, the mapping relationship between camera pixel coordinates and geographic coordinates is calculated according to the UAV's spatial position and attitude angles. Specifically, a rotation matrix representing the camera's attitude is constructed using the attitude angles, and the camera's extrinsic parameter matrix is ​​calculated in conjunction with the UAV's spatial position. Then, the camera's extrinsic parameter matrix is ​​combined with the camera's intrinsic parameter matrix, and under the assumption that the ground surface is planar, a homography transformation matrix is ​​constructed from the camera pixel coordinate system to the geographic coordinate system. Using this homography transformation matrix, pixels in the aerial image can be projected onto geographic space, thereby estimating the geographic coverage area corresponding to the aerial image.

[0027] For example, a rotation matrix representing the camera attitude is constructed based on the attitude angles (roll angle, pitch angle, and yaw angle) corresponding to the aerial image.

[0028] First, calculate the basic rotation matrices about the X, Y, and Z axes respectively: , , , Then, the three basic rotation matrices of the X, Y, and Z axes are combined to obtain the rotation matrix describing the camera's pose: .

[0029] In the formula, , , These represent the basic rotation matrices about the X, Y, and Z axes, respectively. Indicates the attitude angle.

[0030] Then, the spatial position vector of the UAV in the geographic coordinate system was combined. (The vectors representing spatial positions) further determine the camera's extrinsic parameter matrix. Among them, the translation vector... The spatial position vector of the camera center in the world coordinate system Rotation matrix with camera pose The relationship between them is calculated and determined: .

[0031] camera extrinsic matrix With camera intrinsic parameter matrix By combining these elements, a projection matrix from the world coordinate system to the camera coordinate system can be constructed: Assuming the ground is a local plane, the homogeneous coordinates of a point on the ground plane in the world coordinate system are: Its pixel coordinates in the image plane are The two satisfy a projection relationship: ,Right now Extract the rotation matrix of the camera pose. The first two sub-matrices Furthermore, we obtain: , ( This represents the horizontal coordinate of a pixel on the image plane. This represents the vertical coordinate of a pixel on the image plane. This represents the homogeneous coordinate vector of a point on the ground plane in the world coordinate system. Represents the spatial coordinate components of a point on a ground plane in the world coordinate system. (T represents the pixel coordinates of a pixel on the image plane, and T represents the transpose matrix.) From this, a homography matrix between the ground plane and the image plane can be constructed. And perform the inversion operation on the homography matrix. This allows us to obtain the homography transformation matrix from image pixel coordinates to geographic coordinates. This establishes a mapping relationship between aerial images and geospatial data.

[0032] After obtaining the homography transformation matrix, the pixels in the aerial image can be mapped to the geographic coordinate system, thereby estimating the ground coverage area corresponding to the aerial image. Specifically, by projecting the four corner points of the aerial image, the actual coverage area of ​​the image in geographic space can be estimated, and its corresponding geographic boundary can be determined. Considering that UAV attitude measurement errors and positioning errors may affect the projection results, the area is further expanded proportionally based on the calculated geographic boundary to construct a geographic coverage area including redundant boundaries.

[0033] Based on the calculated geographic coverage, tile images that spatially intersect with the region are selected from the satellite image tile database and stitched together according to their geospatial relationships to construct a satellite image. Simultaneously, a mapping relationship is established between the pixel coordinates and geographic coordinates of the stitched satellite image.

[0034] Finally, based on the projected coordinates of the four corner points of the aerial image in geographic space (i.e., the geographic coordinates of the corner points), these coordinates are mapped to the pixel coordinate system of the satellite image. A perspective transformation matrix is ​​then calculated, and the satellite image undergoes a corresponding perspective transformation to generate a satellite image with the same perspective as the aerial image. Through this process, an aerial image and a satellite image with consistent geographic mapping are ultimately obtained, forming an aerial-satellite image pair, such as... Figure 3 Aerial images and Figure 4 Satellite images can form a pair of aerial satellite images.

[0035] In addition, the geographic coverage area can be expanded by a fixed ratio (or distance) to construct a geographic coverage area with redundant boundaries, avoiding omissions in subsequent satellite image matching.

[0036] Step 102: Extract matching points from the aerial satellite image pairs using a dense matching model to obtain first matching point pairs containing confidence scores; determine the spatial divergence of the aerial satellite image pairs based on the first matching point pairs to quantify the matching difficulty of the aerial satellite image pairs. In this embodiment, the aerial satellite image pairs are divided into multiple grids according to a set grid size; the number of first matching point pairs in each grid is counted to determine the spatial coverage of the first matching point pairs in the aerial / satellite images; the proportion of the first matching point pairs in each grid to the total number of matching points is determined, and the grid uniformity of the aerial / satellite images is determined based on this proportion; a weighted average is performed on the spatial coverage and grid uniformity of the aerial / satellite images to determine the spatial divergence of the corresponding aerial satellite image pairs.

[0037] Specifically, for the paired aerial satellite image pairs obtained in step 101, a dense matching model (such as RoMA, a dense feature matching model) is uniformly used to extract the matching point pairs (e.g., ...). Figure 5As shown in the image, RoMA is a cutting-edge model in the field of dense feature matching, with its core advantage being its strong adaptability to complex changes in the real world. RoMA innovatively uses a frozen DINOv2 (second-generation self-supervised visual model) base model to extract coarse-grained features, combined with a dedicated ConvNet (convolutional neural network) to generate fine-grained features, constructing a feature pyramid that combines robustness and precise localization capabilities. This effectively solves the problem of insufficient generalization ability caused by traditional models relying on 3D supervised data. RoMA performs matching calculations on the feature pyramids of aerial and satellite image pairs, outputting pixel-level precision coordinates of matching points (including aerial image points and corresponding satellite image points), and simultaneously outputting the confidence / similarity index for each matching point pair.

[0038] In addition, before inputting aerial satellite images into RoMA, RoMA needs to be preprocessed, including mean normalization, unifying the image input format, and eliminating pixel value scale differences, to ensure the consistency of the input image format.

[0039] Furthermore, due to the characteristics of RoMA, for images that are difficult to match, such as those with large scale and weak texture, the extracted feature points are relatively clustered and not uniformly distributed in space. Therefore, in order to filter out these difficult-to-match images, this application uses spatial divergence to describe the spatial distribution of the first matching point pair.

[0040] First, the aerial and satellite images are divided into multiple grids of equal size according to a set grid size. (For example, the grid size is set to 10×10; the grid size can be flexibly adjusted according to image resolution and actual needs). The number of first matching point pairs within each grid is counted. If the number of first matching point pairs in a grid is greater than or equal to 1, the grid is determined to be a valid grid, and its spatial coverage is calculated. The spatial coverage is the ratio of the number of valid grids to the total number of grids.

[0041] Subsequently, the grid uniformity is calculated. Specifically, the proportion of the number of first matching point pairs in each effective grid to the total number of first matching point pairs is calculated, and then the information entropy of the proportion values ​​of all effective grids is calculated. The information entropy of each effective grid is divided by the maximum entropy value under ideal uniform distribution to complete the normalization and obtain the grid uniformity. The grid uniformity is in the interval [0,1]. The higher the grid uniformity, the more uniform the distribution of the first matching point pairs in the effective grid.

[0042] Finally, a weighted average of spatial coverage and grid uniformity is calculated to obtain spatial divergence, where the weight of spatial coverage is set to 0.4 and the weight of grid uniformity is set to 0.6 (the weights can be adjusted according to the actual scenario). The higher the value of spatial divergence, the more widely and evenly the first matching point pairs of the aerial satellite image pair are distributed, and the lower the matching difficulty. Conversely, the lower the value of spatial divergence, the higher the matching difficulty.

[0043] Aerial and satellite images are differentiated based on the calculated spatial divergence. Images with larger spatial divergence are easier to match, and the first matching point pairs extracted by RoMA are accurate. These are directly saved as txt files as pseudo-labels for weakly matched data. Images with smaller spatial divergence are more difficult to match, and the first matching point pairs extracted by RoMA may have larger errors. In such cases, steps 103 to 105 are required for further optimization.

[0044] For example, the formulas for calculating spatial coverage, grid uniformity, and spatial divergence are as follows: , , , .

[0045] In the formula, C represents spatial coverage, reflecting the spatial coverage range of the first matching point pair. This indicates the total number of grids in the aerial / satellite image. This indicates the number of valid grid cells with the first matching point pair. This indicates grid uniformity, reflecting the evenness of the distribution of the first matching point pair within the effective grid. It is the number of first matching point pairs within the i-th valid grid. The first matching point pair in the i-th valid grid represents the proportion of the total number of first matching point pairs, and S represents the spatial divergence. , These represent the weights of spatial coverage and grid uniformity, respectively. For example, , .

[0046] Step 103: Based on the matching difficulty, weakly paired images in the aerial satellite image pairs are identified and input into multiple models for matching point extraction, resulting in second matching point pairs output by each model, including confidence scores. In this embodiment, the multiple models include LoFTR, ELoFTR, MatchFormer, and D2-Net.

[0047] Specifically, a preset threshold for spatial divergence is set (for example, the preset threshold can be set to 0.8, and the specific value can be adjusted according to the matching requirements of the actual scenario and the experimental verification results), and aerial satellite image pairs with spatial divergence lower than the preset threshold are judged as weakly paired images.

[0048] Traditional single-model approaches often suffer from performance bottlenecks when dealing with complex scenes involving scale variations, lighting perturbations, texture loss, and large viewpoint differences. To address this, this application employs multiple models, including LoFTR (a Transformer-based local feature matching model), ELoFTR (an enhanced LoFTR edge-aware feature matching model), MatchFormer (a hierarchical Transformer feature matching model), and D2-Net (a convolutional network for joint feature detection and description). Through the complementary use of multiple models, the fusion of their strengths, and an intelligent decision-making mechanism, the performance limitations of single models are overcome, achieving high-precision and robust extraction of matching point pairs in complex scenes.

[0049] Furthermore, the weakly matched images selected in step 102 that are difficult to match are preprocessed, including mean normalization, to ensure that the input format of each model is consistent. Then, a multi-model parallel inference engine is launched, and the weakly matched images are input into four models—LoFTR, ELoFTR, MatchFormer, and D2-Net—for independent matching point extraction. LoFTR uses a self-attention mechanism to mine the global correlation between weakly matched images, outputting sparse second matching point pairs with high confidence. ELoFTR adds an edge-aware module to LoFTR, enhancing its ability to match features in image edge regions. MatchFormer utilizes the Transformer architecture combined with a multi-scale feature pyramid to achieve accurate cross-scale feature alignment. D2-Net detects dense keypoints in the image and generates descriptors, outputting dense second matching point pairs. Each model's output second matching point pair contains the corresponding pixel coordinates of the aerial and satellite images, as well as the confidence level of the point pair, providing multi-source matching data support for subsequent pseudo-label generation.

[0050] LoFTR is a detectorless local image feature matching model based on Transformer, employing a coarse-to-fine matching process. Its overall architecture includes a local feature extraction module, a feature transformation module, and a coarse-to-fine matching module. The local feature extraction module utilizes a ResNet-18 convolutional neural network backbone, adding pixel position encoding to the first convolutional layer to extract more precise multi-scale feature maps from the input weakly paired image, generating coarse feature maps at 1 / 8 of the original resolution and fine feature maps at 1 / 2 and 1 / 4 of the original resolution. The feature transformation module applies a Transformer encoder to the coarse feature map, modeling the global context through an alternating mechanism of self-attention and cross-attention to enhance feature discrimination capabilities. The coarse-to-fine matching module first establishes initial correspondences on the coarse feature map, then uses the fine feature map to refine the matching to sub-pixel precision through expectation calculation based on relevant blocks. The model takes a grayscale weakly paired image of arbitrary size as input and outputs a semi-dense set of matching points consisting of second matching point pairs. The learning rate of the LoFTR model is 1×10⁻⁶. -3 .

[0051] ELoFTR builds upon LoFTR by introducing a convergent attention mechanism and a two-stage correlation refinement layer. It is based on a convolutional neural network feature extraction module and a Transformer encoder, aiming to balance matching efficiency and accuracy, and overcoming the limitations of previous methods through a series of key innovations. The two-stage correlation refinement layer first locates pixel-level matches in fine feature blocks using nearest neighbors, and then refines the matches to sub-pixel level within small local blocks through correlation and expectation calculations. Furthermore, the feature extraction backbone network employs a multi-branch residual network during training, and then fuses it into a single-branch network during inference through reparameterization to improve efficiency. The model takes a weakly paired grayscale image of arbitrary size as input and outputs a semi-dense set of matching points consisting of second-pair matching points. The learning rate of LoFTR is 4×10⁻⁶. 3 .

[0052] MatchFormer is a hierarchical extraction-matching Transformer model for local feature matching, designed to address the shortcomings of traditional detector-based models in low-texture scenes, the underutilization of encoder matching capabilities in the extraction-matching pipeline, and the excessive burden on the decoder. Its core innovation lies in interweaving self-attention and cross-attention at each stage of the hierarchical encoder to construct a match-aware encoder, simultaneously learning the image's own features and the similarity between image pairs. Specifically, shallow layers primarily extract texture information using self-attention, while deeper layers primarily explore feature similarity using cross-attention. Simultaneously, it enhances low-level feature detection and location information embedding through position embedding, and employs spatially efficient attention to reduce computational complexity. A lightweight decoder fuses multi-scale features to support subsequent coarse-to-fine matching. The model has a feature dimension of 256, takes a weakly paired grayscale image as input, and outputs a semi-dense set of matching points consisting of second-pair matching points. MatchFormer's learning rate is 3×10⁻⁶. -4 .

[0053] D2-Net is a convolutional neural network that jointly learns feature detection and description, employing a "description-then-detection" paradigm. Its architecture is based on a 22-layer VGG-like convolutional neural network. The input image is processed by the network to output a 3D feature tensor. The n-dimensional vector at each spatial location serves as a dense descriptor, which is normalized by the L2 function and used for comparison. Detection is achieved through soft local maxima and channel selection; that is, the most significant detector response is selected at each location, and its local peak value is verified. During the testing phase, an image pyramid (resolution scales of 0.5, 1, and 2) is constructed for multi-scale feature extraction and fusion. The detection results are then propagated back to the original resolution to enhance scale invariance. The model takes a grayscale weakly paired image as input and outputs a semi-dense set of matching points consisting of second-pair matching points. The learning rate of the D2-Net model is 1×10⁻⁶. 3 .

[0054] Step 104: Based on the second matching point pairs output by each model and their confidence levels, remove low-value point pairs to obtain a high-quality set of matching point pairs. In this embodiment, the Ransac algorithm is used to identify mismatched point pairs in the second matching point pairs output by each model and remove them; based on the confidence levels of the remaining second matching point pairs, a secondary screening is performed using a screening threshold to retain high-confidence second matching point pairs; wherein, the screening threshold is higher than the confidence level within each model.

[0055] Specifically, in the second matching point pairs output by each model, outliers inevitably exist, i.e., incorrect matches caused by occlusion, duplicate textures, lighting variations, or model prediction errors. The core objective of the RANSAC (Random Sample Consensus) algorithm is to robustly estimate the optimal geometric transformation model from a matching point set containing a large number of outliers (i.e., low-quality points), while simultaneously eliminating outliers, ultimately obtaining a high-precision set of interior point matches. Furthermore, although each model internally performs confidence screening when obtaining the second matching point pairs, the confidence screening threshold within the model is relatively low, and there are still many mismatched point pairs in the second matching point pairs after model screening. Therefore, this application first uses the RANSAC algorithm to screen and remove mismatched point pairs, and then uses a higher screening threshold for more rigorous screening in the remaining second matching point pairs, aiming to select more accurate matching point pairs and obtain a high-quality set of matching point pairs.

[0056] Furthermore, for each model's output second matching point pair, the RANSAC algorithm is executed independently to remove mismatched point pairs. Specifically, the inlier threshold for pixel error is set to 10 pixels (this threshold can be flexibly adjusted according to image resolution and matching accuracy requirements). The homography transformation matrix is ​​estimated by random sampling, and second matching point pairs whose deviation from the transformation matrix prediction result exceeds the inlier threshold are identified as mismatched point pairs and removed. Subsequently, for the remaining second matching point pairs, a secondary screening is performed based on their confidence level. That is, a differentiated high confidence threshold (i.e., screening threshold) is set according to the confidence output range of different models. For example, the screening threshold for LoFTR (confidence range 0-1) is set to 0.2, the screening threshold for ELoFTR is set to 0.2, the screening threshold for MatchFormer is set to 0.15, and D2-Net selects the top 2000 second matching point pairs based on their confidence level.

[0057] In this embodiment, the second matching point pairs output by each model are merged into a matching point coordinate array; the coordinates of each second matching point pair in the matching point coordinate array are converted into integer type and deduplicated row by row; each deduplicated second matching point pair is traversed and other second matching point pairs within its first radius are removed to obtain a high-quality matching point pair set.

[0058] Specifically, this step can be performed based on the second matching point pairs output by each model, maximizing the retention of effective feature points and learning the characteristics of each model. To select higher-quality second matching point pairs, this step can be performed based on the second matching point pairs filtered by the RANSAC algorithm and confidence level.

[0059] Furthermore, the second pair of matching points is first converted into a two-dimensional array of matching point coordinates, for example: [(aerial image point x1, aerial image point y1, satellite image point x1, satellite image point y1), (Aerial image point x2, aerial image point y2, satellite image point x2, satellite image point y2) (Aerial image point x3, aerial image point y3, satellite image point x3, satellite image point y3) ..., (Aerial image point xn, aerial image point yn, satellite image point xn, satellite image point yn)).

[0060] Next, the coordinates of all second matching point pairs in the matching point coordinate array are rounded to integers to ensure consistency in coordinate representation. Then, deduplication is performed row by row (i.e., along the 0-axis): if two second matching point pairs have coordinates identical to those of points in the aerial image and satellite image, they are considered duplicate pairs, and only the pair with the higher confidence is retained. After deduplication, a first radius of 3 pixels is set (this radius can be flexibly adjusted according to the resolution of the aerial / satellite image and the density of the second matching point pairs). Each deduplicated second matching point pair is iterated through, and its pixel Euclidean distance to other second matching point pairs in the aerial image is calculated. If the calculated Euclidean distance is less than the first radius, it is removed (or, alternatively, only the pair with the highest built-in confidence within each first radius can be retained). Through these steps, the final high-quality set of matching point pairs eliminates redundant and overlapping pairs while ensuring the representativeness and high confidence of each retained pair, laying a reliable foundation for subsequent pseudo-label generation.

[0061] Using the first radius to filter can reduce the number of second matching point pairs, remove duplicate second matching point pairs, and at the same time ensure that the remaining second matching point pairs are representative and the overall representativeness remains unchanged.

[0062] Step 105: Extract two sets of second matching point pairs from the high-quality matching point pair set, extract the aerial images corresponding to the two sets of second matching point pairs to form aerial image pairs, and extract the matching points of the aerial image pairs to obtain the third matching point pairs; filter the satellite image points that exist simultaneously in the two sets of second matching point pairs. If the satellite image points match two aerial image points in the third matching point pairs, then a strongly paired pseudo-label dataset is obtained.

[0063] Specifically, this application utilizes the geometric relationships and matching information between three independent observation perspectives to cross-validate and filter out high-quality, intrinsically consistent matching point pairs, thereby effectively eliminating erroneous matches and obtaining highly accurate matching point pairs. For example... Figure 6As shown, the core idea of ​​this method is that for a real-world 3D point, its projections (i.e., feature points on aerial images) from different viewpoints must satisfy a specific consistency relationship. Specifically, if point A matches point B in viewpoint 1 and point B matches point C in viewpoint 2, and point B matches point C in viewpoint 2 and point C in viewpoint 3, then point A in viewpoint 1 must match point C in viewpoint 3. This method utilizes this transitivity principle for consistency verification.

[0064] For the same spatial point in a real 3D scene, its projection in images from different viewpoints should satisfy the multi-view geometric consistency constraint. This constraint utilizes the temporal continuity between sequential aerial images to introduce multi-view matching between cross-viewpoint matching and temporal matching. Figure 1 This ensures consistent relationships, effectively eliminating false correspondences caused by mismatches or noise, and improving the reliability and geometric consistency of the final matching point set.

[0065] Specifically, let the current aerial image With satellite imagery The set of matching points between them is: ,in, Indicates the current aerial image Aerial image points (i.e., matching points) in the image. This represents a satellite image point (i.e., a matching point) in satellite image S.

[0066] Current aerial images The next frame of the aerial image With satellite imagery The set of matching points between them is: ,in, Indicates the current aerial image The next frame of the aerial image Aerial image points (i.e., matching points).

[0067] Current aerial images With the next aerial image The set of matching points between them is: Among them, the current aerial images With the next frame of aerial image All images are aerial images corresponding to the second matching point pair in a high-quality set of matching point pairs.

[0068] First, filter out those points that appear simultaneously in the matching point set. and matching point set The set of satellite image points: , Then, the intersection of the two is calculated to obtain the set of matching points that appear simultaneously. and matching point set The set of satellite image points: For any If both exist as well as Then according to multiple views Figure 1 Consistency constraints, points in these two aerial images The matching set should appear simultaneously in the aerial images. In, that is, satisfying Based on the above constraints, the matching set can be... After screening, those that meet the three requirements are obtained. Figure 1 Consistent high-confidence matching point set: .

[0069] This method requires three images as input each time: the current aerial image, a satellite image, and the next frame of the current aerial image. The time-frequency consistency filtering starts with the satellite image, searching for matching points in the satellite image and comparing them with matching points in the current and next frames of the aerial image. It then checks if the matching points found in the current and next frames also match. If they do, a three-way match is established, proving that the matching point pair between the current and satellite images is a strong match with very high accuracy. Retaining all such three-way consistent matching point pairs yields a highly accurate pseudo-label dataset of matching point pairs.

[0070] It should be noted that the two aerial images input here are the current frame and the next frame that are consecutive in time during sampling. The time interval between the two aerial images can be determined by the "preset interval" when uniformly sampling the actual flight aerial video according to the preset interval in step 101.

[0071] Finally, all strongly paired pseudo-labeled samples that meet the criteria are aggregated to form a strongly paired pseudo-labeled dataset. This dataset can be used to train or optimize matching models for weakly paired images, improving the model's matching performance for weakly paired images in complex scenes.

[0072] While this application provides the method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-inventive labor. The order of steps listed in this embodiment is merely one possible execution order among many and does not represent the only execution order. In actual device or client product execution, the methods shown in this embodiment or the accompanying drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment).

[0073] like Figure 2As shown in the figure, this application embodiment also provides a pseudo-labeling device 200 for weakly matched images. The device includes: a construction module 201, a first matching module 202, a second matching module 203, an optimization module 204, and a filtering module 205, as detailed below.

[0074] The construction module 201 is used to align aerial images extracted from live aerial video with satellite images to construct aerial-satellite image pairs with consistent geographic mapping.

[0075] The first matching module 202 is used to extract matching points from aerial satellite image pairs using a dense matching model to obtain a first matching point pair containing confidence; and to determine the spatial divergence of the aerial satellite image pair based on the first matching point pair in order to quantify the matching difficulty of the aerial satellite image pair.

[0076] The second matching module 203 is used to determine the weakly matched images in the aerial satellite image pairs according to the matching difficulty, and input them into multiple models to extract matching points, so as to obtain the second matching point pairs with confidence levels output by each model.

[0077] The optimization module 204 is used to remove low-value point pairs based on the second matching point pairs output by each model and their confidence level to obtain a high-quality matching point pair set.

[0078] The filtering module 205 is used to extract two sets of second matching point pairs from the high-quality matching point pair set, extract the aerial images corresponding to the two sets of second matching point pairs to form aerial image pairs, and extract the matching points of the aerial image pairs to obtain third matching point pairs. Satellite image points that exist simultaneously in the two sets of second matching point pairs are filtered. If a satellite image point matches two aerial image points in the third matching point pair, a strongly paired pseudo-label dataset is obtained.

[0079] Some modules in the apparatus described in this application can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0080] The apparatus or module described in the above embodiments can be implemented by a computer chip or physical entity, or by a product with a certain function. For ease of description, the above apparatus is described by dividing it into various modules according to their functions. When implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware. Of course, a module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0081] The methods, apparatus, or modules described in this application can be implemented in a computer-readable program code manner. The controller can be implemented in any suitable manner, such as a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of a memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code manner, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included within it for implementing various functions can also be considered as structures within the hardware component. Alternatively, the device used to implement various functions can be viewed as either a software module that implements the method or a structure within a hardware component.

[0082] This application also provides an apparatus, the apparatus comprising: a processor; a memory for storing processor-executable instructions; wherein, when the processor executes the executable instructions, it implements the method described in this application.

[0083] This application also provides a non-volatile computer-readable storage medium storing a computer program or instructions thereon, which, when executed, enables the method described in this application embodiment to be implemented.

[0084] Furthermore, in the various embodiments of the present invention, each functional module can be integrated into a processing module, or each module can exist independently, or two or more modules can be integrated into a single module.

[0085] The aforementioned storage media include, but are not limited to, Random Access Memory (RAM), Read-Only Memory (ROM), Cache, Hard Disk Drive (HDD), or Memory Card. The memory can be used to store computer program instructions.

[0086] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, or it can be embodied in the process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, mobile terminal, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0087] The various embodiments described in this specification are presented in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. All or part of this application can be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, mobile communication terminals, multiprocessor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices, etc.

[0088] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of this application.

Claims

1. A pseudo-labeling method for weakly paired images, characterized in that, include: Aerial images extracted from actual flight aerial footage will be aligned with satellite images to construct aerial-satellite image pairs with consistent geographic mapping. The matching points in the aerial satellite image pairs are extracted using a dense matching model to obtain the first matching point pair containing confidence. The spatial divergence of the aerial satellite image pair is determined based on the first matching point pair in order to quantify the matching difficulty of the aerial satellite image pair; Weakly matched images in the aerial satellite image pairs are determined based on the matching difficulty, and then input into multiple models to extract matching points, resulting in second matching point pairs with confidence levels output by each model. Based on the second matching point pair output by each model and its confidence level, low-value point pairs are removed to obtain a set of high-quality matching point pairs. Two sets of second matching point pairs are extracted from the high-quality matching point pair set. The aerial images corresponding to the two sets of second matching point pairs are extracted to form aerial image pairs. The matching points of the aerial image pairs are extracted to obtain third matching point pairs. Satellite image points that exist simultaneously in the two sets of second matching point pairs are selected. If the satellite image points match two aerial image points in the third matching point pairs, a strongly paired pseudo-label dataset is obtained.

2. The method according to claim 1, characterized in that, The process of aligning aerial images extracted from live-flight aerial video with satellite images to construct aerial-satellite image pairs with consistent geographic mapping includes: Acquire a real-flight dataset and a satellite image tile database; wherein, the real-flight dataset includes real-flight aerial video and flight parameter recording files; The aerial video footage was sampled uniformly at preset intervals to obtain an aerial image sequence. Based on the timestamps of the aerial image sequence, flight parameter records are matched to them in the flight parameter record file; wherein, the flight parameter records include the spatial position and attitude angle of the UAV; The geographic coordinates of the four corner points of the corresponding aerial image are determined based on the spatial position and attitude angle of the drone, so as to obtain the geographic coverage area corresponding to the aerial image. Based on the geographical coverage, the corresponding satellite images are matched in the satellite image tile database to obtain aerial satellite image pairs with consistent geographical mapping relationships.

3. The method according to claim 2, characterized in that, The step of matching corresponding satellite images in the satellite image tile database according to the geographical coverage area to obtain aerial satellite image pairs with consistent geographical mapping relationships includes: Based on the geographical coverage, extract tile images with spatial overlap from the satellite image tile database, and stitch them together into a satellite image according to geospatial relationships; The perspective transformation matrix is ​​determined based on the geographic coordinate projection of the four corner points of the aerial image; The satellite image is projected onto the aerial viewpoint using the perspective transformation matrix, forming the aerial satellite image pair with the corresponding aerial image.

4. The method according to claim 1, characterized in that, The determination of the spatial divergence of the aerial satellite image pair based on the first matching point pair includes: The aerial satellite image pair is divided into multiple grids according to a set grid size; The number of first matching point pairs in each grid is counted to determine the spatial coverage of the first matching point pairs in aerial / satellite images; Determine the proportion of the first matching point pair in each grid to the total number of matching points, and determine the grid uniformity of the aerial / satellite image based on this proportion; The spatial coverage and grid uniformity of the aerial / satellite images are weighted and averaged respectively to determine the spatial divergence of the corresponding aerial / satellite image pair.

5. The method according to claim 1, characterized in that, The multiple models include LoFTR, ELoFTR, MatchFormer, and D2-Net.

6. The method according to claim 1, characterized in that, The high-quality set of matching point pairs is obtained by removing low-value point pairs from the second matching point pairs output by each model and their confidence scores, including: The Ransac algorithm is used to identify and remove mismatched point pairs in the second matching point pairs output by each model. Based on the confidence level of the remaining second matching point pairs, a second screening is performed using a screening threshold to retain the second matching point pairs with high confidence; wherein, the screening threshold is higher than the confidence level within each model.

7. The method according to claim 1, characterized in that, The high-quality set of matching point pairs is obtained by removing low-value point pairs from the second matching point pairs output by each model and their confidence scores, including: The second matching point pairs output by each model are merged into a matching point coordinate array; Convert the coordinates of each second matching point pair in the matching point coordinate array to an integer type, and remove duplicates row by row; Traverse each deduplicated second matching point pair, remove other second matching point pairs within its first radius, and obtain the set of high-quality matching point pairs.

8. A pseudo-labeling apparatus for weakly paired images to implement the method described in any one of claims 1-7, characterized in that, include: A module is built to align aerial images extracted from live aerial video with satellite images to construct aerial-satellite image pairs with consistent geographic mapping. The first matching module is used to extract matching points from the aerial satellite image pairs using a dense matching model to obtain a first matching point pair containing confidence. The spatial divergence of the aerial satellite image pair is determined based on the first matching point pair in order to quantify the matching difficulty of the aerial satellite image pair; The second matching module is used to determine the weakly paired images in the aerial satellite image pairs according to the matching difficulty, and input them into multiple models to extract matching points, so as to obtain the second matching point pairs with confidence levels output by each model. The optimization module is used to remove low-value point pairs from the second matching point pairs output by each model and their confidence scores to obtain a set of high-quality matching point pairs. The filtering module is used to extract two sets of second matching point pairs from the high-quality matching point pair set, extract the aerial images corresponding to the two sets of second matching point pairs to form aerial image pairs, and extract the matching points of the aerial image pairs to obtain third matching point pairs; filter the satellite image points that exist simultaneously in the two sets of second matching point pairs, and if the satellite image points match two aerial image points in the third matching point pairs, then a strongly paired pseudo-label dataset is obtained.

9. An apparatus for performing a pseudo-labeling method for weakly paired images, characterized in that, include: processor; Memory used to store processor-executable instructions; When the processor executes the executable instructions, it implements the method as described in any one of claims 1 to 7.

10. A non-volatile computer-readable storage medium, characterized in that, Includes storage of computer programs or instructions that, when executed, cause the method as described in any one of claims 1 to 7 to be implemented.

Images