A semantic-driven cross-view matching method for air-to-ground unmanned aerial vehicle video

Abstract

The application relates to the field of image matching, in particular to a semantic-driven cross-view matching method for air-to-ground unmanned aerial vehicle video, which comprises the following steps: collecting image frames and reference images and performing pretreatment; generating image features and preset category text features by using a CLIP pre-training model and calculating the possibility of target appearance; generating a semantic mask according to the CLIP indefinite feature and the trigger category text feature after triggering; performing weighted enhancement on coarse-level feature maps according to weights to obtain coarse-scale features; and obtaining accurate matching point coordinates and confidence by using a coarse-to-fine matching strategy. The application has the beneficial effects that: the process of "semantic mask + local feature matching" is adopted, coarse-to-fine registration is carried out only in the semantic relevant area, and the misleading caused by the non-intersecting view angle and background clutter is reduced; the same target can be stably positioned in a traffic scene, the false matching rate is reduced, the accuracy and stability of cross-view alignment are improved; and redundant calculation is reduced, and the robustness of the method is improved.

Description

Technical Field

[0001] This invention relates to the field of image matching, and more particularly to a semantic-driven cross-view matching method for air-to-ground UAV videos. Background Technology

[0002] Air-to-ground cross-view image matching is a challenging problem in computer vision. Due to the significant differences in shooting perspectives, the same object appears drastically different in aerial drone videos and ground images viewed from the ground. For example, in traffic scenes, a vehicle appears as its roof from an aerial viewpoint, but as its side or front view from a ground-level perspective. The significant differences in shape, scale, lighting conditions, and background environment brought about by the change in perspective make direct matching extremely complex.

[0003] Currently used local feature matching methods such as SIFT and ORB, as well as some deep learning feature matching algorithms, can achieve good results under conditions where the viewpoint changes little. However, they often fail when faced with extreme viewpoint differences such as air-to-ground. Furthermore, these feature matching methods are generally time-consuming, making them difficult to use for real-time video matching. On the one hand, ground-view images are often affected by dynamic target occlusion, lighting changes, and similar backgrounds at different locations, while drone aerial images cover a wide area with complex and cluttered backgrounds. Directly performing global feature matching on both can easily lead to incorrect correspondences. On the other hand, existing technologies typically perform full-frame feature extraction and matching processing on every frame of the video, without distinguishing whether the frame contains the target object to be matched. In other words, the algorithm performs matching calculations on the entire frame regardless of whether vehicles or other targets are present. This untargeted "target-independent" matching strategy leads to a significant increase in computational redundancy: many frames without targets or with small target regions are also processed completely, greatly wasting computational resources. At the same time, including irrelevant features such as backgrounds in the matching may introduce noise, further reducing matching accuracy and efficiency. For large-scale video processing tasks that require high real-time performance, the aforementioned defects severely restrict system performance and make it difficult to meet the efficiency requirements of practical applications.

[0004] Therefore, a new air-to-ground cross-view matching method is urgently needed to solve the above problems. Summary of the Invention

[0005] To overcome the above-mentioned shortcomings, the present invention aims to provide a semantic-driven cross-view matching method for air-to-ground UAV video, so as to overcome the problems of low accuracy, poor efficiency and computational redundancy in the existing cross-view matching technology.

[0006] This invention achieves the above objective through the following approach: a semantic-driven cross-view matching method for air-to-ground UAV video, comprising the following steps:

[0007] (1) Acquire image frames and reference images, and perform preprocessing such as scale alignment and color format conversion, and output the preprocessed image;

[0008] (2) Input the pre-processed image and the text of the preset category into the pre-trained CLIP model to generate image features and text features of the preset category. Calculate the cosine similarity between the two features as the confidence score. When the highest confidence score is greater than or equal to the threshold... At that time, output the semantic trigger signal and the position of the trigger frame image;

[0009] (3) Patch features are extracted from the trigger frame image and reference image based on CLIP. The similarity between the patch features and the trigger category text features is calculated to generate a patch heatmap. The patch heatmap is upsampled to the LoFTR coarse resolution to obtain the semantic mask.

[0010] (4) Extract coarse and fine feature maps for both the trigger frame image and the reference image. In the coarse matching stage: use a weighting factor to apply the semantic mask to the coarse feature map to obtain the enhanced feature map, and then filter the coarse matching points based on double normalization and mutual nearest neighbors. In the fine matching stage: map the coarse matching points to the fine feature map, perform correlation estimation and expectation solution within the local window to obtain the sub-pixel level position, and then perform visualization processing.

[0011] Preferably, step (1) specifically includes: acquiring each frame of the UAV video stream and a reference image as input; scaling by the short side and filling with equal sides if necessary to achieve scale alignment; unifying the input from BGR or grayscale to RGB; and outputting the preprocessed image.

[0012] Preferably, the reference image is a frame of image containing the target that is manually selected.

[0013] Preferably, the preset category text in step (2) includes one or more predefined target categories.

[0014] Preferably, the threshold Customizable.

[0015] Preferably, step (3) specifically includes the following steps:

[0016] (3.1) CLIP divides the trigger frame image and the reference image into several patches respectively, and outputs patch features for each patch;

[0017] (3.2) Obtain the text features of the trigger category;

[0018] (3.3) Calculate the similarity between the trigger category text features and each patch feature to form a patch heatmap. Upsample the patch heatmap to the LoFTR coarse resolution to obtain a semantic mask. The LoFTR coarse resolution is 1 / 8 of the downsampled trigger frame image.

[0019] Preferably, in step (4), the step of downsampling the trigger frame image and the reference image by 1 / 8 to extract a coarse feature map and downsampling by 1 / 2 to extract a fine feature map, the corresponding two sets of coarse feature maps are represented as follows: , The fine-level feature map is represented as , .

[0020] Preferably, the specific steps of the coarse-level matching stage in step (4) include:

[0021] (4.1a) Applying a weighting factor to the semantic mask on the coarse feature map , To obtain attention-enhanced feature maps , ;

[0022] (4.1b) Calculation , The similarity matrix S is then normalized to obtain the matching confidence matrix;

[0023] (4.1c) After filtering out coarse-level matching points with high confidence based on the matching confidence matrix, apply the mutual nearest neighbor constraint to remove inconsistent pairs, and record the center of the coarse-level matching point and its confidence.

[0024] Preferably, step (4.1a) specifically includes: placing , After flattening the features and adding sinusoidal positional encoding, the input is fed into a coarse-level Transformer encoder. Self-attention and cross-attention mechanisms are used to perform global aggregation within the same graph and cross-graph alignment enhancement, outputting a context-enhanced feature map. , .

[0025] Preferably, the specific steps of the fine-level matching stage in step (4) include:

[0026] (4.2a) Map the coarse-level matching points to the fine-level feature map, and crop two sets of fixed-size windows with the position of the coarse-level matching points as the center anchor point to obtain the local feature map;

[0027] (4.2b) The local feature map is fed into the fine-level Transformer encoder, and the local fine feature map is obtained by using self-attention and cross-attention mechanisms. and ;

[0028] (4.2c) Calculation center vector and The correlation of all vectors yields a local probability heatmap, and the expected value of this heatmap is used to obtain the sub-pixel offset. The coordinates are then superimposed onto the finer-level center coordinates to obtain the finer-level coordinates.

[0029] (4.2d) Restore the fine-level coordinates to the original coordinates according to the feature pyramid ratio, output the fine-level matching points and their confidence scores, and draw the matching visualization effect.

[0030] The beneficial effects of this invention are as follows: Combining the advantages of visual semantic recognition and feature point matching, it achieves intelligent detection and on-demand matching triggering of targets in videos; by suppressing the interference of large-area background and irrelevant textures on matching, it reduces computational redundancy and improves robustness under occlusion, scale and illumination changes; it skips calculations when there are no targets or the targets are significantly insufficient in the image frame, avoiding indiscriminate processing of the entire frame, thereby significantly reducing average computing power and latency, and meeting the real-time requirements of video streaming scenarios; it adopts a "semantic mask + local feature matching" process, performing coarse-to-fine registration only in semantically relevant areas, reducing misleading information caused by non-intersecting viewpoints and background clutter; in traffic scenarios, it can stably locate the same target and reduce the mismatch rate, improving the accuracy and stability of cross-viewpoint alignment. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the steps of the method of the present invention;

[0032] Figure 2 This is a schematic diagram of the network flow of the method of the present invention;

[0033] Figure 3 This is a flowchart illustrating step (2) of the present invention;

[0034] Figure 4 This is a flowchart illustrating the steps of the coarse-level matching stage;

[0035] Figure 5 This is a semantic region visualization diagram generated by the semantic-driven module in this invention;

[0036] Figure 6 This is a visualization of the video frame matching effect of the method of the present invention. Detailed Implementation

[0037] The present invention will be further described below with reference to specific implementation examples, but the scope of protection of the present invention is not limited thereto:

[0038] Example: Figure 1 , Figure 2 As shown, a semantic-driven cross-view matching method for air-to-ground UAV video includes the following steps:

[0039] (1) Acquire image frames and reference images, and perform preprocessing such as scale alignment and color format conversion, and output preprocessed images. Specifically: acquire video data from the drone's aerial perspective, obtain each frame of the drone video stream and a reference image as input; scale the reference frame and video frame by the short side and fill the equilateral side if necessary to achieve scale alignment; unify the input from BGR or grayscale to RGB; output preprocessed images.

[0040] (2) Input the pre-processed image and the text of the preset category into the pre-trained CLIP model to generate image features and text features of the preset category. Calculate the cosine similarity between the two features as the confidence score. When the highest confidence score is greater than or equal to the threshold... At that time, output the semantic trigger signal and the position of the trigger frame image;

[0041] like Figure 3 Specifically, a pre-trained CLIP model (ViT-B / 32 structure) is used to map images and text to the same vector space. Each frame of image is input into the CLIP visual encoder to obtain image feature vectors. Each predefined target category (such as "person", "car", etc.) is input into the CLIP text encoder to obtain the text feature vector of the predefined category. The semantic-driven module calculates the cosine similarity between the image feature vector and the text feature vectors of each preset category as the confidence score. If the highest confidence score is higher than the threshold... If the target semantics are detected, the semantic trigger signal, the trigger frame image, and the location of the reference image are output. This threshold... It can be adjusted according to different application scenarios to balance recall and precision, and can be combined with external semantic segmentation or gradient methods to extract corresponding semantic regions. Through CLIP for semantic detection, this module can leverage the generalization ability of large-scale multimodal pre-training to achieve zero-shot detection for multiple target categories, providing semantic prior guidance for the subsequent LoFTR module.

[0042] (3) Patch features are extracted from the trigger frame image and reference image based on CLIP. The similarity between the patch features and the trigger category text features is calculated to generate a patch heatmap. The patch heatmap is upsampled to the LoFTR coarse resolution to obtain the semantic mask.

[0043] Specifically: After trigger matching, the system uses the CLIP visual encoder to further extract local patch features from the image. CLIP divides the input trigger image into multiple patches, including the trigger frame image and a reference image, and outputs corresponding patch features for each patch. The system obtains the trigger category text features and calculates the similarity with each patch feature to form a patch-level semantic response heatmap. The patch heatmap is then interpolated and upsampled at the coarse feature scale (the same resolution as the LoFTR coarse features, i.e., downsampled 1 / 8 of the trigger frame image) to obtain the semantic mask. .

[0044] (4) Extract coarse and fine feature maps for both the trigger frame image and the reference image. In the coarse matching stage: use a weighting factor to apply the semantic mask to the coarse feature map to obtain the enhanced feature map, and then filter the coarse matching points based on double normalization and mutual nearest neighbors. In the fine matching stage: map the coarse matching points to the fine feature map, perform correlation estimation and expectation solution within the local window to obtain the sub-pixel level position, and then perform visualization processing.

[0045] Specifically:

[0046] The system extracts coarse feature maps by downsampling 1 / 8 of the two images (the trigger frame image and the reference image) and extracts fine feature maps by downsampling 1 / 2 of the two images. The corresponding two sets of coarse feature maps are represented as follows: , The fine-level feature map is represented as , .

[0047] In the coarse-level matching stage, such as Figure 4 As shown, , After flattening the features and adding sinusoidal positional encoding, the input is fed into a coarse-level Transformer encoder. Self-attention and cross-attention mechanisms are used to perform global aggregation within the same graph and cross-graph alignment enhancement, outputting a coarse-scale feature map with enhanced context. , Semantic masks are weighted by factors. The coarse-level feature map is applied to the extraction. and In this case, the semantic mask is only applied in the coarse-level matching stage and is not used in subsequent fine-level matching stages. For example... Figure 5 As shown, the left side of the image is a frame with relatively distinct target features extracted from the video frame as a reference frame, and the right side is the response map for the trigger type "car". The diagonally covered area in the image represents the background suppressed by semantic masking, and the area near the vehicle body is a high-response region. High values ​​amplify the features of the region near the target after modulation, while low-preservation regions are weakened. The system then calculates the similarity matrix S of all coarse-level locations in the two images and performs double normalization to obtain the matching confidence matrix. Based on this matching confidence matrix, high-confidence coarse-level matching points are selected, and inconsistent pairs are removed by applying the mutual nearest neighbor constraint. At the same time, the center of the coarse-level matching point and its confidence are recorded for further fine-level matching.

[0048] In the fine-level matching process, the coarse-level matching points are first mapped onto the fine-level feature map. For each coarse-level matching point... First, determine its location on the fine-level feature map. , by location Two sets of fixed-size windows are cropped from the central anchor point to obtain local feature maps. These local feature maps are then fed into a fine-level transformer, where self-attention and cross-attention are performed again within a small receptive field to refine the local matching discrimination, resulting in two transformed fine-grained local feature maps. and Based on this, the system calculates... center vector and The correlation of all vectors yields a local probability heatmap, and the expected value of this heatmap is used to obtain the sub-pixel offset. The fine-grained matching positions are obtained by overlaying the fine-grained coordinates onto the center coordinates. The system then restores the original image coordinates to the fine-grained coordinates according to the feature pyramid ratio, outputting the fine-grained matching points and their confidence scores. This allows the `make_matching_plot_fast` function to be used to visualize the matching results. Additionally, ffmpeg can be used to create matching videos from time-series images. Figure 6 As shown, Figure 6 This shows the cross-view matching results of two drone-view images under the trigger category "car". Each line in the image represents a pair of matching points between the two images, and the colors are based on the matching confidence level. Linear coloring, The higher the confidence level, the closer it is to a warmer color. For the current image pair, the system actually generated 410 coarse matching point pairs. However, for clearer visualization, we omitted the work of drawing matching point pairs with a confidence level below the threshold of 0.5. Regarding inference time, after extensive inference, we found that CLIP mask generation takes approximately 0.06 seconds, while the semantically triggered local matching module takes around 0.15 seconds, with the total process time within 0.2 seconds. In addition, ffmpeg can be used to create matching videos from time-series images.

[0049] The above description describes specific embodiments of the present invention and the technical principles employed. If changes are made in accordance with the concept of the present invention, and the resulting functions do not exceed the spirit covered by the specification and drawings, they should still fall within the protection scope of the present invention.

Claims

1. A semantic-driven cross-view matching method for air-to-ground UAV video, characterized in that... Includes the following steps: (1) Acquire image frames and reference images, and perform preprocessing such as scale alignment and color format conversion, and output the preprocessed image; (2) Input the pre-processed image and the text of the preset category into the pre-trained CLIP model to generate image features and text features of the preset category. Calculate the cosine similarity between the two features as the confidence score. When the highest confidence score is greater than or equal to the threshold... At that time, output the semantic trigger signal and the position of the trigger frame image; (3) Patch features are extracted from the trigger frame image and reference image based on CLIP. The similarity between the patch features and the trigger category text features is calculated to generate a patch heatmap. The patch heatmap is upsampled to the LoFTR coarse resolution to obtain the semantic mask. (4) Extract coarse and fine feature maps for both the trigger frame image and the reference image. In the coarse matching stage: use a weighting factor to apply the semantic mask to the coarse feature map to obtain the enhanced feature map, and then filter the coarse matching points based on double normalization and mutual nearest neighbors. In the fine matching stage: map the coarse matching points to the fine feature map, perform correlation estimation and expectation solution within the local window to obtain the sub-pixel level position, and then perform visualization processing.

2. The semantic-driven cross-view matching method for air-to-ground UAV video according to claim 1, characterized in that... The specific steps (1) include: acquiring each frame of the UAV video stream and a reference image as input; scaling by the short side and filling with equal sides if necessary to achieve scale alignment; unifying the input from BGR or grayscale to RGB; and outputting the preprocessed image.

3. A semantically driven cross-view matching method for air-to-ground UAV video according to claim 1 or 2, characterized in that... The reference image is a manually selected frame containing the target.

4. The semantic-driven cross-view matching method for air-to-ground UAV video according to claim 1, characterized in that... The preset category text in step (2) includes one or more predefined target categories.

5. A semantically driven cross-view matching method for air-to-ground UAV video according to claim 1, characterized in that... The threshold Customizable.

6. A semantically driven cross-view matching method for air-to-ground UAV video according to claim 1, characterized in that... Step (3) specifically includes the following steps: (3.1) CLIP divides the trigger frame image and the reference image into several patches respectively, and outputs patch features for each patch; (3.2) Obtain the text features of the trigger category; (3.3) Calculate the similarity between the trigger category text features and each patch feature to form a patch heatmap. Upsample the patch heatmap to the LoFTR coarse resolution to obtain a semantic mask. The LoFTR coarse resolution is 1 / 8 of the downsampled trigger frame image.

7. A semantically driven cross-view matching method for air-to-ground UAV video according to claim 1, characterized in that... In step (4), the process of downsampling the trigger frame image and the reference image by 1 / 8 to extract a coarse feature map and downsampling by 1 / 2 to extract a fine feature map, the corresponding two sets of coarse feature maps are represented as follows: , The fine-level feature map is represented as , .

8. A semantically driven cross-view matching method for air-to-ground UAV video according to claim 1, characterized in that... In step (4), the specific steps of the coarse-level matching stage include: (4.1a) Applying a weighting factor to the semantic mask on the coarse feature map , To obtain attention-enhanced feature maps , ; (4.1b) Calculation , The similarity matrix S is then normalized to obtain the matching confidence matrix; (4.1c) After filtering out coarse-level matching points with high confidence based on the matching confidence matrix, apply the mutual nearest neighbor constraint to remove inconsistent pairs, and record the center of the coarse-level matching point and its confidence.

9. A semantically driven cross-view matching method for air-to-ground UAV video according to claim 8, characterized in that, The specific steps (4.1a) include: ... , After flattening the features and adding sinusoidal positional encoding, the input is fed into a coarse-level Transformer encoder. Self-attention and cross-attention mechanisms are used to perform global aggregation within the same graph and cross-graph alignment enhancement, outputting a context-enhanced feature map. , .

10. A semantically driven cross-view matching method for air-to-ground UAV video according to claim 1, characterized in that... In step (4), the specific steps of the fine-level matching stage include: (4.2a) Map the coarse-level matching points to the fine-level feature map, and crop two sets of fixed-size windows with the position of the coarse-level matching points as the center anchor point to obtain the local feature map; (4.2b) The local feature map is fed into the fine-level Transformer encoder, and the local fine feature map is obtained by using self-attention and cross-attention mechanisms. and ; (4.2c) Calculation center vector and The correlation of all vectors yields a local probability heatmap, and the expected value of this heatmap is used to obtain the sub-pixel offset. The coordinates are then superimposed onto the finer-level center coordinates to obtain the finer-level coordinates. (4.2d) Restore the fine-level coordinates to the original coordinates according to the feature pyramid ratio, output the fine-level matching points and their confidence scores, and draw the matching visualization effect.

Images