A cross-modal template super field of view target matching and positioning method and device

Abstract

The application discloses a kind of cross-modal template super field target matching and positioning method and device, the application adopts adaptive convolution feature point network+cross attention matching network combination, by convolution network is responsible for mining modal invariant local topological features, attention network eliminates ambiguity matching, the whole architecture does not need to collect scene visible light-infrared pair sample, can be directly deployed in the field, to solve the problem of high fine-tuning cost, long landing cycle.The application adopts global homography matrix super field coordinate mapping mechanism: abandon single-point landmark calculation logic, rely on image global matching topological constraint to establish two-dimensional plane projection relationship, directly map target priori coordinate to real-time imaging coordinate system, offset most inertial navigation attitude error from algorithm level, weaken high-precision navigation hardware dependence.

Description

Technical Field

[0001] This invention relates to the field of optoelectronic guidance technology, and in particular to a method and apparatus for cross-modal template over-field target matching and positioning. Background Technology

[0002] In air-to-ground optoelectronic observation missions, existing image-matching-based over-field-of-view target localization schemes generally employ template traversal matching logic. This involves offline-prepared target local templates being traversed and matched pixel-by-pixel within real-time infrared images. If a match is successful, target localization is directly achieved; if a match fails, a backup local template is switched and the traversal is repeated. Ultimately, the target coordinates are calculated based on the fixed relative geometric relationship between reference points within the template and the target. Practice has shown that this localization method suffers from poor robustness and weak anti-interference capability. Therefore, improving the robustness and anti-interference capability of localization matching has become an urgent problem to be solved. Summary of the Invention

[0003] This invention provides a cross-modal template beyond-field target matching and localization method and apparatus to solve the problems of poor matching robustness and weak anti-interference ability of existing localization methods.

[0004] In a first aspect, the present invention provides a cross-modal template beyond-field-of-view target matching and localization method, the method comprising: performing cross-modal association between a preset visible light template and an acquired real-time infrared image, wherein the visible light template is a ground visible light remote sensing image labeled with a target; extracting feature points of the visible light template and the real-time infrared image in parallel using an adaptive convolutional feature point description network; based on the extracted feature points, performing bidirectional matching of cross-modal feature points using a cross-attention feature point matching network to obtain matching feature points, wherein the cross-attention feature point matching network adopts a hierarchical self-attention layer + cross-attention layer cascaded architecture; using the RANSAC random sample consensus algorithm to remove outliers based on the matched feature points, and solving for the optimal homography matrix H, wherein the homography matrix is ​​used to characterize the global planar projection mapping relationship between the visible light template and the real-time infrared image; using the homography matrix to convert the target coordinates in the visible light template into the target's mapped coordinates in the infrared coordinate system, and performing target localization.

[0005] Optionally, the cross-modal association between the preset visible light template and the acquired real-time infrared image includes: marking the target on the preset visible light remote sensing image of the Earth to obtain the visible light template; and aligning the visible light template with the observation orientation of the real-time infrared image through correction processing based on the acquired real-time infrared image.

[0006] Optionally, the step of aligning the visible light template with the observation orientation of the infrared real-time image through correction processing based on the acquired infrared real-time image includes: based on continuously acquired ground infrared real-time images, synchronously reading the three-axis real-time attitude angles output by the airborne inertial navigation system, calculating the initial projection transformation parameters according to the attitude angles, and performing global rotation and translation coarse correction on the visible light template to offset the image viewing angle deviation caused by platform flight jitter and heading offset, so that the visible light template is aligned with the observation orientation of the infrared real-time image.

[0007] Optionally, the parallel extraction of feature points from the visible light template and the real-time infrared image using an adaptive convolutional feature point description network includes: the adaptive convolutional feature point description network comprising a shared feature encoder, an interest point detector, and a descriptor generator, wherein... The shared feature encoder adopts a VGG convolutional structure, uniformly receives normalized single-channel grayscale images, and outputs a 128-dimensional shared feature map. The shared feature map is subjected to a two-stage convolution transformation by the interest point detector. The first stage increases the dimension to 256 to enhance local texture differences. The second stage outputs the channel feature response map. After normalizing the response probability by Softmax, the background channel is removed and stable corner points are retained as effective feature points. At the same time, the maximum feature point extraction limit of a single image is globally adjusted according to the computing power margin. The descriptor generator encodes local features of key points through interpolation and convolution based on the shared feature map, generates a fixed 256-dimensional vector for each effective feature point, and performs L2 normalization to eliminate the interference of the difference in grayscale values ​​between the infrared real-time image and the visible light image on feature similarity calculation. It also performs direct comparison of cross-modal descriptors to extract feature points of the visible light template and the infrared real-time image.

[0008] Optionally, the cross-modal bidirectional feature point matching using the cross-attention feature point matching network includes: modeling the context association of feature points within a single image through the self-attention layer, where the image is the visible light template or the infrared real-time image; sorting out the topological constraints between local landmarks and eliminating redundant and repeated feature points within the single image; establishing cross-modal feature interaction between the visible light template and the infrared real-time image through the cross-attention layer, so that the feature points of the single image perceive the global feature differences of all feature points in the other modality; calculating the bidirectional matching matrix through cosine similarity, and obtaining a set of matched feature point pairs based on the bidirectional matching matrix.

[0009] Optionally, each layer of the cross-attention feature point matching network has a built-in confidence discriminator at its end. The confidence discriminator prunes feature points that are below a preset matching confidence threshold to reduce subsequent computation.

[0010] Optionally, the matching feature point pairs are repeatedly randomly selected to fit the projection model, and the pixel points with a proportion greater than the pixel point threshold are selected to obtain the optimal pixel point set. Based on the optimal pixel point set, the global optimal homography matrix is ​​obtained by least squares convergence. The mapping formula for the homography matrix is: ,in, Let be the homogeneous coordinates of a point in the visible light template. for The corresponding homogeneous coordinates in the real-time infrared image are provided, with all homogeneous coordinates padded with trailing 1s. RGB represents visible light, and IR represents infrared. Let x be the x-coordinate of the target in the visible light image. The vertical coordinate of the target in the visible light image is denoted as . is the x-coordinate of the target in the infrared image. y is the vertical coordinate of the target in the infrared image.

[0011] Optionally, the target coordinates in the visible light template are converted into the mapped coordinates of the target in the infrared coordinate system using the homography matrix, and the target is located, including: The target reference coordinates in the visible light template are converted to homogeneous coordinates. Multiplying this by the homography matrix H yields the target's mapped coordinates in the infrared coordinate system. ; Let the width of the real-time infrared image be W and the height be H. If the following conditions are met... or If the target is determined to be outside the infrared field of view, then... As a field-of-view positioning coordinate, it is used by the equipment to adjust the detection direction and locate the target.

[0012] Secondly, the present invention provides a cross-modal template beyond-field target matching and localization device, the device comprising: A cross-modal association unit is used to perform cross-modal association between a preset visible light template and a acquired real-time infrared image, wherein the visible light template is a ground visible light remote sensing image with a target marked. The extraction unit is used to extract feature points of the visible light template and the real-time infrared image in parallel through an adaptive convolutional feature point description network; The matching unit is used to perform bidirectional matching of cross-modal feature points based on the extracted feature points using a cross-attention feature point matching network to obtain matching feature points. The cross-attention feature point matching network adopts a hierarchical serial architecture of self-attention layer + cross-attention layer. The processing unit is used to remove outliers based on the matched feature points using the RANSAC random sample consensus algorithm, and to solve for the optimal homography matrix H. The homography matrix is ​​used to characterize the global planar projection mapping relationship between the visible light template and the infrared real-time image. The target coordinates in the visible light template are converted into the target's mapped coordinates in the infrared coordinate system using the homography matrix, and the target is located.

[0013] Thirdly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-described cross-modal template over-field target matching and localization methods.

[0014] The beneficial effects of this invention are as follows: This invention employs a combination of an adaptive convolutional feature point network and a cross-attention matching network. The convolutional network is responsible for mining modality-invariant local topological features, while the attention network eliminates ambiguous matches. The entire architecture can be directly deployed in the field without collecting visible light-infrared paired samples within the scene, thus solving the problems of high fine-tuning costs and long deployment cycles in existing systems. Furthermore, this invention adopts a global homography matrix super-field-of-view coordinate mapping mechanism: abandoning the logic of single-point landmark inference, it establishes a two-dimensional planar projection relationship based on global image matching topological constraints, directly mapping the target's prior coordinates to the real-time imaging coordinate system. This algorithmically offsets most of the inertial navigation attitude errors, reducing the reliance on high-precision navigation hardware.

[0015] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0016] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a flowchart illustrating a cross-modal template over-field target matching and localization method provided in an embodiment of the present invention; Figure 2 This is a flowchart illustrating another cross-modal template beyond-field target matching and localization method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of a visible light template provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of a real-time infrared image provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the corrected visible light template provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the feature points of the corrected visible light template extracted by the adaptive convolutional feature point description network provided in this embodiment of the invention; Figure 7 This is a schematic diagram of feature points extracted from a real-time infrared image by an adaptive convolutional feature point description network provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of feature point matching between the corrected visible light template and the real-time infrared image provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of the structure of a cross-modal template beyond-field target matching and positioning device provided in an embodiment of the present invention. Detailed Implementation

[0017] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of the invention.

[0018] In air-to-ground optoelectronic observation missions, visible light templates (satellite remote sensing images, aerial images) have low acquisition thresholds and rich imaging texture details, capable of completely restoring structural information such as target shape, surrounding road network, and building topology, making them the mainstream data source for target prior information acquisition. However, visible light imaging is highly susceptible to interference from day and night lighting, clouds, fog, dust, and shadows, and imaging fails at night and in severe weather conditions. Infrared imaging, on the other hand, relies on target thermal radiation imaging, does not depend on external lighting, has stronger cloud and fog penetration and environmental anti-interference capabilities, and its all-weather observation stability is far superior to visible light. Therefore, constructing a robust cross-modal association between visible light prior templates and real-time infrared images is a core requirement for achieving long-distance beyond-field-of-view target localization under all-weather and complex backgrounds. Existing image matching-based beyond-field-of-view target localization schemes generally adopt template traversal matching logic: offline bound target local templates are traversed and matched pixel by pixel in real-time infrared images. If a match is successful, the target localization is directly completed; if a match fails, a backup local template is switched and the traversal is repeated, ultimately relying on the fixed relative geometric relationship between the reference points within the template and the target to calculate the target coordinates. Practice has shown that current engineering applications combining visible light and infrared cross-modal air-to-ground observation are not suitable for complex field environments, mainly due to two problems. First, traditional manually designed features and shallow grayscale / gradient template matching algorithms have extremely poor modal generalization capabilities. Visible light relies on surface reflected light for imaging, while infrared relies on thermal radiation. The pixel grayscale distribution, edge texture, local contrast, and target representation logic of these two types of images have fundamentally different mechanisms, resulting in completely unrelated visual features for the same ground object in both types of images. Traditional algorithms cannot mine deep features shared across modalities, ultimately leading to a high mismatch rate and difficulty in establishing pixel-level image correspondences. Second, the problem of amplified attitude errors in beyond-field-of-view target localization is prominent. Existing solutions all adopt a single-point landmark geometric extrapolation mode, relying solely on a set of matching points combined with the platform's inertial navigation attitude to complete coordinate conversion. This mode places stringent requirements on the accuracy of the platform's pitch, roll, and yaw three-axis attitude measurements; even small attitude measurement errors will be linearly amplified into target localization deviations through geometric transformations. To control positioning errors, a high-precision inertial navigation system must be installed, which significantly increases the overall cost of the optoelectronic payload. When a low-precision inertial navigation system is used in conjunction with single-point calculation, the positioning deviation beyond the field of view is likely to exceed the engineering allowable threshold, directly leading to target loss.

[0019] In other words, the differences between existing visible light and infrared imaging mechanisms lead to a high cross-modal mismatch rate, poor matching robustness, and weak anti-interference ability; furthermore, existing beyond-field-of-view scenarios lack a global image mapping and localization mechanism, resulting in low single-point calculation fault tolerance and sensitivity to attitude errors.

[0020] To address the aforementioned issues, this invention constructs a three-tiered technical chain: "adaptive convolutional feature extraction - cross-attention cross-modal matching - robust homography matrix estimation." Relying on an open-source pre-trained deep learning model, it can complete the description of visible and infrared feature points without the need for fine-tuning by acquiring paired, homologous visible and infrared images. Through a cross-attention mechanism, it achieves robust cross-modal feature matching. Based on the global matching point pairs, it solves the planar projection transformation relationship, reducing the dependence on the platform's inertial navigation attitude accuracy. Finally, it completes the mapping of target coordinates outside the field of view through the homography matrix.

[0021] Specifically, embodiments of the present invention provide a cross-modal template beyond-field target matching and localization method, see [link to relevant documentation]. Figure 1 and Figure 2 The method includes: S101. Perform cross-modal correlation between the preset visible light template and the acquired real-time infrared image; In this embodiment of the invention, the visible light template is a visible light remote sensing image of the Earth marked with a target; In a specific implementation, the present invention involves marking targets on a preset visible light remote sensing image of the Earth to obtain the visible light template; and based on the acquired real-time infrared image, a correction process is performed to align the visible light template with the observation orientation of the real-time infrared image.

[0022] It should be noted that, in specific implementation, those skilled in the art can also use the visible light template and the acquired real-time visible light image to process the method described in the embodiments of the present invention, thereby achieving the positioning of the target.

[0023] The correction process in this embodiment of the invention is based on continuously acquired real-time infrared images of the ground, synchronously reading the three-axis real-time attitude angles output by the airborne inertial navigation system, calculating the initial projection transformation parameters based on the attitude angles, and performing global rotation and translation coarse correction on the visible light template to offset the image viewing angle deviation caused by platform flight jitter and heading offset, so that the visible light template is aligned with the observation orientation of the real-time infrared image.

[0024] Specifically, embodiments of the present invention use high-resolution visible light remote sensing images of the Earth as offline reference templates, as detailed below. Figure 3 As shown, the image sources include satellite remote sensing and aerial images. The template needs to cover the target to be located and the surrounding background area, with priority given to densely textured backgrounds such as road networks and building complexes to provide sufficient and stable features for cross-modal matching. An image pixel coordinate system is established with the top left corner of the template as the origin. The center pixel coordinates of the stationary ground target are manually labeled as the prior coordinates of the visible light target, completing the offline binding and storage of the target template.

[0025] The airborne infrared photoelectric payload of this invention continuously acquires real-time grayscale images of the ground, specifically as follows: Figure 4 As shown, time-series infrared video frames are output; the real-time three-axis attitude angles output by the airborne inertial navigation system are read synchronously, allowing for an initial measurement error of less than 15°. Based on the attitude angle coarse calculation of the initial projection transformation parameters, global rotation and translation coarse correction are performed on the offline visible light template to offset the image viewing angle deviation caused by platform flight jitter and heading offset, so that the observation orientation of the visible light template is initially aligned with that of the real-time infrared image, specifically as follows. Figure 5 As shown, this reduces the difficulty of subsequent feature matching.

[0026] It should be noted that this correction is only a coarse matching and does not require high-precision attitude iteration optimization.

[0027] S102. The feature points of the visible light template and the real-time infrared image are extracted in parallel using an adaptive convolutional feature point description network. Specifically, the adaptive convolutional feature point description network of this embodiment includes a shared feature encoder, an interest point detector, and a descriptor generator, wherein... The shared feature encoder adopts a VGG convolutional structure, uniformly receives normalized single-channel grayscale images, and outputs a 128-dimensional shared feature map. The shared feature map is subjected to a two-stage convolution transformation by the interest point detector. The first stage increases the dimension to 256 to enhance local texture differences. The second stage outputs the channel feature response map. After normalizing the response probability by Softmax, the background channel is removed and stable corner points are retained as effective feature points. At the same time, the maximum feature point extraction limit of a single image is globally adjusted according to the computing power margin. The descriptor generator encodes local features of key points through interpolation and convolution based on the shared feature map, generates a fixed 256-dimensional vector for each effective feature point, and performs L2 normalization to eliminate the interference of the difference in grayscale values ​​between the infrared real-time image and the visible light image on feature similarity calculation. It also performs direct comparison of cross-modal descriptors to extract feature points of the visible light template and the infrared real-time image.

[0028] In other words, this embodiment of the invention employs a self-supervised pre-trained adaptive convolutional feature point description network to extract, in parallel, sparse feature point pixel coordinates and 256-dimensional local feature descriptors from coarsely corrected visible light templates and real-time infrared images, as detailed below. Figure 6 and Figure 7As shown. Practice has proven that, compared to traditional SIFT and ORB handcrafted features, this network, through self-supervised pre-training, has a natural adaptability to illumination reversal, texture loss, and modal grayscale reversal; compared to the Transformer global feature network, the pure convolutional architecture has lower computational overhead and lower inference latency, meeting the low-latency real-time processing requirements of airborne embedded platforms, while also incorporating a multi-scale feature encoding module to adapt to image scale changes caused by continuous platform degradation and long-distance zoom.

[0029] S103. Based on the extracted feature points, a cross-attention feature point matching network is used to perform bidirectional matching of feature points across modalities to obtain matching feature points. The cross-attention feature point matching network in this embodiment of the invention adopts a hierarchical self-attention layer + cross-attention layer serial architecture; Specifically, this embodiment of the invention models the contextual association of feature points within a single image through the self-attention layer. This image can be either the visible light template or the real-time infrared image. It sorts out the topological constraints between local landmarks and removes redundant and repetitive feature points within the single image. Furthermore, it establishes cross-modal feature interaction between the visible light template and the real-time infrared image through the cross-attention layer, enabling feature points in a single image to perceive the global feature differences of all feature points in another modality. Finally, it calculates a bidirectional matching matrix using cosine similarity and obtains a set of matched feature point pairs based on the bidirectional matching matrix. See details below. Figure 8 .

[0030] In this embodiment of the invention, each layer of the cross-attention feature point matching network has a built-in confidence discriminator at its end. The confidence discriminator prunes feature points that are below a preset matching confidence threshold to reduce the amount of subsequent computation.

[0031] In simple terms, this invention involves inputting the feature point coordinates and normalized descriptors of two sets of images into a lightweight cross-attention feature point matching network to achieve bidirectional cross-modal feature point matching. The network employs a hierarchical self-attention + cross-attention concatenated architecture, possessing two lightweight optimization capabilities: dynamic pruning and early termination of inference, requiring no additional computational overhead. First, a self-attention layer is used to model the contextual association of feature points within a single image, clarifying the topological constraints between local landmarks and eliminating redundant and repetitive feature points within the image. Then, a cross-attention layer is used to establish cross-modal feature interaction between visible light and infrared images, allowing feature points in one image to perceive the global feature differences of all feature points in another modality. Each layer of the network incorporates a lightweight confidence discriminator at its end, permanently pruning feature points with low matching confidence to reduce subsequent computation. If the matching confidence of a shallow network layer meets the threshold, deep inference is terminated directly, further compressing latency. Finally, a bidirectional matching matrix is ​​calculated using cosine similarity, outputting a preliminary set of matched feature point pairs.

[0032] S104. Based on the matched feature points, the RANSAC random sample consensus algorithm is used to remove outliers, and the optimal homography matrix H is solved. The homography matrix is ​​used to characterize the global planar projection mapping relationship between the visible light template and the infrared real-time image. The target coordinates in the visible light template are converted into the target's mapped coordinates in the infrared coordinate system using the homography matrix, and the target is located.

[0033] Specifically, in this embodiment of the invention, the projection model is fitted by repeatedly randomly sampling matching feature point pairs, and the pixel in-pixel points with a proportion greater than the pixel in-pixel threshold are selected to obtain the optimal pixel in-pixel point set. Based on the optimal pixel in-pixel point set, the global optimal homography matrix is ​​obtained by least squares convergence. The mapping formula for the homography matrix in this embodiment of the invention is as follows: ,in, Let be the homogeneous coordinates of a point in the visible light template. for The corresponding homogeneous coordinates in the real-time infrared image are provided, with all homogeneous coordinates padded with trailing 1s. RGB represents visible light, and IR represents infrared. Let x be the x-coordinate of the target in the visible light image. The vertical coordinate of the target in the visible light image is denoted as . is the x-coordinate of the target in the infrared image. y is the vertical coordinate of the target in the infrared image.

[0034] In practice, due to outdoor occlusion and noise interference, the initial matched point pairs contain a large number of mismatched outliers. The RANSAC random sample consensus algorithm is used to remove outliers and solve for the optimal 3×3 homography matrix H. The iteration count is set to 100 times, and the inlier threshold is 1 pixel. Matched point pairs are repeatedly randomly selected to fit the projection model, and the optimal set of points with an inlier ratio higher than 90% is selected. Based on the optimal inlier set, the global optimal homography matrix is ​​converged using the least squares method. The homography matrix represents the global planar projection mapping relationship between two two-dimensional images, and the mapping formula is: ,in Let these be the homogeneous coordinates of a point in the visible light template. This represents the homogeneous coordinates of the point in the infrared image. Homogeneous coordinates are uniformly padded with trailing 1s. RGB represents visible light, and IR represents infrared. Let x be the x-coordinate of the target in the visible light image. The vertical coordinate of the target in the visible light image is denoted as . is the x-coordinate of the target in the infrared image. Here, represents the vertical coordinate of the target in the infrared image. Because this invention uses matrix fusion to integrate topological constraints from hundreds of matching points globally, unlike single-point geometric transformations, it possesses extremely strong fault tolerance.

[0035] It should be noted that, in this embodiment of the invention, the target reference coordinates in the visible light template are converted into homogeneous coordinates. Multiplying this by the homography matrix H yields the target's mapped coordinates in the infrared coordinate system. Let the width of the real-time infrared image be W and the height be H. If the following conditions are met... or If the target is determined to be outside the infrared field of view, then... As a field-of-view positioning coordinate, it is used by the equipment to adjust the detection direction and locate the target.

[0036] It is understood that the embodiments of this invention adopt a cross-modal lightweight dual-network linkage matching architecture: a combination of a pre-trained adaptive convolutional feature point network with unpaired modal data fine-tuning and a cross-attention matching network. The convolutional network is responsible for mining modality-invariant local topological features, and the attention network is responsible for eliminating ambiguous matches. The entire architecture can be directly deployed in the field without collecting visible light-infrared paired samples in the scene, solving the problems of high cost and long deployment cycle of traditional domain adaptive fine-tuning solutions. Global homography matrix super-field-of-view coordinate mapping mechanism: abandoning the logic of single-point landmark inference, relying on the topological constraints of global image matching to establish a two-dimensional plane projection relationship, directly mapping the target's prior coordinates to the real-time imaging coordinate system, offsetting most of the inertial navigation attitude error from the algorithm level, and weakening the dependence on high-precision navigation hardware.

[0037] In specific implementation, this invention uses a pre-trained adaptive convolutional feature point network combined with a lightweight cross-attention matching network in the field of air-to-ground photoelectric cross-modal observation to achieve the application of cross-modal feature matching without fine-tuning of visible light-infrared heterogeneous images; and realizes the mapping of visible light template target coordinates to infrared real-time image coordinate system based on global homography matrix to complete a general positioning method for over-field-of-view target pointing prediction.

[0038] Practice has proven that, firstly, this invention, through an adaptive convolutional feature network, learns modality-invariant geometric features such as edges, contours, and relative distances using self-supervised learning, rather than pixel grayscale features, thus avoiding the grayscale distribution differences between visible light and infrared light caused by different imaging mechanisms. Cross-attention, through the interaction of dual-modal features, supplements background context constraints and distinguishes ambiguous matches between similar-looking objects. With these two elements working together, the matching basis shifts from shallow pixel information to deep geometric and topological information, thus significantly improving the cross-high-quality inlier rate. Secondly, the homography matrix of this invention is constrained by dozens to hundreds of matching points, possessing data redundancy tolerance and capable of offsetting random matching noise and coarse pose errors. Simultaneously, the two-dimensional planar mapping does not depend on the platform's three-dimensional spatial pose, relying solely on the image's own features for alignment, completely reducing the rigid requirements for high-precision inertial navigation and adapting to the large-scale deployment of low-cost optoelectronic platforms.

[0039] See Figure 9 This invention also provides a cross-modal template beyond-field target matching and localization device, the device comprising: A cross-modal association unit is used to perform cross-modal association between a preset visible light template and a acquired real-time infrared image, wherein the visible light template is a ground visible light remote sensing image with a target marked. Specifically, in this embodiment of the invention, the cross-modal association unit performs target annotation on a preset visible light remote sensing image of the Earth to obtain the visible light template; based on the acquired real-time infrared image, the visible light template is aligned with the observation orientation of the real-time infrared image through correction processing.

[0040] The extraction unit is used to extract feature points of the visible light template and the real-time infrared image in parallel through an adaptive convolutional feature point description network; The adaptive convolutional feature point description network of this invention includes a shared feature encoder, an interest point detector, and a descriptor generator. The shared feature encoder uses a VGG convolutional structure, uniformly receiving normalized single-channel grayscale images and outputting a 128-dimensional shared feature map. The interest point detector performs a two-stage convolutional transformation on the shared feature map: the first stage increases the dimensionality to 256 to enhance local texture differences; the second stage outputs a channel feature response map. After normalizing the response probabilities using Softmax, the background channel is removed, and stable corner points are retained as effective feature points. Simultaneously, the maximum feature point extraction limit for a single image is globally adjusted based on available computing power. The descriptor generator, based on the shared feature map, performs local feature encoding of key points through interpolation convolution, generating a fixed 256-dimensional vector for each effective feature point. L2 normalization is then performed to eliminate the interference of grayscale value differences between the infrared real-time image and the visible light image on feature similarity calculation. Direct comparison of cross-modal descriptors is then performed to extract feature points from the visible light template and the infrared real-time image.

[0041] The matching unit is used to perform bidirectional matching of cross-modal feature points based on the extracted feature points using a cross-attention feature point matching network to obtain matching feature points. The cross-attention feature point matching network adopts a hierarchical serial architecture of self-attention layer + cross-attention layer. Specifically, in this embodiment of the invention, the context association modeling of feature points within a single image is performed through the self-attention layer. The image is either the visible light template or the real-time infrared image. The topological constraints between local landmarks are sorted out, and redundant and repeated feature points within a single image are eliminated. Cross-modal feature interaction between the visible light template and the real-time infrared image is established through the cross-attention layer, enabling feature points in a single image to perceive the global feature differences of all feature points in another modality. A bidirectional matching matrix is ​​calculated using cosine similarity, and a set of matching feature point pairs is obtained based on the bidirectional matching matrix.

[0042] The processing unit is used to remove outliers based on the matched feature points using the RANSAC random sample consensus algorithm, and to solve for the optimal homography matrix H. The homography matrix is ​​used to characterize the global planar projection mapping relationship between the visible light template and the infrared real-time image. The target coordinates in the visible light template are converted into the target's mapped coordinates in the infrared coordinate system using the homography matrix, and the target is located.

[0043] In specific implementation, the embodiments of the present invention involve repeatedly and randomly sampling matching feature point pairs through the processing unit to fit a projection model, selecting pixel points whose pixel point ratio is greater than the pixel point threshold to obtain the optimal pixel point set, and then using the least squares method to converge the optimal homography matrix based on the optimal pixel point set. The mapping formula for the homography matrix is ​​as follows: , Let be the homogeneous coordinates of a point in the visible light template. for The corresponding homogeneous coordinates in the real-time infrared image are provided, with all homogeneous coordinates padded with trailing 1s. RGB represents visible light, and IR represents infrared. Let x be the x-coordinate of the target in the visible light image. The vertical coordinate of the target in the visible light image is denoted as . is the x-coordinate of the target in the infrared image. y is the vertical coordinate of the target in the infrared image.

[0044] By converting the target reference coordinates in the visible light template into homogeneous coordinates Multiplying this by the homography matrix H yields the target's mapped coordinates in the infrared coordinate system. ; Let the width of the real-time infrared image be W and the height be H. If the following conditions are met... or If the target is determined to be outside the infrared field of view, then... As a field-of-view positioning coordinate, it is used by the equipment to adjust the detection direction and locate the target.

[0045] In other words, in the field of air-to-ground photoelectric cross-modal observation, the embodiments of the present invention use a pre-trained adaptive convolutional feature point network combined with a lightweight cross-attention matching network to achieve the application of cross-modal feature matching without fine-tuning of visible light-infrared heterogeneous images, and realize the mapping of visible light template target coordinates to infrared real-time image coordinate system based on global homography matrix to complete the general positioning of target pointing prediction beyond the field of view.

[0046] Meanwhile, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-described cross-modal template over-field target matching and localization methods.

[0047] The relevant content of the device embodiment and storage medium embodiment of the present invention can be understood by referring to the method embodiment of the present invention, and will not be discussed in detail here.

[0048] Although preferred embodiments of the invention have been disclosed for illustrative purposes, those skilled in the art will recognize that various modifications, additions, and substitutions are possible, and therefore the scope of the invention should not be limited to the embodiments described above.

Claims

1. A cross-modal template beyond-field target matching and localization method, characterized in that, The method includes: Cross-modal correlation is performed between a preset visible light template and a real-time infrared image, wherein the visible light template is a ground visible light remote sensing image with a target marked. The visible light template and the real-time infrared image are extracted in parallel using an adaptive convolutional feature point description network. Based on the extracted feature points, a cross-attention feature point matching network is used to perform bidirectional matching of feature points across modalities to obtain matching feature points. The cross-attention feature point matching network adopts a hierarchical self-attention layer + cross-attention layer cascaded architecture. Based on the matched feature points, outliers are eliminated using the RANSAC random sample consensus algorithm, and the optimal homography matrix H is solved. The homography matrix is ​​used to characterize the global planar projection mapping relationship between the visible light template and the infrared real-time image. The target coordinates in the visible light template are converted into the target's mapped coordinates in the infrared coordinate system using the homography matrix, and then the target is located.

2. The method according to claim 1, characterized in that, The cross-modal correlation of the preset visible light template and the acquired real-time infrared image includes: The target is marked on a preset visible light remote sensing image of the Earth to obtain the visible light template; Based on the acquired real-time infrared image, the visible light template is aligned with the observation orientation of the real-time infrared image through correction processing.

3. The method according to claim 2, characterized in that, The process of aligning the visible light template with the observation orientation of the infrared real-time image through correction processing, based on the acquired infrared real-time image, includes: Based on continuously acquired real-time infrared images of the ground, the three-axis real-time attitude angles output by the airborne inertial navigation system are read synchronously. The initial projection transformation parameters are calculated based on the attitude angles, and the visible light template is subjected to global rotation and translation coarse correction to offset the image viewing angle deviation caused by platform flight jitter and heading offset, so that the visible light template is aligned with the observation orientation of the real-time infrared image.

4. The method according to claim 1, characterized in that, The step of extracting feature points from the visible light template and the real-time infrared image in parallel using an adaptive convolutional feature point description network includes: The adaptive convolutional feature point description network includes a shared feature encoder, an interest point detector, and a descriptor generator, wherein... The shared feature encoder adopts a VGG convolutional structure, uniformly receives normalized single-channel grayscale images, and outputs a 128-dimensional shared feature map. The shared feature map is subjected to a two-stage convolution transformation by the interest point detector. The first stage increases the dimension to 256 to enhance local texture differences. The second stage outputs the channel feature response map. After normalizing the response probability by Softmax, the background channel is removed and stable corner points are retained as effective feature points. At the same time, the maximum feature point extraction limit of a single image is globally adjusted according to the computing power margin. The descriptor generator encodes local features of key points through interpolation and convolution based on the shared feature map, generates a fixed 256-dimensional vector for each effective feature point, and performs L2 normalization to eliminate the interference of the difference in grayscale values ​​between the infrared real-time image and the visible light image on feature similarity calculation. It also performs direct comparison of cross-modal descriptors to extract feature points of the visible light template and the infrared real-time image.

5. The method according to claim 1, characterized in that, The method of using a cross-attention feature point matching network for bidirectional feature point matching across modalities includes: The self-attention layer is used to model the context association of feature points within a single image, which is either the visible light template or the real-time infrared image. This process sorts out the topological constraints between local landmarks and removes redundant and repetitive feature points within the single image. Furthermore, the cross-attention layer establishes cross-modal feature interaction between the visible light template and the real-time infrared image, enabling feature points in a single image to perceive the global feature differences of all feature points in another modality. A bidirectional matching matrix is ​​calculated using cosine similarity, and a set of matching feature point pairs is obtained based on the bidirectional matching matrix.

6. The method according to claim 5, characterized in that, Each layer of the cross-attention feature point matching network has a built-in confidence discriminator at its end. The confidence discriminator prunes feature points that are below a preset matching confidence threshold to reduce the amount of subsequent computation.

7. The method according to any one of claims 1-6, characterized in that, By repeatedly randomly selecting matching feature point pairs to fit the projection model, the optimal set of intra-pixel points is obtained by screening intra-pixel points whose intra-pixel point ratio is greater than the intra-pixel point threshold. Based on the optimal set of intra-pixel points, the globally optimal homography matrix is ​​obtained by convergence through the least squares method. The mapping formula for the homography matrix is: ,in, Let be the homogeneous coordinates of a point in the visible light template. for The corresponding homogeneous coordinates in the real-time infrared image are provided, with all homogeneous coordinates padded with trailing 1s. RGB represents visible light, and IR represents infrared. Let x be the x-coordinate of the target in the visible light image. The vertical coordinate of the target in the visible light image is denoted as . is the x-coordinate of the target in the infrared image. y is the vertical coordinate of the target in the infrared image.

8. The method according to claim 7, characterized in that, The target coordinates in the visible light template are converted into mapped coordinates in the infrared coordinate system using the homography matrix, and target localization is performed, including: The target reference coordinates in the visible light template are converted to homogeneous coordinates. Multiplying this by the homography matrix H yields the target's mapped coordinates in the infrared coordinate system. ; Let the width of the real-time infrared image be W and the height be H. If the following conditions are met... or If the target is determined to be outside the infrared field of view, then... As a field-of-view positioning coordinate, it is used by the equipment to adjust the detection direction and locate the target.

9. A cross-modal template beyond-field target matching and positioning device, characterized in that, The device includes: A cross-modal association unit is used to perform cross-modal association between a preset visible light template and a acquired real-time infrared image, wherein the visible light template is a ground visible light remote sensing image with a target marked. The extraction unit is used to extract feature points of the visible light template and the real-time infrared image in parallel through an adaptive convolutional feature point description network; The matching unit is used to perform bidirectional matching of cross-modal feature points based on the extracted feature points using a cross-attention feature point matching network to obtain matching feature points. The cross-attention feature point matching network adopts a hierarchical serial architecture of self-attention layer + cross-attention layer. The processing unit is used to remove outliers based on the matched feature points using the RANSAC random sample consensus algorithm, and to solve for the optimal homography matrix H. The homography matrix is ​​used to characterize the global planar projection mapping relationship between the visible light template and the infrared real-time image. The target coordinates in the visible light template are converted into the target's mapped coordinates in the infrared coordinate system using the homography matrix, and the target is located.

10. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the cross-modal template over-field target matching and localization method according to any one of claims 1-8.

Images