A sar and infrared image globally optimal similarity target registration method

Abstract

This invention discloses a global optimal similarity target registration method for SAR and infrared images, comprising: applying trained infrared detection models and SAR detection models to the infrared image and SAR image to be registered, respectively, to obtain infrared target detection sets and SAR target detection sets, performing spatial coordinate system unification, and then constructing a similarity matrix through comprehensive similarity calculation; using a global optimization algorithm to solve the maximum similarity matching optimization problem based on the similarity matrix to obtain a cross-modal target matching set; using interior point screening to obtain an interior point matching set that satisfies geometric consistency, using the least squares method to estimate the homography matrix from the SAR image to the infrared image, and using the homography matrix to perform geometric transformation on the SAR image to complete target registration in cross-modal images with high accuracy and good stability.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a cross-modal registration method for global optimal similarity at the SAR (Synthetic Aperture Radar) and infrared target levels. Background Technology

[0002] In modern remote sensing and intelligent perception systems, SAR and infrared images are crucial data sources for acquiring target information. SAR images can acquire structural and geometric information of targets under all-weather conditions, offering advantages such as strong penetration and resistance to weather interference. Infrared images reflect the thermal characteristics and radiation information of targets, possessing unique value for target identification at night or in low-visibility conditions. Effective registration of these two modalities to achieve information fusion can fully leverage their complementary advantages, providing a reliable data foundation for target detection, identification, and situational awareness in complex environments. However, due to significant differences in imaging mechanisms, radiation characteristics, and texture representation between SAR and infrared images, the appearance features of the same target vary greatly across different modalities, making traditional registration methods based on grayscale information or local features difficult to apply directly. Furthermore, in real-world complex scenarios, target occlusion, background clutter, and viewing angle differences further increase the difficulty of target registration across modalities. Therefore, achieving reliable registration of SAR and infrared images while ensuring high accuracy and robustness has become a critical technical problem that urgently needs to be solved in the field of cross-modal image processing.

[0003] Existing image registration methods mainly include grayscale-based methods and feature-based methods. Grayscale-based methods typically match the entire image using metrics such as maximum cross-correlation and mutual information, achieving good alignment results under homogeneous or optical-infrared conditions, as exemplified by the overall grayscale consistency metric commonly used in multimodal remote sensing registration research [Zhang et al., Remote Sensing, 2021]. However, when applied to SAR-infrared cross-modal images, these methods are prone to failure or significant decrease in registration accuracy due to substantial differences in radiometric characteristics and grayscale distribution caused by different sensor imaging mechanisms. Furthermore, grayscale-based methods are sensitive to noise, geometric distortion, and contrast variations in complex scenes, making robustness difficult to guarantee. Feature-based methods, on the other hand, achieve registration by extracting and matching local features such as points, lines, edges, or regions in the image. These methods are relatively robust in optical-infrared or homologous image registration. For example, Che et al. proposed an infrared-visible image registration method that combines phase consistency and gradient magnitude for stable feature point extraction and designs rotation-invariant feature descriptors to improve matching robustness [Che et al., Remote Sensing, 2024]. However, for SAR-infrared cross-modal images, the significant differences in the local feature representation of the target in different modalities lead to unstable feature matching and a high probability of mismatch, thereby reducing registration accuracy and stability. Furthermore, it is difficult to meet the high robustness requirements in complex scenes.

[0004] Therefore, it has become an important issue to provide an image registration method that has high registration accuracy and stability under complex scenes and strong modal differences, while also having good versatility and engineering feasibility. Summary of the Invention

[0005] To address the aforementioned problems in the existing technology, this invention provides a global optimal similarity target registration method for SAR and infrared images.

[0006] The technical problem to be solved by this invention is achieved through the following technical solution: In a first aspect, the present invention provides a method for global optimal similarity target registration between SAR and infrared images, the method comprising: The trained infrared detection model is applied to the infrared image to be registered to obtain the infrared target detection set, and the trained SAR detection model is applied to the SAR image to be registered to obtain the SAR target detection set. After aligning the infrared target detection set and the SAR target detection set to a single spatial coordinate system, a similarity matrix is ​​constructed through a comprehensive similarity calculation operation; A global optimization algorithm is used to solve the maximum similarity matching optimization problem constructed based on the similarity matrix, resulting in a cross-modal target matching set. The maximum similarity matching optimization problem aims to maximize the overall similarity, with the constraint that each SAR target and each infrared target participate in at most one matching. The cross-modal target matching set is filtered by interior points to obtain an interior point matching set that satisfies geometric consistency; Based on the inlier matching set, the homography matrix from the SAR image to the infrared image is estimated using the least squares method, and the SAR image is geometrically transformed using the homography matrix to complete target registration in the cross-modal image.

[0007] Secondly, the present invention provides a global optimal similarity target registration device for SAR and infrared images, the device comprising: The application module is used to apply the trained infrared detection model to the infrared image to be registered to obtain an infrared target detection set, and to apply the trained SAR detection model to the SAR image to be registered to obtain a SAR target detection set. The calculation module is used to construct a similarity matrix by combining the infrared detection target set and the SAR target detection set into a spatial coordinate system and then performing a comprehensive similarity calculation operation. The solution module is used to solve the maximum similarity matching optimization problem constructed based on the similarity matrix using a global optimization algorithm, and to obtain a cross-modal target matching set; the maximum similarity matching optimization problem takes maximizing the comprehensive similarity as the optimization objective and is constrained to allow each SAR target and each infrared target to participate in at most one matching. The filtering module is used to filter the cross-modal target matching set to obtain an interior point matching set that satisfies geometric consistency. The transformation module is used to estimate the homography matrix from the SAR image to the infrared image using the least squares method based on the inlier matching set, and to perform geometric transformation on the SAR image using the homography matrix to complete target registration in the cross-modal image.

[0008] Thirdly, the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a computer program stored in memory, it implements the steps described in any of the above-mentioned SAR and infrared image global optimal similarity target registration methods.

[0009] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in any of the above-mentioned SAR and infrared image global optimal similarity target registration methods.

[0010] This invention provides a globally optimal similarity-based target registration method for SAR and infrared images. It does not rely on the consistency of grayscale, texture, or local features between SAR and infrared images. Instead, it uses the target object in the target detection result as the basic unit of registration, establishing cross-modal correspondences through target-level spatial relationships, fundamentally reducing the impact of modal differences on registration performance. By unifying the spatial coordinates of targets in SAR and infrared images, a comparable target spatial representation under cross-modal conditions is constructed, providing a unified geometric basis for subsequent target matching. A similarity matrix is ​​constructed through comprehensive similarity calculation operations, effectively improving the stability and discriminative ability of the matching.

[0011] This invention models the cross-modal target matching problem as a maximum similarity matching optimization problem and employs a global optimization algorithm to solve for the one-to-one correspondence between targets. This avoids mismatches caused by greedy matching or locally optimal matching, ensuring that the target matching result is optimal globally. Therefore, this invention maintains high registration accuracy and good stability even in complex scenarios and under conditions of strong modal differences, while also possessing strong versatility and engineering feasibility.

[0012] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating a global optimal similarity target registration method for SAR and infrared images provided in an embodiment of the present invention; Figure 2 These are schematic diagrams of infrared image detection results and SAR image detection results; Figure 3 This is a schematic diagram of cross-modal image target matching results and similarity matrices in two scenarios; Figure 4 It is a stitched image of cross-modal target matching in two scenarios; Figure 5 These are schematic diagrams of cross-modal image homography matrices in two scenarios; Figure 6 These are schematic diagrams of the registration results in two scenarios; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0014] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0015] To address the issues of poor registration accuracy and stability in existing image registration methods under complex scenes and conditions with strong modal differences, as well as low versatility and engineering feasibility, this invention provides a globally optimal similarity target registration method for SAR and infrared images. (See [link to relevant documentation]). Figure 1 , Figure 1 This is a flowchart illustrating a global optimal similarity target registration method for SAR and infrared images provided by an embodiment of the present invention, specifically including the following steps: Step S101: Apply the trained infrared detection model to the infrared image to be registered to obtain an infrared target detection set, and apply the trained SAR detection model to the SAR image to be registered to obtain a SAR target detection set.

[0016] First, the training process of the infrared detection model and the SAR detection model will be explained, as follows: The training methods for the infrared detection model include: Obtain a sample infrared image dataset; the sample infrared image dataset includes multiple sample infrared images and the first target ground truth bounding box, the first target ground truth confidence score, and the first target category label for each sample infrared image; The sample infrared image dataset is input into the initial infrared detection model to obtain the sample infrared target detection set; the sample infrared target detection set includes the first target prediction bounding box, the first target prediction confidence, and the first target prediction category corresponding to each sample infrared image; Based on the differences between the true bounding box and the predicted bounding box of the first target in each sample infrared image, the differences between the true confidence and the predicted confidence of the first target, and the differences between the first target category label and the predicted category of the first target, the first loss function is used for iterative training until the first loss function converges or the number of iterations reaches the preset number, and the trained infrared detection model is obtained. Training methods for SAR detection models include: Obtain a sample SAR image dataset; the sample SAR image dataset includes multiple sample SAR images and the true bounding box of the second target, the true confidence of the second target, and the category label of the second target for each sample SAR image; The sample SAR image dataset is input into the initial SAR detection model to obtain the sample SAR target detection set; the sample SAR target detection set includes the second target prediction bounding box, the second target prediction confidence, and the second target prediction category corresponding to each sample SAR image; Based on the differences between the true bounding box and the predicted bounding box of the second target in each sample SAR image, the differences between the true confidence and the predicted confidence of the second target, and the differences between the second target category label and the predicted category of the second target, the second loss function is used for iterative training until the second loss function converges or the number of iterations reaches the preset number, thus obtaining the trained SAR detection model.

[0017] Specifically, let the sample infrared image dataset be... The sample SAR image dataset is ,in Indicates the first One sample infrared image, Indicates the first Zhang sample SAR images, For the first The set of true bounding boxes of the first target in the sample infrared images For the first The set of true bounding boxes of the second target in a sample SAR image. For the first The first target category label set in the sample infrared images For the first The second target category label set in the sample SAR images, where the true confidence of all first targets and the true confidence of second targets are both 1; This represents the total number of sample infrared images; This represents the total number of sample SAR images.

[0018] The target detection model is trained separately for sample SAR images and sample infrared images. The training objective is to learn the detector. parameters This makes it applicable to any input image Output target detection set: ; Among them, when the input image When the sample is an infrared image, This represents the initial infrared detection model. Indicates the first The first target prediction bounding box of each sample of infrared targets. Indicates the first The first target prediction category for each infrared target. Indicates the first The first target prediction confidence of each sample of infrared targets. This indicates the total number of infrared targets in the sample. Represents the set of infrared target detection samples; Among them, when the input image When the sample SAR image is used, This represents the initial SAR detection model. Indicates the first The second target prediction bounding box of each sample SAR target Indicates the first The second target prediction category for each sample SAR target. Indicates the first The second target prediction confidence of each sample SAR target This represents the total number of SAR targets in the sample. Represents the sample SAR target detection set; During training, the general loss function for: ; in, This represents the classification loss, used to optimize target category prediction; This represents the bounding box regression loss, used to optimize the position and shape weight coefficients of the detection boxes; This represents the target confidence loss, used to determine whether each predicted bounding box contains the target; , , This represents the weights used to balance the impact of the three loss components on model training. It's important to note that this general loss function is the first loss function when training an infrared detection model, and the second loss function when training a SAR detection model.

[0019] ; in, This represents the cross-entropy between the calculated prediction result and the labeled information; when Indicates the first The first target prediction category for each sample of infrared targets. Indicates the first When labeling the first target category of each sample infrared target... This represents the difference between the first target category label and the first target predicted category in the sample infrared image; when Indicates the first The second target prediction category for each sample SAR target. Indicates the first When labeling the second target category of SAR targets in a sample, This represents the difference between the second target category label and the predicted second target category in the sample SAR image. ; in, This indicates that the target exists. Indicates background; when Indicates the first The first target prediction confidence of each sample of infrared targets. Indicates the first When the first target true confidence level of each sample of infrared targets is... This represents the difference between the true confidence level of the first target and the predicted confidence level of the first target in the sample infrared image; when Indicates the first The second target prediction confidence of each sample SAR target Indicates the first When the second target true confidence of a sample SAR target is... This represents the difference between the true confidence level of the second target and the predicted confidence level of the second target in the sample SAR image; ; in, The intersection-union ratio (IU) of the predicted bounding box and the ground truth bounding box. Distance from the center point The length of the envelope diagonal. For aspect ratio consistency, These are the weighting coefficients; when Indicates the first The first target prediction bounding box of each sample of infrared targets. Indicates the first When the first true bounding box of each sample infrared target is obtained... This represents the difference between the true bounding box of the first target and the predicted bounding box of the first target in the sample infrared image; when Indicates the first The second target prediction bounding box of each sample SAR target Indicates the first When the second true bounding box of a sample SAR target is obtained, This represents the difference between the true bounding box of the second target and the predicted bounding box of the second target in the sample SAR image.

[0020] During the training phase, the infrared detection model is trained on both the sample infrared image dataset and the sample SAR image dataset until the first and second loss functions converge or the preset number of iterations is reached. and SAR detection model .

[0021] During the detection phase, the trained infrared detection model is applied to the infrared image to be registered. The infrared target detection set is obtained, including: ; in, Represents the set of infrared target detections in the infrared image to be registered; Indicates the bounding box of an infrared target; Indicates the category of infrared targets; Indicates the confidence level of an infrared target; Indicates the number of infrared targets; The trained SAR detection model is applied to the SAR map to be registered. To obtain a SAR target detection set, including: ; in, This represents the set of SAR target detections for the SAR image to be registered; Represents the SAR target bounding box; Indicates the SAR target category; Indicates the SAR target confidence level; Indicates the number of SAR targets; bounding box , This indicates the coordinates of the top-left corner of the bounding box. This represents the coordinates of the bottom right corner of the bounding box. For each bounding box, calculate the coordinates of its center point. : ; ; in, This indicates rounding down. By calculating the center point coordinates, a set of infrared target detections containing only the center point, category, and confidence level is obtained. and SAR target detection set Each goal Including the coordinates of its image center point Target category and confidence score ,Right now .

[0022] In this embodiment of the invention, in addition to using a target detection model based on the YOLO series of networks, other deep learning target detection frameworks can also be used as alternatives, such as target detection models based on Faster R-CNN, SSD, RetinaNet, or Transformer structures. As long as the target's position, scale, and confidence information can be extracted from SAR and infrared images, they can all be used as inputs for subsequent target-level registration.

[0023] Step S102: After aligning the infrared target detection set and the SAR target detection set with a spatial coordinate system, a similarity matrix is ​​constructed through a comprehensive similarity calculation operation.

[0024] In this embodiment of the invention, due to the significant differences in spatial information between SAR and infrared images, it is necessary to unify the spatial coordinates of targets of the two modalities to the same coordinate system. For the same pair of infrared and SAR images, the image center is already guaranteed to be the scene center during sensor imaging. The spatial resolutions of the infrared and SAR images are obtained separately, and the geometric center point of each image is used as a unified scene center reference point. For each infrared target in the infrared target detection set, based on the spatial resolution of the infrared image, its center point coordinates in pixels are converted into spatial coordinates with the scene center of the infrared image as the origin and physical length as the unit. For each SAR target in the SAR target detection set, based on the spatial resolution of the SAR image, its center point coordinates in pixels are converted into spatial coordinates with the scene center of the SAR image as the origin and physical length as the unit. After the conversion, the spatial coordinates of the infrared and SAR targets are represented based on a unified physical length unit and a coordinate system with the scene center of their respective images as the origin, as follows: Taking infrared images as an example, let the image size be... Then the coordinates of the center point : ; ; Let the spatial resolution of the infrared image be... If the unit is m / pixel, then the spatial coordinates of the infrared target in the spatial coordinate system are: ; in, This represents the vector transpose. The same principle applies to SAR images, which will not be elaborated here. After coordinate transformation, the targets of the two modes ultimately fall into the same spatial coordinate system, achieving a unified spatial coordinate system for the infrared target detection set and the SAR target detection set.

[0025] In this embodiment of the invention, by mapping the detected targets in SAR images and infrared images to the same reference coordinate system and performing spatial normalization based on the scene center, a target spatial representation that is comparable under cross-modal conditions is constructed, providing a unified geometric basis for subsequent target matching.

[0026] In this embodiment of the invention, after aligning the infrared target detection set and the SAR target detection set to a single spatial coordinate system, a similarity matrix is ​​constructed through a comprehensive similarity calculation operation, including: After structuring the infrared target detection set and the SAR target detection set into a spatial coordinate system, for each pair of infrared and SAR targets in the spatial coordinate system, the cosine similarity, distance similarity, and confidence similarity are calculated. The cosine similarity is calculated based on the spatial coordinate vectors of the two targets, the distance similarity is calculated based on the Euclidean distance between the two targets and its ratio to the global maximum distance, and the confidence similarity is calculated based on the class consistency and confidence score of the two targets. The cosine similarity, distance similarity, and confidence similarity between each pair of targets are weighted and fused according to preset weights to obtain the comprehensive similarity of each pair of targets and construct a similarity matrix.

[0027] In one implementation, for infrared targets and SAR targets in the spatial coordinate system, the cosine similarity, distance similarity, and confidence similarity between each pair of targets are calculated as follows: In this embodiment of the invention, the coordinates of the center points of the two targets in the same spatial coordinate system are used, and the vector representation of the coordinates is given as follows: ,but Indicates infrared target The corresponding spatial coordinate vector, Indicates SAR target The corresponding spatial coordinate vectors. Calculate the cosine similarity of their spatial coordinate vectors. include: ; set up express and The Euclidean distance between them for With all SAR targets The maximum distance between them, i.e., the global maximum distance. : ; ; Then distance similarity for: ; set up for and The confidence similarity between them is calculated using the following formula: ; in and Infrared targets The category and confidence score, and For SAR target The category and confidence score. When the infrared target category and the SAR target category are the same, ,otherwise .

[0028] In this embodiment of the invention, the cosine similarity, distance similarity, and confidence similarity between each pair of targets are weighted and fused according to preset weights to obtain the comprehensive similarity of each pair of targets, including: ; in, Indicates the overall similarity, for They believed the two targets were too far apart in location, which directly led to... ; , This represents the weights used to determine the three similarity levels, satisfying the following constraints: ; The similarity matrix can be obtained by calculating the similarity between all targets in the two modes. .

[0029] In this embodiment of the invention, the spatial positional relationship, geometric distribution characteristics and detection confidence information of the targets are comprehensively utilized to construct a target matching similarity matrix that includes cosine similarity, distance similarity and confidence similarity. The comprehensive similarity between targets is obtained by weighted fusion, which effectively improves the stability and discrimination ability of matching.

[0030] In one implementation, during the cross-modal target similarity modeling stage, in addition to constructing a similarity matrix based on the spatial geometric positional relationship of the targets and the detection confidence, other attributes that can characterize the consistency of the targets can be introduced as alternatives or supplements, such as the target area ratio, target orientation information, relative topological relationship between targets, or similarity learning of target features through shallow neural networks or metric learning models, as long as a similarity metric or scoring matrix that can be formed for target matching can be formed in the end.

[0031] Step S103: The global optimization algorithm is used to solve the maximum similarity matching optimization problem based on the similarity matrix to obtain the cross-modal target matching set. The maximum similarity matching optimization problem takes maximizing the comprehensive similarity as the optimization objective and each SAR target and each infrared target participate in the matching at most once as the constraint.

[0032] In this embodiment of the invention, after obtaining the similarity matrix, the matching relationship between targets is globally optimized to determine the optimal matching relationship across modal targets. Let the matching relationship be represented as a binary variable. ,in express and Matched. The following maximum similarity matching optimization problem is solved by maximizing the overall similarity: ; in, A binary variable representing the relationship between infrared targets and SAR targets; Indicates overall similarity; Indicates the total number of SAR targets; Indicates the total number of infrared targets; Indicates the infrared target index; This represents the SAR target index.

[0033] The above constraints ensure that each SAR target and each infrared target participate in a matching process at most once, thereby achieving a one-to-one correspondence at the target level.

[0034] In this embodiment of the invention, a global optimization algorithm is used to solve the maximum similarity matching optimization problem based on the similarity matrix, resulting in a cross-modal target matching set, including: The similarity matrix is ​​converted into a cost matrix, and the Hungarian algorithm is used to solve the maximum similarity matching optimization problem to obtain the target matching matrix. Successfully matched infrared and SAR targets are extracted from the target matching matrix to form a cross-modal target matching set.

[0035] In one implementation, when the similarity matrix When the number of infrared targets differs from the number of SAR targets, the similarity matrix is ​​padded with zeros (introducing virtual targets) to form a square matrix, and then the cost matrix is ​​calculated based on this square matrix. The square matrix... for: ; In this embodiment of the invention, the cost matrix include: ; in, This represents the maximum similarity in the similarity matrix. The cost matrix represents the cost value. By transforming the similarity matrix into the cost matrix, the maximum optimization problem is converted into a minimum optimization problem. Based on the cost matrix, the Hungarian algorithm is used to solve the minimum optimization problem, including: ; Thus, the globally optimal target matching matrix is ​​obtained. According to the optimal solution Extract to satisfy Target pairs form a cross-modal target matching set. : ; This matching set serves as input for subsequent geometric transformation parameter estimation.

[0036] In this embodiment of the invention, by introducing virtual targets or zero-padding mechanisms in the target matching modeling stage, matching and registration can still be completed through a unified optimization framework even when the number of targets in SAR images and infrared images is inconsistent, thus enhancing the applicability of the method in practical application scenarios.

[0037] In one implementation, during the target matching solution stage, although the embodiment of the present invention preferably uses the Hungarian algorithm based on the maximum similarity criterion to solve the globally optimal one-to-one matching relationship, under the premise that the target matching is modeled as a global optimization problem, other global or near-global optimization algorithms can also be used as alternatives, such as linear programming, integer programming, maximum weight matching, minimum cost maximum flow or graph matching algorithms, etc. As long as the optimal or suboptimal matching relationship between targets can be obtained under the overall constraints, the registration purpose of the present invention can be achieved.

[0038] Step S104: Obtain a set of interior point matches that satisfy geometric consistency by filtering the cross-modal target matching set through interior points.

[0039] In this embodiment of the invention, obtaining a geometrically consistent set of interior point matches from the cross-modal target matching set through interior point filtering includes: Extract the geometric center coordinates of each pair of matched targets in the cross-modal target matching set; A random sampling consensus algorithm is used to filter the cross-modal target matching set based on the geometric center coordinates of each pair of matching targets, thus obtaining the interior point matching set.

[0040] In this embodiment of the invention, for ease of description, The matching target pairs in the data are renumbered as follows: ; in, This indicates the number of target pairs in the cross-modal target matching set.

[0041] Based on the aforementioned center point coordinate extraction method, the geometric center coordinates of each pair of matched targets in the cross-modal target matching set are extracted in their respective images: ; ; in, and These represent the homogeneous coordinate vectors of the target center point in the infrared image and the target center point in the SAR image under the same matching pair, respectively, which are used for subsequent homography transformation matrix estimation and reprojection error calculation; and These represent the horizontal and vertical coordinates of the target center point in the infrared image under the same matching pair; and These represent the x and y coordinates of the target center point in the SAR image under the same matching pair; superscript Indicates the transpose operation; To eliminate anomalous correspondences introduced by mismatched targets, a Random Sample Consensus (RANSAC) algorithm is used for robust screening of matching point pairs. The process includes: a) From Four sets of matching target center points are randomly selected as the minimum sample set. b) Based on the four selected sets of corresponding points, estimate the candidate homography transformation matrix between the infrared image and the SAR image. ; c) Calculate the reprojection error for all matching point pairs using the candidate single-transformation matrix; d) If the reprojection error of a certain matching point pair is less than a preset threshold If it is an interior point, then it is determined to be an interior point. e) Repeat the above process and select the candidate homography transformation matrix with the most interior points as the optimal homography model.

[0042] set up Indicates the first In the nth RANSAC iteration, the candidate homography transformation matrix estimated from the corresponding points of random sampling is then... For the matching point, the following conditions must be met: ; Among them, the symbol " "" indicates the proportional relationship in homogeneous coordinates. If we denote: ; in, , , Indicates the first In the nth RANSAC iteration, the th The homogeneous coordinates of the SAR matching points, after being mapped by the candidate homography transformation matrix, consist of three components; after normalization, their corresponding two-dimensional predicted coordinates are... Then the first The reprojection error of the matching points is defined as: ; If the reprojection error of a certain matching point pair satisfies: ; The matching point is then determined to be an interior point. Wherein, This indicates the preset error threshold.

[0043] Repeat the above process, selecting the candidate homography transformation matrix with the largest number of interior points as the optimal model. After RANSAC processing, a set of interior point matches that satisfy geometric consistency is obtained: ; in, This represents the set of interior point matches.

[0044] In one implementation, in the geometric transformation estimation and robustness enhancement stages, in addition to using the Randomized Consensus Algorithm (RANSAC) for interior point screening and homography matrix estimation, robust estimation methods such as M-estimator, LMDS (Least Mean Squared), and weighted least squares can also be used. Alternatively, affine transformation models and similarity transformation models can be used as alternative forms of geometric models to meet the registration requirements of different scenarios, provided that the application scenario is met.

[0045] Step S105: Based on the inlier matching set, the homography matrix from the SAR image to the infrared image is estimated using the least squares method, and the homography matrix is ​​used to perform geometric transformation on the SAR image to complete target registration in the cross-modal image.

[0046] Interior point matching set obtained from RANSAC Based on this, the least squares method is used to accurately estimate the homography matrix between cross-modal images, wherein the homography matrix to be estimated includes: ; in, , , , It mainly describes the linear transformation relationship of planar coordinates. , Indicates the translation component. , Describe the perspective projection effect. This is the scaling normalization parameter.

[0047] Based on the set of interior point matching pairs, the optimal homography matrix is ​​solved by minimizing the sum of squared reprojection errors of all interior point matching pairs: ; Solve this optimization problem to obtain the optimal homography matrix. And use this matrix to perform geometric transformations on the SAR image: ; This enables target-level alignment from SAR images to infrared image coordinates, completing target registration across modal images.

[0048] In this embodiment of the invention, instead of relying on the consistency of grayscale, texture, or local features between SAR and infrared images, the target object in the target detection result is used as the basic unit for registration. A cross-modal correspondence is established through target-level spatial relationships, fundamentally reducing the impact of modal differences on registration performance. By unifying the spatial coordinates of targets in SAR and infrared images, a comparable target spatial representation under cross-modal conditions is constructed, providing a unified geometric basis for subsequent target matching. A similarity matrix is ​​constructed through comprehensive similarity calculation operations, effectively improving the stability and discriminative ability of the matching.

[0049] This invention models the cross-modal target matching problem as a maximum similarity matching optimization problem and employs a global optimization algorithm to solve for the one-to-one correspondence between targets. This avoids mismatches caused by greedy matching or locally optimal matching, ensuring that the target matching result is optimal globally. Therefore, this invention maintains high registration accuracy and good stability even in complex scenarios and under conditions of strong modal differences, while also possessing strong versatility and engineering feasibility.

[0050] Furthermore, at the overall technical process level, the above steps can be executed sequentially, or parallel processing or iterative optimization mechanisms can be introduced in some steps. For example, joint optimization or multiple rounds of iterative updates can be performed between target matching and geometric transformation estimation. As long as cross-modal image registration based on target-level information is ultimately achieved, it should be regarded as a reasonable extension of the technical solution of this invention.

[0051] The simulation experiment of the global optimal similarity target registration method for SAR and infrared images provided by the embodiments of the present invention is as follows: First, see the two sets of infrared image detection results and SAR image detection results. Figure 2 , Figure 2 These are schematic diagrams of infrared image detection results and SAR image detection results. Figure 2 (a) in the image shows the infrared image detection results for scene one. Figure 2(b) in the image shows the SAR image detection results for scene one. Figure 2 (c) in the image represents the infrared image detection result for scene one. Figure 2 In the image (d), the SAR image detection result for scene two is shown. Here, "car" represents a car, "digger" represents an excavator, and "truck" represents a truck.

[0052] See the cross-modal image target matching results and similarity matrices for both scenarios. Figure 3 , Figure 3 These are schematic diagrams showing cross-modal image target matching results and similarity matrices in two scenarios. Figure 3 (a) in the image shows the spatial distribution of the target in the infrared image of Scene 1. Figure 3 In Figure (c), the spatial distribution of targets in the SAR image of Scene 2 is shown. Gray lines connecting two targets indicate that the similarity between them is greater than 0. The similarity between the targets connected by gray lines in (a) and (c) is calculated respectively, and the similarity matrices are shown in (e) and (f). Figure 3 In the matrix (e), the similarity matrix for scene one is... Figure 3 In the matrix (b), the similarity matrix for scenario two is shown. Figure 3 (b) in the figure represents the optimal matching result for scenario one obtained by solving the global optimality problem. Figure 3 In the diagram, (d) represents the optimal matching result for scenario two obtained by solving the global optimal problem, and the gray line connecting the two targets indicates that they are the same matching pair.

[0053] See Figure 4 , Figure 4 It is a stitched image of cross-modal target matching from two different scenarios. Figure 4 (a) in the image is a stitched image of cross-modal target matching in Scene 1. Figure 4 (b) in the figure is a cross-modal target matching stitching diagram of scenario 2, showing a stitching visualization diagram of target matching in infrared and SAR images of scenario 1 and scenario 2. Colored lines indicate that the two targets are the same matching pair. Figure 5 These are schematic diagrams of cross-modal image homography matrices in two scenarios. Figure 5 In the diagram, (a) represents the homography matrix of the cross-modal image in scene one. Figure 5 (b) in the figure is the homography matrix of the cross-modal image in scene 2. It shows the homography matrix obtained by geometric transformation estimation through matching results, which reflects the mapping relationship between any pixel of the SAR image and the infrared image. Figure 6 These are schematic diagrams of the registration results in two scenarios. Figure 6 (a) in the image shows the infrared image registration result for scene one. Figure 6 (b) in the image shows the infrared image registration result for scene two. Figure 6 (c) in the image shows the SAR image registration result for scene one. Figure 6(d) in the figure represents the SAR image registration result for scene 2.

[0054] Compared to existing image registration methods that primarily rely on grayscale or local feature information, this invention presents a SAR-infrared cross-modal image registration method based on target-level information. By using target detection results as the basic registration unit, it avoids dependence on the consistency of low-level grayscale distribution or local features between different modal images, fundamentally reducing the impact of differences in imaging mechanisms and radiometric characteristics between SAR and infrared images on registration performance. By mapping targets in both modal images to a unified spatial reference coordinate system and fusing the spatial positional relationships, geometric distribution characteristics, and detection confidence information of the targets to construct a cross-modal target similarity model, and then solving for the globally optimal target matching relationship based on the maximum similarity criterion, it effectively reduces the probability of mismatches caused by local matching instability in complex scenes, improving the reliability of target correspondences. Furthermore, by combining random consistency constraints to perform robust interior point screening on matched target pairs and estimating the geometric transformation relationship between images based on the interior point set, it further enhances the stability of the registration results under conditions of noise interference, occlusion, and complex target distribution. Meanwhile, since this invention performs similarity calculation and matching only on the target-level set, it significantly reduces computational complexity and improves registration efficiency compared to pixel-level or dense feature matching methods, making it more suitable for practical engineering applications. In summary, under complex scenes and strong modal differences, this invention can significantly improve the accuracy, robustness, and adaptability of SAR-infrared cross-modal image registration while ensuring computational efficiency.

[0055] Based on the same inventive concept, embodiments of the present invention also provide a global optimal similarity target registration device for SAR and infrared images, the device comprising: The application module is used to apply the trained infrared detection model to the infrared image to be registered to obtain an infrared target detection set, and to apply the trained SAR detection model to the SAR image to be registered to obtain a SAR target detection set. The calculation module is used to construct a similarity matrix by combining the infrared detection target set and the SAR target detection set into a spatial coordinate system and then performing a comprehensive similarity calculation operation. The solution module is used to solve the maximum similarity matching optimization problem constructed based on the similarity matrix using a global optimization algorithm, and to obtain a cross-modal target matching set; the maximum similarity matching optimization problem takes maximizing the comprehensive similarity as the optimization objective and is constrained to allow each SAR target and each infrared target to participate in at most one matching. The filtering module is used to filter the cross-modal target matching set to obtain an interior point matching set that satisfies geometric consistency. The transformation module is used to estimate the homography matrix from the SAR image to the infrared image using the least squares method based on the inlier matching set, and to perform geometric transformation on the SAR image using the homography matrix to complete target registration in the cross-modal image.

[0056] In this embodiment of the invention, instead of relying on the consistency of grayscale, texture, or local features between SAR and infrared images, the target object in the target detection result is used as the basic unit for registration. A cross-modal correspondence is established through target-level spatial relationships, fundamentally reducing the impact of modal differences on registration performance. By unifying the spatial coordinates of targets in SAR and infrared images, a comparable target spatial representation under cross-modal conditions is constructed, providing a unified geometric basis for subsequent target matching. A similarity matrix is ​​constructed through comprehensive similarity calculation operations, effectively improving the stability and discriminative ability of the matching.

[0057] This invention models the cross-modal target matching problem as a maximum similarity matching optimization problem and employs a global optimization algorithm to solve for the one-to-one correspondence between targets. This avoids mismatches caused by greedy matching or locally optimal matching, ensuring that the target matching result is optimal globally. Therefore, this invention maintains high registration accuracy and good stability even in complex scenarios and under conditions of strong modal differences, while also possessing strong versatility and engineering feasibility.

[0058] This invention also provides an electronic device, such as... Figure 7 As shown, it includes a processor 701, a communication interface 702, a memory 703, and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704. Memory 703 is used to store computer programs; When the processor 701 executes the program stored in the memory 703, it implements the method steps of any of the above-mentioned SAR and infrared image global optimal similarity target registration methods.

[0059] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0060] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0061] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0062] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0063] The present invention also provides a computer-readable storage medium. A computer program is stored in the computer-readable storage medium, and when executed by a processor, the computer program implements the method steps of any of the above-described SAR and infrared image global optimal similarity target registration methods.

[0064] Optionally, the computer-readable storage medium may be non-volatile memory (NVM), such as at least one disk storage device.

[0065] Optionally, the aforementioned computer-readable storage medium may also be at least one storage device located remotely from the aforementioned processor.

[0066] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the steps of the method described in any of the above-described SAR and infrared image global optimal similarity target registration methods.

[0067] It should be noted that the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention.

[0068] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0069] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings and the disclosure in carrying out the claimed invention. In the description of the invention, the word "comprising" does not exclude other components or steps, "a" or "an" does not exclude a plurality, and "a plurality" means two or more, unless otherwise explicitly specified. Furthermore, while different embodiments may describe certain measures, this does not mean that these measures cannot be combined to produce good results.

[0070] The method provided in this invention can be applied to electronic devices. Specifically, the electronic device can be a desktop computer, a portable computer, a smart mobile terminal, a server, etc. No limitation is made herein; any electronic device that can implement this invention falls within the protection scope of this invention.

[0071] For the embodiments of the device / electronic device / storage medium, since they are basically similar to the method embodiments, the description is relatively simple, and relevant parts can be referred to in the description of the method embodiments.

[0072] It should be noted that the device, electronic device and storage medium in the embodiments of the present invention are respectively the device, electronic device and storage medium for applying the above-mentioned global optimal similarity target registration method for SAR and infrared images. Therefore, all embodiments of the above-mentioned global optimal similarity target registration method for SAR and infrared images are applicable to the device, electronic device and storage medium, and can achieve the same or similar beneficial effects.

[0073] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A method for global optimal similarity target registration between SAR and infrared images, characterized in that, The method includes: The trained infrared detection model is applied to the infrared image to be registered to obtain the infrared target detection set, and the trained SAR detection model is applied to the SAR image to be registered to obtain the SAR target detection set. After aligning the infrared target detection set and the SAR target detection set to a single spatial coordinate system, a similarity matrix is ​​constructed through a comprehensive similarity calculation operation; A global optimization algorithm is used to solve the maximum similarity matching optimization problem constructed based on the similarity matrix, resulting in a cross-modal target matching set. The maximum similarity matching optimization problem aims to maximize the overall similarity, with the constraint that each SAR target and each infrared target participate in at most one matching. The cross-modal target matching set is filtered by interior points to obtain an interior point matching set that satisfies geometric consistency; Based on the inlier matching set, the homography matrix from the SAR image to the infrared image is estimated using the least squares method, and the SAR image is geometrically transformed using the homography matrix to complete target registration in the cross-modal image.

2. The method according to claim 1, characterized in that, After aligning the infrared target detection set and the SAR target detection set to a single spatial coordinate system, a similarity matrix is ​​constructed through a comprehensive similarity calculation operation, including: After converting the infrared target detection set and the SAR target detection set into a spatial coordinate system, for each pair of infrared and SAR targets in the spatial coordinate system, the cosine similarity, distance similarity, and confidence similarity are calculated. The cosine similarity is calculated based on the spatial coordinate vectors of the two targets, the distance similarity is calculated based on the Euclidean distance between the two targets and its ratio to the global maximum distance, and the confidence similarity is calculated based on the class consistency and confidence score of the two targets. The cosine similarity, distance similarity, and confidence similarity between each pair of targets are weighted and fused according to preset weights to obtain the comprehensive similarity of each pair of targets and construct a similarity matrix.

3. The method according to claim 1, characterized in that, A global optimization algorithm is used to solve the maximum similarity matching optimization problem constructed based on the similarity matrix, resulting in a cross-modal target matching set, including: The similarity matrix is ​​converted into a cost matrix, and the maximum similarity matching optimization problem is solved using the Hungarian algorithm to obtain the target matching matrix; Successfully matched infrared and SAR targets are extracted from the target matching matrix to form a cross-modal target matching set.

4. The method according to claim 1, characterized in that, The cross-modal target matching set is filtered by interior points to obtain an interior point matching set that satisfies geometric consistency, including: Extract the geometric center coordinates of each pair of matched targets in the cross-modal target matching set; The random sampling consensus algorithm is used to filter the cross-modal target matching set based on the geometric center coordinates of each pair of matching targets to obtain the interior point matching set.

5. The method according to claim 1, characterized in that, The maximum similarity matching optimization problem includes: ； in, A binary variable representing the relationship between infrared targets and SAR targets; Indicates overall similarity; Indicates the total number of SAR targets; Indicates the total number of infrared targets; Indicates the infrared target index; This represents the SAR target index.

6. The method according to claim 1, characterized in that, The training methods for the infrared detection model include: Obtain a sample infrared image dataset; the sample infrared image dataset includes multiple sample infrared images and the first target ground truth bounding box, the first target ground truth confidence score, and the first target category label for each sample infrared image; The sample infrared image dataset is input into the initial infrared detection model to obtain a sample infrared target detection set; the sample infrared target detection set includes the first target prediction bounding box, the first target prediction confidence, and the first target prediction category corresponding to each sample infrared image; Based on the differences between the true bounding box and the predicted bounding box of the first target in each sample infrared image, the differences between the true confidence and the predicted confidence of the first target, and the differences between the first target category label and the predicted category of the first target, the first loss function is used for iterative training until the first loss function converges or the number of iterations reaches the preset number, and the trained infrared detection model is obtained. The training methods for the SAR detection model include: Obtain a sample SAR image dataset; the sample SAR image dataset includes multiple sample SAR images and the second target ground truth bounding box, second target ground truth confidence, and second target category label for each sample SAR image; The sample SAR image dataset is input into the initial SAR detection model to obtain a sample SAR target detection set; the sample SAR target detection set includes the second target prediction bounding box, the second target prediction confidence, and the second target prediction category corresponding to each sample SAR image; Based on the differences between the true bounding box and the predicted bounding box of the second target in each sample SAR image, the differences between the true confidence and the predicted confidence of the second target, and the differences between the second target category label and the predicted category of the second target, the second loss function is used for iterative training until the second loss function converges or the number of iterations reaches the preset number, thus obtaining the trained SAR detection model.

7. A global optimal similarity target registration device for SAR and infrared images, characterized in that, The device includes: The application module is used to apply the trained infrared detection model to the infrared image to be registered to obtain an infrared target detection set, and to apply the trained SAR detection model to the SAR image to be registered to obtain a SAR target detection set. The calculation module is used to construct a similarity matrix by combining the infrared detection target set and the SAR target detection set into a spatial coordinate system and then performing a comprehensive similarity calculation operation. The solution module is used to solve the maximum similarity matching optimization problem constructed based on the similarity matrix using a global optimization algorithm, and to obtain a cross-modal target matching set; the maximum similarity matching optimization problem takes maximizing the comprehensive similarity as the optimization objective and is constrained to allow each SAR target and each infrared target to participate in at most one matching. The filtering module is used to filter the cross-modal target matching set to obtain an interior point matching set that satisfies geometric consistency. The transformation module is used to estimate the homography matrix from the SAR image to the infrared image using the least squares method based on the inlier matching set, and to perform geometric transformation on the SAR image using the homography matrix to complete target registration in the cross-modal image.

8. The apparatus according to claim 7, characterized in that, The calculation module is specifically used to systematize the infrared detection target set and the SAR target detection set into a spatial coordinate system, and then calculate the cosine similarity, distance similarity, and confidence similarity between each pair of infrared and SAR targets in the spatial coordinate system. The cosine similarity is calculated based on the spatial coordinate vectors of the two targets, the distance similarity is calculated based on the Euclidean distance between the two targets and its ratio to the global maximum distance, and the confidence similarity is calculated based on the class consistency and confidence score of the two targets. The cosine similarity, distance similarity, and confidence similarity between each pair of targets are then weighted and fused according to preset weights to obtain the comprehensive similarity of each pair of targets, thereby constructing a similarity matrix.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a computer program stored in memory, implements the SAR and infrared image global optimal similarity target registration method according to any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the SAR and infrared image global optimal similarity target registration method according to any one of claims 1-6.

Images