Dual-modality image registration method based on deep learning and GPU parallel optimization

Abstract

This invention discloses a dual-modal image registration method based on deep learning and GPU parallel optimization, belonging to the field of computer vision and image processing. Binary edge maps are extracted from infrared and visible light images respectively, and distance transform maps are calculated. At multiple scales, the infrared binary edge maps are used as templates, and sliding coarse registration is performed on the distance transform maps to obtain multiple initial affine transformation matrices. For each initial affine transformation matrix, iterative fine registration is performed, including block translation matching, outlier block removal, and weighted least squares fitting, to obtain corresponding candidate fine registration matrices, and a final affine transformation matrix is ​​selected. Based on the continuity between the final affine transformation matrices of adjacent frames in the video sequence, temporal consistency optimization processing is performed on the registration results. This invention improves the accuracy, robustness, and computational efficiency of dual-modal image registration, and is suitable for automatic registration scenarios of infrared and visible light images, such as security monitoring and remote sensing fusion.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing, specifically relating to a dual-modal image registration method based on deep learning and GPU parallel optimization. Background Technology

[0002] Dual-modal image registration refers to the geometric alignment of images of the same scene acquired in different imaging bands (such as infrared and visible light) to ensure that corresponding physical locations in the two images precisely overlap at the pixel level. This technology has wide applications in fields such as security monitoring, remote sensing fusion, and autonomous driving. However, due to the differences in imaging mechanisms between infrared and visible light images, directly matching image grayscale values ​​is extremely difficult. Existing technologies for dual-modal image registration still have the following shortcomings:

[0003] 1. Difficulty in cross-modal feature matching:

[0004] Infrared images reflect the thermal radiation characteristics of a target, while visible light images reflect its reflected light characteristics. The two differ significantly in grayscale distribution and texture features. Traditional registration methods based on grayscale similarity (such as cross-correlation) experience a significant drop in matching accuracy in cross-modal scenes. Feature-point-based methods (such as SIFT and SURF) extract a limited number of feature points from infrared images with low repeatability, resulting in a high registration failure rate.

[0005] 2. Scale-based search is inefficient:

[0006] When the field of view of infrared and visible light cameras differs, the registration process requires simultaneous estimation of scaling and translation parameters. Existing multi-scale search methods typically employ pixel-by-pixel exhaustive search or pyramid methods, which incur significant computational overhead and are prone to getting trapped in local optima.

[0007] 3. Insufficient robustness of fine registration:

[0008] Fine-grained registration after coarse localization typically employs gradient descent or direct search methods, which are susceptible to noise in scenarios with sparse edges or partial occlusion, leading to increased registration errors. Existing methods lack effective outlier removal and adaptive weighting mechanisms.

[0009] 4. Insufficient parallelism in batch processing:

[0010] In video frame sequence registration scenarios, a large number of image pairs need to be registered. Most existing methods process each pair sequentially, failing to fully utilize the parallel computing power of GPUs and resulting in low processing efficiency.

[0011] 5. Lack of utilization of timing consistency:

[0012] The registration results of adjacent frames in a video sequence have temporal continuity, but existing methods usually process each frame independently, failing to utilize temporal information to detect and correct abnormal registration results.

[0013] Therefore, current technology urgently needs a bimodal image registration method that can overcome cross-modal feature differences, efficiently search for optimal registration parameters, possess robust fine registration capabilities, and have high parallel computing efficiency. Summary of the Invention

[0014] The purpose of this invention is to overcome the problems in the prior art, such as difficulty in cross-modal feature matching, low efficiency of scale search, insufficient robustness of fine registration, insufficient parallelism of batch processing, and failure to consider temporal consistency. It proposes a dual-modal image registration method based on deep learning and GPU parallel optimization.

[0015] The technical solution adopted in this invention is as follows:

[0016] In a first aspect, this invention proposes a dual-modal image registration method based on deep learning and GPU parallel optimization, comprising the following steps:

[0017] S1. For each frame in the video sequence, extract the corresponding infrared binary edge map and visible light binary edge map respectively.

[0018] S2. Calculate the range transformation map based on the visible light binary edge map. Use the infrared binary edge map as a template at multiple scales and perform sliding coarse registration on the range transformation map to obtain multiple initial affine transformation matrices.

[0019] S3. Perform iterative fine registration on each initial affine transformation matrix to obtain the corresponding candidate fine registration matrix;

[0020] The fine registration includes: dividing the infrared binary edge map transformed based on the current affine transformation matrix into multiple blocks; calculating the local offset of each block on the visible light binary edge map and the matching weight of each block; removing blocks with abnormal offsets; fitting an affine correction matrix based on the offsets and weights of the remaining blocks and updating the current affine transformation matrix; after the iteration is completed, selecting the optimal matrix from all candidate fine registration matrices based on the local edge overlap quality score as the final affine transformation matrix from the infrared image to the visible light image;

[0021] S4. Based on the continuity between the final affine transformation matrices of adjacent frames in the video sequence, perform temporal consistency optimization processing on the registration results, including marking abnormal frames and correcting the final affine transformation matrix of low-quality frames.

[0022] Furthermore, a common downsampling step is included before calculating the distance transformation graph:

[0023] A common downsampling factor is calculated based on the maximum edge length of the visible light binary edge map and the preset maximum edge length for searching. The infrared binary edge map and the visible light binary edge map are downsampled according to the common downsampling factor to establish a coarse search coordinate system in the coarse matching stage.

[0024] The initial affine transformation matrix is ​​calculated in the coarse search coordinate system and then converted back to the original image resolution coordinate system.

[0025] Furthermore, a distance transformation map is obtained based on the downsampled infrared binary edge map and the downsampled and inverted visible light binary edge map, and Gaussian smoothing is performed; each pixel value in the distance transformation map represents the distance from the pixel position to the nearest visible light edge pixel.

[0026] Furthermore, in S2, the sliding coarse registration includes:

[0027] The fitting scale is calculated based on the size ratio of the infrared binary edge map and the visible light binary edge map. The infrared template scale search range is determined based on the fitting scale. Multiple scale values ​​are sampled within the search range as the infrared template scale.

[0028] For each infrared template scale, the downsampled infrared binary edge map is further scaled to an infrared template, and the infrared template is slid-matched on the distance transformation map to calculate the matching score;

[0029] All infrared template scales are sorted in ascending order of matching score. Several of the top infrared template scales are selected as candidates to obtain the scaling-translation matrix at each candidate scale. The scaling-translation matrix is ​​then transformed back to the original resolution coordinate system based on the common downsampling factor to obtain the initial affine transformation matrix from the infrared image coordinate system to the visible light image coordinate system at each candidate scale.

[0030] Furthermore, the calculation process for the matching score is as follows:

[0031] The template matching score is obtained by multiplying and summing the pixel values ​​of the infrared template position in the distance transformation map and the pixel values ​​of the infrared template itself, and then dividing by the total number of edge pixels in the infrared template.

[0032] Furthermore, in S3, the local offset is achieved through block template matching, including:

[0033] Transform the infrared binary edge image to the visible light coordinate system using the current affine transformation matrix, calculate the bounding box of the transformed infrared binary edge image, and expand it with a preset fill value.

[0034] The expanded region is uniformly divided into 3×3 block grids. For each block, the block region in the transformed infrared binary edge map is used as the template, and the distance transformation map corresponding to the block region in the visible light binary edge map is used as the search region. The normalized cross-correlation response map is calculated based on the normalized template and the search region.

[0035] The position of the maximum value in the normalized cross-correlation response graph is selected, and the difference between this position and the original center coordinate of the block is used as the local offset.

[0036] Furthermore, in S3, the calculation process of the matching weight of each block includes:

[0037] For each block, extract the corresponding visible light edge region in the visible light binary edge map based on the position corresponding to the local offset;

[0038] Calculate the intersection-union ratio (IUU) between the template region where the segment is located and the visible light edge region, and use the square of the IUU as the matching weight of the segment.

[0039] Furthermore, in S3, the process of updating the current affine transformation matrix includes:

[0040] The RANSAC algorithm is used to analyze the local offsets of multiple blocks. Blocks with abnormal offsets are identified as outliers and removed, while the remaining blocks are identified as inliers.

[0041] A design matrix is ​​constructed using the source point coordinates of each internal point block and the target point coordinates obtained by adding the source point coordinates with their local offsets.

[0042] Multiply each row of the design matrix by the square root of the matching weight of the corresponding block to obtain the weighted design matrix;

[0043] Based on the weighted design matrix and the target point coordinates, the affine correction matrix is ​​calculated by solving the weighted least squares problem.

[0044] Concatenate the affine correction matrix with the current affine transformation matrix to update the current affine transformation matrix.

[0045] Furthermore, the calculation process for the edge overlap quality fraction includes:

[0046] The infrared binary edge map is transformed to the visible light image coordinate system using matrix transformation;

[0047] Define a local window that covers the transformed infrared edge bounding box and is expanded by a preset pixel value;

[0048] Within the local window, the cross-over ratio (CUP) of the transformed infrared edge and the visible light edge is calculated as the local edge overlap quality score corresponding to the matrix; wherein, the CUP is the ratio of the number of intersections to the number of unions of non-zero pixels in the two edge images.

[0049] Furthermore, the fine registration process in S3 is executed using GPU batch parallel acceleration technology. Based on the GPU's currently available video memory, the estimated video memory required for a single image registration, and the system's reserved video memory, the number of image pairs that can be processed concurrently in the same batch is dynamically calculated.

[0050] Furthermore, S4 includes:

[0051] For each frame, the final affine transformation matrix is ​​decomposed into linear components and translation components. The deviations between the linear components and translation components and the median reference of the corresponding components in each frame in the temporal neighborhood are calculated. The anomaly degree is calculated based on the deviations. If the anomaly degree is greater than or equal to a preset anomaly threshold, it is marked as an abnormal frame.

[0052] Calculate the median of the local edge overlap quality score of all frames in the video sequence and set a dynamic quality threshold; for low-quality frames whose local edge overlap quality score is lower than the dynamic quality threshold, use the final affine transformation matrix of their neighboring high-quality frames whose local edge overlap quality score is higher than the dynamic quality threshold to generate a replacement matrix and correct the final affine transformation matrix of the low-quality frames.

[0053] Secondly, the present invention proposes a computer program product, including a computer program / instruction, which, when executed by a processor, can implement the dual-modal image registration method based on deep learning and GPU parallel optimization as described above.

[0054] The beneficial effects of this invention are:

[0055] This invention transforms the cross-modal grayscale matching problem into modality-independent geometric edge matching by extracting edge features, effectively overcoming the feature differences between infrared and visible light images caused by their different imaging mechanisms. It employs a multi-scale coarse localization strategy based on distance transform, generating and retaining multiple initial transform matrix candidates, significantly improving the efficiency of the global search and the ability to avoid getting trapped in local optima. In the fine registration stage, block-based translation matching, dynamic weight calculation based on matching quality, and robust fitting are introduced. The combination of these three approaches greatly enhances the local alignment accuracy and robustness of the algorithm in scenarios with sparse edges, occlusion, or noise. Finally, through temporal consistency post-processing, abnormal frames are automatically detected and low-quality frames are corrected, ensuring the overall smoothness, stability, and temporal consistency of the video stream registration results. Attached Figure Description

[0056] Figure 1 This is an overall flowchart of a dual-modal image registration method based on deep learning and GPU parallel optimization;

[0057] Figure 2 This is a schematic diagram of multi-scale coarse localization based on distance transformation;

[0058] Figure 3 This is a schematic diagram of fine registration based on block translation;

[0059] Figure 4 This is a visual representation of the block offset;

[0060] Figure 5 These are schematic diagrams of the dual-modal images before and after registration, where (a) and (b) are the original infrared image and the original visible light image, respectively; (c) and (d) are the registered infrared image and the registered visible light image, respectively; and (e) and (f) are the edge overlay images of the infrared image and the visible light image and the final overlay image of the registered image, respectively.

[0061] Figure 6 This is a framework diagram of a dual-modal image registration system based on deep learning and GPU parallel optimization;

[0062] Figure 7 This is a schematic diagram of the components of a computer electronic device. Detailed Implementation

[0063] The present invention will be further described and illustrated below with reference to specific embodiments. The embodiments described are merely examples of the content of this disclosure and do not limit the scope of the invention. The technical features of each embodiment in the present invention can be combined accordingly, provided that there is no mutual conflict.

[0064] The accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0065] The flowchart shown in the attached diagram is merely an illustrative example and does not necessarily include all steps. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0066] The proposed dual-modal image registration method based on deep learning and GPU parallel optimization has the following main implementation process: First, edge feature maps are extracted and binarized from infrared and visible light images using a deep learning edge detection model. Then, multiple initial affine transformation matrix candidates are obtained through multi-scale coarse localization based on distance transform. Next, 3×3 block translation fine registration is performed on each candidate, and a final affine transformation matrix is ​​selected based on the local edge overlap quality score. Fine registration of multiple image pairs is accelerated on the GPU through CUDA streaming and batch computing. Subsequently, abnormal frames are marked and low-quality frames are corrected using temporal consistency. Finally, the registered infrared and visible light images are cropped and fused according to the distortion-free effective region mask.

[0067] like Figure 1 As shown, the main steps include:

[0068] (1) Edge feature extraction

[0069] This invention utilizes a neural network edge detection model (NBED) based on an encoder-decoder architecture to extract edge features from infrared and visible light images, respectively. The encoder employs a DUL-M36 multi-scale dilated convolutional architecture, capturing multi-scale edge features through convolutional layers with different dilation rates; the decoder employs a UNETP progressive upsampling structure, fusing features from each layer of the encoder through skip connections to output an edge probability map with the same resolution as the input image.

[0070] In one specific embodiment of the present invention, the input image undergoes ImageNet standard normalization preprocessing (mean [0.485, 0.456, 0.406], standard deviation [0.229, 0.224, 0.225]), and the edge probability values ​​output by the model are truncated to the range of [0, 1] and saved as a grayscale edge map. The output edge map is then binarized, with pixels having a value greater than or equal to 128 set to 255 (edge), and the rest set to 0 (background), resulting in a binary edge map.

[0071] Let the size of the infrared binary edge map be... The visible light binary edge map size is To improve search efficiency, a common downsampling factor is calculated. :

[0072]

[0073] Where L is the parameter for searching the maximum side length. The downsampling factor used by both the infrared binary edge map and the visible light binary edge map is applied uniformly to both binary edge maps; when At that time, both binary edge maps were processed according to... Downsampling to the coarse search coordinate system; when hour, This indicates that public downsampling will not be performed.

[0074] Public downsampling It applies to both infrared binary edge maps and visible light binary edge maps.

[0075] (2) Multi-scale coarse localization based on distance transformation

[0076] like Figure 2 As shown, this invention achieves coarse localization of infrared binary edge maps within visible light binary edge maps through distance transformation and multi-scale template matching. In this step, the infrared binary edge map is used as a local image, and the visible light binary edge map is used as a global image.

[0077] The infrared binary edge map is processed as described above. Downsampling, and the visible light binary edge map is processed as described above. After downsampling and inversion, the distance transformation map can be calculated based on the two. After performing 5×5 Gaussian smoothing, the final distance transformation map is obtained, which is denoted as D. Each pixel value in D represents the distance from that pixel to the nearest visible light edge pixel, which is used for calculations later.

[0078] Calculate the adaptation scale of an infrared image placed into a visible light image:

[0079]

[0080] Determine the infrared template scale search range based on the adaptation scale:

[0081]

[0082]

[0083] 36 scale values ​​were uniformly sampled within the above range. .in, This represents the scaling scale of the infrared binary edge map when used as a template in the coarse search coordinate system; when no additional template scaling is required, the corresponding scale is denoted as 1. Infrared Template Scale It only applies to the already downsampled infrared binary edge map. For each scale... The downsampled infrared binary edge map is then processed according to... Scale to template If the template size exceeds the size of the downsampled visible light image, or the template width and height are too small, or the number of pixels at the template edge is less than a preset threshold, then that scale is skipped.

[0084] Template on the distance transformation graph Perform template matching and select the position with the minimum score as the optimal position at that scale. The template matching score is defined as:

[0085]

[0086] in, template The number of edge pixels in the middle. template The value at pixel (x, y). The optimal position of the top left corner of the template in the coarse search coordinate system is determined by the point where the template matching score is minimized. The smaller the value, the closer the edge of the infrared template is to the edge of the visible light.

[0087] All effective scale candidates are sorted by Sort the candidates from smallest to largest and select the top 3 candidates. Then, insert finer scale values ​​near these 3 candidates to continue the evaluation. Reorder the valid candidates obtained from the coarse search and supplementary search and select the top N candidates with the smallest scores.

[0088] For each candidate Construct a scaling-translation matrix in the coarse search coordinate system:

[0089]

[0090] Transform the matrix in the coarse search coordinate system back to the original image resolution coordinate system to obtain the initial affine transformation matrix from the infrared image coordinate system to the visible light image coordinate system:

[0091]

[0092] in, The right side This means first transforming the original infrared coordinates to the coarse search coordinate system, on the left side. This means restoring the visible light coordinates in the coarse search coordinate system to the original visible light coordinate system.

[0093] The above steps yield several candidate initial affine transformation matrices. These candidates serve as initial values ​​for subsequent fine registration, optimization, and selection.

[0094] (3) Fine registration based on block translation

[0095] like Figure 3 As shown, this invention uses a 3×3 block translation iteration algorithm to perform fine registration on the coarse positioning results, with a maximum number of iterations set to 3.

[0096] In each iteration, the original infrared binary edge map at the original resolution is first transformed to the visible light coordinate system using the current affine transformation matrix. The bounding box of non-zero pixels after transformation is calculated, and it is expanded outward by 50 pixels as search fill. The expanded region is then uniformly divided into 9 blocks of 3×3, and the center coordinates of each block are calculated.

[0097] For each block, the pixels within that block's region of the transformed infrared binary edge image are used as templates, and the distance transformation map of the corresponding region in the visible light binary edge image is used as the search space. If the number of non-zero pixels in the template is less than a preset threshold, the block is skipped. The optimal offset for each block is obtained through template matching. .

[0098] In this embodiment, the optimal offset Obtained through block template matching, this process is an internal step of fine registration. When implemented on a GPU, only the computation scheduling method is changed, without altering the geometric meaning of block matching, RANSAC elimination, and weighted fitting.

[0099] During block template matching, the GPU generates a transformed infrared binary edge map based on the current affine transformation matrix, calculates the bounding box of the transformed infrared binary edge map, divides the region after expanding outward by 50 pixels into 3×3 blocks, and calculates the matching response in the corresponding search region of the visible light binary edge map at the original resolution. The block offset is obtained by taking the position of the optimal response. The visualization results are as follows Figure 4 As shown.

[0100] Specifically, regarding the segmented template and search area First, the template Perform zero-mean normalization, then use the template The search region is calculated using convolution kernels of the same size. moving average and sliding variance Calculate the response graph:

[0101]

[0102] in, It is the template after zero-mean normalization. It's a template. standard deviation It is the pixel value of the search area. It's a template. Size, This is to avoid the minimum value where the denominator is 0. Take... The location of the maximum value is taken as the optimal response location. The difference between the optimal response location and the original center coordinates of the block is the local offset of the block. The local offsets of multiple blocks together constitute the data source for fitting the affine correction matrix.

[0103] Calculate the dynamic weight of each block: Extract the corresponding region of the visible light binary edge map at the matched position, and calculate the intersection-union ratio (IUU) of the template and the edge of that region. :

[0104]

[0105] in, Indicates the number of non-zero pixels. and These represent the intersection and union of pixels, respectively. The square of the value is used as the weight of the block:

[0106]

[0107] Squared weights give higher weights to high-quality matching blocks, effectively suppressing interference from sparse or partially occluded edges.

[0108] The source coordinates, target coordinates, and weights of the nine blocks are fed into the RANSAC algorithm to estimate the affine transformation matrix. The RANSAC algorithm is then used to automatically remove blocks with abnormal offsets, and weighted least squares fitting is performed on the remaining interior points.

[0109] In one specific embodiment of the present invention, the weighted least squares fitting process is as follows:

[0110] Let the source coordinates of the k-th interior point block be... The target coordinates are The weight is First, construct the design matrix based on the target coordinates. :

[0111]

[0112] Where n is the number of blocks remaining after removing blocks with abnormal offsets, and in this embodiment n≤9; A is the least squares fitting matrix;

[0113] Will Each row is multiplied by the square root of its corresponding weight. The weighted design matrix is ​​obtained. Press each inner point After weighting, the least squares solution is performed to obtain the affine correction matrix H:

[0114]

[0115]

[0116]

[0117] Concatenate the affine correction matrix H with the current matrix M to obtain a new matrix. :

[0118]

[0119] Perform a convergence check on the old and new matrices. Let the set of the four corner points in the visible light coordinate system be... Calculate its value in the old matrix M and the new matrix M. Maximum displacement of the lower projected coordinates :

[0120]

[0121] like If the pixel size is less than 0.5 pixels, stop iterating.

[0122] The aforementioned fine registration process is performed on each initial affine transformation matrix candidate, resulting in multiple candidate fine registration matrices. The infrared binary edge image is then transformed to the visible light coordinate system using each candidate fine registration matrix. Within a local window extending 50 pixels outward from the transformed infrared edge bounding box, the overlap ratio between the infrared and visible light edges is calculated, yielding a local edge overlap quality score. The candidate fine registration matrix with the highest quality score is selected as the unique final affine transformation matrix for the current image pair.

[0123] Specifically, the calculation process for the local edge overlap quality fraction is as follows:

[0124] The infrared binary edge image is transformed to the visible light coordinate system using a candidate fine registration matrix. A local window is then expanded by 50 pixels outside the transformed infrared edge bounding box. The overlap ratio between the infrared edge and the visible light edge is calculated to obtain the local edge overlap quality fraction. :

[0125]

[0126] in, It is a local window The set of edge pixels within.

[0127] This score evaluates edge overlap only near the actual projection area of ​​the infrared image, avoiding interference from background edges far from the registration area on candidate selection and timing correction.

[0128] This invention utilizes GPU batch parallelism to accelerate fine registration processing of multiple image pairs. GPU acceleration does not change the mathematical process of coarse localization and fine registration, but organizes multiple image pairs, multiple candidate matrices, and multiple block search tasks into parallel computational tasks to reduce frame-by-frame serial processing time; when the GPU is unavailable or video memory is insufficient, it can fall back to the CPU path to execute the same fine registration logic.

[0129] In one specific embodiment of the present invention, the dynamic concurrency number is automatically calculated based on the available GPU memory, image size, and reserved memory. This concurrency number is used to determine the number of image pairs that can simultaneously enter the fine registration stage within the same batch.

[0130] First, calculate the estimated video memory required for a single image pair during the fine registration stage. The calculation formula is as follows:

[0131]

[0132] in, It is the estimated video memory required to process a pair of H×W images during the GPU fine registration stage, in GB.

[0133] Memory is reserved for the framework and intermediate tensors, and constrained by the upper limit of the number of CUDA streams to avoid insufficient memory during batch image transformation, block search, and candidate evaluation stages. The formula for calculating the concurrency is as follows:

[0134]

[0135] in, It is a dynamic concurrency level. It rounds down. This refers to the GPU's currently available video memory. It is the GPU memory reserved for deep learning frameworks, CUDA contexts, and intermediate tensors.

[0136] In one specific embodiment of this invention, when implementing fine registration on a GPU, the serial computation process is restructured into batch parallel computation without changing the mathematical essence of the algorithm. The specific process is as follows:

[0137] Perform batch image transformation. For a batch of affine transformation matrices, the GPU generates a sampling grid in the target visible light coordinate system, and reads the corresponding infrared image pixels at once according to the inverse of each matrix. The batch transformation is completed by normalizing the coordinates, and the transformed infrared edge maps under all candidate matrices are obtained at the same time.

[0138] Parallel block template matching is performed. The GPU utilizes the batch transformation results to perform 3×3 block partitioning and search in parallel on the edge map under each candidate matrix. Normalized cross-correlation responses are calculated in batches within the distance transformation map region corresponding to each block, and the local offsets of all blocks are output efficiently.

[0139] The process involves fitting and selection. Offsets obtained in batches from the GPU are used as input for subsequent RANSAC anomaly removal and weighted least-squares fitting processes to fit candidate registration matrices. The optimal candidate is selected by calculating and comparing the local edge overlap quality scores of each matrix. GPU parallel acceleration technology significantly reduces the time required for frame-by-frame, candidate-by-candidate serial processing through task reorganization.

[0140] (4) Frame processing based on timing consistency

[0141] This invention utilizes the temporal consistency of video frame sequences to post-process the registration results, including abnormal frame detection and low-quality frame correction. Abnormal frames describe abrupt changes in the matrix relative to its temporal neighborhood, while low-quality frames describe insufficient quality scores for local edge overlap. These two types of frames have different definitions and are processed differently.

[0142] Abnormal frame detection: Extract the matrix components of the affine transformation matrix of the i-th frame and separate them into linear and translational components. Within the temporal radius, take the element-wise median of the linear components of neighboring frames. Element-wise median of translation components As a benchmark, the formula is expressed as follows:

[0143]

[0144]

[0145] in, It is the set of frames within the temporal radius of the i-th frame. yes The linear components of the affine transformation matrix in the j-th frame, yes The translation component of the affine transformation matrix of the j-th frame.

[0146] Calculate the deviation between the current frame matrix components and the neighborhood reference, and then calculate the anomaly degree:

[0147]

[0148]

[0149]

[0150]

[0151]

[0152] in, These are the deviations of the linear and translational components relative to the temporal neighborhood reference, respectively. These are the relative deviations of the linear and translational components, respectively, and Q is the matrix anomaly degree. The larger Q is, the greater the deviation of the current frame's affine matrix from the temporal neighborhood reference. If Q is greater than or equal to a preset anomaly threshold, the frame is marked as an anomalous frame. Anomalous frames are handled by outputting an anomaly flag to indicate that the registration result of that frame has a temporal stability risk; the anomaly flag itself does not directly replace the frame matrix.

[0153] Low-quality frame correction: Low-quality frames are those whose local edge overlap quality score is lower than the good frame threshold. First, the local edge overlap quality score of all frames is calculated. The median of the local edge overlap quality scores of all frames is multiplied by a quality coefficient to obtain the good frame threshold. A minimum threshold is also set, and a dynamic threshold is calculated. :

[0154]

[0155] in, Let be the local edge overlap quality score of the i-th frame. If it reaches... If the number of good frames is insufficient, skip the timing correction and retain the original matrices of each frame; otherwise, for Below For low-quality frames, take the final affine transformation matrix of the K nearest good frames, and take the median of each corresponding element of the matrix to obtain the replacement matrix.

[0156] The focus of processing the above-mentioned abnormal frames is to mark the risk of temporal mutations, while the focus of processing low-quality frames is to correct the low-quality results based on the good frame matrix.

[0157] (5) Output of results

[0158] After completing the above steps, output the final affine transformation matrix, local edge overlap quality score, and abnormal frame marker for the current image pair.

[0159] The infrared image is transformed to the visible light coordinate system based on the final affine transformation matrix. Then, the registered infrared and visible light images are cropped and fused according to the distortion-reduced effective region mask. The distortion-reduced effective region mask refers to the effective display area on the visible light image corresponding to the infrared image after the final affine transformation matrix. It is obtained by mapping the boundary regions of the infrared image to the visible light image coordinate system according to the final affine transformation matrix.

[0160] Figure 5The image shows the registration effect of infrared and visible light images. (a) and (b) are the original infrared and visible light images, respectively; (c) and (d) are the registered infrared and visible light images, respectively; and (e) and (f) are the edge overlay images of the infrared and visible light images and the final overlay image of the registered image, respectively. It can be seen that the original infrared and visible light images differ significantly in field of view, scale, and location, making direct pixel-level comparison impossible. After registration, the infrared and visible light images are mapped to a unified common area, maintaining consistency in the main scene structure. The edge overlay results show that the contours of roads, buildings, tree canopies, and waterfronts have good overlap, indicating good geometric alignment. The final overlay image shows that the two modalities are effectively fused within the same spatial range, and the visible light texture information corresponds to the infrared thermal response information, providing a reliable foundation for subsequent bimodal analysis and dataset construction.

[0161] It should also be noted that the dual-modal image registration method based on deep learning and GPU parallel optimization in the above embodiments can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a dual-modal image registration system based on deep learning and GPU parallel optimization, corresponding to the dual-modal image registration method based on deep learning and GPU parallel optimization provided in the above embodiments, such as... Figure 6 As shown, it includes:

[0162] The edge detection module is used to extract the corresponding infrared binary edge map and visible light binary edge map for each pair of infrared and visible light images corresponding to each frame in the video sequence.

[0163] The coarse localization module is used to calculate the range transformation map based on the visible light binary edge map. It uses the infrared binary edge map as a template at multiple scales and performs sliding coarse registration on the range transformation map to obtain multiple initial affine transformation matrices.

[0164] The fine registration module is used to perform iterative fine registration on each initial affine transformation matrix to obtain the corresponding candidate fine registration matrix. The fine registration includes: dividing the infrared binary edge map transformed based on the current affine transformation matrix into multiple blocks, calculating the local offset of each block on the visible light binary edge map and the matching weight of each block, removing blocks with abnormal offsets, fitting an affine correction matrix based on the offset and weight of the remaining blocks and updating the current affine transformation matrix; after the iteration is completed, selecting the optimal matrix from all candidate fine registration matrices based on the local edge overlap quality score as the final affine transformation matrix from the infrared image to the visible light image.

[0165] The timing processing module is used to perform timing consistency optimization on the registration results based on the continuity between the final affine transformation matrices of adjacent frames in the video sequence, including marking abnormal frames and correcting the final affine transformation matrices of low-quality frames.

[0166] In one specific embodiment of the present invention, the system further includes:

[0167] The GPU acceleration module is used to achieve parallel fine registration of multiple image pairs by utilizing CUDA streaming concurrency mechanism and GPU batch image transformation, block search and candidate evaluation.

[0168] The output module is used to crop and fuse the registered infrared and visible light images according to the distortion-reduced effective region mask. The distortion-reduced effective region mask refers to the effective display area on the visible light image after the infrared image is transformed by the final affine transformation matrix. It is obtained by mapping the boundary region of the infrared image to the visible light image coordinate system according to the final affine transformation matrix.

[0169] It should also be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the embodiments provided in this application, the division of steps or modules in the system and method is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules or steps may be combined or integrated together, and a module or step may also be split.

[0170] It is understood that the dual-modal image registration method based on deep learning and GPU parallel optimization in the above embodiments can essentially be implemented by a computer program. Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer program product corresponding to the dual-modal image registration method based on deep learning and GPU parallel optimization provided in the above embodiments. This product includes a computer program / instruction that, when executed by a processor, can implement the dual-modal image registration method based on deep learning and GPU parallel optimization as described in the above embodiments.

[0171] Similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the dual-modal image registration method based on deep learning and GPU parallel optimization provided in the above embodiments, such as... Figure 7 As shown, it includes a memory and a processor;

[0172] The memory is used to store computer programs;

[0173] The processor is configured to implement the dual-modal image registration method based on deep learning and GPU parallel optimization in the above embodiments when executing the computer program.

[0174] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0175] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the dual-modal image registration method based on deep learning and GPU parallel optimization provided in the above embodiments. The storage medium stores a computer program, which, when executed by a processor, can realize the dual-modal image registration method based on deep learning and GPU parallel optimization in the above embodiments.

[0176] It is understood that the computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0177] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A dual-modality image registration method based on deep learning and GPU parallel optimization, characterized in that, The method comprises the following steps: S1, for each pair of infrared image and visible light image corresponding to each frame in the video sequence, extract the corresponding infrared binary edge map and visible light binary edge map respectively; S2, calculate the distance transform map based on the visible light binary edge map, and perform sliding coarse registration on the distance transform map with the infrared binary edge map as a template at multiple scales to obtain multiple initial affine transformation matrices; S3, perform iterative fine registration on each initial affine transformation matrix to obtain a corresponding candidate fine registration matrix; The fine registration comprises: dividing the infrared binary edge map transformed based on the current affine transformation matrix into multiple blocks, calculating the local offset of each block on the visible light binary edge map and the matching weight of each block, eliminating the blocks with abnormal offset, fitting an affine correction matrix based on the offset and weight of the remaining blocks and updating the current affine transformation matrix; after the iteration is completed, the optimal matrix is selected from all candidate fine registration matrices based on the local edge overlap quality score, as the final affine transformation matrix of the infrared image to the visible light image; S4, based on the continuity between the final affine transformation matrices of adjacent frames in the video sequence, perform time sequence consistency optimization processing on the registration result, including marking abnormal frames and correcting the final affine transformation matrix of low-quality frames.

2. The dual-modality image registration method based on deep learning and GPU parallel optimization according to claim 1, characterized in that, Before calculating the distance transform map, a common downsampling step is further included: A common downsampling factor is calculated according to the maximum edge length of the visible light binary edge map and a preset maximum edge length for searching, and the infrared binary edge map and the visible light binary edge map are downsampled according to the common downsampling factor to establish a coarse search coordinate system in the coarse matching stage; The initial affine transformation matrix is calculated in the coarse search coordinate system and converted back to the original image resolution coordinate system.

3. The dual-modality image registration method based on deep learning and GPU parallel optimization according to claim 2, characterized in that, The distance transform map is obtained from the downsampled infrared binary edge map and the downsampled and inverted visible light binary edge map and is subjected to Gaussian smoothing processing; each pixel value in the distance transform map represents the distance from the pixel position to the nearest visible light edge pixel.

4. The dual-modality image registration method based on deep learning and GPU parallel optimization according to claim 2, characterized in that, In S2, the sliding coarse registration comprises: An adaptive scale is calculated according to the size ratio of the infrared binary edge map and the visible light binary edge map, the infrared template scale search range is determined according to the adaptive scale, and multiple scale values are sampled as infrared template scales in the search range; For each infrared template scale, the downsampled infrared binary edge map is further scaled to an infrared template, the infrared template is matched by sliding on the distance transform map, and a matching score is calculated; All infrared template scales are arranged in ascending order of the matching score, and the top several infrared template scales are selected as candidates to obtain a scaling-translation matrix under each candidate scale, and the scaling-translation matrix is converted back to the original resolution coordinate system based on the common downsampling factor to obtain an initial affine transformation matrix from the infrared image coordinate system to the visible light image coordinate system under each candidate scale.

5. The dual-modality image registration method based on deep learning and GPU parallel optimization according to claim 4, characterized in that, The calculation process of the matching score is as follows: The pixel values corresponding to the infrared template position in the distance transform map and the pixel values of the infrared template itself are multiplied and summed, and then divided by the total number of edge pixels in the infrared template to obtain the template matching score.

6. The dual-modality image registration method based on deep learning and GPU parallel optimization according to claim 1, characterized in that, In S3, the local offset is achieved through block template matching, including: Transform the infrared binary edge image to the visible light coordinate system using the current affine transformation matrix, calculate the bounding box of the transformed infrared binary edge image, and expand it with a preset fill value. The expanded region is uniformly divided into 3×3 block grids. For each block, the block region in the transformed infrared binary edge map is used as the template, and the distance transformation map corresponding to the block region in the visible light binary edge map is used as the search region. The normalized cross-correlation response map is calculated based on the normalized template and the search region. The location of the maximum value in the normalized cross-correlation response graph is selected, and the difference between this location and the original center coordinates of the block is used as the local offset.

7. The dual-modality image registration method based on deep learning and GPU parallel optimization according to claim 6, characterized in that, In S3, the calculation process for the matching weight of each block includes: For each block, extract the corresponding visible light edge region in the visible light binary edge map based on the position corresponding to the local offset; Calculate the intersection-union ratio (IUU) between the template region where the segment is located and the visible light edge region, and use the square of the IUU as the matching weight of the segment.

8. The dual-modality image registration method based on deep learning and GPU parallel optimization according to claim 1, characterized in that, The fine registration process in S3 is executed using GPU batch parallel acceleration technology. Based on the GPU's currently available video memory, the estimated video memory required for a single image registration, and the system's reserved video memory, the number of image pairs that can be processed concurrently in the same batch is dynamically calculated.

9. The dual-modality image registration method based on deep learning and GPU parallel optimization according to claim 1, characterized in that S4 include: For each frame, the final affine transformation matrix is ​​decomposed into linear components and translation components. The deviations between the linear components and translation components and the median reference of the corresponding components in each frame in the temporal neighborhood are calculated. The anomaly degree is calculated based on the deviations. If the anomaly degree is greater than or equal to a preset anomaly threshold, it is marked as an abnormal frame. Calculate the median of the local edge overlap quality score of all frames in the video sequence and set a dynamic quality threshold; for low-quality frames whose local edge overlap quality score is lower than the dynamic quality threshold, use the final affine transformation matrix of their neighboring high-quality frames whose local edge overlap quality score is higher than the dynamic quality threshold to generate a replacement matrix and correct the final affine transformation matrix of the low-quality frames.

10. A computer program product comprising computer programs / instructions, characterized in that, When the computer program / instruction is executed by the processor, it can implement the dual-modal image registration method based on deep learning and GPU parallel optimization as described in any one of claims 1 to 9.

Images