Infrared-visible light dual optical registration method and device

Abstract

This application belongs to the field of image processing technology, and particularly relates to an infrared and visible light dual-light registration method and apparatus. The method acquires infrared and visible light images simultaneously captured by a dual-light imaging device. After preprocessing both images, they are input into a lightweight convolutional neural network for feature extraction to obtain key points, feature descriptors, and reliability information. Based on the feature descriptors, cross-modal feature matching is performed to obtain an initial set of matching point pairs. An initial spatial transformation relationship between the infrared and visible light images is estimated based on the initial set of matching point pairs. Then, based on the initial spatial transformation relationship, the initial set of matching point pairs is further filtered and optimized to obtain an optimized spatial transformation relationship. Finally, perspective transformation is performed on the visible light image based on the optimized spatial transformation relationship to obtain a registered visible light image aligned with the infrared image. This method can improve the registration accuracy and robustness of infrared and visible light images.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an infrared-visible dual-light registration method and apparatus. Background Technology

[0002] Infrared and visible light image registration is a crucial foundational step in dual-light fusion imaging, widely applied in security monitoring, autonomous driving, industrial inspection, and machine vision. Infrared imaging reflects the thermal radiation distribution of a target, while visible light imaging reflects scene texture, color, and illumination information. Significant differences exist between the two in their imaging mechanisms, grayscale distributions, and feature representations, making it difficult to directly establish stable correspondences between cross-modal images. Existing technologies, such as traditional manual feature-based registration methods, are susceptible to variations in illumination, differences in thermal target distribution, and scene occlusion, resulting in limited robustness and registration accuracy. While deep learning-based registration methods improve feature extraction capabilities to some extent, they generally suffer from complex model structures, large parameter sets, and high computational costs, making them unsuitable for real-time deployment on embedded terminals. Furthermore, existing solutions generally lack dedicated training data support for infrared-visible dual-light registration, and significant differences in model output formats across different deployment platforms further complicate engineering applications.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Application content

[0004] In view of at least one of the above technical problems, this application provides an infrared-visible dual-light registration method and apparatus.

[0005] In a first aspect, this application provides an infrared-visible dual-light registration method, the method comprising: Acquire a dual-light image pair, which includes an infrared image and a visible light image synchronously acquired by a dual-light camera device of the same scene. Perform preprocessing on the infrared image and the visible light image respectively to obtain a preprocessed infrared image and a preprocessed visible light image. The preprocessed infrared image and the preprocessed visible light image are input into a lightweight convolutional neural network for feature extraction, resulting in the first key point and the first feature descriptor of the preprocessed infrared image, and the second key point and the second feature descriptor of the preprocessed visible light image. Cross-modal feature matching is performed on the first and second feature descriptors to obtain an initial set of matching point pairs; Based on the initial set of matching point pairs, the spatial transformation relationship between the preprocessed infrared image and the preprocessed visible light image is estimated to obtain the initial spatial transformation relationship; Based on the initial spatial transformation relationship, the initial set of matching point pairs is further filtered and optimized to obtain the optimized spatial transformation relationship. Based on the optimized spatial transformation relationship, a perspective transformation is performed on the preprocessed visible light image to obtain a registered visible light image aligned with the preprocessed infrared image. The registered visible light image and the preprocessed infrared image constitute a registered two-light image pair.

[0006] This method preprocesses infrared and visible light images and extracts key points and feature descriptors from the dual-modal images using a lightweight convolutional neural network, achieving efficient cross-modal feature representation between infrared and visible light images. By performing reliability screening on key points and combining cross-modal feature matching, spatial transformation relationship estimation, and secondary screening optimization of matching point pairs, the method can effectively reduce the impact of false and abnormal matching on the registration results, and improve the registration accuracy and robustness of dual-light images.

[0007] In some possible implementations, the infrared image and the visible light image are preprocessed separately, including: Gaussian smoothing denoising was performed on the infrared image and the visible light image respectively to obtain the denoised infrared image and the denoised visible light image. Based on the resolution of the denoised infrared image, the denoised visible light image is scaled to the same resolution as the denoised infrared image through bilinear interpolation to complete the scale normalization process, resulting in a scale-normalized infrared image and a scale-normalized visible light image. The scale-normalized image is converted from RGB three-channel to grayscale single-channel to maintain the same channel dimension as the scale-normalized infrared image, resulting in a preprocessed infrared image and a preprocessed visible light image.

[0008] In some possible implementations, before inputting the preprocessed infrared image and preprocessed visible light image into the lightweight convolutional neural network for feature extraction, the following steps are taken: The preprocessed infrared image and the preprocessed visible light image are downscaled according to a preset scaling factor.

[0009] In some possible implementations, the lightweight convolutional neural network uses depthwise separable convolutions to construct a four-level downsampled backbone layer. The lightweight convolutional neural network includes a keypoint detection branch, a feature descriptor generation branch, and a reliability map output branch. The reliability map output branch and the feature descriptor generation branch share the features of the four-level downsampled backbone layer. A reliability map with the same resolution as the input image is generated through 1×1 convolution. Each pixel value in the reliability map represents the reliability score of the corresponding keypoint.

[0010] In some possible implementations, after obtaining the first and second keypoints, the following are included: Non-maximum suppression was used to screen the first and second key points respectively, retaining the first high-response key point and the second high-response key point; Based on the reliability map, the reliability scores of the first and second highest response key points are calculated. The reliability scores are obtained by weighting the normalized value of the response intensity of the first or second highest response key point, the normalized variance of the feature descriptor, and the neighborhood consistency score of the feature point. Keypoints with reliability scores below a preset reliability threshold are removed. The first keypoint with the highest response is retained as the new first keypoint, and the second keypoint with the highest response is retained as the new second keypoint. The feature descriptor corresponding to the new first keypoint is used as the first feature descriptor, and the feature descriptor corresponding to the new second keypoint is used as the second feature descriptor.

[0011] In some possible implementations, cross-modal feature matching is performed on the first and second feature descriptors to obtain an initial set of matching point pairs, including: The similarity between the first feature descriptor and the second feature descriptor is calculated based on cosine similarity. Keypoint pairs with similarity higher than a preset cosine similarity threshold are retained, while low-confidence matching pairs with similarity lower than the cosine similarity threshold are removed, resulting in an initial set of matching point pairs. Each element in the initial set of matching point pairs is the corresponding coordinate pair between the first keypoint and the second keypoint.

[0012] In some possible implementations, based on the initial set of matching point pairs, the spatial transformation relationship between the preprocessed infrared image and the preprocessed visible light image is estimated to obtain the initial spatial transformation relationship, including: The USAC-MAGSAC robust estimation algorithm is adopted, with the goal of minimizing the reprojection error. The small-scale homography matrix of the downscaled preprocessed visible light image to the downscaled preprocessed infrared image is calculated iteratively based on the initial set of matching point pairs. Based on the scaling factor and image edge offset, scale compensation and offset compensation are performed on the small-scale homography matrix to restore the homography matrix at the original scale. The homography matrix serves as the initial spatial transformation relation.

[0013] In some possible implementations, the initial set of matching point pairs is further filtered and optimized based on the initial spatial transformation relationship to obtain an optimized spatial transformation relationship, including: Based on the initial spatial transformation relationship, the visible light feature points in the initial set of matching point pairs are mapped to the infrared image coordinate system, the reprojection error of each matching point pair is calculated, and abnormal matching point pairs with reprojection errors exceeding the preset error threshold are removed. The preprocessed infrared image is divided into regular grids, and the matching points in each grid are sorted in descending order of reliability score. A preset number of matching point pairs are retained in each grid. The optimized spatial transformation relationship is obtained by iterative optimization of the matching point pairs after uniform distribution.

[0014] In some possible implementations, after dividing the preprocessed infrared image into a regular grid, the following methods are included: Fill empty grids with neighboring point pairs; For an empty grid without a valid matching point pair, select the matching point pair with the highest reliability score and the closest distance to the empty grid from the non-empty grids within the 3×3 neighborhood of the empty grid and add it to the empty grid; If there are no valid matching point pairs within the 3×3 neighborhood, the neighborhood search range is expanded to 5×5, and the matching point pairs with the highest reliability score and the closest distance to the empty grid are selected to supplement the empty grid. If there are still no valid matching point pairs within the 5×5 neighborhood, then the replenishment of empty grids is abandoned.

[0015] Secondly, this application provides an infrared-visible dual-light registration device, the device comprising: The preprocessing module is used to acquire dual-light image pairs, which include infrared images and visible light images synchronously acquired by dual-light camera devices of the same scene. The infrared images and visible light images are preprocessed separately to obtain preprocessed infrared images and preprocessed visible light images. The feature extraction module is used to input the preprocessed infrared image and the preprocessed visible light image into a lightweight convolutional neural network for feature extraction, to obtain the first key point and the first feature descriptor of the preprocessed infrared image, and to obtain the second key point and the second feature descriptor of the preprocessed visible light image. The feature matching module is used to perform cross-modal feature matching on the first feature descriptor and the second feature descriptor to obtain an initial set of matching point pairs; The initial estimation module is used to estimate the spatial transformation relationship between the preprocessed infrared image and the preprocessed visible light image based on the initial set of matching point pairs, and to obtain the initial spatial transformation relationship. The optimization module is used to perform secondary filtering and optimization on the initial set of matching points based on the initial spatial transformation relationship, so as to obtain the optimized spatial transformation relationship. The registration module is used to perform perspective transformation on the preprocessed visible light image according to the optimized spatial transformation relationship, so as to obtain a registered visible light image that is aligned with the preprocessed infrared image. The registered visible light image and the preprocessed infrared image form a registered two-light image pair.

[0016] The present application will be further described below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic flowchart of the infrared-visible dual-light registration method provided in the embodiments of this application; Detailed Implementation

[0019] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the specific embodiments of this application are described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0020] This application provides an infrared-visible light dual-light registration method, applicable to the registration processing of infrared and visible light images simultaneously acquired by dual-light imaging devices. This method can be deployed in security monitoring terminals, industrial inspection terminals, machine vision terminals, and other devices requiring cross-modal registration of infrared and visible light images. This infrared-visible light dual-light registration method preprocesses the infrared and visible light images and extracts key points and feature descriptors from the dual-modal images based on a lightweight convolutional neural network, achieving efficient cross-modal feature representation between the infrared and visible light images. By performing reliability screening on key points and combining cross-modal feature matching, spatial transformation relationship estimation, and secondary screening optimization of matching point pairs, the method effectively reduces the impact of false and abnormal matching on the registration results, improving the registration accuracy and robustness of dual-light image registration.

[0021] like Figure 1 As shown, this embodiment provides an infrared-visible dual-light registration method, which includes steps S100 to S600.

[0022] Step S100: Obtain a dual-light image pair, which includes an infrared image and a visible light image synchronously acquired by a dual-light camera device of the same scene. Perform preprocessing on the infrared image and the visible light image respectively to obtain a preprocessed infrared image and a preprocessed visible light image. First, a dual-light image pair is acquired. This pair consists of an infrared image and a visible light image simultaneously captured by a dual-light camera device of the same scene. It can be understood that the dual-light camera device is a dual-light camera integrated with an infrared imaging module and a visible light imaging module. The two modules maintain time synchronization during acquisition to ensure that the acquired infrared and visible light images reflect the same scene content at the same time.

[0023] For example, using a dual-light device with fully aligned infrared and visible light, infrared and visible light images of the same scene are acquired under different lighting conditions and at different distances (near distance 0-5m, medium distance 5-20m, far distance 20-50m). A total of 20,000 pairs of aligned infrared and visible light data are collected. A VIT-based infrared and visible light dual-light registration model is used to annotate the pairs, obtain information such as the position of feature point pairs in the dual-light images, and generate npz data, including image address, point pair coordinates, and path index.

[0024] After acquiring the two-light image pair, the infrared and visible light images are preprocessed separately to obtain preprocessed infrared and visible light images. Specifically, the preprocessing may include Gaussian smoothing for noise reduction, scale normalization, and channel adaptation.

[0025] In some embodiments, Gaussian smoothing denoising is performed on the infrared image and the visible light image respectively to obtain a denoised infrared image and a denoised visible light image. It is understood that Gaussian filtering is applied to the infrared image and the visible light image for smoothing and denoising to suppress acquisition noise and local random interference, thereby improving the stability of subsequent keypoint detection. The kernel size and standard deviation used for Gaussian smoothing can be set according to the actual application scenario; for example, a 3×3 or 5×5 Gaussian kernel can be selected.

[0026] After denoising, using the resolution of the denoised infrared image as a benchmark, the denoised visible light image is scaled to the same resolution as the denoised infrared image through bilinear interpolation, completing the scale normalization process. After scale normalization, scale-normalized infrared and visible light images are obtained. Since infrared images are typically single-channel grayscale images, while visible light images are typically RGB three-channel color images, the scale-normalized visible light image is further converted from RGB three-channel to grayscale single-channel to maintain the same channel dimension as the scale-normalized infrared image, resulting in preprocessed infrared and visible light images.

[0027] In some embodiments, before inputting the preprocessed infrared image and the preprocessed visible light image into the lightweight convolutional neural network, the preprocessed infrared image and the preprocessed visible light image can be downscaled according to a preset scaling factor to obtain small-scale infrared images and small-scale visible light images, thereby reducing the computational load of subsequent feature extraction and matrix estimation and improving inference efficiency. The scaling factor can be set according to the computing platform and image resolution.

[0028] In some embodiments, the sets of preprocessed infrared images and preprocessed visible light images are divided into a training set: validation set: test set in a ratio of 8:1:1 for subsequent training and testing.

[0029] Step S200: Input the preprocessed infrared image and the preprocessed visible light image into a lightweight convolutional neural network for feature extraction to obtain the first key point and the first feature descriptor of the preprocessed infrared image, and the second key point and the second feature descriptor of the preprocessed visible light image. The lightweight convolutional neural network employs depthwise separable convolutions to construct a four-level downsampling backbone layer, effectively reducing the number of parameters and computational cost while maintaining feature extraction capabilities. The lightweight convolutional neural network can include a keypoint detection branch, a feature descriptor generation branch, and a reliability map output branch. The reliability map output branch shares the feature map output from the four-level downsampling backbone layer with the feature descriptor generation branch, and generates a reliability map with the same resolution as the input image through 1×1 convolutions. Each pixel value in the reliability map represents the reliability score of the corresponding keypoint.

[0030] This lightweight convolutional neural network can employ the XFeat lightweight convolutional neural network model, which natively includes a keypoint detection branch, a 64-dimensional descriptor generation branch, and a reliability map output branch, with a total of 12 layers. The convolutional kernels can include 3×3 kernels and 1×1 kernels. After processing by this network, it outputs the keypoint locations and 64-dimensional first feature descriptors corresponding to the infrared image, and the keypoint locations and 64-dimensional second feature descriptors corresponding to the visible light image.

[0031] After obtaining the first and second key points, the key points and corresponding features can be further filtered to improve the accuracy and robustness of subsequent cross-modal matching.

[0032] Specifically, non-maximum suppression is used to filter the first and second keypoints separately, retaining the first and second high-response keypoints with larger local response values. The threshold for non-maximum suppression can be set according to the richness of image texture and the complexity of the target scene.

[0033] Based on the reliability map, reliability scores are calculated for the first and second highest-response keypoints. These scores are weighted by the normalized value of the response intensity of either the first or second highest-response keypoint, the normalized variance of the feature descriptor, and the feature point's neighborhood consistency score. This method comprehensively measures keypoint saliency, descriptor stability, and local neighborhood structure consistency, thereby filtering out low-quality keypoints that, while having strong responses, suffer from unstable descriptions or poor neighborhood consistency.

[0034] Keypoints with reliability scores below a preset reliability threshold are removed. The first keypoint with the highest response is retained as the new first keypoint, and the second keypoint with the highest response is retained as the new second keypoint. The feature descriptor corresponding to the new first keypoint is used as the first feature descriptor, and the feature descriptor corresponding to the new second keypoint is used as the second feature descriptor.

[0035] For example, firstly, non-maximum suppression (NMS) is used to screen high-quality keypoints, with the NMS threshold Tnms=0.4, to obtain the feature sets of infrared and visible light images: Infrared image feature set:

[0036] Visible light image feature set:

[0037] in, After the two-dimensional coordinates of key points in the preprocessed infrared image are filtered by NMS thresholding, high-response key points are retained, which are the first key points.

[0038] The two-dimensional coordinates of key points in the preprocessed visible light image are filtered by NMS thresholding to retain high-response key points, which are the second key points.

[0039] This is the 64-dimensional high-discrimination feature descriptor corresponding to the first keypoint, also known as the first feature descriptor.

[0040] This is the 64-dimensional high-discrimination feature descriptor corresponding to the second keypoint, also known as the second feature descriptor.

[0041] This represents the number of valid key points after filtering the preprocessed infrared image.

[0042] This represents the number of valid key points after filtering the preprocessed infrared image.

[0043] Secondly, the descriptor reliability map output by the lightweight convolutional neural network is a grayscale image with the same resolution as the small-scale preprocessed infrared image and the small-scale preprocessed visible light image. Its pixel value is the reliability score of the key point at the corresponding location. The score range is [0,1] and is calculated by weighting three indicators: key point response intensity, descriptor normalized variance, and feature point neighborhood consistency. The calculation formula is as follows: In the formula, The final reliability score for critical point p. The normalized result (0-1) of the keypoint response values. The normalized variance (0-1) of the 64-dimensional descriptor corresponding to the keypoints. Assign a consistency score (0-1) to features within a 3×3 neighborhood of the keypoint; set a reliability score threshold. Reliability rating Feature points are directly removed, and only high-reliability feature points are retained, which further improves feature quality and robustness of subsequent matching.

[0044] Step S300: Perform cross-modal feature matching on the first feature descriptor and the second feature descriptor to obtain an initial set of matching point pairs; Specifically, the similarity between the first and second feature descriptors is calculated based on cosine similarity. Keypoint pairs with similarity higher than a preset cosine similarity threshold are retained, while low-confidence matching pairs with similarity lower than the cosine similarity threshold are removed, resulting in an initial set of matching point pairs. Each element in the initial set of matching point pairs represents the corresponding coordinate pair between the first and second keypoints. By using cosine similarity for cross-modal matching, the directional consistency of feature descriptors between different modalities can be effectively measured, thus maintaining good matching robustness even when there are significant differences between infrared and visible light image imaging.

[0045] In some embodiments, the initial set of matching point pairs is defined as:

[0046] in, Here is the formula for calculating cosine similarity. For the L2 norm, Using a cosine similarity threshold, only feature pairs with similarity higher than the threshold are retained, while low-confidence pseudo-matches are removed. This is the initial set of matching pairs, where each element is a one-to-one coordinate pair of infrared and visible light keypoints.

[0047] Step S400: Based on the initial set of matching point pairs, estimate the spatial transformation relationship between the preprocessed infrared image and the preprocessed visible light image to obtain the initial spatial transformation relationship; Specifically, after downscaling, the initial spatial transformation relationship is estimated based on the downscaled preprocessed infrared image and the downscaled preprocessed visible light image. The USAC-MAGSAC robust estimation algorithm is used, aiming to minimize the reprojection error, and iteratively calculates the small-scale homography matrix from the downscaled preprocessed visible light image to the downscaled preprocessed infrared image based on the initial set of matching points. Based on the scaling factor and image edge offset, scale compensation and offset compensation are performed on the small-scale homography matrix to restore the homography matrix at the original scale. The homography matrix serves as the initial spatial transformation relation.

[0048] By adopting the above method, the computational cost of directly estimating spatial transformation relationships on high-resolution images can be reduced, while ensuring the registration accuracy after restoration to the original scale.

[0049] For example, the USAC-MAGSAC robust estimation algorithm is used, based on the initial set of matching pairs M. init Solve for the homography matrix H of a small-scale two-light image. s And through the scale factor s and the edge offset The original scale two-light registration matrix H is obtained by restoration.

[0050] More specifically, with the goal of minimizing the reprojection error, the homography matrix H of the small-scale visible light image to infrared image is solved iteratively using the USAC-MAGSAC algorithm. s ∈R 3×3 Combining the small-scale scaling factor s with the image padding offset o, for H s Scale and offset compensation are performed to obtain the two-light registration matrix at the original scale. :

[0051] in, , These are the scaling transformation matrix and the offset transformation matrix, respectively, which adapt the coordinate mapping relationship of the original scaled image. The final result, H, is the registration matrix from the visible light image to the infrared image coordinate system.

[0052] Step S500: Based on the initial spatial transformation relationship, perform secondary screening and optimization on the initial matching point pair set to obtain the optimized spatial transformation relationship; Specifically, based on the initial spatial transformation relationship, the visible light feature points in the initial set of matched point pairs are mapped to the infrared image coordinate system. The reprojection error of each matched point pair is calculated, and abnormal matched point pairs whose reprojection errors exceed a preset error threshold are removed. The preset error threshold can be 2 pixels. Through this process, residual erroneous matched point pairs in the initial matching can be eliminated.

[0053] The preprocessed infrared image is divided into regular grids, and the matching points in each grid are sorted in descending order of reliability score. A preset number of matching point pairs are retained in each grid.

[0054] Understandably, after removing abnormal matching point pairs, to further prevent matching points from becoming overly concentrated in local areas and causing the registration matrix to be dominated by local features, the preprocessed infrared image can be divided into regular grids. For an infrared image with a resolution of 384×288, it can be divided into a regular grid of 16×12, with each grid measuring 24×24 pixels. Then, the matching points in each grid are sorted according to their reliability scores from high to low, and a preset number of matching point pairs are retained in each grid, for example, a maximum of 5 matching point pairs, to suppress redundant points in locally overly dense areas.

[0055] For empty grid cells without valid matching point pairs, neighboring point pairs can be used for supplementation. Specifically, first, non-empty grid cells are searched within a 3×3 neighborhood of the empty grid cell, and the matching point pair with the highest reliability score and closest distance to the empty grid cell is selected to supplement the empty grid cell. If no valid matching point pair is found within the 3×3 neighborhood, the neighborhood search range is expanded to 5×5, and matching point pairs are reselected for supplementation according to the same rules. If no valid matching point pair is found within the 5×5 neighborhood, the supplementation of the empty grid cell is abandoned. Through the above grid homogenization and neighboring point supplementation mechanism, the matching point pairs participating in the optimization can be more evenly distributed on the image plane.

[0056] The optimized spatial transformation relationship is obtained through iterative optimization of the uniformly distributed matching point pairs. Compared with the initial spatial transformation relationship, the optimized spatial transformation relationship has better geometric robustness and registration stability.

[0057] Step S600: According to the optimized spatial transformation relationship, the preprocessed visible light image is subjected to perspective transformation to obtain a registered visible light image aligned with the preprocessed infrared image. The registered visible light image and the preprocessed infrared image constitute a registered two-light image pair.

[0058] Understandably, based on the homography matrix corresponding to the optimized spatial transformation relationship, a perspective projection transformation can be performed on the visible light image at the original scale, mapping the pixel coordinates in the visible light image to the coordinate system of the infrared image, thereby achieving spatial alignment of the two images in a unified coordinate system. After alignment, the registered visible light image and infrared image can be used for subsequent image fusion, target detection, scene perception, or multimodal display.

[0059] For example, based on the registration matrix H, preprocessed visible light images Perform a perspective transformation to achieve precise spatial alignment between the visible light image and the infrared image coordinate system, resulting in the aligned visible light image. The perspective transformation formula is: , in, The element in the a-th row and b-th column of the registration matrix H , To preprocess the pixel coordinates of the visible light image, The pixel coordinates of the infrared image pixel coordinate system are aligned and then the visible light image is preprocessed. With infrared images The pixel coordinates are perfectly matched to achieve accurate registration of the two-light images, resulting in a registered two-light image pair.

[0060] The following section provides a further explanation of lightweight convolutional neural networks and their deployment methods.

[0061] In this embodiment, the lightweight convolutional neural network model can be pruned and quantized after training to further reduce the number of model parameters, storage overhead, and inference latency. Specifically, a structured channel pruning method based on the L2 norm of the convolutional kernel weights can be used to differentiate the pruning of the core convolutional blocks of the model. For BatchNorm layers and upstream / downstream dependent layers that exhibit dimension mismatch after pruning, inter-layer dimension adaptation can be achieved by adjusting the corresponding parameters and adding a 1×1 convolutional mapping layer.

[0062] Furthermore, different quantization strategies can be adopted for different deployment platforms. For example, W8A8 quantization can be used on the RKNN platform, combined with the MMSE algorithm to optimize quantization error; on the NCNN platform, FP32 model conversion can be completed first, followed by INT8 quantization; on the TensorRT platform, INT8 quantization conversion can be performed after operator compatibility optimization. The output results from different platforms can be further unified and converted into an OpenCV-compatible format to reuse post-processing workflows such as keypoint selection, feature matching, and spatial transformation relationship estimation.

[0063] Corresponding to the above method embodiments, this embodiment provides an infrared-visible dual-light registration device, the device comprising: The preprocessing module is used to acquire dual-light image pairs, which include infrared images and visible light images synchronously acquired by dual-light camera devices of the same scene. The infrared images and visible light images are preprocessed separately to obtain preprocessed infrared images and preprocessed visible light images. The feature extraction module is used to input the preprocessed infrared image and the preprocessed visible light image into a lightweight convolutional neural network for feature extraction, to obtain the first key point and the first feature descriptor of the preprocessed infrared image, and to obtain the second key point and the second feature descriptor of the preprocessed visible light image. The feature matching module is used to perform cross-modal feature matching on the first feature descriptor and the second feature descriptor to obtain an initial set of matching point pairs; The initial estimation module is used to estimate the spatial transformation relationship between the preprocessed infrared image and the preprocessed visible light image based on the initial set of matching point pairs, and to obtain the initial spatial transformation relationship. The optimization module is used to perform secondary filtering and optimization on the initial set of matching points based on the initial spatial transformation relationship, so as to obtain the optimized spatial transformation relationship. The registration module is used to perform perspective transformation on the preprocessed visible light image according to the optimized spatial transformation relationship, so as to obtain a registered visible light image that is aligned with the preprocessed infrared image. The registered visible light image and the preprocessed infrared image form a registered two-light image pair.

[0064] It should be understood that the above-described device is used to execute the methods in the above embodiments. The implementation principle and technical effect of the corresponding program modules in the device are similar to those described in the above methods. The working process of the device can be referred to the corresponding process in the above methods, and will not be repeated here.

[0065] In the description of the embodiments of this application, unless otherwise stated, " / " means "or", for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0066] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0067] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0068] The terms "coupled," "connected," or "connected" in the instruction manual include both direct and indirect connections. An indirect connection is a connection made through an intermediate medium, such as a connection made through an electrically conductive medium, which may have parasitic inductance or capacitance. An indirect connection may also include connections made through other active or passive devices to achieve the same or similar functional purpose, such as connections made through circuits or components such as switches or follower circuits.

[0069] In the detailed description of this specification, reference is made to the accompanying drawings, which form a part thereof, wherein like reference numerals always denote like parts, and wherein exemplary embodiments are shown by way of example that may be implemented. It should be understood that other embodiments may be utilized, and structural or logical changes may be made, without departing from the scope of this application. Therefore, the following detailed description should not be considered limiting.

[0070] The various operations in the specification may be described sequentially as multiple discrete actions or operations in a manner most conducive to understanding the claimed subject matter. However, the order of description should not be construed as implying that these operations must be sequentially related. Specifically, these operations may not be performed in the order presented. The described operations may be performed in a different order than in the described embodiments. Various additional operations may be performed in additional embodiments and / or the described operations may be omitted.

[0071] Various components and devices may be referred to or shown in the singular (e.g., “MOS transistor”, “transistor”, “switch”, etc.) in this document, but only for the convenience of discussion, and any element referred to in the singular may include multiple such elements as taught herein.

[0072] The above are merely preferred embodiments of this application and do not constitute any limitation on this application. Any person skilled in the art can make many possible variations and modifications to the technical solution of this application, or modify it into equivalent embodiments, without departing from the scope of the technical solution of this application. Therefore, all equivalent changes made based on the shape, structure, and principle of this application without departing from the content of the technical solution of this application should be covered within the protection scope of this application.

Claims

1. An infrared-visible dual-light registration method, characterized in that, The method includes: A dual-light image pair is acquired, comprising an infrared image and a visible light image synchronously acquired by a dual-light camera device of the same scene. The infrared image and the visible light image are preprocessed respectively to obtain a preprocessed infrared image and a preprocessed visible light image. The preprocessed infrared image and the preprocessed visible light image are input into a lightweight convolutional neural network for feature extraction to obtain the first key point and the first feature descriptor of the preprocessed infrared image, and the second key point and the second feature descriptor of the preprocessed visible light image. Cross-modal feature matching is performed on the first feature descriptor and the second feature descriptor to obtain an initial set of matching point pairs; Based on the initial set of matching point pairs, the spatial transformation relationship between the preprocessed infrared image and the preprocessed visible light image is estimated to obtain the initial spatial transformation relationship; Based on the initial spatial transformation relationship, the initial set of matching point pairs is further filtered and optimized to obtain the optimized spatial transformation relationship; Based on the optimized spatial transformation relationship, the preprocessed visible light image is subjected to perspective transformation to obtain a registered visible light image aligned with the preprocessed infrared image. The registered visible light image and the preprocessed infrared image constitute a registered two-light image pair.

2. The infrared-visible dual-light registration method according to claim 1, characterized in that, The preprocessing of the infrared image and the visible light image includes: Gaussian smoothing denoising is performed on the infrared image and the visible light image respectively to obtain a denoised infrared image and a denoised visible light image; Using the resolution of the denoised infrared image as a reference, the denoised visible light image is scaled to the same resolution as the denoised infrared image through bilinear interpolation to complete the scale normalization process, resulting in a scale-normalized infrared image and a scale-normalized visible light image. The scale-normalized image is converted from RGB three-channel to grayscale single-channel to maintain the same channel dimension as the scale-normalized infrared image, thus obtaining the preprocessed infrared image and the preprocessed visible light image.

3. The infrared-visible dual-light registration method according to claim 2, characterized in that, Before inputting the preprocessed infrared image and preprocessed visible light image into the lightweight convolutional neural network for feature extraction, the process includes: The preprocessed infrared image and the preprocessed visible light image are downscaled according to a preset scaling factor.

4. The infrared-visible dual-light registration method according to claim 3, characterized in that, The lightweight convolutional neural network uses depthwise separable convolutions to construct a four-level downsampling backbone layer. The lightweight convolutional neural network includes a keypoint detection branch, a feature descriptor generation branch, and a reliability map output branch. The reliability map output branch shares the features of the four-level downsampling backbone layer with the feature descriptor generation branch. A reliability map with the same resolution as the input image is generated through 1×1 convolution. Each pixel value in the reliability map represents the reliability score of the corresponding keypoint.

5. The infrared-visible dual-light registration method according to claim 4, characterized in that, After obtaining the first key point and the second key point, the process includes: Non-maximum suppression is used to filter the first key point and the second key point respectively, retaining the first high-response key point and the second high-response key point; Based on the reliability map, the reliability scores of the first high-response key point and the second high-response key point are calculated. The reliability scores are obtained by weighting the normalized value of the response intensity of the first high-response key point or the second high-response key point, the normalized variance of the feature descriptor, and the neighborhood consistency score of the feature point. Key points with reliability scores below a preset reliability threshold are removed. The first high-response key point is retained as the new first key point, and the second high-response key point is retained as the new second key point. The feature descriptor corresponding to the new first key point is used as the first feature descriptor, and the feature descriptor corresponding to the new second key point is used as the second feature descriptor.

6. The infrared-visible dual-light registration method according to claim 1, characterized in that, The step of performing cross-modal feature matching on the first feature descriptor and the second feature descriptor to obtain an initial set of matching point pairs includes: The similarity between the first feature descriptor and the second feature descriptor is calculated based on cosine similarity. Keypoint pairs with similarity higher than the preset cosine similarity threshold are retained, and low-confidence matching pairs with similarity lower than the cosine similarity threshold are removed to obtain the initial matching point pair set; each element in the initial matching point pair set is the corresponding coordinate pair between the first keypoint and the second keypoint.

7. The infrared-visible dual-light registration method according to claim 3, characterized in that, The step of estimating the spatial transformation relationship between the preprocessed infrared image and the preprocessed visible light image based on the initial set of matching point pairs to obtain the initial spatial transformation relationship includes: The USAC-MAGSAC robust estimation algorithm is used to minimize the reprojection error. Based on the initial set of matching point pairs, the small-scale homography matrix from the downscaled preprocessed visible light image to the downscaled preprocessed infrared image is calculated iteratively. Based on the scaling factor and image edge offset, scale compensation and offset compensation are performed on the small-scale homography matrix to restore the homography matrix at the original scale. The homography matrix serves as the initial spatial transformation relation.

8. The infrared-visible dual-light registration method according to claim 1, characterized in that, The step of performing secondary filtering and optimization on the initial set of matching point pairs based on the initial spatial transformation relationship to obtain an optimized spatial transformation relationship includes: Based on the initial spatial transformation relationship, the visible light feature points in the initial set of matching point pairs are mapped to the infrared image coordinate system, the reprojection error of each matching point pair is calculated, and abnormal matching point pairs with reprojection errors exceeding a preset error threshold are removed. The preprocessed infrared image is divided into regular grids, and the matching points in each grid are sorted in descending order of reliability score. A preset number of matching point pairs are retained in each grid. The optimized spatial transformation relationship is obtained by iterative optimization of the matching point pairs after uniform distribution.

9. The infrared-visible dual-light registration method according to claim 8, characterized in that, After dividing the preprocessed infrared image into a regular grid, the process includes: Fill empty grids with neighboring point pairs; For an empty grid without a valid matching point pair, select the matching point pair with the highest reliability score and the closest distance to the empty grid from the non-empty grids within the 3×3 neighborhood of the empty grid and add it to the empty grid; If there are no valid matching point pairs within the 3×3 neighborhood, the neighborhood search range is expanded to 5×5, and the matching point pair with the highest reliability score and the closest distance to the empty grid is selected to supplement the empty grid. If there are still no valid matching point pairs within the 5×5 neighborhood, then the addition of the empty grid is abandoned.

10. An infrared-visible dual-light registration device, characterized in that, The device includes: The preprocessing module is used to acquire a dual-light image pair, which includes an infrared image and a visible light image synchronously acquired by a dual-light camera device of the same scene. The infrared image and the visible light image are preprocessed respectively to obtain a preprocessed infrared image and a preprocessed visible light image. The feature extraction module is used to input the preprocessed infrared image and the preprocessed visible light image into a lightweight convolutional neural network for feature extraction, to obtain the first key point and the first feature descriptor of the preprocessed infrared image, and to obtain the second key point and the second feature descriptor of the preprocessed visible light image. The feature matching module is used to perform cross-modal feature matching on the first feature descriptor and the second feature descriptor to obtain an initial set of matching point pairs; The initial estimation module is used to estimate the spatial transformation relationship between the preprocessed infrared image and the preprocessed visible light image based on the initial set of matching point pairs, and obtain the initial spatial transformation relationship; The optimization module is used to perform secondary filtering and optimization on the initial set of matching points based on the initial spatial transformation relationship to obtain an optimized spatial transformation relationship; The registration module is used to perform perspective transformation on the preprocessed visible light image according to the optimized spatial transformation relationship to obtain a registered visible light image aligned with the preprocessed infrared image. The registered visible light image and the preprocessed infrared image constitute a registered two-light image pair.

Images