Cascade infrared and visible image registration method and apparatus for autonomous driving

Abstract

The application provides a cascaded infrared and visible light image registration method and device for automatic driving, and belongs to the technical field of image processing.The method comprises the following steps: constructing a dual-light data set containing physical parallax and non-rigid deformation; the data set comprises an infrared image and a visible light image pair; taking the visible light image as a reference to perform geometric correction on the infrared image to obtain a registered infrared image; the specific process is as follows: the visible light image and the infrared image are mapped into four-level multi-scale pyramid features through two parallel encoding branches, and a deformation field is recursively regressed from the fourth level with the lowest spatial resolution to the first level with the highest resolution through a coarse-to-fine recursive strategy, and the original infrared image is spatially transformed by using the deformation field of the first level with the highest resolution; and the registered infrared image and the visible light image features are fused to output a fused image.Based on the method, a corresponding device is further provided.The image registration process is divided into two stages of geometric correction and feature fusion, and the accuracy and robustness of registration are improved.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and specifically relates to a cascaded infrared and visible light image registration method and device for autonomous driving. Background Technology

[0002] In autonomous driving systems, environmental perception is crucial. Image fusion technology improves the robustness and accuracy of perception systems in harsh environments by combining infrared and visible light images. Currently, single visible light sensors cannot provide sufficiently effective information in low-light or drastically changing lighting environments, while infrared sensors, although unaffected by lighting conditions, lack sufficient texture detail. To address these issues, a joint perception method combining infrared and visible light images is commonly employed.

[0003] However, physical parallax exists between infrared and visible light sensors, and vibrations generated during vehicle movement cause non-rigid deformations in the image, making image alignment difficult. Existing technologies mainly rely on traditional image registration methods, which often fail to achieve efficient image alignment when dealing with non-rigid deformations and parallax. For example, some existing registration methods rely on feature point matching and optical flow algorithms, but these methods are susceptible to noise and motion artifacts in dynamic scenes, resulting in low alignment accuracy. Traditional optimization methods, such as gradient-based minimization methods, often encounter local extrema problems when dealing with complex deformations, thus affecting the quality of image registration. Furthermore, while some methods reduce the limitations of a single sensor through multimodal fusion, they often neglect the spatiotemporal alignment problem between sensors, leading to insufficient image alignment accuracy in dynamic scenes.

[0004] Therefore, how to achieve accurate image alignment in dynamic scenes and perform efficient feature fusion on this basis remains a challenge for current technology. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes a cascaded infrared and visible light image registration method and device for autonomous driving. By dividing registration and fusion into independent geometric correction and feature fusion stages, the problem of physical parallax and non-rigid deformation is effectively solved, improving the accuracy and robustness of image registration.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention proposes a cascaded infrared and visible light image registration method for autonomous driving, comprising the following steps: Construct a two-light dataset that includes physical parallax and non-rigid deformation; the dataset contains pairs of infrared and visible light images; Using the visible light image as a reference, the infrared image is geometrically corrected to obtain a registered infrared image. The geometric correction is achieved in the following way: the visible light image and the infrared image are mapped to four-level multi-scale pyramid features with spatial resolution halved at each level and channel number doubled at each level through two parallel coding branches. The deformation field is regressed step by step from the fourth level with the lowest spatial resolution through a coarse-to-fine recursive strategy. The original infrared image is spatially transformed using the first level deformation field with the highest spatial resolution. The registered infrared image and the visible light image are feature-fused to output a fused image.

[0007] Furthermore, the process of acquiring physical parallax data and non-rigid deformation data includes: A visible light camera module and an infrared camera module with identical parameters are fixedly connected; By using the visible light camera module and the infrared camera module that are fixedly connected, infrared images and visible light images are simultaneously acquired in a road scene that includes both distant and close views in order to obtain physical parallax data. During synchronous acquisition, the fixedly connected visible light camera module and infrared camera module are randomly vibrated to cause non-rigid deformation in the synchronously acquired infrared and visible light images, so as to obtain non-rigid deformation data.

[0008] Furthermore, in geometric correction, the visible light image and the infrared image are mapped to four-level multi-scale pyramid features with spatial resolution halved at each level and the number of channels doubled at each level through two parallel coding branches, specifically: The visible light image is input into the first coding branch, and the infrared image is input into the second coding branch; The first coding branch and the second coding branch share weights and perform sequential downsampling on the input image to extract four-level multi-scale features; The spatial resolution of the first-level features is equal to the original image resolution. The number of feature channels is the number of basic channels. The second-level features are obtained by downsampling the first-level features through a convolution with a stride of 2, resulting in a spatial resolution of [missing information]. The number of feature channels is The third-level features are obtained by downsampling the second-level features through a convolution with a stride of 2, resulting in a spatial resolution of [missing information]. The number of feature channels is The fourth-level features are obtained by downsampling the third-level features through a convolution with a stride of 2, resulting in a spatial resolution of [missing information]. The number of feature channels is .

[0009] Furthermore, in the geometric correction, at each level Visible light image features Features of infrared images After concatenation, feature extraction is performed to form a combined semantic feature body. The expression is: ; in, Indicates feature concatenation operation; These correspond to the pyramid levels from the shallowest to the deepest. This indicates a feature extraction operation.

[0010] Furthermore, a recursive strategy from coarse to fine is employed, starting with the fourth level (lowest spatial resolution) and progressively regressing the deformation field. The first level deformation field (highest spatial resolution) is then used to perform a spatial transformation on the original infrared image. Specifically: In the fourth level, which has the lowest spatial resolution, the combined semantic feature volume of the fourth level is input into the deformation update block to directly calculate the deformation field of the fourth level. For the third level to the first level, the deformation field of the previous level is upsampled and then superimposed with the deformation information estimated by the deformation update block of the current level based on the combined semantic feature body of the current level to obtain the deformation field of the current level. Repeat the above steps to obtain the deformation fields of the third, second, and first levels in sequence; The original infrared image is spatially transformed using the first-level deformation field with the highest spatial resolution to generate a registered infrared image.

[0011] Furthermore, at the fourth level, the deformation field Directly from the combined semantic feature body The expression obtained through the deformation update block is as follows: ; in, This indicates a fourth-level deformation update operation; This represents the combined semantic feature body of the fourth level.

[0012] Furthermore, for Hierarchical, using a residual learning strategy to calculate the deformation field Its expression is: ; in, Indicates according to the first Hierarchical combined semantic feature body Extract deformation update information; Indicates the first The result of upsampling the hierarchical deformation field.

[0013] Furthermore, the deformation update block includes: The stitching operation layer is used to stitch visible light image features with infrared image features; Multiple convolutional blocks are connected in sequence. Each convolutional block includes a convolutional layer, an instance normalization layer, and a LeakyReLU activation function layer. An unbiased two-dimensional convolutional layer is connected to the output of the multiple convolutional blocks to output a dense displacement residual map.

[0014] Furthermore, the training of the geometric correction employs locally normalized cross-correlation loss. With spatial gradient smoothing regularization loss The total loss function is: ; in, This represents the balance coefficient.

[0015] Secondly, the present invention proposes a cascaded infrared and visible light image registration device for autonomous driving, comprising: At least one processor and memory, The memory stores a computer program that, when executed by the at least one processor, implements the cascaded infrared and visible light image registration method for autonomous driving.

[0016] The effects described in the invention are merely those of the embodiments, and not all the effects of the invention. One of the above technical solutions has the following advantages or beneficial effects: This invention proposes a cascaded infrared and visible light image registration method and device for autonomous driving, belonging to the field of image processing technology. The method includes the following steps: constructing a dual-light dataset containing physical parallax and non-rigid deformation; the dataset contains pairs of infrared and visible light images; geometrically correcting the infrared image using the visible light image as a reference to obtain a registered infrared image; the geometric correction is achieved by mapping the visible light and infrared images to four-level multi-scale pyramid features with progressively halved spatial resolution and progressively doubled channel number through two parallel coding branches; and regressing the deformation field step by step from the fourth level (lowest spatial resolution) using a coarse-to-fine recursive strategy, using the first level (highest spatial resolution) deformation field to perform spatial transformation on the original infrared image; and fusing the registered infrared image and the visible light image to output a fused image. Based on this method, this invention also proposes a corresponding device. This invention divides the image registration process into two independent stages—geometric correction and feature fusion—through a cascaded framework, effectively solving the problems of physical parallax and non-rigid deformation, and improving the accuracy and robustness of infrared and visible light image registration.

[0017] This invention utilizes multi-scale pyramid feature extraction and a coarse-to-fine recursive deformation field regression strategy to simultaneously handle large displacement physical parallax and local small non-rigid deformations, adapting to complex deformations in dynamic scenarios.

[0018] This invention employs a deformation update block with a pure convolutional structure, avoiding the computational overhead of Transformer or attention mechanisms. It achieves real-time processing capability while ensuring registration accuracy, thus meeting the real-time requirements of autonomous driving systems.

[0019] This invention uses local normalized cross-correlation loss and spatial gradient smoothing regularization loss as optimization objectives, which effectively solves the problem of nonlinear radiation differences between multimodal images, and makes the network focus on geometric structure rather than pixel intensity. Attached Figure Description

[0020] Figure 1 This is a flowchart of the cascaded infrared and visible light image registration method for autonomous driving proposed in Embodiment 1 of the present invention; Figure 2 This is a diagram illustrating the overall framework for geometric correction of images using the static feature recursive registration network proposed in Embodiment 1 of the present invention. Figure 3 This is a schematic diagram of a cascaded infrared and visible light image registration device for autonomous driving proposed in Embodiment 2 of the present invention. Detailed Implementation

[0021] To clearly illustrate the technical features of this solution, the invention will be described in detail below through specific embodiments and in conjunction with the accompanying drawings. The following disclosure provides many different embodiments or examples for implementing different structures of the invention. To simplify the disclosure of the invention, components and arrangements of specific examples are described below. Furthermore, reference numerals and / or letters may be repeated in different examples. This repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed. It should be noted that the components illustrated in the drawings are not necessarily drawn to scale. Descriptions of well-known components, processing techniques, and processes are omitted in this invention to avoid unnecessarily limiting the invention.

[0022] Example 1 Embodiment 1 of this invention proposes a cascaded infrared and visible light image registration method for autonomous driving, which solves the problem that infrared and visible light images are difficult to align accurately in autonomous driving scenarios due to physical parallax and non-rigid deformation caused by vehicle vibration.

[0023] Figure 1 This is a flowchart of the cascaded infrared and visible light image registration method for autonomous driving proposed in Embodiment 1 of the present invention; In step S1, a dual-light dataset containing physical parallax and non-rigid deformation is constructed; the dataset contains pairs of infrared and visible light images. A visible light camera module and an infrared camera module with identical parameters are fixedly connected; By using the visible light camera module and the infrared camera module that are fixedly connected, infrared images and visible light images are simultaneously acquired in a road scene that includes both distant and close views in order to obtain physical parallax data. During synchronous acquisition, the fixedly connected visible light camera module and infrared camera module are randomly vibrated to induce non-rigid deformation in the synchronously acquired infrared and visible light images, thereby obtaining non-rigid deformation data. The visible light camera module and infrared camera module in this application have identical parameters. The KM2010-3660 industrial-grade camera module, equipped with a 1 / 2.7-inch OV2710 CMOS sensor, can be selected. The visible light camera captures ambient color and texture, while the infrared camera operates in the 700-1100nm wavelength range to extract thermal radiation. Both have a pixel size of 3.0μm, an output resolution of 1920×1080, and are fixedly connected by a rigid bracket.

[0024] The data acquisition scenarios cover complex real-world driving environments, including complex road conditions, moving vehicles, dynamic lighting changes, and random vibrations caused by vehicle bumps. The acquisition process is conducted in road scenes containing both distant and close-up views to obtain image offset data caused by sensor physical parallax. Simultaneously, during the synchronous acquisition process, the vehicle's travel on uneven road surfaces causes random vibrations in the fixedly connected camera module, resulting in non-rigid deformation of the synchronously acquired infrared and visible light images.

[0025] To verify the alignment reliability, salient landmarks were manually annotated on 100 representative image pairs using a manual annotation protocol, and the average registration error was verified to be 1.26 pixels.

[0026] The scope of protection of this invention is not limited to the specific models and values ​​listed in Example 1. Those skilled in the art can make reasonable selections based on the actual situation.

[0027] In step S2, the infrared image is geometrically corrected using the visible light image as a reference to obtain a registered infrared image; Among them, geometric correction adopts a static feature recursive registration network, which is implemented in the following way: the visible light image and the infrared image are mapped to four-level multi-scale pyramid features with spatial resolution halved and channel number doubled step by step through two parallel coding branches respectively. Then, the deformation field is regressed step by step from the fourth level with the lowest spatial resolution through a coarse-to-fine recursive strategy. The original infrared image is spatially transformed using the first level deformation field with the highest spatial resolution. Figure 2 This is a diagram illustrating the overall framework for geometric correction of images using the static feature recursive registration network proposed in Embodiment 1 of the present invention. 1. Encoder Structure The static feature recursive registration network adopts an encoder-decoder structure, which includes two parallel encoding branches and a coarse-to-fine recursive decoding branch.

[0028] The visible light image is input into the first coding branch, and the infrared image is input into the second coding branch; The first coding branch and the second coding branch share weights and perform sequential downsampling on the input image to extract four-level multi-scale features. The spatial resolution of the first-level features is equal to the original image resolution. The number of feature channels is the number of basic channels. (In this embodiment) =16); The second-level features are obtained by downsampling the first-level features through a convolution with a stride of 2, resulting in a spatial resolution of [missing information]. The number of feature channels is , =32; The third-level features are obtained by downsampling the second-level features through a convolution with a stride of 2, resulting in a spatial resolution of [missing information]. The number of feature channels is , =64; The fourth-level features are obtained by downsampling the third-level features through a convolution with a stride of 2, resulting in a spatial resolution of [missing information]. The number of feature channels is , =128.

[0029] Each residual convolutional block contains a 3×3 convolutional layer, an instance normalization layer, and a LeakyReLU activation function layer. This design effectively filters out mode-specific radiation fluctuations and extracts stable geometric features.

[0030] 2. Decoder Structure The decoding phase employs a coarse-to-fine recursive strategy, starting from the fourth level with the lowest spatial resolution and progressively regressing the deformation field upwards. At each level, visible light image features and infrared image features are concatenated to form a combined semantic feature body, which is input into the Deformation Update Block (DUB) to calculate the deformation field of the current level. The fourth level directly calculates the deformation field from the combined semantic feature body; from the third to the first level, the deformation field of the previous level is upsampled and then input together with the combined semantic feature body of the current level into the Deformation Update Block, and a residual learning strategy is used to output the deformation field of the current level. Finally, the deformation field of the first level, which has the highest spatial resolution, is used to perform a spatial transformation on the original infrared image to generate a registered infrared image.

[0031] The decoder employs a coarse-to-fine recursive strategy, starting from the fourth level with the lowest spatial resolution and progressively regressing the deformation field upwards, ultimately outputting a registered infrared image. The core processing flow of the decoder includes four stages: fusion feature construction, semantic feature extraction, deformation update, and updated feature output.

[0032] ① Construction of fusion features At each level, the visible light image features extracted by the encoder are first fused with the infrared image features to form the fused features for that level, specifically: In the geometric correction, at each level Visible light image features Features of infrared images After concatenation, feature extraction is performed to form a combined semantic feature body. The expression is: ; in, Indicates feature concatenation operation; These correspond to the pyramid levels from the shallowest to the deepest. This indicates a feature extraction operation.

[0033] The fused features after splicing integrate the geometric information of the two modes, providing a basis for subsequent deformation field regression.

[0034] ② Semantic feature extraction The fused features contain rich geometric and modal information, but direct use in deformation field regression may be redundant. Therefore, further processing of the fused features using convolutional blocks is needed to extract semantic features related to geometric deformation.

[0035] Combine semantic feature bodies The input deformation update block consists of multiple convolutional blocks, each including a convolutional layer, an instance normalization layer, and a LeakyReLU activation function layer. Through layer-by-layer convolution processing, deep semantic features related to displacement field regression are extracted, while mode-specific radiation fluctuations are filtered out.

[0036] ③ Deformation and Update Based on semantic features, the decoder regresses the deformation field of the current layer through Deformation Update Blocks (DUBs). The deformation update employs a residual learning strategy, using different computation methods depending on the current layer's position. The specific operation is as follows: Fourth-level processing: Due to the lack of prior information, at the fourth level, the deformation field... Directly from the combined semantic feature body The expression obtained through the deformation update block is as follows: ; in, This indicates a fourth-level deformation update operation; This represents the combined semantic feature body of the fourth level.

[0037] Third to first level processing: For The hierarchy employs a residual learning strategy. The deformed field from the previous level is then... Upsample to the resolution of the current layer as the base guide; then combine the semantic features of the current layer. Input deformation update block Estimate the incremental deformation; superimpose the two values ​​to obtain the deformation field at the current level. Its expression is: ; in, Indicates according to the first Hierarchical combined semantic feature body Extract deformation update information; Indicates the first The result of upsampling the hierarchical deformation field.

[0038] The deformation update block is a pure convolutional structure, including: Multiple convolutional blocks are connected in sequence. Each convolutional block includes a convolutional layer, an instance normalization layer, and a LeakyReLU activation function layer. An unbiased 2D convolutional layer, connected to the output of the multiple convolutional blocks, is used to output a dense displacement residual map. The output dimension is... ,in, and These represent the height and width of the current layer feature map, respectively, and the number of channels 2 corresponds to the displacement in the X and Y directions, respectively.

[0039] ④ Update feature output The deformation field calculated at the current level As updated features, they are passed to the next (shallower) decoder layer as the basis for residual learning at that layer. Before passing them, an upsampling operation is required to match the resolution of the previous layer.

[0040] Utilizing the deformation field of the first level (the finest level) Perform spatial transformation on the original infrared image to generate a registered infrared image: ; in, Represented as the original infrared image, Transform represents a spatial transformation operation based on the deformation field, achieving pixel-level geometric correction through bilinear interpolation.

[0041] The input to the deformation update block is the concatenated combined semantic feature body, with the number of channels being the [number of channels]. The number of layers is twice the number of feature channels in that layer, or the number of channels in the fused, reduced-dimensional version. The output dimension is... The dense displacement residual diagram.

[0042] After calculating the deformed field at each level, a regular sampling grid is constructed using a grid generation mechanism. The predicted flow field is then superimposed on the regular grid to obtain the target coordinates. To facilitate subsequent bilinear interpolation operations (such as PyTorch's grid_sample function), the target coordinates need to be normalized to the [-1, 1] interval.

[0043] The normalization formula is: ; in, This represents the normalized target coordinates; Indicates the original target coordinates; This indicates the size of the current layer feature map in the corresponding direction; specifically, when calculating the normalized coordinate sequence in the horizontal direction (X-axis), size is the width of the current layer feature map; when calculating the vertical direction (Y-axis), size is the height of the current layer feature map.

[0044] For multimodal images with large nonlinear radiation differences, the training phase does not use absolute pixel intensity differences, but instead uses Local Normalized Cross-Correlation Loss (LNCC) to maximize structural similarity, so that the network focuses on geometric patterns; at the same time, spatial gradient smoothing regularization loss is applied to prevent topological folding of the deformation field and ensure the physical rationality of deformation.

[0045] The training of geometric correction uses locally normalized cross-correlation loss. With spatial gradient smoothing regularization loss The total loss function is: ; in, This represents the balance coefficient.

[0046] In step S3, the registered infrared image and the visible light image are feature-fused to output a fused image.

[0047] First, the registered infrared and visible light images are input into the encoder of the feature fusion network to extract multi-scale features. The encoder of the feature fusion network adopts a pyramid structure symmetrical to the geometric correction network, mapping the input image into multi-level features. The spatial resolution of each feature layer is halved at each level, while the number of channels is doubled at each level, thereby simultaneously capturing global structural information and local detail information.

[0048] Secondly, the extracted multi-scale features of the visible light image and the multi-scale features of the registered infrared image are fused at the corresponding levels. This embodiment adopts a stitching fusion method: the visible light image features and the registered infrared image features are stitched together in the channel dimension, and then dimensionality reduction fusion is performed through a convolutional layer to obtain the fused multi-scale features.

[0049] Then, the fused multi-scale features are input into the decoder of the feature fusion network to restore the spatial resolution step by step. The decoder uses skip connections, connecting the features of the corresponding level of the encoder with the features of the current level of the decoder to preserve multi-scale information and avoid loss of details.

[0050] Finally, the fused image is output. This fused image retains the rich texture details of the visible light image while incorporating the thermal radiation information of the infrared image, providing clear and complete environmental perception information in complex environments such as low light and severe weather.

[0051] The cascaded infrared and visible light image registration method for autonomous driving proposed in Embodiment 1 of this invention divides the image registration process into two independent stages: geometric correction and feature fusion. This effectively solves the problems of physical parallax and non-rigid deformation, improving the accuracy and robustness of infrared and visible light image fusion. This method can preserve useful information from cross-modal images to the greatest extent while ensuring geometric consistency, ensuring high-quality performance of the fused image in terms of detail and global structure. Compared with traditional methods, this invention has greater adaptability and flexibility, and can be widely applied to autonomous driving and other environmental perception systems.

[0052] The cascaded infrared and visible light image registration method for autonomous driving proposed in Embodiment 1 of this invention achieves a mutual information (MI) score of 6.321, an improvement of approximately 13.8% compared to the base network; a visual information fidelity (VIF) score of 1.305; a structural similarity (SSIM) score of 0.910; and a correlation coefficient (CC) score of 0.958. These data demonstrate the significant advantages of this method in preserving the geometric structure of road scenes, lane line edges, and vehicle contours. After removing the static feature recursive registration (SFRR) module (without geometric correction), the SSIM and CC scores decreased significantly by approximately 44% and 29%, respectively, fully demonstrating the core role of the registration module in eliminating ghosting artifacts caused by vehicle vibration and physical parallax. When processing 640×480 image pairs on an NVIDIA RTX 3090 GPU, the total number of parameters is only 5.98M, and the single-frame inference time is 0.031s (approximately 32FPS), fully meeting the real-time requirements of continuous perception in autonomous driving systems.

[0053] Table 1: Comparison of Horizontal Methods

[0054] Table 1 shows the performance comparison of the present invention with six existing mainstream image fusion methods (MMIF, DenseFuse, MemoryFusion, U2Fusion, SwinFusion, RFNNest) on four evaluation metrics.

[0055] Mutual Information (MI): A metric that measures the amount of information inherited by the fused image from the source images. A higher MI value indicates that the fused image retains more information from the source images. This invention achieves a MI of 6.321, approximately 13.8% higher than the second-ranked SwinFusion (5.553), demonstrating a significant advantage in information preservation.

[0056] Visual Information Fidelity (VIF): A metric that measures the consistency between the fused image and human visual perception. A higher VIF value indicates better visual quality of the fused image. This invention achieves a VIF of 1.305, far exceeding other methods (the second-ranked MMIF is 1.151), indicating that the fused image generated by this invention has better visual perception quality.

[0057] Structural Similarity (SSIM): Measures the degree of similarity between the fused image and the source image in terms of structural information. The closer the SSIM value is to 1, the more complete the structure is preserved. Our invention achieves 0.910, the highest among all methods, indicating that our invention has significant advantages in preserving the geometry of road scenes, lane line edges, and vehicle contours.

[0058] Correlation coefficient (CC): Measures the degree of linear correlation between the fused image and the source image. The higher the CC value, the stronger the correlation between the fused image and the source image. This invention achieves a CC of 0.958, the highest among all methods, indicating that this invention performs excellently in maintaining overall image consistency.

[0059] Conclusion: This invention outperforms existing methods in all four evaluation metrics, verifying the effectiveness of the proposed cascaded registration and fusion framework.

[0060] Table 2: Module ablation experiment.

[0061]

[0062] Table 2 shows ablation experiments conducted by removing or replacing the core modules of this invention to verify the contribution of each module. The variations are described below: w / o SFRR: Removes the static feature recursive registration module, i.e., without geometric correction, directly fuses the original infrared image with the visible light image.

[0063] SFRR-Single+HIBF: Replaces the static feature recursive registration network with a single-scale registration network, while retaining the HIBF module.

[0064] w / o HIBF: Remove the HIBF (High Frequency Information Enhancement Fusion) module.

[0065] w / o SFS: Removes the spatial feature selection module.

[0066] w / o IB: Remove the information bottleneck module.

[0067] Ours: This invention is a complete method, including all modules.

[0068] Key comparative analysis: The Importance of Geometric Correction: Comparing "w / o SFRR" and "Ours," it is evident that after removing the SFRR module, SSIM decreased from 0.910 to 0.812 (a decrease of approximately 10.8%), and CC decreased from 0.958 to 0.884 (a decrease of approximately 7.7%). This fully demonstrates the core role of the registration module in eliminating "ghosting artifacts" caused by vehicle vibration and physical parallax.

[0069] Advantages of multi-scale registration: Compared with “SFRR-Single+HIBF” and “Ours”, the single-scale registration variant is inferior to the present invention in all indicators, indicating that multi-scale pyramid feature extraction and the coarse-to-fine recursive strategy make important contributions to registration accuracy.

[0070] Contributions of each module: After removing any of the HIBF, SFS, or IB modules, all indicators decreased to varying degrees, indicating that each module made a positive contribution to the overall performance, and that the modules of this invention have a good synergistic effect.

[0071] Conclusion: Ablation experiments verified the effectiveness and necessity of each core module (SFRR, HIBF, SFS, IB) of the present invention, and proved the overall superiority of the cascaded registration and fusion framework.

[0072] Table 3: Horizontal comparison of inference time on NVIDIA RTX 3090 GPU

[0073] Table 3 shows the comparison between the present invention and six comparative methods in terms of model parameter count, computational cost (FLOPs), inference time, and frame rate (FPS).

[0074] Parameters: The number of parameters the model needs to train, measured in M ​​(millions). This invention has 5.98M parameters, which is moderate, less than MMIF (14.50M) and MemoryFusion (8.12M), slightly larger than Swin Fusion (2.85M), but much smaller than RFN-Nest (6.24M). This indicates that this invention has a reasonable design in terms of model complexity.

[0075] Computational cost (FLOPs): The number of floating-point operations required for the model to process a single image, measured in gigabytes (G). The computational cost of this invention is 78.4 G, lower than RFN-Nest (115.8 G), MemoryFusion (102.6 G), and MMIF (88.3 G), indicating that this invention has high computational efficiency.

[0076] Inference Time: The time required for the model to process a single image, measured in seconds. The inference time of this invention is 0.031s, second only to DenseFuse (0.015s) and U2Fusion (0.028s), and superior to RFN-Nest (0.072s), MMIF (0.051s), MemoryFusion (0.065s), and SwinFusion (0.124s).

[0077] Frame rate (FPS): The number of image frames that the model can process per second. This invention achieves 32.2 FPS, exceeding the real-time processing threshold of 30 FPS, second only to DenseFuse (66.6 FPS) and U2Fusion (35.7 FPS), and outperforming other methods.

[0078] Key Analysis: Real-time performance is met: This invention, with a processing speed of 32.2 FPS, fully meets the real-time requirements of continuous perception for autonomous driving systems (typically requiring ≥30 FPS).

[0079] Balancing accuracy and efficiency: While slightly slower than DenseFuse and U2Fusion, this invention leads in all fusion quality metrics (MI, VIF, SSIM, CC). Compared to high-precision methods such as RFN-Nest and MMIF, this invention maintains high accuracy while significantly improving inference speed (approximately 2.3 times that of RFN-Nest and 1.6 times that of MMIF).

[0080] Lightweight design: The number of parameters in this invention is only 5.98M, and the computational load is 78.4G. It achieves lightweight design while ensuring high precision, making it easy to deploy in automotive embedded devices.

[0081] Conclusion: This invention achieves a good balance between accuracy and efficiency, outperforming existing methods in terms of fusion quality while meeting the real-time requirements of autonomous driving systems, and has practical application value.

[0082] Example 2 The present invention also proposes a device, Figure 3 This invention relates to a cascaded infrared and visible light image registration method and device for autonomous driving, as proposed in Embodiment 2 of the present invention.

[0083] At the hardware level, electronic device 300 includes a processor 310, and optionally, an internal bus 320, a network interface 330, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or it may also include non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for its functions. The processor 310, network interface 330, and memory can be interconnected via an internal bus 320. This internal bus 320 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be categorized as an address bus, data bus, control bus, etc. For ease of illustration, only a single bidirectional arrow is used in this diagram, but this does not imply that there is only one bus or one type of bus. The memory is used to store programs. Specifically, the program can include program code, which includes computer operation instructions. The memory can include main memory 340 and non-volatile memory 350, and provides instructions and data to the processor 310. Processor 310 reads the corresponding computer program from non-volatile memory 350 into memory 340 and then runs it, forming a device for locating the target user at the logical level. Processor 310 executes the program stored in memory and specifically performs the following: Step S1: Construct a dual-light dataset containing physical parallax and non-rigid deformation; the dataset contains pairs of infrared and visible light images; Step S2: Using the visible light image as a reference, perform geometric correction on the infrared image to obtain a registered infrared image; the geometric correction is achieved in the following way: the visible light image and the infrared image are mapped to four-level multi-scale pyramid features with spatial resolution halved at each level and channel number doubled at each level through two parallel coding branches, and the deformation field is regressed step by step from the fourth level with the lowest spatial resolution through a recursive strategy from coarse to fine, and the original infrared image is spatially transformed using the first level deformation field with the highest spatial resolution; Step S3: Perform feature fusion between the registered infrared image and the visible light image to output a fused image.

[0084] Figure 1It can be applied to processor 310, or implemented by processor 310. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit in the processor or by instructions in the form of software. The processor mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. Of course, in addition to software implementation, the electronic device of this application does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0085] Example 3 The present invention also proposes a readable storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, implements the following method steps: Step S1: Construct a dual-light dataset containing physical parallax and non-rigid deformation; the dataset contains pairs of infrared and visible light images; Step S2: Using the visible light image as a reference, perform geometric correction on the infrared image to obtain a registered infrared image; the geometric correction is achieved in the following way: the visible light image and the infrared image are mapped to four-level multi-scale pyramid features with spatial resolution halved at each level and channel number doubled at each level through two parallel coding branches, and the deformation field is regressed step by step from the fourth level with the lowest spatial resolution through a recursive strategy from coarse to fine, and the original infrared image is spatially transformed using the first level deformation field with the highest spatial resolution; Step S3: Perform feature fusion between the registered infrared image and the visible light image to output a fused image.

[0086] Embodiment 3 of this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory that stores a computer program, which can be executed by a processor to complete the steps described in the aforementioned method. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0087] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks. Alternatively, if the integrated units of this application are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0088] The description of the relevant parts of the cascaded infrared and visible light image registration device for autonomous driving provided in Embodiment 2 of this application and the cascaded infrared and visible light image registration storage medium for autonomous driving provided in Embodiment 3 of this application can be found in the detailed description of the corresponding parts of the cascaded infrared and visible light image registration method for autonomous driving provided in Embodiment 1 of this application, and will not be repeated here.

[0089] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that the elements inherent in a process, method, article, or apparatus that includes a list of elements are included. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Additionally, portions of the technical solutions provided in the embodiments of this application that are consistent with the implementation principles of corresponding technical solutions in the prior art have not been described in detail to avoid excessive elaboration.

[0090] While specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art can make other modifications or variations based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A cascaded infrared and visible light image registration method for autonomous driving, characterized in that, Includes the following steps: Construct a two-light dataset that includes physical parallax and non-rigid deformation; the dataset contains pairs of infrared and visible light images; Using the visible light image as a reference, the infrared image is geometrically corrected to obtain a registered infrared image. The geometric correction is achieved in the following way: the visible light image and the infrared image are mapped to four-level multi-scale pyramid features with spatial resolution halved at each level and channel number doubled at each level through two parallel coding branches. The deformation field is regressed step by step from the fourth level with the lowest spatial resolution through a coarse-to-fine recursive strategy. The original infrared image is spatially transformed using the first level deformation field with the highest spatial resolution. The registered infrared image and the visible light image are feature-fused to output a fused image.

2. The cascaded infrared and visible light image registration method for autonomous driving according to claim 1, characterized in that, The process of acquiring physical parallax data and non-rigid deformation data includes: A visible light camera module and an infrared camera module with identical parameters are fixedly connected; By using the visible light camera module and the infrared camera module that are fixedly connected, infrared images and visible light images are simultaneously acquired in a road scene that includes both distant and close views in order to obtain physical parallax data. During synchronous acquisition, the fixedly connected visible light camera module and infrared camera module are randomly vibrated to cause non-rigid deformation in the synchronously acquired infrared and visible light images, so as to obtain non-rigid deformation data.

3. The cascaded infrared and visible light image registration method for autonomous driving according to claim 1, characterized in that, In geometric correction, the visible light image and the infrared image are mapped to a four-level multi-scale pyramid feature with spatial resolution halved at each level and the number of channels doubled at each level through two parallel coding branches, specifically: The visible light image is input into the first coding branch, and the infrared image is input into the second coding branch; The first coding branch and the second coding branch share weights and perform sequential downsampling on the input image to extract four-level multi-scale features; The spatial resolution of the first-level features is equal to the original image resolution. The number of feature channels is the number of basic channels. The second-level features are obtained by downsampling the first-level features through a convolution with a stride of 2, resulting in a spatial resolution of [missing information]. The number of feature channels is The third-level features are obtained by downsampling the second-level features through a convolution with a stride of 2, resulting in a spatial resolution of [missing information]. The number of feature channels is The fourth-level features are obtained by downsampling the third-level features through a convolution with a stride of 2, resulting in a spatial resolution of [missing information]. The number of feature channels is .

4. The cascaded infrared and visible light image registration method for autonomous driving according to claim 3, characterized in that, In the geometric correction, at each level Visible light image features Features of infrared images After concatenation, feature extraction is performed to form a combined semantic feature body. The expression is: ； in, Indicates feature concatenation operation; These correspond to the pyramid levels from the shallowest to the deepest. This indicates a feature extraction operation.

5. The cascaded infrared and visible light image registration method for autonomous driving according to claim 1, characterized in that, A recursive strategy, progressing from coarse to fine, is employed to progressively regress the deformation field, starting from the fourth level with the lowest spatial resolution. The first level deformation field, with the highest spatial resolution, is then used to perform a spatial transformation on the original infrared image. Specifically: In the fourth level, which has the lowest spatial resolution, the combined semantic feature volume of the fourth level is input into the deformation update block to directly calculate the deformation field of the fourth level. For the third level to the first level, the deformation field of the previous level is upsampled and then superimposed with the deformation information estimated by the deformation update block of the current level based on the combined semantic feature body of the current level to obtain the deformation field of the current level. Repeat the above steps to obtain the deformation fields of the third, second, and first levels in sequence; The original infrared image is spatially transformed using the first-level deformation field with the highest spatial resolution to generate a registered infrared image.

6. The cascaded infrared and visible light image registration method for autonomous driving according to claim 4 or 5, characterized in that, At the fourth level, the deformation field Directly from the combined semantic feature body The expression obtained through the deformation update block is as follows: ； in, This indicates a fourth-level deformation update operation; This represents the combined semantic feature body of the fourth level.

7. The cascaded infrared and visible light image registration method for autonomous driving according to claim 4 or 5, characterized in that, for Hierarchical, using a residual learning strategy to calculate the deformation field Its expression is: ； in, Indicates according to the first Hierarchical combined semantic feature body Extract deformation update information; Indicates the first The result of upsampling the hierarchical deformation field.

8. The cascaded infrared and visible light image registration method for autonomous driving according to claim 5, characterized in that, The deformation update block includes: The stitching operation layer is used to stitch visible light image features with infrared image features; Multiple convolutional blocks are connected in sequence. Each convolutional block includes a convolutional layer, an instance normalization layer, and a LeakyReLU activation function layer. An unbiased two-dimensional convolutional layer is connected to the output of the multiple convolutional blocks to output a dense displacement residual map.

9. The cascaded infrared and visible light image registration method for autonomous driving according to claim 1, characterized in that, The geometric correction is trained using locally normalized cross-correlation loss. With spatial gradient smoothing regularization loss The total loss function is: ； in, This represents the balance coefficient.

10. A cascaded infrared and visible light image registration device for autonomous driving, characterized in that, include: At least one processor and memory, The memory stores a computer program that, when executed by the at least one processor, implements the cascaded infrared and visible light image registration method for autonomous driving as described in any one of claims 1 to 9.

Images