Visible light-to-infrared image translation method based on circulation consistency

Abstract

The invention discloses a visible light-to-infrared image translation method based on circulation consistency, and belongs to the technical field of image processing, and the method comprises the steps: firstly training a reverse generator based on a multi-granularity U-shaped generator to complete the reconstruction from an infrared image to a visible light image; then freezing the generator, training a forward generator with the same structure to translate a visible light image to an infrared image, and introducing a cyclic consistency constraint; and finally, optimizing the model by combining the structural similarity loss and the semantic consistency loss. The core of the generator is that an encoder of the generator adopts a two-dimensional convolutional layer and a visual state space block which are sequentially connected in series, and a decoder integrates a transposed convolutional layer and a multi-scale visual state space block. According to the method, efficient fusion of long-range dependence modeling and local feature extraction is realized, and the fidelity, detail richness and semantic consistency of cross-modal image translation are remarkably improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a visible light to infrared image translation method based on cycle consistency. BACKGROUND

[0002] For intelligent traffic monitoring or automatic driving systems, the image translation task of visible light to infrared image is involved, and the existing technology mainly relies on a model based on a convolutional neural network (CNN) to realize cross-modal mapping. Such a method can effectively extract local detail features of an image through its local receptive field characteristics, maintain high local consistency and texture authenticity in the image-to-image conversion process, has been applied in various practical scenarios, and has achieved certain technical effects. With the popularization of visual Transformer in the field of image processing, some researches begin to introduce attention mechanisms to enhance the modeling ability of the model to global information, capture long-range dependencies in the image through its self-attention structure, and further improve the adaptability of structural features and geometric transformation in the image translation task.

[0003] However, the multi-modal visual perception requirements of intelligent traffic monitoring or automatic driving systems still have obvious limitations in the existing technology. The CNN-based method is limited by its local receptive field and cannot effectively model the global semantic information and long-range spatial dependencies in the image, resulting in structure distortion or content inconsistency in cross-modal conversion in complex scenes. On the other hand, although the Transformer-based method performs well in global information modeling, the computational complexity of its self-attention mechanism grows quadratically with the image resolution. Therefore, when processing multi-modal perception tasks of intelligent traffic monitoring or automatic driving systems, the Transformer-based method faces significant computational burden and scalability bottleneck in the process of converting high-resolution aerial visible light images to infrared images, thereby limiting its widespread deployment in practical systems. SUMMARY

[0004] To solve the above technical problems, the application provides a visible light to infrared image translation method based on cycle consistency to solve the problems existing in the above-mentioned prior art.

[0005] To achieve the above-mentioned purpose, the application provides a visible light to infrared image translation method based on cycle consistency, comprising: S1, training an inverse generator, the inverse generator being constructed based on a multi-granularity U-shaped generator, and being used for translating an input infrared image into a visible light image; S2. Freeze the inverse generator and train the forward generator, which is built based on the multi-granularity U-shaped generator and is used to translate visible light images acquired by intelligent traffic monitoring or autonomous driving systems into infrared images; wherein, the output of the forward generator is input to the frozen inverse generator to reconstruct the visible light image, and the cycle consistency loss is calculated based on the reconstructed visible light image and the original input visible light image. The encoder of the multi-granularity U-shaped generator includes at least one multi-granularity encoder block, which sequentially includes a two-dimensional convolutional layer for extracting local features and a visual state space block for capturing global dependencies; the decoder of the multi-granularity U-shaped generator includes at least one multi-granularity decoder block, which includes a transposed convolutional layer for upsampling and a multi-scale visual state space block for multi-scale global feature fusion. S3. Calculate a loss function based on the infrared image generated by the forward generator and the real infrared image. The loss function includes at least structural similarity loss and semantic consistency loss.

[0006] Preferably, the visual state space block is a two-dimensional visual state space block based on a selective scanning mechanism.

[0007] Preferably, the processing of the two-dimensional visual state space block includes: performing layer normalization, linear projection, depth convolution, SiLU activation function, two-dimensional selective scanning operation, and another linear projection on the input features in sequence, and adding the input features to the output features of the two-dimensional selective scanning operation through residual connection.

[0008] Preferably, the processing of the multi-granularity decoder block includes: sequentially applying the transposed convolutional layer, layer normalization, the multi-scale visual state space block, another layer normalization, and the multi-scale feedforward aggregation network to the input features, and adding the output features of the multi-scale visual state space block to the output features of the multi-scale feedforward aggregation network through residual connections.

[0009] Preferably, the structural similarity loss is calculated based on the brightness, contrast, and structural information of the generated fake infrared image and the real infrared image.

[0010] Preferably, the semantic consistency loss is obtained by extracting features of the generated fake infrared image and the real infrared image through a pre-trained deep network, and calculating the error between the features.

[0011] Preferably, the multi-granularity U-shaped generator has a skip connection between the encoder and the decoder, which is used to pass the features extracted by the encoder to the decoder at the corresponding level.

[0012] Preferably, the bottleneck layer of the multi-granularity U-shaped generator consists of a multi-granularity encoder block and a multi-granularity decoder block.

[0013] Preferably, the two-dimensional selective scanning operation employs a four-directional scanning mechanism.

[0014] Preferably, the multi-scale feedforward aggregation network extracts multi-scale features through parallel convolution operations with multiple kernel sizes.

[0015] Compared with the prior art, the present invention has the following advantages and technical effects: This invention employs a technical solution that integrates a multi-granularity encoder block comprising a two-dimensional convolutional layer and a visual state space block sequentially within the encoder of a multi-granularity U-shaped generator, and integrates a transposed convolutional layer and a multi-scale visual state space block in the decoder. This enables the network to synchronously and iteratively integrate local detail information and global contextual dependencies of an image during image translation. This structure, on the one hand, utilizes the two-dimensional convolutional layer to ensure effective extraction of local features, and on the other hand, leverages the linear computational complexity advantage of the visual state space block to overcome the computational bottleneck faced with high-resolution images, thereby achieving efficient and robust modeling of the global geometric structure of the image.

[0016] This invention constructs a closed-loop image translation system by employing a two-stage training strategy that includes freezing the inverse generator and a cycle consistency constraint. This mechanism mandates that the infrared image translated by the forward generator can be accurately reconstructed back into the original visible light image via the inverse generator. This cycle consistency constraint effectively strengthens the consistency between the generated image and the source image in terms of content and structure, significantly improving the visual realism of the output and the accuracy of cross-modal mapping.

[0017] This invention, by jointly optimizing structural similarity loss and semantic consistency loss in the loss function, further constrains the generation process from the overall image structure level and the deep semantic level, building upon pixel-level matching. This enables the generated infrared image to not only reduce blur and artifacts but also better preserve the texture details, structural contours, and high-level semantic information of the input image, thereby improving both visual quality and applicability to downstream tasks. Attached Figure Description

[0018] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of a visible light to infrared image translation method based on cycle consistency, according to an embodiment of the present invention. Detailed Implementation

[0019] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0021] Example 1 like Figure 1 As shown, this embodiment provides a visible light to infrared image translation method based on cycle consistency, which overcomes the limitations of existing image conversion methods in terms of long-range dependency modeling and computational complexity. Specifically, it includes: S1. Train the reverse generator, which is built based on a multi-granularity U-shaped generator and is used to translate infrared images acquired by intelligent traffic monitoring or autonomous driving systems into visible light images. Specifically, an inverse generator (infrared → visible light) is trained to learn the mapping relationship between infrared and visible light images through adversarial training.

[0022] To meet the multimodal perception needs of intelligent traffic monitoring or autonomous driving systems, if the spatially paired visible light images are Infrared image is Then first use MGU ( This involves translating an infrared image into a visible light image to obtain the generated visible light image. Secondly, a discriminator is used to determine the currently generated visible light. Determining the authenticity of an image. This process can be represented as: (1) (2) in, To pass The feature map generated by the discriminator, where the value at each location represents Is the content real or generated, or something similar? For the discriminator pair The discriminant's feature map after discrimination. The loss of the discriminator can then be calculated using these two feature maps.

[0023] It includes two processes: encoding and decoding. The encoding stage mainly consists of a multi-granular encoder block (MGEB) that combines the advantages of Mamba and CNN, and an encoder block consisting only of a regular CNN. Composition, in which This indicates the number of encoder blocks. The decoding process includes multi-granularity decoder blocks (MGDB) with multi-scale parallel convolutions and ordinary decoder blocks. . It is a PatchGAN structure that judges whether an image patch is true or false. This structure classifies each patch on an image to determine whether it is true or false, then averages all the results and finally outputs the result. The network structure is exactly the same as that of the forward generator in step S2.

[0024] S2. Freeze the inverse generator and train the forward generator, which is built based on the multi-granularity U-shaped generator and is used to translate the input visible light image into an infrared image; wherein, the output of the forward generator is input to the frozen inverse generator to reconstruct the visible light image, and the cycle consistency loss is calculated based on the reconstructed visible light image and the original input visible light image. The encoder of the multi-granularity U-shaped generator includes at least one multi-granularity encoder block, which sequentially includes a two-dimensional convolutional layer for extracting local features and a visual state space block for capturing global dependencies; the decoder of the multi-granularity U-shaped generator includes at least one multi-granularity decoder block, which includes a transposed convolutional layer for upsampling and a multi-scale visual state space block for multi-scale global feature fusion.

[0025] Furthermore, the processing of the multi-granularity decoder block includes: sequentially applying the transposed convolutional layer, layer normalization, the multi-scale visual state space block, another layer normalization, and the multi-scale feedforward aggregation network to the input features, and adding the output features of the multi-scale visual state space block to the output features of the multi-scale feedforward aggregation network through residual connections.

[0026] Furthermore, the multi-granularity U-shaped generator has a skip connection between the encoder and the decoder, which is used to pass the features extracted by the encoder to the decoder at the corresponding level.

[0027] Furthermore, the bottleneck layer of the multi-granularity U-shaped generator is composed of a multi-granularity encoder block and a multi-granularity decoder block.

[0028] Furthermore, the multi-scale feedforward aggregation network extracts multi-scale features through parallel convolution operations with multiple kernel sizes.

[0029] Furthermore, the visual state space block is a two-dimensional visual state space block based on a selective scanning mechanism.

[0030] Furthermore, the processing of the two-dimensional visual state space block includes: performing layer normalization, linear projection, depth convolution, SiLU activation function, two-dimensional selective scanning operation, and another linear projection on the input features in sequence, and adding the input features to the output features of the two-dimensional selective scanning operation through residual connection.

[0031] Furthermore, the two-dimensional selective scanning operation employs a four-directional scanning mechanism.

[0032] Specifically, the inverse generator is frozen, the forward generator (visible light → infrared) is trained, and the model is optimized under the cycle consistency constraint to ensure a high degree of consistency between the generated images and the real data.

[0033] After obtaining the reverse generator Subsequently, in the current step, only the forward generator for visible-to-infrared image translation is trained. ,and The parameters are frozen. The network structure is exactly the same as the MGU structure in step S1. For the input visible light image... pass Output composite image ,Then Used for reconstruction The generated infrared image as The corresponding visible light image at input The reconstruction loss is then calculated using the input. This process can be represented as: (3) (4) (5) Specifically in In the input image In the encoding process, features are first obtained through two consecutive MGEB operations. In each MGEB, features are first extracted locally through convolution, and then captured globally by two consecutive visual state space blocks (VSS) with global receptive fields and linear complexity to obtain the final features. This process can be represented as: (6) (7) in, This represents a 2D convolution operation. This represents two consecutive state space blocks. Taking the second MGEB as an example, the input features... First, local features are extracted through two-dimensional convolution. : (8) Then, After undergoing a layer normalization process, the data enters the SS2D block and utilizes a unique four-directional scanning mechanism to extract global information and obtain features. Specifically, in the SS2D module, the input features... First, the layer passes through a linear layer and a depthwise convolutional layer, followed by activation using the SiLU activation function. Then, a four-directional selective scanning mechanism (SS2D) is used to capture long-range dependencies. Finally, the output is obtained after passing through another normalization layer and a linear layer. The process is as follows: (9) in, Representation layer normalization, This represents a linear projection operation used to double the number of channels. It is a deep convolutional layer with a kernel size of 3. It is the SiLU activation function. Indicates 2D selective scan operation. This represents another linear projection operation used to halve the number of channels. Simultaneously, a residual connection is used to compensate for information loss during processing within the SS2D block, adding the input and output of the SS2D block to obtain... The process is as follows: (10) in, These are intermediate features obtained after residual connection.

[0034] To further enhance the features, After layer normalization, the data enters the feedforward aggregation network (FFN), and then the output features are summed. The final output is generated by adding the residuals. This can be expressed as a formula: (11) Subsequently, features Input encoder blocks Until the bottleneck layer is reached, the features are obtained. ,Right now: (12) in, This is the final output of the encoding process in the bottleneck layer.

[0035] The bottleneck layer consists of an MGEB and an MGDB, characterized by Features are obtained after the bottleneck layer. This process at the bottleneck layer can be represented as: (13) (14) in, This represents the intermediate features of the bottleneck layer. This is the output of the bottleneck layer. The number of channels and spatial resolution remain constant in the bottleneck layer.

[0036] During the decoding process, skip links are used to introduce features from the encoding process into the decoding process, helping the network generate an infrared image from highly compressed features. Specifically, features from the encoding process are first concatenated with features input to the decoding module along the channel dimension, and the concatenated features are then fed into... In each decoder, the resolution of the features is gradually restored through transposed convolutions until the features are incorporated into our shallow MGDB. This process can be represented as: (15) (16) in, This means concatenating two features along the channel dimension. These are features obtained during the decoding process through skip connections, which are then used as input to the next module. These are the features processed by the decoder, used to concatenate with the output of the corresponding module along the channel dimension during the encoding process, serving as the input to the next module. MGDB includes transposed convolutions and multi-scale visual state space blocks (MSVSS) to improve feature resolution, which are then processed by the decoder blocks. Features after In general, the infrared image is obtained after two consecutive MGDB operations. This process can be represented as: (17) (18) in, This represents the output features after MGDB processing. Represents a spatial block of multi-scale visual states. This represents a transposed convolution. Taking the last MGDB module as an example, it processes the input features... First, transpose convolution is used to restore the image and improve its spatial resolution, thus obtaining features. Then, layer normalization is performed, and the SS2D two-dimensional scanning mechanism is used. We perform global modeling to provide a global receptive field and utilize residual connections to obtain enhanced features. This is followed by a layer normalization layer and a multi-scale feedforward aggregation network (MS-FFN). MS-FFN extracts image features at different scales by utilizing three parallel convolution operations with different kernel sizes. Simultaneously, residual connections are used to sum the input and output of MS-FFN to obtain the final output infrared image. This process can be represented as: (19) (20) (twenty one) S3. Calculate a loss function based on the infrared image generated by the forward generator and the real infrared image. The loss function includes at least structural similarity loss and semantic consistency loss.

[0037] Furthermore, the structural similarity loss is calculated based on the brightness, contrast, and structural information of the generated fake infrared image and the real infrared image.

[0038] Furthermore, the semantic consistency loss is obtained by extracting features of the generated fake infrared image and the real infrared image through a pre-trained deep network, and calculating the error between the features.

[0039] Building upon the smoothing loss that reduces blur, structural similarity loss and semantic consistency loss, which focus on both local details and global features, are added to fully preserve high-level features and ensure cross-modal structural and content consistency. First, the structural similarity between the real and generated images is calculated. The calculation of structural similarity can be represented as: (twenty two) in, and It is a constant. and Let the mean and variance of the real image and the generated fake image be represented, respectively. It is their covariance. Thus, the structural similarity loss is obtained: (twenty three) The semantic consistency loss is achieved through a pre-trained network. (e.g., VGG16) After extracting image features, the error between the real image and the generated image can be calculated as follows: (twenty four) in, This represents the total number of elements in the feature map. This indicates that the pre-trained network The extracted feature vector of the th Each element is used to minimize this loss, making the feature representation in the generated image as close as possible to the feature representation in the real image.

[0040] To verify the effectiveness of the method proposed in this embodiment, it was validated on AVIID-1. To ensure experimental consistency, all input images were uniformly adjusted to a resolution of 256×256 pixels. Seven standard metrics were used: Mean Absolute Error (MAE), Mean Squared Error (MSE), Structural Similarity (SSIM), Multi-Scale Structural Similarity (MS-SSIM), and LPIPS. All experiments were performed on an NVIDIA GeForce GTX 3090 GPU with a batch size of 4. Regarding experimental settings, Pix2Pix, BicycleGAN, CycleGAN, GCGAN, CUT, DCLGAN, and CycleMamba all used the Adam optimizer with a learning rate of 0.0002 and were trained for 1000 epochs; DRIT and MSGAN also used the Adam optimizer with a learning rate of 0.0001 and were trained for 1200 epochs; KAN-CUT maintained its default parameter configuration. This unified experimental design ensured the fairness and reliability of the comparison results. To ensure the robustness of the evaluation, a comprehensive testing protocol was implemented: 150 test images were randomly selected from each dataset, and this process was repeated five times to construct five independent test sets. For single-input, multiple-output methods, one output was uniformly and randomly selected for the final evaluation. The final results are reported as mean ± standard deviation, reflecting both performance and the measurement reliability of multiple trials.

[0041] The quantitative comparison results on AVIID-1 are shown in Table 1; the quantitative comparison results on VEDAI are shown in Table 2. As can be seen from the results in Tables 1 and 2, the method proposed in this embodiment achieves optimal performance across all metrics, significantly improving the quality of the generated infrared images.

[0042] Table 1

[0043] Table 2

[0044] The beneficial effects of this embodiment: This embodiment addresses the scene visual perception requirements of intelligent traffic monitoring and autonomous driving systems under low light or severe weather conditions. First, it trains an inverse generator to reconstruct infrared to visible light images. Then, it freezes the generator and strengthens the cyclic consistency through the cyclic conversion of visible light → infrared → visible light, ensuring that the generated images are highly consistent with the real data. This embodiment, based on the effective suppression of image blur by the basic pixel-level loss, further introduces structural similarity loss (SSIM) and semantic consistency loss to fully preserve high-level features and ensure cross-modal structural and content consistency. This embodiment overcomes the problems of existing methods, such as the difficulty of capturing long-range dependencies based on CNN and the high computational complexity based on Transformer, by combining the advantages of Mamba model and CNN, and achieves efficient cross-modal image mapping.

[0045] The verification process demonstrates that this embodiment outperforms existing methods in generalization under different scenarios and imaging conditions. Experimental results on multiple public datasets consistently prove its superiority, especially in cross-modal image conversion tasks, where it can adapt to diverse input conditions and generate high-quality output.

[0046] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A visible light to infrared image translation method based on cycle consistency, characterized in that, Includes the following steps: S1. Train the inverse generator, which is built based on a multi-granularity U-shaped generator and is used to translate infrared images acquired by intelligent traffic monitoring or autonomous driving systems into visible light images. S2. Freeze the inverse generator and train the forward generator, which is built based on the multi-granularity U-shaped generator and is used to translate the input visible light image into an infrared image; wherein, the output of the forward generator is input to the frozen inverse generator to reconstruct the visible light image, and the cycle consistency loss is calculated based on the reconstructed visible light image and the original input visible light image. The encoder of the multi-granularity U-shaped generator includes at least one multi-granularity encoder block, which sequentially includes a two-dimensional convolutional layer for extracting local features and a visual state space block for capturing global dependencies; the decoder of the multi-granularity U-shaped generator includes at least one multi-granularity decoder block, which includes a transposed convolutional layer for upsampling and a multi-scale visual state space block for multi-scale global feature fusion. S3. Calculate a loss function based on the infrared image generated by the forward generator and the real infrared image. The loss function includes at least structural similarity loss and semantic consistency loss.

2. The method according to claim 1, characterized in that, The visual state space block is a two-dimensional visual state space block based on a selective scanning mechanism.

3. The method according to claim 2, characterized in that, The processing of the two-dimensional visual state space block includes: performing layer normalization, linear projection, depth convolution, SiLU activation function, two-dimensional selective scanning operation, and another linear projection on the input features in sequence, and adding the input features to the output features of the two-dimensional selective scanning operation through residual connection.

4. The method according to claim 1, characterized in that, The processing of the multi-granularity decoder block includes: sequentially applying the transposed convolutional layer, layer normalization, the multi-scale visual state space block, another layer normalization, and the multi-scale feedforward aggregation network to the input features, and adding the output features of the multi-scale visual state space block to the output features of the multi-scale feedforward aggregation network through residual connections.

5. The method according to claim 1, characterized in that, The structural similarity loss is calculated based on the brightness, contrast, and structural information of the generated fake infrared image and the real infrared image.

6. The method according to claim 1, characterized in that, The semantic consistency loss is obtained by extracting features from the generated fake infrared image and the real infrared image through a pre-trained deep network, and calculating the error between the features.

7. The method according to claim 1, characterized in that, The multi-granularity U-shaped generator has a skip connection between the encoder and the decoder, which is used to pass the features extracted by the encoder to the decoder at the corresponding level.

8. The method according to claim 1, characterized in that, The bottleneck layer of the multi-granularity U-shaped generator consists of a multi-granularity encoder block and a multi-granularity decoder block.

9. The method according to claim 3, characterized in that, The two-dimensional selective scanning operation employs a four-directional scanning mechanism.

10. The method according to claim 4, characterized in that, The multi-scale feedforward aggregation network extracts multi-scale features through parallel convolution operations with multiple kernel sizes.

Images