Zero-shot infrared image colorization method and system based on multi-level representation fusion

Through the multi-level characterization and fusion method, the frequency domain feature reconstruction of infrared images is combined with global and instance-level features, which solves the problem of dependency and color confusion in infrared image shading, and realizes high-quality infrared image shading and cross-modality Adaptability.

CN118823175BActive Publication Date: 2024-11-19NANJING UNIV OF SCI & TECH
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411298137.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-18
Publication Date
2024-11-19
Estimated Expiration
2044-09-18

AI Technical Summary

Technical Problem

Existing infrared image shading methods are dependent in cross-modal data processing, especially the coloring quality of long-wave infrared images is poor, and it is difficult to avoid confusion between background and object colors in complex scenes.

Method used

Using the zero-sample infrared image shading method based on multi-level characterization fusion, the combination of global image shading network, instance shading module and global re-shading module is used to perform end-to-end instance-level shading using visible light images to eliminate dependence on infrared data, and Color smoothing is achieved through frequency domain feature reconstruction.

Benefits of technology

It significantly improves the performance of infrared image shading algorithm, can extract color distributions that meet the instance characteristics in complex scenes, improves the shading quality of long-wave infrared images, and reduces the color confusion between background and object.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118823175B_ABST
    Figure CN118823175B_ABST
Patent Text Reader

Abstract

The present invention discloses a zero-shot infrared image colorization method and system based on multi-level representation fusion, the method comprising: using a global image colorization network to colorize the entire infrared image; using a pre-trained instance segmentation model to extract instance-level features from the infrared image, and generate corresponding instance segmentation pseudo-labels, which are sent to the instance colorization network to achieve end-to-end instance colorization; the instance colorization results and the overall colorization results are input into a global recoloring module for coupled reconstruction to obtain an optimized infrared colorization result. The present invention only uses visible light images for training, realizes cross-modal zero-shot learning, and eliminates dependence on infrared data sets; the present invention can further optimize the performance of infrared colorization algorithms and extract color distributions that meet instance characteristics from complex scenes containing multiple objects.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of infrared colorization technology, and in particular relates to a zero-sample infrared image colorization method and system based on multi-level representation fusion. Background Art

[0002] In the field of digital image processing today, infrared colorization technology has attracted much attention and is profoundly affecting the development direction of security monitoring, medical imaging, and assisted driving. The core principle of this technology is to convert infrared spectral information into visible spectrum, making it easier for humans to perceive and understand. Therefore, infrared image colorization is considered to be an effective way to improve scene perception and interpretation capabilities. However, there is an irregular mapping relationship between the brightness and chromaticity of a given single-channel infrared image and a visible light image, which leads to non-unique results of infrared colorization. In addition, the infrared colorization task has a strong dependence on cross-modal data. Therefore, the infrared colorization task remains a challenging and interesting research problem.

[0003] Traditional image colorization methods mainly include interactive colorization techniques and reference example-based colorization techniques. In interactive colorization techniques, users provide rough color indications by scribbling inside the region. Using these user-provided constraints will automatically propagate the color to other pixels in the image sequence. The main disadvantage of this type of method is that there is a lot of user intervention in the colorization process, and the colorization quality depends largely on the user's scribbling experience. In contrast, in reference example-based colorization techniques, users only provide reference images with similar color distribution to the target image. Although this saves a lot of user effort, it still depends on the color availability of the reference example and requires fine tuning to achieve a more ideal colorization effect. However, this type of method still has a more obvious color bleeding phenomenon. Traditional image colorization is limited to certain assumptions, such as directly using the grayscale of the input single-channel image as the brightness of the colored image and only estimating the chromaticity part of the image. However, traditional image colorization methods are limited by people's subjective perception of specific scenes and are difficult to expand. In addition, infrared image colorization requires simultaneous estimation of brightness and chromaticity, so traditional image colorization methods are no longer applicable to this situation. With the continuous development of deep learning, methods for infrared image colorization combined with deep learning networks have emerged in an endless stream. These methods can be roughly divided into infrared colorization based on paired data and unpaired data. The training process of infrared colorization based on paired data is relatively direct, which is essentially an input-to-output reconstruction process. However, this one-to-one mapping relationship makes the model generalization poor, and it is difficult to obtain a large amount of paired data in practical applications. In contrast, infrared colorization based on unpaired data further integrates the one-to-one mapping relationship between images into a mapping relationship between two set features, which reduces the pressure of paired data acquisition and improves the generalization performance of the model. Although the above methods have achieved good results, there is still a problem of dependence on infrared data during training, which directly leads to poor model performance in cross-modal colorization. The zero-shot infrared colorization network first proposed a cross-modal zero-shot learning infrared colorization method. This method gets rid of the dependence on infrared data during training, so that infrared data only participates in the model reasoning stage. This method has good cross-modal adaptability and also has a wider spectrum colorization capability. Although the effectiveness of this method has been verified on a multi-band infrared dataset, the quality of long-wave infrared colorization is poor compared with the colorization effects of near-infrared and mid-wave infrared. Although the structural details of the background and object instances are preserved to the greatest extent during the long-wave infrared colorization process, the color representation of the object instances is disturbed by the contextual information in the global image, especially in complex scenes containing multiple objects (such as the green of trees appearing on a car, etc.). At the same time, because the long-wave infrared images have low contrast and blurred edges, there are obvious differences between them and visible light images. The colorization effect of the object instances cannot be improved by decoupling and reconstructing the frequency domain features alone.How to further exploit the similarities between visible and infrared modality features to achieve accurate instance-level colorization and avoid color confusion with the background remains underexplored. Summary of the invention

[0004] The object of the present invention is to provide a zero-sample infrared image colorization method and system based on multi-level representation fusion.

[0005] To achieve the above object, the present invention provides the following technical solution: a zero-sample infrared image colorization method based on multi-level representation fusion, comprising:

[0006] Use the global image colorization network to colorize the entire infrared image;

[0007] Using the pre-trained instance segmentation model, instance-level features are extracted from infrared images and corresponding instance segmentation pseudo labels are generated, which are then sent to the instance colorization network to achieve end-to-end instance colorization.

[0008] The instance shading results and the overall shading results are input into the global reshading module for coupled reconstruction to obtain the optimized infrared shading results.

[0009] A zero-sample infrared image colorization system based on multi-level representation fusion is used to implement the above method. The system includes:

[0010] Global image colorization network, used for preliminary colorization of global infrared images;

[0011] An instance colorization module is used to extract instance features of objects from infrared images and feed the generated instance masks to the instance colorization network to supplement the local coloring details of the object instances;

[0012] The global recoloring module couples the output results of the two branches of the global coloring network and the instance coloring module, and reconstructs the color distribution of the object instance in the global image.

[0013] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the program.

[0014] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention combines the contextual information of the image, couples the global color representation after colorization with the color representation of the object instance, and extracts the coupled frequency domain features for further reconstruction to obtain a color-smooth infrared colorization result. The present invention only uses visible light images for training, realizes cross-modal zero-sample learning, and eliminates the dependence on infrared data sets. Experimental results show that the method of the present invention can further optimize the performance of the infrared colorization algorithm and extract color distribution that conforms to instance characteristics from complex scenes containing multiple objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for describing the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 Schematic diagram of the limitations of existing infrared image colorization methods.

[0018] Figure 2 Schematic diagram of an overview of the method of the present invention.

[0019] Figure 3 Schematic diagram of the network for coloring the global image.

[0020] Figure 4 Colorize the network diagram for the example.

[0021] Figure 5 This is a schematic diagram of the instance generator structure.

[0022] Figure 6 Schematic diagram of the multi-scale discriminator structure.

[0023] Figure 7 This is a schematic diagram of the global recoloring module.

[0024] Figure 8 It is a schematic diagram for visual comparison between the existing infrared image colorization method and the method of the present invention.

[0025] Fig. 9 Schematic diagram of the coloring optimization results of different instance coloring networks in the instance coloring module.

[0026] Fig.10 A schematic diagram of the shading effect before and after processing by the global shading network in the global recoloring module.

[0027] Fig.11 A visual comparison of the vehicle detection results of the YOLOv5 model on the FLIR dataset.

[0028] Fig.12 Schematic diagram of the visual comparison before and after optimization of near-infrared modal coloring on the MSOD dataset.

[0029] Fig.13 This is a visual comparison of the vehicle detection results of the YOLOv5 model on the MSOD dataset. DETAILED DESCRIPTION

[0030] In order to solve the problems existing in the existing methods in the background technology, the present invention proposes a zero-shot infrared image colorization method and system based on multi-level representation fusion. The core idea of ​​this method is to learn the color representation of the image global and object instance separately under the premise of zero-shot learning, and further reconstruct the decoupled frequency domain features in combination with the context information of the image to achieve a significant improvement in the infrared instance colorization performance. First, the zero-shot infrared colorization network has proved the effectiveness of mining the similarity between visible light and infrared modal features for infrared colorization tasks (Wei C, Chen H, Bai L, et al. Infrared colorization with cross-modality zero-shot learning[J]. Neurocomputing, 2024: 127449.). To this end, we extract the high-frequency feature components of the visible light modality, reconstruct its low-frequency components through the network, and then learn the global color representation of the image, and use it for the preliminary colorization of the global infrared image. Secondly, we believe that foreground-background separation can effectively avoid color confusion between background and object. Compared with the complex method of coloring the global image, learning to colorize instances is easier to implement because it does not need to deal with complex background interference, but only needs to focus on coloring specific instances, which improves learning efficiency. At the same time, with the increasing maturity of instance segmentation algorithms, object instance features are easier to obtain as invariant features between modalities. Finally, we combine the contextual information of the image to couple the colorized global color representation and the color representation of the object instance, and extract the coupled frequency domain features for further reconstruction to obtain a color-smooth infrared colorization result. In order to get rid of the dependence on infrared data during training, we still use the zero-shot learning method to colorize infrared instance targets. Specifically, our algorithm framework consists of three parts: (1) a global image colorization network: preliminarily colorizes the global infrared image; (2) an instance colorization module: used to extract instance features of objects from infrared images and feed the generated instance masks to the instance colorization network to supplement the local coloring details of object instances; (3) a global recolorization module: couples the output results of the two branches of the global colorization network and the instance colorization module, and reconstructs the color distribution of object instances in the global image. We adopt a two-step training to learn the instance colorization module and the global colorization network separately, and directly inherit the global image colorization network as the main part in the global image recolorization module.

[0031] like Figure 1As shown in Figure 2, existing learning-based infrared colorization methods cannot provide reasonable color distribution for multiple object instances in complex scenes. For example, Pix2pix (Isola P, Zhu JY, Zhou T, et al. Image-to-image translation with conditional adversarial networks[C] / / Proceedings ofthe IEEE conference on computer vision and pattern recognition. 2017: 1125-1134.) and CycleGan (Zhu JY, Park T, Isola P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C] / / Proceedings ofthe IEEE international conference on computer vision. 2017: 2223-2232.) both suffer from varying degrees of object instance loss. Due to the lack of effective separation of object instances and background, the colorization results of the zero-shot infrared colorization network are confused by context information (the color of the vehicle is biased towards the color of the ground).

[0032] Experimental results show that our work demonstrates its uniqueness on the FLIR dataset and can significantly improve the colorization quality of instance targets in infrared colorization methods compared with the main methods in related fields. Figure 1 The infrared colorization example optimization effect of the method of the present invention on the test set is demonstrated.

[0033] By analyzing the dependence of existing infrared colorization methods on infrared data, we propose a zero-shot infrared image colorization algorithm based on multi-level representation fusion. This method only requires visible light image data in the training phase and achieves infrared instance colorization optimization in the inference phase.

[0034] The present invention further improves the similarity of features between modalities through the strategy of feature dimensionality reduction, and reconstructs the decoupled frequency domain features and instance-level features in conjunction with contextual information, thereby achieving further "cross-modal feature mining".

[0035] like Figure 2As shown in the figure, we propose a zero-shot infrared image colorization algorithm based on multi-level representation fusion, which consists of three main parts: a global image colorization network, an instance colorization module, and a global recolorization module. The overall framework takes the infrared image as input and goes through two main branches. On the one hand, we use the global image colorization network to colorize the entire infrared image. On the other hand, we use the existing pre-trained instance segmentation model to extract instance-level features from the infrared image and generate corresponding instance segmentation pseudo-labels, which are fed into the instance colorization network to achieve end-to-end instance colorization. Finally, the instance colorization results and the overall colorization results are input into the global recolorization module for coupled reconstruction to obtain the optimized infrared colorization results.

[0036] This method realizes cross-modal zero-shot learning and trains each sub-network in sequence using only the visible light modality dataset to significantly improve the colorization quality of infrared instance targets.

[0037] (1) Global Image Colorization Network

[0038] This paper proposes a simple and effective self-supervised method for infrared image colorization, which can effectively explore the correlation between frequency domain information of different modalities by combining the concept of frequency domain feature decoupling and reconstruction with mask image modeling. Figure 3 This paper summarizes the infrared image colorization method we proposed, which consists of four parts: image preprocessing, frequency domain processing, mask image reconstruction, and inference image post-processing. This method achieves good colorization results while simplifying the network structure. To this end, we embed it into the overall algorithm framework as a global image colorization network.

[0039] Specifically, during the network training phase, the preprocessing process is to grayscale the visible light image to enhance the network's color reconstruction capability by removing the color information in the input image.

[0040] After image preprocessing is completed, we input the image into the frequency domain processing module. In the frequency domain processing module, we first perform a single-channel image Perform a two-dimensional discrete Fourier transform as shown in formula (1). and Represents the height and width of the image respectively, represents the coordinates in the spatial domain, represents the coordinates in the frequency domain; is a two-dimensional discrete signal in the spatial domain, In the frequency domain The complex frequency value at the coordinate; represents the Euler number, represents the imaginary part; formula The role of is to traverse the sum of the function values ​​of each pixel in the image, indicating that a specific spatial frequency in the frequency domain depends on all the pixels in the image.

[0041]

[0042] Then, a circular mask of the same resolution is generated by the frequency domain mask generator by default. , whose value is determined by the hyperparameter mask radius The relevant threshold function is determined as shown in formula (2). represents the center coordinates of the image, Represents the Euclidean distance between any coordinate point in the image and the center coordinate of the image. , the mask value at that coordinate is 0, otherwise it is 1.

[0043]

[0044] Next, the mask is used to implement high-pass filtering of the image, remove the low-frequency components in the image, and highlight the edge texture details in the image. The processed frequency domain image is subjected to a two-dimensional discrete inverse Fourier transform, as shown in formula (3). The definition of the symbols corresponds to that in equation (1). The frequency domain high-pass filtering method degrades the original image information and further improves the similarity between the visible light and infrared modes.

[0045]

[0046] Combining the above steps, we can easily get the output image of the frequency domain processing module , as shown in formula (4). represents the Hadamard inner product of the matrix, and Represents Fourier transform and inverse Fourier transform respectively. By adjusting the mask The mask radius can flexibly control the degree of retention of frequency domain information in the image, and plays a key role in balancing cross-modal similarity and image reconstruction difficulty.

[0047]

[0048] Finally, we perform mask image reconstruction on the output results of the frequency domain processing module. Different from the masking mechanism in the spatial domain, the method of the present invention reduces the image information in the frequency domain, so the algorithm framework is more flexible. This method shows cross-modal adaptability. We use the standard ViT network as the encoder, combined with the same simple decoder as the mask image self-encoding task, so that the mask image reconstruction network is more focused on capturing the low-frequency components of the image. Our reconstruction loss function consists of three parts, namely the MSE loss function supervised in the spatial domain, the frequency domain reconstruction loss function supervised in the frequency domain, and the chrominance reconstruction loss function to enhance the color quality of the reconstructed image. The MSE loss function is the most common reconstruction loss function, as shown in formula (5). Among them, and Represents the height and width of the image respectively, and Respectively Pixel values ​​of the visible light image and the network reconstructed image at the coordinate location.

[0049]

[0050] In the frequency domain reconstruction loss function, we first obtain the definition of the image signal amplitude based on the definition of two-dimensional discrete Fourier transform as shown in formula (6). and Respectively The real and imaginary parts of .

[0051]

[0052] At the same time, for two input image signals and , the corresponding frequency domain space Frequency distance between spectral coordinate positions The definition is shown in formula (7). Where, Represents the exponent that controls the sharpness of the distance function. The default setting is 1. The definitions of other symbols are the same as those in formula (6).

[0053]

[0054] Therefore, combining formulas (6) and (7), we can get the frequency domain reconstruction loss function: As shown in formula (8), we calculate the input image pixel by pixel in the frequency domain and reconstruct the image This allows for better consistency in the reconstruction results and eliminates the negative effects of image post-processing.

[0055]

[0056] In order to obtain better color reconstruction effect, we further introduce a chromaticity reconstruction loss function. The image is converted to YUV format and the input image is calculated on the UV channel. and reconstruct the image The mean square error between is used to obtain the chrominance reconstruction loss function, as shown in formula (9). and Respectively represent images On UV Channel The pixel value at the coordinate position, the definitions of other symbols are the same as formula (5). The experimental results show that after introducing the chromaticity reconstruction loss function, the reconstruction results have a significant improvement in some color details.

[0057]

[0058] In summary, the final loss function is shown in formula (10). Where λ is the weight coefficient of the frequency domain reconstruction loss function. If the weight of the frequency domain reconstruction loss function is too large, the color of the reconstructed image will be abnormal. In order to prevent this negative impact, The value of must be less than 1, and we set it to .

[0059]

[0060] In the inference stage, due to the significant difference between the long-wave infrared modality and the visible light modality, in order to obtain a better infrared colorization effect during reconstruction, we performed a grayscale inversion operation on the long-wave infrared image, assuming that the grayscale value of each pixel in the source image is , then the grayscale inversion value is , the grayscale values ​​before and after flipping are both range. This preprocessing enhances the naturalness of colors in the reconstructed image, especially in the sky area. After image preprocessing, the network shows effective infrared colorization over a wide range of the infrared spectrum. However, based on the concept of frequency domain decoupling, we can further improve the colorization effect of this method through image post-processing during the inference phase. Specifically, we integrate the high-frequency components of the input image with the low-frequency components of the reconstructed image, thereby preserving key details such as the edges of the original input image and ultimately generating a higher quality output. In the frequency domain processing module, we perform a Using frequency domain masks For inference image post-processing, we use the same frequency domain mask Acts on the output of the mask image modeling module , and get the final global coloring result , as shown in formula (11).

[0061]

[0062] Among them, the input image of the mask image modeling module and output image The corresponding frequency domain mask are complementary, is the Hadamard inner product of the same matrix in equation (4). This post-processing of frequency domain feature fusion benefits from the frequency domain reconstruction loss function in the loss function, which can improve the quality of the final result without causing any negative impact.

[0063] (2) Instance Coloring Module

[0064] The effectiveness of mining the similarity between visible and infrared modal features for infrared colorization has been demonstrated in the global colorization network. At the same time, instance segmentation algorithms have become increasingly mature, which can effectively separate the foreground and background of images, replace the high-frequency features of visible-infrared instance-level features, and further improve the similarity of features between modalities through feature degradation strategies to achieve a more stable "feature bridging". To this end, we use the existing instance segmentation network MaskR-CNN, and on this basis, combine the backbone network ConvNeXt V2 as our instance segmentor. It is used for instance-level feature extraction of visible-infrared and generates corresponding instance segmentation pseudo-labels for training and testing of instance colorization networks.

[0065] The goal of this method is to use the semantic-level object feature maps obtained by segmentation to further optimize the coloring quality of object instances. PixpixHD is a conditional GAN ​​framework for high-resolution image-to-image translation. We use the same multi-scale discriminator as PixpixHD to train the generator to verify the effectiveness of our instance coloring module. The instance coloring network structure is as follows: Figure 4 As shown in Figure 2. Compared with the panoptic segmentation label synthesis of realistic images, the semantic-level object feature map we input is simpler. Therefore, at the network level, unlike PixpixHD's way of decomposing the generator into a global generator network and a local enhancer network, we removed the heavy local enhancer network, while ensuring the generation quality, further realizing the lightweight of the network and optimizing more hyperparameters for computational time.

[0066] The instance coloring network includes an instance generator and a multi-scale discriminator;

[0067] For our task, the role of the generator is to convert semantic-level object feature maps into realistic instance target images. We call the generator the instance generator network. , the instance generator network structure is as follows Figure 5As shown in Figure 2. To synthesize images at higher resolutions, the instance generator network is fed with a 512×512 resolution input. The instance generator network consists of three parts: (1) a convolutional front end : The input image tensor is downsampled three times. Each downsampling reduces the spatial size of the feature map by half while increasing the number of channels to capture features of different scales. (2) A set of residual blocks :Each residual block contains a series of convolution operations, normalization operations and activation functions, which help the instance generator network learn more complex representation information, thereby improving network performance and accelerating convergence; (3) A transposed convolution backend : The feature tensor output by the residual block is upsampled three times to increase the spatial size of the feature map and reduce the number of channels of the feature map so that the final output has a similar spatial dimension to the input.

[0068] In order to distinguish high-resolution generated images from real images, the discriminator needs to have a larger receptive field. Usually, this will require a deeper network or a larger convolution kernel. Since both methods will increase the capacity of the network, it is likely to cause overfitting problems during model training. In addition, both methods will take up more video memory resources during training. In order to solve the problem of limited receptive field of the discriminator, we abandon the single-scale discriminator and use a multi-scale discriminator instead. .like Figure 6 As shown in Figure 2, the discriminator consists of three identical network structures, but each sub-network of the discriminator acts on different image scales. We refer to each sub-network of the multi-scale discriminator as , and Specifically, we downsample the high-resolution generated images and real images by two and four times to create an image pyramid containing three scales. We then train a multi-scale discriminator under , and The generated images and real images of three different scales are judged respectively. Although each sub-network of the multi-scale discriminator has the same structure, The discriminator operates on the largest scale image, which has a global image receptive field and can train the generator to generate globally consistent images. On the other hand, the discriminator operates on the smallest scale. Used to train the generator to generate finer detail representations. Without a multi-scale discriminator, the generated images often suffer from color blur and loss of structural details.

[0069] The training set of the instance colorization network is a set of image pairs. The network is input in the form of represents the semantic-level object feature map, represents the corresponding real target image, Indicates the calculation of all image pairs The instance coloring network is a conditional GAN ​​network that aims to achieve the following The game models the conditional distribution of semantic-level object feature maps given an input and the true target image, the instance generator The purpose of generating images is to be as close to real images as possible, and the multi-scale discriminator The purpose is to determine the instance generator as much as possible The generated image of is fake, as shown in formula (12).

[0070]

[0071]

[0072] in, The expansion of is shown in formula (13). represents the first The layer discriminator determines the probability of whether the real image is real, because is a real image, so the closer the real probability is to the discriminator The better. represents the first Layer Discriminator Judgement Instance Generator The probability that the generated image is real. In other words, the instance generator hope As big as possible, It approaches the minimum value, so we have formula (12) The stronger the multi-scale discriminator is, The bigger, It should be smaller. Approaches the maximum value, so formula (12) is the solution for the multi-scale discriminator. .

[0073]

[0074] We combine the feature matching loss function based on the multi-scale discriminator , further optimize the conditional GAN ​​loss in formula (13), as shown in formula (14). In this loss function, It is only used as a feature extractor and does not participate in the optimization of the loss function. Represents the first The first The network first extracts the features of the real image and the original image at each layer. and , then calculate the mean absolute error between them, and finally divide by , Indicates The number of elements in the feature map of the layer. We arrive The same calculation is performed on the layers of and summed. In general, the feature matching loss function is calculated by matching The extracted multi-scale features make the generated image and the real image as close as possible in the feature space.

[0075]

[0076] We also define a loss function using the VGG19 network, which is often referred to as the perceptual loss function, as shown in formula (15). The given generated image and the corresponding real image are input into the VGG19 network, their feature mapping results at different network layers are compared, and the mean absolute error is calculated, and finally the results of each layer are weighted summed. The smaller the value, the smaller the difference between the generated image and the real image. In formula (15), represents the weight value of the corresponding layer, Represents the first Layer. Since this loss function uses the VGG19 pre-trained network to extract the feature map of the image, it takes into account the feature-level differences between images while calculating the pixel-level differences. Therefore, using the perceptual loss function in this task can help the network generate images that are more likely to be visually closer to real images, with better global consistency and clearer structural textures.

[0077] We weighted sum the above formula to get the total loss function of the instance coloring network , as shown in formula (16). Where, Represents the feature matching loss function And the perceptual loss function The weighting coefficient of .

[0078]

[0079] (3) Global recoloring module

[0080] The output results of the instance shading module and the global shading network are reasonably coupled to achieve better shading effects, such as Figure 7As shown. Specifically, we noticed that as the scene becomes more and more complex and the illumination intensity gradually decays, the global colorization result of the far-infrared band still has obvious geometric distortion and color loss in the foreground color. Therefore, we believe that the foreground part after global colorization cannot directly provide effective visual information. At the same time, we are still concerned about the generalization performance of the model. To this end, we directly use the semantic-level labels segmented in the instance colorization module as pixel indexes in the spatial domain, perform instance extraction on the corresponding areas of the generated image of the instance colorization module and the reconstructed image of the global colorization network, and recouple the extracted parts, as shown in formula (17).

[0081]

[0082] in, represents the generated image of the instance colorization module, represents the reconstructed image of the global coloring network, A mask representing the permuted region, Represents the output image after coupling of the target region.

[0083] However, we found that the output image after coupling the target region , there is a big difference in the color distribution between the target area and the background area, which does not match the visual perception. Looking back at the global image colorization network, it focuses on capturing the low-frequency components of the image and has excellent color reconstruction capabilities, especially for the colored image. Therefore, we couple the image after the target area Feed the global image shading network again to get smoother infrared shading results.

[0084] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0085] Example

[0086] In this embodiment, we experimented and analyzed the proposed method from both qualitative and quantitative perspectives, aiming to verify the zero-shot infrared image colorization algorithm based on multi-level representation fusion proposed in this invention. The experimental results show that compared with the main methods in the related field, this method can significantly improve the colorization quality of instance targets in different infrared colorization methods, especially showing its uniqueness in the long-wave infrared modality. In addition, the ablation experiment part also shows the contribution of different modules to the method. Through the target detection experiment, we verified the potential of this method in cross-modal recognition tasks. Finally, we briefly discussed the performance of this method in the near-infrared modality. Rich experiments verify the effectiveness of this method and its universal value in fields such as infrared image colorization and target recognition.

[0087] This embodiment uses three datasets for training and evaluation. We obtain 19,998 images (excluding segmentation labels) from the Cityscapes dataset. Due to its excellent data quality, we directly feed them into the instance segmentation network to obtain the corresponding foreground pseudo labels. We use two infrared modality datasets describing vehicle-mounted scenes for testing and evaluation. One is the FLIR alignment dataset of the long-wave infrared modality, and the other is the MSOD dataset of the near-infrared modality. Our main dataset is the FLIR alignment dataset. Researchers screened 5,142 pairs of well-paired long-wave infrared and visible light images from more than 9,000 pairs of original datasets. The resolution of both visible and long-wave infrared modalities is 640×512. The MSOD dataset contains image data of four modalities, namely, visible light modality, near-infrared modality, medium-wave infrared modality, and long-wave infrared modality. The dataset contains 7,521 pairs of unpaired images, of which the resolution of visible light and long-wave infrared modalities is 640×480, while the resolution of near-infrared and medium-wave infrared modalities is 320×256. We only use the near-infrared modality for evaluation. More details about the dataset settings in the experiments will be provided in each subsequent experiment.

[0088] In order to provide a reasonable evaluation of the method of the present invention, we analyze it from both qualitative and quantitative perspectives, with qualitative evaluation as the main evaluation criterion and quantitative evaluation indicators as quantitative references. First, from the perspective of visualization, we define the criterion for qualitative evaluation: it is required that after colorization, a color distribution that conforms to the target characteristics is provided from a complex scene containing multiple objects while maintaining the details of the image structure, and the overall picture conforms to the visual perception of the human eye. Then, we use the same quantitative evaluation indicators as the existing infrared colorization methods. Specifically, we use PSNR and SSIM to quantify the colorization quality in the experiment. PSNR is mainly used to measure the degree of pixel-level distortion of the image, while SSIM pays more attention to the consistency of structural information. Therefore, the combined use of these two indicators can evaluate the image quality more comprehensively and objectively. We believe that an ideal infrared instance colorization method should retain image content at all levels, from scene level layout to fine-grained edges. In order to further evaluate the performance of the method of the present invention, we use the target detection task to indirectly evaluate it on the premise of satisfying visualization, and explore the potential of the method of the present invention in cross-modal recognition tasks. For the evaluation of the target detection task, we analyze it from a quantitative perspective. Typically, Average Precision (AP) is defined as the average detection accuracy at different recall rates. In the multi-category case, we use the mean average precision (mAP) across all object categories as the final measure of performance.

[0089] In the experiment, the higher the PSNR and SSIM values, the smaller the image quality loss and the greater the similarity between the colorized image and its corresponding visible light original image. The larger the mAP value, the higher the accuracy in detecting the object.

[0090] This embodiment adopts two-step training, and the specific training process is as follows.

[0091] (1) Instance colorization module: First, we initialize the instance segmentation network using the pre-trained weights provided by mmdetection. Then, we feed the Cityscapes dataset directly into the instance segmentation network to obtain the corresponding foreground pseudo-labels, which are then fed into the instance colorization network as a training set to be trained from scratch using the parameters The Adam optimizer was trained for 200 epochs with a learning rate of 2e-4, keeping the same learning rate in the first 100 epochs and linearly decaying the learning rate to zero in the next 100 epochs to obtain the instance coloring model.

[0092] (2) Global image colorization network: During the training phase of the global image colorization network, the parameters The AdamW optimizer was trained for 5000 epochs with a learning rate of 1e-3 and a warm-up of the learning rate in the first 40 epochs.

[0093] We directly inherit the baseline global image colorization network as the main part in the global image recolorization module.

[0094] The above training process is implemented based on the PyTorch framework. We resize all images to a resolution of 256×256. Although the method of the present invention supports higher resolutions, we keep the resolution at 256 for comparison with other methods.

[0095] To highlight the effectiveness and versatility of our proposed optimization method for infrared image instance colorization, we publish a quantitative comparison of different methods before and after optimization in Table 1, summarizing the performance by averaging all images in the FLIR data validation set.

[0096] Table 1 shows the mean quantitative metrics of image colorization in the FLIR data validation set. We compare the metrics of each method after infrared colorization. We use bold font to highlight the best metrics and underline to emphasize the suboptimal metrics.

[0097] Table 1

[0098]

[0099] We selected 4000 pairs of daytime image data with different light intensities from the FLIR aligned dataset and randomly assigned 3600 pairs as training sets and 400 pairs as validation sets. The zero-shot infrared colorization network and the proposed method use cross-modal zero-shot learning and are trained only on visible light images. The remaining methods are trained on visible light-long wave infrared image pairs.

[0100] The results show that Pix2pix achieves the best results in the evaluation indicators, followed by CycleGan. This experimental result can be attributed to the inherent differences between the zero-shot infrared colorization network and the method of the present invention and the image-to-image conversion method. Since we cannot obtain long-wave infrared modal images during the training phase, the purpose of the method of the present invention is not to directly convert long-wave infrared images into corresponding visible light paired images, but to use the learned visible light image color representation to colorize long-wave infrared images. Compared with the zero-shot infrared colorization network, the method of the present invention has achieved significant performance improvement in the SSIM indicator. While optimizing the instance colorization, the structural details of the instance target can be retained to the greatest extent. This progress highlights the contribution of instance colorization optimization to performance improvement. However, the performance of the method of the present invention in the PSMR indicator is slightly lower than that of the zero-shot infrared colorization network, which is caused by the global recoloring module in the method of the present invention. Although this module makes the color distribution in the picture more coordinated, it brings about pixel loss of the overall image during the color reconstruction process, resulting in an increase in MSE. Since PSNR is negatively correlated with MSE, the presentation of the PSNR indicator is further degraded.

[0101] like Figure 8 As shown in the figure, the coloring effects of Pix2pix, CycleGan, zero-sample infrared coloring network and the method of the present invention are compared on the FLIR dataset. RGB refers to the visible light image in the image pair, while FIR represents the long-wave infrared image. The method of the present invention achieves further optimization of instance-level target coloring and can provide color distribution that meets the target characteristics from a complex scene containing multiple objects.

[0102] The coloring results of the experiment are as follows Figure 8 As shown, the coloring optimization effects of three different lighting scenes are included. It can be seen from the image that Pix2pix and CycleGan are better than the zero-sample infrared coloring network and the method of the present invention in the color performance without instance targets, but are not as good as the zero-sample infrared coloring network and the method of the present invention in the local structural details. Specifically, Pix2pix and CycleGan both have different degrees of instance-level target loss in the process of global infrared image coloring, especially in complex scenes containing multiple objects, there are obvious geometric distortions and pixel blurs. Although the zero-sample infrared coloring network retains the structural details of the scene and instance targets to the greatest extent in the process of global infrared image coloring, the color representation of the instance target is disturbed by the bleeding of other colors in the scene, such as the green of trees appearing on the car, which is not conducive to human eye observation. It is worth noting that the method of the present invention realizes instance-level optimization of the infrared coloring algorithm. Under the premise of maintaining the geometric features of the instance-level target, a color distribution that meets the target characteristics is provided from a complex scene containing multiple objects, and the overall picture meets the visual perception of the human eye. Due to the particularity of the training process, the method of the present invention does not aim to provide the same color representation as the visible light image, but explores its application in downstream tasks on the premise of satisfying visualization, and taps the potential of the method of the present invention in cross-modal recognition tasks. Relevant details will be provided in subsequent experiments.

[0103] The present invention conducts a series of ablation experiments to verify the core components of the proposed method, including the instance colorization module and the global recolorization module. The instance colorization module determines the colorization quality of the instance target, and the global recolorization module has an important influence on the colorization effect of the overall image. In all ablation experiments, we use the FLIR validation dataset.

[0104] First, we conduct ablation experiments on the instance colorization network in the instance colorization module and study the impact of four different methods on the quality of colorization optimization. Fig. 9 Coloring optimization results for different instance coloring networks in the instance coloring module, including Pix2pix, CycleGan, Pix2pixHD, and SPADE.

[0105] For details on the Pix2pixHD method, see: Wang TC, Liu MY, Zhu JY, et al. High-resolution image synthesis and semantic manipulation with conditional gans[C] / / Proceedings of the IEEE conference on computer vision and patternrecognition. 2018: 8798-8807.

[0106] For details on the SPADE method, see: Park T, Liu MY, Wang TC, et al. Semantic images synthesis with spatially-adaptive normalization[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019: 2337-2346.

[0107] Fig. 9 The coloring optimization results obtained using different networks are shown in . By comparison, it is found that the coloring quality of Pix2pix and CycleGan for instance targets is poor, with obvious geometric distortion and blurred details, which is because these two methods only constrain at a single scale. The difference is that the multi-scale discriminator of Pix2pixHD and SPADE combines perceptual loss to guide the generator to produce finer details while ensuring the maximum receptive field. The indicator evaluation results of different networks are published in Table 2. SPADE achieved the best results in the evaluation indicators, followed by Pix2pixHD. Compared with the advantages of SPADE in the evaluation indicators, it is slightly insufficient in visual perception. On the contrary, Pix2pixHD provides a more reasonable and more visually consistent coloring optimization effect. Therefore, we prefer to use Pix2pixHD as the instance coloring network in the instance coloring module to participate in the composition design of the overall baseline algorithm.

[0108] Table 2 shows the impact of different instance coloring networks in the instance coloring module on the mean coloring indicators of the validation dataset. For each indicator, we use bold font to highlight the best indicator and underline to emphasize the suboptimal indicator.

[0109] Table 2

[0110]

[0111] Next, we conducted an ablation experiment on the global recoloring module to study the impact of the global shading network directly inherited in this module on the shading optimization quality before and after processing. The global recoloring module aims to instance-couple the output results of the two branches of the instance shading module and the global shading network to achieve better shading effects. Fig.10 The shading effect before and after the global recoloring module is shown in Figure 1. It can be seen that without the global shading network in the global recoloring module, the color distribution between the instance coupling part and the background is very different, which is not conducive to the human eye's perception of the target in the scene. However, after the global shading network in the global recoloring module is processed, the color distribution between the instance coupling part and the background is reconstructed, and a shading optimization result that is more in line with the human eye's perception is obtained.

[0112] Table 3 shows the impact of the global coloring network in the global recoloring module on the mean coloring metrics of the validation dataset. For each metric, we use bold font to highlight the best metric.

[0113] Table 3

[0114]

[0115] In addition, we also tested the impact of the global coloring network processing in the global recoloring module on the coloring results from various indicators. Table 3 gives the corresponding results. As can be seen from the table, after processing by this module, a significant performance improvement is achieved in the SSIM indicator, which can maximize the structural similarity between the result and the original image. Considering that the global recoloring module is more focused on capturing the low-frequency components of the image during color reconstruction. While optimizing the color perception, it inevitably brings about pixel loss of the overall image. Even so, on our baseline model algorithm, this module is effective on all evaluation indicators at the same time, and has different degrees of improvement.

[0116] In order to further verify the effectiveness and versatility of our proposed method, the advantages of cross-modal infrared instance colorization under zero-shot learning conditions are highlighted. We conducted target detection experiments on the FLIR validation dataset. The FLIR alignment dataset provides detection box annotations for visible-infrared image pairs. Therefore, we used YOLOv5 to verify the instance colorization performance of the colorization optimization method. It was tested only under the detection model trained on 3600 RGB images, and 400 images were used as the test set. Based on the zero-shot learning characteristics of the zero-shot infrared colorization network, the infrared colorization results of RGB, FIR, and zero-shot infrared colorization networks and the colorization results optimized by the method of the present invention on this basis were compared.

[0117] From the detection results of four different modalities published in Table 4, it can be seen that the method of the present invention has the best detection performance (the test results of RGB images are only for reference and are not included in the index evaluation). Although our colorization results lack the most realistic colors and structures, under the premise of zero-shot learning, we optimize the colorization based on the zero-shot infrared colorization network and generate vehicles that the detector can recognize, while the zero-shot infrared colorization network is lower than the direct detection accuracy of FIR.

[0118] We are Fig.11 The figure shows a visual comparison of the vehicle detection results of the YOLOv5 model on the FLIR dataset, presented at the original resolution of the image (640×512). Compared with the detection results of FIR and zero-shot infrared colorization networks, the proposed method has better detection rate and confidence.

[0119] The method of the present invention does not rely on the annotation of infrared data, and can directly apply the trained RGB modality detection model to the colored infrared image, and has a significant improvement in detection performance. The method provides new possibilities for cross-modality recognition tasks.

[0120] Fig.11 This is a visual comparison of the vehicle detection results of the YOLOv5 model on the FLIR dataset. The infrared colorization results of FIR, RGB, and zero-shot infrared colorization networks are compared, as well as the detection effects corresponding to the colorization results optimized by the method of the present invention on this basis. All vehicle categories are integrated into one category, and the detection category of the target and the corresponding confidence level are displayed in the image.

[0121] Table 4 shows the detection results of different modal data, and mAP is calculated with an IoU of 0.5. For each indicator, we use bold font to highlight the best indicator and underline to emphasize the suboptimal indicator.

[0122] Table 4

[0123]

[0124] In order to further verify the performance of our proposed method on wide-spectrum infrared modal data, we supplemented the experiments of the near-infrared modality, which mainly characterizes the radiation characteristics of objects, with the experiments of the long-wave infrared modality, which mainly characterizes the radiation characteristics of objects. Specifically, we selected 3850 sets of daytime image data with different light intensities from the MSOD dataset, using only the near-infrared modality and the visible light modality. We randomly assigned 3500 sets as training sets and 350 sets as validation sets, which were trained only on the visible light modality.

[0125] First, we verified the effect of near-infrared modality colorization before and after optimization on the baseline model. Because it is an unpaired dataset, we only compare the colorization effect by visualization. Fig.12 As shown. It can be seen that although the zero-shot infrared colorization network adopts cross-modal zero-shot learning, its colorization guarantees the structural similarity to the greatest extent and the color distribution that conforms to the visual perception of the human eye. Although the method of the present invention improves the color performance of the instance target to varying degrees, it is structurally inferior to the direct coloring effect of the zero-shot infrared colorization network. We analyzed that this is due to the high similarity of the brightness mapping relationship between the near-infrared modality and the visible light modality, and the zero-shot infrared colorization network reconstructs the low-frequency components of the image based on the high-frequency information after the visible light image is grayed. Therefore, the near-infrared modality after colorization has excellent color and structural details.

[0126] At the same time, we conducted target detection experiments on this dataset. From the detection results of the four different modalities published in Table 5 (the test results of RGB images are only for reference and are not included in the index evaluation), it can be seen that the near-infrared modality has the best detection performance, followed by the zero-sample infrared colorization network after colorization. Fig.13 The visual comparison results shown in the figure focus on analyzing the cases where the method of the present invention misses detections. This is because the general instance colorization module can only obtain rough instance colorization results on images with lower resolutions. We believe that by adding richer scene data to the instance colorization module, the method of the present invention can achieve further performance improvements.

[0127] In general, we believe that the zero-shot infrared colorization network has provided an excellent cross-modal zero-shot learning colorization paradigm in the near-infrared modality, and the introduction of the method of the present invention has complicated the colorization process.

[0128] Fig.12 Visual comparison of near-infrared modality colorization before and after optimization on the MSOD dataset. On the MOSD dataset, the performance of the zero-shot infrared colorization network before and after optimization of infrared instance colorization is compared. * indicates the optimized baseline method. RGB refers to visible light images, while NIR stands for near-infrared images.

[0129] Fig.13 This is a visual comparison of the vehicle detection results of the YOLOv5 model on the MSOD dataset. The infrared colorization results of NIR, RGB, and zero-shot infrared colorization networks are compared, as well as the detection effects corresponding to the colorization results optimized by the method of the present invention on this basis. All vehicle categories are integrated into one category, and the detection category of the target and the corresponding confidence are displayed in the image.

[0130] Table 5 shows the detection results of different modal data, and mAP is calculated with an IoU of 0.5. For each indicator, we use bold font to highlight the best indicator and underline to emphasize the suboptimal indicator.

[0131] Table 5

[0132]

[0133] In summary, the present invention proposes a zero-shot infrared image colorization algorithm based on multi-level representation fusion. Features are extracted and coupled from the two branches of the global colorization network and the instance colorization module respectively, and then reconstructed through the global recolorization module to achieve a significant improvement in colorization performance. This method eliminates the dependence on infrared image data in the training stage. At the same time, this method achieves further optimization of the infrared colorization algorithm and can provide a color distribution that conforms to the instance characteristics. We also explored its application in downstream tasks and verified the potential of the method of the present invention in cross-modal recognition tasks. However, this method still has some limitations. Although the method of the present invention improves the color performance of the instance target to varying degrees in the infrared modality, it introduces certain complexity in the colorization process. Future work will be committed to further optimizing the instance colorization process and improving its performance in downstream tasks. We hope that this work can bring new ideas to the field of infrared image analysis and recognition.

[0134] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0135] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A zero-shot infrared image colorization method based on multi-level representation fusion, characterized in that: include: The global image colorization network is used to colorize the entire infrared image, including image preprocessing, frequency domain processing, mask image reconstruction and inference image post-processing. In the network training phase, the preprocessing process is to grayscale the visible light image, which enhances the network's color reconstruction ability by removing the color information in the input image; After image preprocessing is completed, the image is input into the frequency domain processing module; In the frequency domain processing module, first a single-channel image of a given input is processed. Perform a two-dimensional discrete Fourier transform, as shown in formula (1): ; in, and Represents the height and width of the image respectively, represents the coordinates in the spatial domain, represents the coordinates in the frequency domain; is a two-dimensional discrete signal in the spatial domain, In the frequency domain The complex frequency value at the coordinate; represents the Euler number, represents the imaginary part; formula The role of is to traverse the sum of the function values ​​of each pixel in the image, indicating that a specific spatial frequency in the frequency domain depends on all the pixels in the image; By default, a circular mask of the same resolution is generated by the frequency domain mask generator. , whose value is determined by the hyperparameter mask radius The relevant threshold function is determined as shown in formula (2): ; in, represents the center coordinates of the image, Represents the Euclidean distance between any coordinate point in the image and the center coordinate of the image. When , the mask value at the coordinate is 0, otherwise it is 1; The mask is used to implement high-pass filtering of the image, remove the low-frequency components in the image, and highlight the edge texture details in the image. The processed frequency domain image is then subjected to a two-dimensional discrete inverse Fourier transform, as shown in formula (3): ; Combining the above steps, we can get the output image of the frequency domain processing module. , as shown in formula (4): ; in represents the Hadamard inner product of the matrix, and Represents Fourier transform and inverse Fourier transform respectively; by adjusting the mask The mask radius controls the degree of retention of frequency domain information in the image; The output result of the frequency domain processing module is used to reconstruct the mask image. The reconstruction loss function consists of three parts, namely, the MSE loss function supervised in the spatial domain, the frequency domain reconstruction loss function supervised in the frequency domain, and the chrominance reconstruction loss function. The MSE loss function is the reconstruction loss function, as shown in formula (5): ; in, and Represents the height and width of the image respectively, and Respectively Pixel values ​​of the visible light image and the network reconstructed image at the coordinate position; In the frequency domain reconstruction loss function, firstly, according to the definition of two-dimensional discrete Fourier transform, the amplitude definition of the image signal is obtained as shown in formula (6); ; in, and Respectively The real and imaginary parts of For two input image signals and , the corresponding frequency domain space Frequency distance between spectral coordinate positions The definition is shown in formula (7): ; in, represents the exponent that controls the sharpness of the distance function; Combining formulas (6) and (7), we get the frequency domain reconstruction loss function: As shown in formula (8), the input image is calculated pixel by pixel in the frequency domain and reconstruct the image The frequency distance between ; Convert the image to YUV format by calculating the input image on the UV channels and reconstruct the image The mean square error between them gives the chrominance reconstruction loss function, as shown in formula (9); ; in and Respectively represent images On UV Channel The pixel value of the coordinate position; The final loss function is shown in formula (10): ; Where λ is the weight coefficient of the frequency domain reconstruction loss function; In the inference stage, the grayscale inversion operation is performed on the long-wave infrared image, assuming that the grayscale value of each pixel in the source image is , then the grayscale inversion value is , the grayscale values ​​before and after flipping are both The high-frequency components of the input image are integrated with the low-frequency components of the reconstructed image, thereby preserving the edge details of the original input image and ultimately generating a higher quality output; in the frequency domain processing module, the input image is Using frequency domain masks ; For inference image post-processing, use the same frequency domain mask Acts on the output of the mask image modeling module , and get the final global coloring result , as shown in formula (11): ; Among them, the input image of the mask image modeling module and output image The corresponding frequency domain mask are complementary, is the Hadamard inner product of the same matrix in equation (4); Using the pre-trained instance segmentation model, instance-level features are extracted from infrared images, and corresponding instance segmentation pseudo labels are generated, which are then sent to the instance coloring network to achieve end-to-end instance coloring. Specifically: The instance colorization network includes an instance generator and a multi-scale discriminator; the instance generator is used to convert the semantic-level object feature map into a realistic instance target image, and the generator is called the instance generator network , the instance generator network is fed with a resolution of 512×512; the instance generator network consists of three parts: a convolutional front end , used to downsample the input image tensor three times, each downsampling will halve the spatial size of the feature map, while increasing the number of channels to capture features of different scales; a set of residual blocks , each residual block contains a convolution operation, a normalization operation, and an activation function; a transposed convolution backend , the feature tensor output by the residual block is upsampled three times to increase the spatial size of the feature map while reducing the number of channels of the feature map; multi-scale discriminator It consists of three identical network structures, each sub-network , , Works at different image scales; The training set of the instance colorization network is a set of image pairs. The network is input in the form of represents the semantic-level object feature map, represents the corresponding real target image, Indicates the calculation of all image pairs The expected value of; the instance coloring network is used as a conditional GAN ​​network through the following The game models the conditional distribution of semantic-level object feature maps given an input and the true target image: ; ; in, The expansion of is shown in formula (13), represents the first The probability that the layer discriminator judges whether the real image is real or not, is a real image, represents the first Layer Discriminator Judgement Instance Generator The probability of whether the generated image is real; ; By combining the feature matching loss function based on multi-scale discriminator , optimizes the conditional GAN ​​loss in formula (13), as shown in formula (14); in this loss function, It is only used as a feature extractor and does not participate in the optimization of the loss function; Represents the first The first The network first extracts the features of the real image and the original image at each layer. and , then calculate the mean absolute error between them, and finally divide by , Indicates The number of elements in the feature map of the layer; in all arrive The same calculation is performed on all layers and summed; ; Define a perceptual loss function using the VGG19 network, as shown in formula (15); input the given generated image and the corresponding real image into the VGG19 network, compare their feature mapping results at different network layers, calculate the mean absolute error, and finally weight the results of each layer; The smaller the value, the smaller the difference between the generated image and the real image; in formula (15), represents the weight value of the corresponding layer, Represents the first layer; The weighted sum of the above formulas gives the total loss function of the instance coloring network: : ; in, Represents the feature matching loss function And the perceptual loss function The weighting coefficient of The instance shading results and the overall shading results are input into the global reshading module for coupled reconstruction to obtain the optimized infrared shading results, which are as follows: Using the semantic-level labels segmented in the instance colorization module as the pixel index in the spatial domain, instance extraction is performed on the corresponding areas of the generated image of the instance colorization module and the reconstructed image of the global colorization network, and the extracted parts are recoupled, as shown in formula (17): ; in, represents the generated image of the instance colorization module, represents the reconstructed image of the global coloring network, A mask representing the permuted region, represents the output image after coupling of the target area; Image after coupling the target area Feed the global image shading network again to get smoother infrared shading results.

2. The zero-shot infrared image colorization method based on multi-level representation fusion according to claim 1 is characterized in that: Set to 1.

3. The zero-shot infrared image colorization method based on multi-level representation fusion according to claim 1 is characterized in that: In the training phase, λ is set to .

4. The zero-shot infrared image colorization method based on multi-level representation fusion according to claim 1 is characterized in that: 。 5. A zero-shot infrared image colorization system based on multi-level representation fusion, characterized in that: For implementing any of the methods described in claims 1 to 4, the system comprises: Global image colorization network, used for preliminary colorization of global infrared images; An instance colorization module is used to extract instance features of objects from infrared images and feed the generated instance masks to the instance colorization network to supplement the local coloring details of the object instances; The global recoloring module couples the output results of the two branches of the global coloring network and the instance coloring module, and reconstructs the color distribution of the object instance in the global image.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Citation Information

Patent Citations

  • Method and system for near infrared-visible light image conversion

    CN117314734A

  • Methods and systems for colorizing infrared images

    US20210400167A1

  • Automatic Colorization of Grayscale Stereo Images

    US20240062425A1