Method and system for near-infrared-visible light image conversion

Abstract

The present application provides a method and system for near-infrared-visible light image conversion, belonging to the technical field of computer vision. The present application divides the original conversion from near-infrared image to visible light image into near-infrared-gray image conversion and gray-visible light image conversion, and converts the original near-infrared single channel to RGB three channel mapping into near-infrared single channel to luminance single channel mapping, solving the problem of establishing one-to-one color mapping for near-infrared 2RGB conversion and simplifying the task complexity. In the near-infrared-gray conversion, frequency domain learning is introduced to help the network retain high-frequency edge information, solving the shortcoming of traditional deep learning network that prefers to learn low-frequency information fitting features, improving the conversion effect and quality, and the obtained image effect is more consistent with the real feeling of human eyes. Through the use of large-scale gray-visible light image dataset for pre-training, the problem of lack of paired near-infrared-visible light image dataset is alleviated, and the color information closer to the real image is recovered.

Description

Technical Field

[0001] This invention relates to a method and system for near-infrared-visible image conversion, specifically a two-step near-infrared-visible image conversion method and system using dual-domain learning, belonging to the field of computer vision technology. Background Technology

[0002] The near-infrared spectral band (700-1000nm) is the closest wavelength to the human visual system, and near-infrared images share many similar properties with visible light images. However, due to its unique characteristics compared to visible light images, such as strong anti-interference ability, sensitivity to surface temperature, and the ability to penetrate obstacles like fog and clouds, near-infrared imaging has wide applications in military, medical, agricultural, and transportation fields. Near-infrared imaging technology is also used in driver assistance and security monitoring systems because of its stable image quality, immunity to lighting conditions, and non-exposed nature within the monitored field of view. These characteristics make near-infrared imaging an ideal choice for situations where lighting conditions may vary or visibility is limited.

[0003] However, because near-infrared images cannot be directly seen by the human eye, and their lack of brightness and colorlessness hinders further observation and analysis, in many cases, it is necessary to convert near-infrared images into visible light images for human understanding and analysis. This has led to the development of near-infrared-visible light image conversion technology. Near-infrared-visible light image conversion technology is of great significance, as it helps people better utilize the information in near-infrared images and improve decision-making efficiency.

[0004] Grayscale image colorization is a classic computer vision task aimed at generating perceptually realistic color information for grayscale images. Currently, grayscale image colorization methods mainly fall into two categories: prior knowledge-guided colorization and automatic colorization. Prior knowledge-guided colorization uses prior knowledge to color the image, such as palette-based, instance-based, and language / text-based approaches. Automatic colorization, on the other hand, automatically colors the image without any additional prerequisites. Current automatic colorization methods primarily explore different network architectures to improve colorization quality, such as deep learning networks, generative adversarial networks, and deep learning models with self-attention mechanisms.

[0005] Compared to visible light images, grayscale images lack only chromaticity information, while near-infrared images lack both luminance and chromaticity information. Furthermore, compared to grayscale images, the non-linear relationship between the luminance distribution and the reflectivity of the target in near-infrared images leads to the loss of shadow and luminance information. Current grayscale image colorization methods are relatively mature compared to near-infrared-visible conversion methods; therefore, near-infrared-visible image conversion methods based on generating grayscale images are theoretically feasible.

[0006] Frequency domain learning, as a machine learning technique, aims to transform signals or / or data from the time domain to the frequency domain for analysis and processing. The frequency domain refers to the representation of a signal in terms of frequency, and a time-domain signal can be converted to a frequency-domain signal using Fourier transform. Frequency domain learning has wide applications in signal processing, image processing, audio processing, natural language processing, and other fields. By analyzing and processing signals or data in the frequency domain, more useful features can be extracted, thereby improving model performance.

[0007] In recent years, the rapid development of deep learning technology has made it possible to automatically generate visible light images with good results from near-infrared images based on generated grayscale images. Deep learning is a new research direction in the field of machine learning. It typically uses deep, nonlinear neural networks to approximate complex functions and trains them with appropriate data and loss functions. By changing the parameters stored in the neural network, it learns statistical laws from a large amount of known data, thereby making predictions about unknown events. Deep learning technology has developed rapidly since its breakthrough in image classification tasks in 2012. With the continuous emergence of new algorithms and ideas, human intelligence has endowed machines with increasingly higher abilities to understand image information, and the problem of near-infrared-visible light image conversion based on generated grayscale images is gradually becoming solvable.

[0008] Existing techniques directly use convolutional neural networks to recover visible light images from single near-infrared images. However, the overlap between the near-infrared spectral range and the spectral sensitivity of a three-channel visible light camera makes it difficult to establish a corresponding color map. A representative work, such as the one proposed by Lei Liu et al. from Shanghai Jiao Tong University in 2022, obtains the optimal near-infrared image by reusing different near-infrared spectra to maximize the differentiation of color variations across the three channels of the visible light image. This method produces good results, but its final conversion performance remains limited due to the lack of paired near-infrared-visible image datasets.

[0009] Therefore, in order to improve the accuracy and sensory quality of automatic near-infrared-visible image conversion, and to solve the overlap between the near-infrared spectral range and the spectral sensitivity of the visible light three-channel camera, as well as the lack of paired matching near-infrared-visible image datasets, there is an urgent need for a near-infrared-visible image conversion technology that can automatically generate visible light images with accurate image detail information and appropriate color style, and can still generate high-quality visible light images under complex real near-infrared image conditions. Summary of the Invention

[0010] The purpose of this invention is to overcome the shortcomings of existing technologies and creatively propose a method and system for near-infrared-visible light image conversion. This invention effectively improves the accuracy and sensory quality of near-infrared-visible light image conversion without requiring special equipment and with a relatively fast generation speed.

[0011] The innovations of this invention include: dividing the original direct conversion from near-infrared images to visible light images into two steps: near-infrared-grayscale image conversion and grayscale-visible light image conversion, and dividing the entire process into a training phase and an application phase.

[0012] During the training phase, the near-infrared-grayscale image conversion module is first trained on a pairwise matched near-infrared-grayscale image training dataset. A Fourier convolutional neural network with frequency domain learning is introduced as the frequency domain extraction module, while a traditional convolutional neural network is used as the spatial domain extraction module. Global and local information of the image is learned by using the frequency and spatial domain extraction modules in parallel. The frequency and spatial domain extraction modules map the input near-infrared image onto the feature spaces of the frequency and spatial domains. Then, grayscale images are generated based on semantic information at different depths, and these are compared with real grayscale images to calculate the loss function and update the parameters in the model accordingly. Next, grayscale-visible light converted images are trained on a pairwise matched grayscale-visible light image training dataset, and compared with real visible light images to calculate the loss function and update the parameters in the model accordingly. The loss function is iteratively calculated repeatedly until a preset termination condition is met, at which point the network structure and model parameters are saved.

[0013] During the usage phase, a network model is created based on the model structure and parameters saved during the training phase. The acquired near-infrared images are input into the model for computation, and the network map generated by the model is stored.

[0014] The performance of the near-infrared-visible light conversion network of the model is evaluated by comparing the generated visible light image with the real visible light image.

[0015] To achieve the above objectives, the present invention adopts the following technical solution:

[0016] A method for near-infrared-visible image conversion includes two phases: training and usage.

[0017] Step 1: Training phase.

[0018] The parameter dictionary of the intelligent conversion algorithm model was iteratively modified using pairwise matched near-infrared-grayscale image datasets and grayscale-visible light datasets.

[0019] Specifically, step 1 includes the following steps:

[0020] Step 1.1: Map the input near-infrared image onto the feature spaces of the frequency domain and spatial domain respectively, and generate a grayscale image.

[0021] Specifically, for all pixels in the input near-infrared image, frequency domain transformation and spatial domain transformation are used to extract global and local information of the image.

[0022] In the frequency domain transformation, Fourier transform and Fourier convolution are used to map pixels onto the feature space of the frequency domain. In the spatial domain transformation, a neural network is used to map pixels onto the feature space of the spatial domain. The depth of the feature space and the number of network parameters are controlled by adjusting the depth of the neural network. Finally, the extracted global and local information are combined to generate a grayscale image.

[0023] Furthermore, since the current near-infrared image dataset is too small, data augmentation and other methods can be used to obtain more paired matching datasets and improve the final conversion effect.

[0024] Step 1.2: Based on the luminance channel information in the generated grayscale image, recover its corresponding color channel information and generate a visible light image.

[0025] Specifically, based on the grayscale image obtained in step 1.1, the luminance channel information is read from the grayscale image. Then, the images are grouped according to their luminance levels, treating the grayscale image colorization task as a classification problem. Based on the color probability distribution corresponding to different luminance levels in the grayscale-visible light image dataset, the color channel information for each pixel is predicted. Finally, the luminance channel information and the color channel information are concatenated to obtain the final visible light image.

[0026] Since the luminance-color channel image representation method used belongs to the LAB color space, while traditional visible light images belong to the RGB color space, the result is converted to the RGB color space using a standard color space conversion function.

[0027] All predicted grayscale and visible light images are stored for use in training.

[0028] Step 1.3: Using the intelligent conversion algorithm model, generate the corresponding grayscale image and visible light image based on the input near-infrared image, compare them with the real grayscale image and visible light image respectively, calculate their loss function, and update the parameters in the intelligent conversion algorithm model accordingly.

[0029] Specifically, the implementation method is as follows:

[0030] Several near-infrared images are randomly selected from the training dataset and input into the intelligent conversion algorithm model. For each near-infrared image, the intelligent conversion algorithm model first outputs a corresponding grayscale prediction image, and then generates the final visible light prediction image based on the predicted grayscale image. The width and height of the grayscale prediction image and the visible light prediction image output by the intelligent conversion algorithm model are consistent with the input near-infrared images.

[0031] The intelligent conversion algorithm model includes a grayscale image generation module and a visible light image generation module. When a near-infrared image is input into the intelligent conversion algorithm model, it first passes through the grayscale image generation module. This module includes frequency domain conversion and spatial domain conversion, both of which are fully convolutional networks. They are based on translation invariance, and the convolutional layers, pooling layers, and activation functions are all operated on the local input region and depend only on the relevant spatial coordinates.

[0032] In this module, each layer's parameters are a three-dimensional array of size b*h*w*d, where b is the batch size for model training, h and w are the spatial dimensions, and d is the number of channels. The first layer is an image with size h*w and d color channels.

[0033] Let X be the vector at position (i, j) in a specific layer of the grayscale image generation module. ij After calculation, it becomes y. ij :

[0034]

[0035] Among them, f ks It determines the layer type, including convolution, average pooling, max pooling, and activation function; This indicates that in the feature map X involved in the calculation, the position is (S i +δ i ,S j +δ j The vector; k is the kernel size; s is the stride of the convolution operation; δ i δ j Let S represent all integers in the range [0, k], and let S be the vector involved in the calculation. i ,S j ) represents all vectors within a convolution kernel with a top-left vertex and a side length of k.

[0036] The grayscale image generation module uses a base loss function that is the sum of the absolute bias losses for each pixel, where the absolute bias loss function L1 is:

[0037]

[0038] Where, x i Represents the actual grayscale pixel data, yi This represents the predicted grayscale pixel data, where n represents the number of pixels in the image. Minimizing the absolute bias loss allows the predicted value to approximate the true distribution as closely as possible.

[0039] Preferably, the grayscale image generation module can select different loss functions, such as the smoothing absolute bias loss function and the mean square error loss function, depending on the specific details and the training dataset.

[0040] The generated grayscale image will then be sent to the visible light image generation module, which will predict the corresponding color channel information based on the brightness channel information in the grayscale image, and finally generate the visible light image.

[0041] The visible light image generation module is a deep learning model using a self-attention mechanism, which utilizes the ground truth of the results in the target domain for supervised learning. The module consists of an encoder and a decoder. The encoder uses a self-attention mechanism to capture long-range dependencies in the input sequence. This mechanism weights words at different positions in the sequence, incorporating all information. Each word in the encoder is compared with other words in the sequence, and an attention score is calculated to determine the weight of each word. The decoder also uses a self-attention mechanism, but introduces an encoder-decoder attention mechanism. This mechanism compares the vector representation of each position in the encoder with the vector representation of the current position in the decoder, thus incorporating information from the encoder. Simultaneously, the decoder introduces positional encoding to represent the relationships between positions in the input sequence.

[0042] The basic loss function used by the visible light image generation module is the sum of the cross-entropy losses for each pixel, where the cross-entropy loss function H is:

[0043]

[0044] Where, x i Let p represent the true color probability distribution and q represent the predicted color probability distribution. Minimizing the cross-entropy loss allows the predicted probability distribution to approximate the true distribution as closely as possible.

[0045] In addition, the visible light image generation module can select different loss functions, such as reconstruction loss function, feature matching loss function, perception loss function, and multi-size discriminator loss function, depending on the specific details of the intelligent conversion algorithm model and the different training datasets.

[0046] Step 1.4: Repeat step 1.3 until the set termination condition is met, then save the network structure and model parameters.

[0047] Specifically, during the training process, near-infrared images are repeatedly input into the intelligent conversion algorithm model, and the parameters in the model are dynamically adjusted according to the loss function until the preset conditions are met (such as: the number of training rounds reaches a certain amount or a certain evaluation index is better than a certain preset value). At this point, the training process stops, and the parameters of the intelligent conversion algorithm model are stored.

[0048] Step 2: Usage Phase. Using the intelligent conversion algorithm model parameters obtained in Step 1 (training phase), the input near-infrared image is converted into a visible light image and stored.

[0049] The near-infrared image read in is in RGB format and its size is preset. If the input near-infrared image is not in this format, it is first cropped and stored. Then, based on the intelligent conversion algorithm model structure and parameters saved during the training phase, an intelligent conversion algorithm model is created. The near-infrared image is input into this model for conversion, generating a corresponding grayscale image. The grayscale image is then processed by the visible light image generation module to obtain the final predicted visible light image, and both the generated grayscale and visible light images are saved. This visible light image is in RGB format and its size is consistent with the input near-infrared image.

[0050] Furthermore, the generated visible light image is evaluated against the real visible light image, and the performance of the intelligent conversion algorithm model in converting the near-infrared image into a visible light image is judged.

[0051] On the other hand, based on the above method, the present invention also proposes a system for near-infrared-visible light image conversion, including a real image processing subsystem, a grayscale image generation subsystem, a visible light image generation subsystem, a loss function calculation subsystem, and a generation result evaluation subsystem.

[0052] The real image processing subsystem includes a real image augmentation module and a real image formatting module.

[0053] The real image augmentation module is used to perform random cropping, rotation, and flipping operations on near-infrared-visible images to increase the paired near-infrared-visible image dataset, making the near-infrared images as diverse as possible, thereby enabling the trained model to have stronger generalization ability.

[0054] The real image formatting module is used to convert near-infrared-visible images stored in bitmap format into tensor format and normalize them to meet the input data format requirements of the grayscale image generation subsystem.

[0055] The grayscale image generation subsystem includes a frequency domain transformation module and a spatial domain transformation module, both of which are composed of convolutional layers, pooling layers, activation layers, and cross-layer connection structures. It is used to extract features from near-infrared images layer by layer, and can fully understand the semantic information corresponding to each pixel, providing important information for generating grayscale images.

[0056] The visible light image generation subsystem includes an encoder and a decoder using a self-attention mechanism. It predicts the corresponding color channel information based on the grayscale brightness channel information obtained from the grayscale image generation subsystem, treating this as a classification problem. The color channels are predicted according to the color probability distribution, ultimately generating a high-quality visible light image.

[0057] The loss function calculation subsystem includes a grayscale generation loss function calculation module and a visible light generation loss function calculation module. The grayscale generation loss function calculation module calculates the absolute deviation loss based on the real grayscale image and the grayscale image converted by the grayscale image generation subsystem. This loss is backpropagated to update the parameters of each neural network layer in the grayscale image generation subsystem, correcting the brightness information converted by the grayscale image generation subsystem. The visible light generation loss function calculation module calculates the cross-entropy loss based on the real visible light image and the final visible light image obtained by the visible light image generation subsystem. This loss is backpropagated to update the parameters of each neural network layer in the visible light image generation subsystem, and indirectly affects the parameters in the grayscale image generation subsystem through the visible light image generation subsystem, correcting the final visible light image generated by the system.

[0058] The generation result evaluation subsystem is used to compare the predicted visible light image with the real visible light image. It can automatically calculate a variety of objective evaluation indicators of the generated image, including PSNR, SSIM, and RMSE, as a supplement to the user's subjective experience and help the user quantitatively understand the quality of the visible light image generated by the system.

[0059] The connections between the above components are as follows:

[0060] The real image processing subsystem is connected to the grayscale image generation subsystem and the loss function calculation subsystem, and provides the grayscale image generation subsystem with processed near-infrared images that meet the requirements.

[0061] The grayscale image generation subsystem is connected to the real image processing subsystem, the visible light image generation subsystem, and the loss function calculation subsystem. It provides the visible light image generation subsystem with the predicted brightness channel information in the image, and at the same time receives the loss value returned by the visible light image generation subsystem for training itself.

[0062] The visible light image generation subsystem is connected to the grayscale image generation subsystem and the loss function calculation subsystem, respectively. It predicts the corresponding color channel information based on the brightness channel information and finally obtains the visible light image.

[0063] The loss function calculation subsystem is connected to the grayscale image generation subsystem, the visible light image generation subsystem, and the real image processing subsystem, respectively. It obtains the real grayscale image and the real visible light image from the real image processing subsystem, then obtains the grayscale image generated by the grayscale image generation subsystem, compares it with the real grayscale image to calculate the loss value, and then feeds the loss value back to the grayscale image generation subsystem for training. Simultaneously, the loss function calculation subsystem obtains the visible light image generated by the visible light image generation subsystem, compares it with the real visible light image to calculate the loss value, and then feeds the loss value back to the visible light image generation subsystem for training.

[0064] The result evaluation subsystem is connected to the real image processing subsystem and the visible light image generation subsystem. It obtains real visible light images from the real image processing subsystem and converted visible light images from the visible light image generation subsystem. By comparing and calculating evaluation indicators, it helps users quantitatively understand the quality of the visible light images converted by the system.

[0065] The working process of the above system is as follows:

[0066] Step 1: Input the near-infrared-visible light image into the real image processing subsystem, perform data augmentation, normalization and formatting, and store the processed near-infrared-visible light image in memory for later use;

[0067] Step 2: The grayscale image generation subsystem reads the near-infrared image, processes it through a neural network layer to generate a grayscale image, and calculates a loss function between the grayscale image and the real grayscale image. The brightness channel information from the grayscale image is then fed to the visible light image generation subsystem. The generated grayscale image is stored for later use.

[0068] Step 3: The visible light image generation subsystem reads the brightness channel information from the grayscale image, processes it through a neural network layer with a self-attention mechanism to generate the final visible light image, and calculates the loss function with the real visible light image; the generated visible light image is stored for later use.

[0069] Step 4: In the loss function calculation subsystem, the grayscale image absolute loss function calculation part reads the real grayscale image and the generated grayscale image, compares them, calculates the loss value, and feeds the loss value back to the grayscale image generation subsystem for training; in the visible light image cross-entropy loss function calculation part, the real visible light image and the generated visible light image are read and compared to calculate the loss value. Then, the loss value is fed back to the visible light generation subsystem for training.

[0070] Step 5: Each neural network layer in the visible light image generation subsystem modifies its parameters according to the loss value returned by the subsystem based on the loss function, completes one training iteration, and returns the loss value of the first layer to the grayscale image generation subsystem.

[0071] Step 6: Each neural network layer in the grayscale image generation subsystem calculates the loss value returned by the subsystem and the network map generation subsystem according to the loss function, modifies the parameters, and completes one training iteration;

[0072] Step 7: Repeat steps 1 to 6. After each iteration of all training data, send the real visible light image and the converted visible light image to the generation result evaluation subsystem to calculate the index; if the index meets the preset conditions, or the total number of iterations reaches the preset upper limit, stop training and proceed to step 8;

[0073] Step 8: Perform steps 1 to 3 to convert and generate the final visible light image.

[0074] Step 9: Send the real visible light image and the converted visible light image to the generation result evaluation subsystem to calculate the final index.

[0075] Beneficial effects

[0076] Compared with the prior art, the present invention has the following advantages:

[0077] 1. This invention decomposes the near-infrared-visible light conversion process into two processes: near-infrared-grayscale image conversion and grayscale-visible light image conversion. It transforms the original near-infrared single-channel to RGB three-channel mapping into a simpler near-infrared single-channel to luminance single-channel mapping, solving the problem of establishing a one-to-one color mapping for near-infrared 2RGB conversion and simplifying the complexity of the target task.

[0078] 2. This invention introduces frequency domain learning during the near-infrared-grayscale conversion process to help the network retain high-frequency edge information. This solves the shortcomings of traditional deep learning networks that prioritize learning low-frequency information to fit features, further improving the final conversion effect and quality, and the resulting actual image effect is more in line with the real perception of the human eye.

[0079] 3. This invention uses a large-scale grayscale-visible light image dataset for pre-training in the grayscale-visible light conversion module, which indirectly alleviates the problem of lacking paired near-infrared-visible light image datasets and helps to recover color information that is more vivid and closer to the real image. Therefore, its generated visible light images are superior to existing methods that directly convert from near-infrared to visible light images in terms of both accuracy and visual effect.

[0080] 4. This invention employs a two-step design approach, consisting of a grayscale image generation module and a visible light generation module. Each module can be updated based on the latest or future semantic segmentation and image generation technologies, while still maintaining a superior generation effect compared to using a single module alone. Therefore, the network model structure proposed in this invention has excellent updability and scalability. Attached Figure Description

[0081] Figure 1 This is a flowchart of the method of the present invention.

[0082] Figure 2 This is a schematic diagram illustrating the training of the core algorithm model described in the method of this invention.

[0083] Figure 3 This is a schematic diagram of the system composition of the present invention. Detailed Implementation

[0084] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0085] Example

[0086] like Figure 1 As shown in the figure. This embodiment discloses a method for near-infrared-visible image conversion, including a training phase and a usage phase.

[0087] During the training phase, the near-infrared-grayscale image conversion module is first trained on a pairwise matched near-infrared-grayscale image training dataset. A Fourier convolutional neural network with frequency domain learning is introduced as the frequency domain extraction module, while a traditional convolutional neural network is used as the spatial domain extraction module. The global and local information of the image is learned by using the frequency and spatial domain extraction modules in parallel. The frequency and spatial domain extraction modules map the input near-infrared image onto the feature spaces of the frequency and spatial domains. Then, grayscale images are generated based on semantic information at different depths, and these grayscale images are compared with real grayscale images to calculate the loss function and update the parameters in the model accordingly. Next, the grayscale-visible light image conversion module is trained on a pairwise matched grayscale-visible light image training dataset to perform grayscale-visible light image conversion. This conversion is then compared with real visible light images to calculate the loss function and update the parameters in the model accordingly. The loss function is iteratively calculated repeatedly until a preset termination condition is met, at which point the network structure and model parameters are saved. During the usage phase, a network model will be created based on the model structure and parameters saved during the training phase. The acquired near-infrared images will be input into the model for computation, and the network map generated by the model will be stored. The system's built-in evaluation method will be used to compare and evaluate the generated visible light images with the real visible light images to determine the performance of the model's near-infrared-visible light conversion network.

[0088] Near-infrared imaging technology offers stable image quality, immunity to lighting conditions, and non-exposure within the monitored field of view, making it an ideal choice for situations where lighting conditions may vary or visibility is limited. However, the lack of brightness and colorlessness of near-infrared images hinders further observation and analysis by users. Therefore, in many cases, it is necessary to convert near-infrared images into visible light images for human understanding and analysis.

[0089] Currently, existing near-infrared-visible image conversion techniques directly use convolutional neural networks to recover visible light images from single near-infrared images. However, the overlap between the near-infrared spectral range and the spectral sensitivity of a three-channel visible light camera makes it difficult to establish a corresponding color map. Recent representative work has explored reusing different near-infrared spectra to obtain optimal near-infrared images, maximizing the differentiation of color variations across the three channels of a visible light image. This method has yielded good results, but its final conversion performance remains limited due to the lack of paired near-infrared-visible image datasets. Grayscale images, compared to visible light images, only lack chromaticity information, and current grayscale image colorization methods are relatively mature compared to near-infrared-visible conversion methods. Furthermore, the rapid development of deep learning technology has made it possible to automatically generate high-quality visible light images from near-infrared images based on grayscale image generation.

[0090] Therefore, the method described in this embodiment proposes a two-step core algorithm, which includes a grayscale image generation module and a visible light image generation module, and cleverly integrates the two modules. This allows the overall network to fully utilize the information in the near-infrared image for training, resulting in a visible light image with high accuracy and a suitable image style. A schematic diagram of a network model for near-infrared-visible light image conversion is shown below. Figure 2 As shown in the figure. The method described in this embodiment is an end-to-end solution that can directly convert and generate a corresponding visible light image from the input near-infrared image during the usage phase, without the need for explicit intermediate steps or manual intervention.

[0091] A method for near-infrared-visible image conversion includes two phases: training and usage.

[0092] Step 1: During the training phase, the near-infrared-visible light paired dataset is used to iteratively modify and store the parameter dictionary of the algorithm model.

[0093] Step 1.1: Through frequency domain and spatial domain transformation, the input near-infrared image is mapped onto the feature spaces of the frequency domain and spatial domain respectively, and a grayscale image is generated.

[0094] For all pixels in the input near-infrared image, frequency domain transformation and spatial domain transformation are performed in parallel to extract global and local information of the image.

[0095] For frequency domain transformation, ResNet network architecture is preferred (see He K, Zhang X, Ren S, et al. Deep presidual learning for image recognition[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition.2016:770-778.), and for spatial domain transformation, U-net network architecture is preferred (see Ronneberger O, Fischer P, Brox TU-net: Convolutional networks for biomedical image segmentation[C] / / Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015:18th International Conference,Munich,Germany,October 5-9,2015,Proceedings,Part III 18.SpringerInternational Publishing,2015:234-241.).

[0096] During frequency domain transformation, Fourier transform and Fourier convolution are used to map pixels onto the feature space of the frequency domain. During spatial domain transformation, a traditional neural network is used to map pixels onto the feature space of the spatial domain. The depth of the feature space and the number of network parameters are controlled by adjusting the depth of the neural network. Finally, the global and local information extracted from the frequency and spatial domain transformations are combined to generate a grayscale image.

[0097] Step 1.2: Based on the luminance channel information in the generated grayscale image, recover its corresponding color channel information and generate a visible light image.

[0098] Specifically, based on the grayscale image obtained in step 1.1, the luminance channel information in the grayscale image is read. The images are grouped according to their luminance magnitude, treating the grayscale image colorization task as a classification problem. When generating visible light images, CT scans are preferred. 2The algorithm (see Weng S, Sun J, Li Y, et al. CT 2: Colorization Transformer via ColorTokens[C] / / Computer Vision–ECCV 2022:17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VII. Cham: Springer Nature Switzerland, 2022:1-16.) predicts the color channel information of each pixel based on the color probability distribution corresponding to different brightness levels in the grayscale-visible light image dataset. Finally, the brightness channel information and color channel information are concatenated to obtain the final visible light image.

[0099] All predicted grayscale and visible light images are stored for use in training.

[0100] Step 1.3: The intelligent conversion algorithm model generates the corresponding grayscale image and visible light image based on the input near-infrared image, compares them with the real grayscale image and visible light image respectively, calculates the loss function and updates the parameters in the model accordingly.

[0101] Specifically, several near-infrared images are randomly selected from the training dataset and input into the intelligent conversion algorithm model. For each near-infrared image, the intelligent conversion algorithm model first outputs a corresponding grayscale prediction image, and then generates the final visible light prediction image based on the predicted grayscale image. The width and height of the grayscale prediction image and the visible light prediction image output by the intelligent conversion algorithm model are consistent with the input near-infrared images.

[0102] The intelligent conversion algorithm model includes two functional modules: a grayscale image generation module and a visible light image generation module.

[0103] When a near-infrared image is input into the intelligent conversion algorithm model, it first passes through a grayscale image generation module. This module includes two conversion modules: frequency domain and spatial domain. Both are fully convolutional networks, which are based on translation invariance. The convolutional layers, pooling layers, and activation functions are all operated on the local input region and depend only on the relevant spatial coordinates.

[0104] Each layer of the grayscale image generation module has model parameters that are three-dimensional arrays of size b*h*w*d, where b is the batch size for model training, h and w are the spatial dimensions, and d is the number of channels. The first layer is an image of size h*w with d color channels. Let X be the vector at position (i, j) in a specific layer of the grayscale image generation module. ij After calculation, it becomes y.ij :

[0105] y ij =f ks ({X si+δi,sj+δj} 0≤δi,δj≤k (1)

[0106] Among them, f ks It determines the layer type, including convolution, average pooling, max pooling, and activation function; This indicates that the position in the feature map X involved in the calculation is (si+δ) i ,s j +δ j The vector; k is the kernel size; s is the stride of the convolution operation; δ i δ j For all integers in [0, k], let represent the vector involved in the calculation, denoted by s. i ,s j ) represents all vectors within a convolution kernel with a top-left vertex and a side length of k.

[0107] The grayscale image generation module uses a base loss function that is the sum of the absolute bias losses for each pixel, where the absolute bias loss function is:

[0108]

[0109] Where, x i Represents the actual grayscale pixel data, y i This represents the predicted grayscale pixel data. Minimizing the absolute bias loss allows the predicted value to approximate the true distribution as closely as possible.

[0110] In addition, the grayscale image generation module can select different loss functions, such as the smoothing absolute bias loss function and the mean square error loss function, depending on the specific details and the different training datasets.

[0111] The generated grayscale image is then sent to the visible light image generation module, which predicts the corresponding color channel information based on the brightness channel information in the grayscale image, and finally generates the visible light image.

[0112] The visible light image generation module is a deep learning model using a self-attention mechanism. It utilizes the ground truth of the target domain for supervised learning and includes an encoder and a decoder. The encoder employs a self-attention mechanism to capture long-range dependencies in the input sequence. This mechanism weights words at different positions in the sequence, incorporating all information. Specifically, each word in the encoder is compared with other words in the sequence, and an attention score is calculated to determine the weight of each word. The decoder also uses a self-attention mechanism, but introduces an additional attention mechanism: the encoder-decoder attention mechanism. This mechanism compares the vector representation of each position in the encoder with the vector representation of the current position in the decoder, thus incorporating information from the encoder. Simultaneously, the decoder introduces positional encoding to represent the relationships between positions in the input sequence. The module uses a base loss function that is the sum of the cross-entropy losses for each pixel, where the cross-entropy loss function is:

[0113]

[0114] Where, x i Let p represent the true color probability distribution and q represent the predicted color probability distribution. Minimizing the cross-entropy loss allows the predicted probability distribution to approximate the true distribution as closely as possible.

[0115] In addition, the grayscale image generation module can select different loss functions, such as reconstruction loss function, feature matching loss function, perceptual loss function, and multi-size discriminator loss function, depending on the specific details of the model and the training dataset.

[0116] Step 1.4: Repeat step 1.3 until the set termination condition is met, then save the network structure and model parameters.

[0117] During training, near-infrared images are repeatedly input into the intelligent conversion algorithm model, and the parameters in the model are dynamically adjusted according to the loss function until the preset conditions are met (such as: the number of training rounds reaches a certain amount or a certain evaluation index is better than a certain preset value). The training process stops and the model parameters are stored.

[0118] Step 2: Usage Phase. Using the model parameters obtained in Step 1 (training phase), the input near-infrared image is converted into a visible light image.

[0119] Furthermore, the generated visible light image and the real visible light image are evaluated, and the performance of the intelligent conversion algorithm model in converting the near-infrared image into a visible light image is judged.

[0120] Step 2.1: Create a network model based on the model structure and parameters saved during the training phase, input the near-infrared image into the intelligent conversion algorithm model, and perform the conversion. Store the generated visible light image.

[0121] The input near-infrared image is typically in RGB format and its size is usually preset. If the input near-infrared image is not in this format, it is first cropped and stored. Then, a network model is created based on the model structure and parameters saved during the training phase, and the near-infrared image is input into the intelligent conversion algorithm model. The intelligent conversion algorithm model generates a corresponding grayscale image through the grayscale image generation module. Then, the grayscale image is processed by the visible light image generation module to obtain the final predicted visible light image, and the generated grayscale and visible light images are automatically saved. This visible light image will be in RGB format, and its size will be consistent with the input near-infrared image.

[0122] Step 2.2: Evaluate the generated visible light image with the real visible light image, and determine the merits of the intelligent conversion algorithm model in converting the near-infrared image into a visible light image.

[0123] Specifically, to objectively assess the effectiveness of the network map generated by the model, objective image evaluation metrics can be generated based on PSNR (Peak Signal-to-Noise Ratio), SSIM (Structural Similarity), and RMSE (Root Mean Square Error). The generated visible light image can be directly compared with the real visible light image for evaluation and metric calculation. The PSNR calculation formula is as follows:

[0124]

[0125] Where H and W represent the length and width of the image, respectively, x(i,j) represents the actual grayscale pixel data, and y(i,j) represents the predicted grayscale pixel data;

[0126] The SSIM calculation formula is:

[0127]

[0128] Where x and y represent the pixel values ​​of the real image and the predicted image, respectively, and μ x μ y These represent the x-mean and y-mean respectively. The variances of x and y are respectively, σ xy Let c1 be the covariance of x and y, and c1 = (k1L). 2 c2 = (k2L) 2It is a constant used to maintain stability. L is the dynamic range of pixel values, for example, k1 = 0.01, k2 = 0.03.

[0129] The formula for calculating RMSE is:

[0130]

[0131] Where, x i Represents actual pixel data, y i This represents the predicted pixel data.

[0132] Furthermore, to better illustrate the purpose and advantages of the present invention, the system of the present invention will be further described below in conjunction with the accompanying drawings.

[0133] Example

[0134] like Figure 3 As shown in the figure. This embodiment discloses a system for near-infrared-visible light image conversion, including a real image processing subsystem, a grayscale image generation subsystem, a visible light image generation subsystem, a loss function calculation subsystem, and a generation result evaluation subsystem.

[0135] The real image processing subsystem includes a real image augmentation module and a real image formatting module.

[0136] The real image augmentation module is used to perform random cropping, rotation, and flipping operations on near-infrared-visible images to increase the paired near-infrared-visible image dataset, making the near-infrared images as diverse as possible, thereby enabling the trained model to have stronger generalization ability.

[0137] The real image formatting module is used to convert near-infrared-visible images stored in bitmap format into tensor format and normalize them to meet the input data format requirements of the grayscale image generation subsystem.

[0138] The grayscale image generation subsystem includes a frequency domain transformation module and a spatial domain transformation module, both of which are composed of convolutional layers, pooling layers, activation layers, and cross-layer connection structures. It is used to extract features from near-infrared images layer by layer, and can fully understand the semantic information corresponding to each pixel, providing important information for generating grayscale images.

[0139] The visible light image generation subsystem includes an encoder and a decoder using a self-attention mechanism. It predicts the corresponding color channel information based on the grayscale brightness channel information obtained from the grayscale image generation subsystem, treating this as a classification problem. The color channels are predicted according to the color probability distribution, ultimately generating a high-quality visible light image.

[0140] The loss function calculation subsystem includes a grayscale generation loss function calculation module and a visible light generation loss function calculation module. The grayscale generation loss function calculation module calculates the absolute deviation loss based on the real grayscale image and the grayscale image converted by the grayscale image generation subsystem. This loss is backpropagated to update the parameters of each neural network layer in the grayscale image generation subsystem, correcting the brightness information converted by the grayscale image generation subsystem. The visible light generation loss function calculation module calculates the cross-entropy loss based on the real visible light image and the final visible light image obtained by the visible light image generation subsystem. This loss is backpropagated to update the parameters of each neural network layer in the visible light image generation subsystem, and indirectly affects the parameters in the grayscale image generation subsystem through the visible light image generation subsystem, correcting the final visible light image generated by the system.

[0141] The generation result evaluation subsystem is used to compare the predicted visible light image with the real visible light image. It can automatically calculate a variety of objective evaluation indicators of the generated image, including PSNR, SSIM, and RMSE, as a supplement to the user's subjective experience and help the user quantitatively understand the quality of the visible light image generated by the system.

[0142] The connections between the above components are as follows:

[0143] The real image processing subsystem is connected to the grayscale image generation subsystem and the loss function calculation subsystem, and provides the grayscale image generation subsystem with processed near-infrared images that meet the requirements.

[0144] The grayscale image generation subsystem is connected to the real image processing subsystem, the visible light image generation subsystem, and the loss function calculation subsystem. It provides the visible light image generation subsystem with the predicted brightness channel information in the image, and at the same time receives the loss value returned by the visible light image generation subsystem for training itself.

[0145] The visible light image generation subsystem is connected to the grayscale image generation subsystem and the loss function calculation subsystem, respectively. It predicts the corresponding color channel information based on the brightness channel information and finally obtains the visible light image.

[0146] The loss function calculation subsystem is connected to the grayscale image generation subsystem, the visible light image generation subsystem, and the real image processing subsystem, respectively. It obtains the real grayscale image and the real visible light image from the real image processing subsystem, then obtains the grayscale image generated by the grayscale image generation subsystem, compares it with the real grayscale image to calculate the loss value, and then feeds the loss value back to the grayscale image generation subsystem for training. Simultaneously, the loss function calculation subsystem obtains the visible light image generated by the visible light image generation subsystem, compares it with the real visible light image to calculate the loss value, and then feeds the loss value back to the visible light image generation subsystem for training.

[0147] The result evaluation subsystem is connected to the real image processing subsystem and the visible light image generation subsystem. It obtains real visible light images from the real image processing subsystem and converted visible light images from the visible light image generation subsystem. By comparing and calculating evaluation indicators, it helps users quantitatively understand the quality of the visible light images converted by the system.

[0148] The working process of the above system is as follows:

[0149] Step 1: Input the near-infrared-visible light image into the real image processing subsystem, perform data augmentation, normalization and formatting, and store the processed near-infrared-visible light image in memory for later use;

[0150] Step 2: The grayscale image generation subsystem reads the near-infrared image, processes it through a neural network layer to generate a grayscale image, and calculates a loss function between the grayscale image and the real grayscale image. The brightness channel information from the grayscale image is then fed to the visible light image generation subsystem. The generated grayscale image is stored for later use.

[0151] Step 3: The visible light image generation subsystem reads the brightness channel information from the grayscale image, processes it through a neural network layer with a self-attention mechanism to generate the final visible light image, and calculates the loss function with the real visible light image; the generated visible light image is stored for later use.

[0152] Step 4: In the loss function calculation subsystem, the grayscale image absolute loss function calculation part reads the real grayscale image and the generated grayscale image, compares them, calculates the loss value, and feeds the loss value back to the grayscale image generation subsystem for training; in the visible light image cross-entropy loss function calculation part, the real visible light image and the generated visible light image are read and compared to calculate the loss value. Then, the loss value is fed back to the visible light generation subsystem for training.

[0153] Step 5: Each neural network layer in the visible light image generation subsystem modifies its parameters according to the loss value returned by the subsystem based on the loss function, completes one training iteration, and returns the loss value of the first layer to the grayscale image generation subsystem.

[0154] Step 6: Each neural network layer in the grayscale image generation subsystem calculates the loss value returned by the subsystem and the network map generation subsystem according to the loss function, modifies the parameters, and completes one training iteration;

[0155] Step 7: Repeat steps 1 to 6. After each iteration of all training data, send the real visible light image and the converted visible light image to the generation result evaluation subsystem to calculate the index; if the index meets the preset conditions, or the total number of iterations reaches the preset upper limit, stop training and proceed to step 8;

[0156] Step 8: Perform steps 1 to 3 to convert and generate the final visible light image.

[0157] Step 9: Send the real visible light image and the converted visible light image to the generation result evaluation subsystem to calculate the final index.

[0158] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for near-infrared-visible light image conversion, characterized in that, It includes two phases: training and use; Step 1: During the training phase, the parameter dictionary of the intelligent conversion algorithm model is iteratively modified using pairwise matched near-infrared-grayscale image datasets and grayscale-visible light datasets. Step 1.1: Map the input near-infrared image onto the feature spaces of the frequency domain and spatial domain respectively, and generate a grayscale image; Step 1.2: Based on the luminance channel information in the generated grayscale image, recover its corresponding color channel information and generate a visible light image; Store all predicted grayscale and visible light images; Step 1.3: Using the intelligent conversion algorithm model, generate the corresponding grayscale image and visible light image based on the input near-infrared image, compare them with the real grayscale image and visible light image respectively, calculate their loss function, and update the parameters in the intelligent conversion algorithm model accordingly. Several near-infrared images are randomly selected from the training dataset and input into the intelligent conversion algorithm model; For each near-infrared image, the intelligent conversion algorithm model first outputs a corresponding grayscale prediction image, and then generates the final visible light prediction image based on the predicted grayscale image. The width and height of the grayscale prediction image and the visible light prediction image output by the intelligent conversion algorithm model are consistent with the input near-infrared image. The intelligent conversion algorithm model includes a grayscale image generation module and a visible light image generation module. When a near-infrared image is input into the intelligent conversion algorithm model, it first passes through the grayscale image generation module. This module includes frequency domain conversion and spatial domain conversion. Both of these are fully convolutional networks, which are based on translation invariance. The convolutional layers, pooling layers, and activation functions are all operated on the local input region and depend only on the relevant spatial coordinates. In this module, each layer of the grayscale image generation module has a three-dimensional array of size b*h*w*d, where b is the batch number for model training, h and w are the spatial dimensions, and d is the number of channels. The first layer is an image with size h*w and d color channels. Let X be the vector at position (i, j) in a specific layer of the grayscale image generation module. ij After calculation, it becomes y. ij : Among them, f ks It determines the layer type, including convolution, average pooling, max pooling, and activation function; Feature maps representing computation In the middle, the position is ( The vector is ; k is the kernel size; s is the stride of the convolution operation; , To take values ​​in All integers, representing vectors involved in the calculation, are given by (...). () represents all vectors within the convolution kernel with side length k and top-left vertex; The grayscale image generation module uses a base loss function that is the sum of the absolute bias loss for each pixel, where the absolute bias loss function... for: in, Represents actual grayscale pixel data. This represents the predicted grayscale pixel data. This represents the number of pixels in the image; minimizing the absolute bias loss allows the predicted value to approximate the true distribution as closely as possible. The generated grayscale image is then sent to the visible light image generation module, which predicts the corresponding color channel information based on the brightness channel information in the grayscale image, and finally generates a visible light image. The visible light image generation module is a deep learning model using a self-attention mechanism, which utilizes the ground truth of the results in the target domain for supervised learning. The module includes an encoder and a decoder. The encoder uses a self-attention mechanism to capture long-range dependencies in the input sequence. This mechanism weights words at different positions in the sequence, incorporating all information. Each word in the encoder is compared with other words in the sequence, and an attention score is calculated to determine the weight of each word. The decoder also uses a self-attention mechanism. Building upon this, an encoder-decoder attention mechanism is introduced, comparing the vector representation of each position in the encoder with the vector representation of the current position in the decoder to incorporate information from the encoder. Simultaneously, the decoder introduces positional encoding to represent the relationships between positions in the input sequence. The basic loss function used by the visible light image generation module is the sum of the cross-entropy losses for each pixel in the image, where the cross-entropy loss function is... for: in, Let p represent the true color probability distribution and q represent the predicted color probability distribution. Minimizing the cross-entropy loss allows the probability distribution of the predicted value to approximate the true distribution as closely as possible. Step 1.4: Repeat step 1.3 until the set termination condition is met, then save the network structure and model parameters; During the training process, near-infrared images are repeatedly input into the intelligent conversion algorithm model, and the parameters in the model are dynamically adjusted according to the loss function until the preset conditions are met. Then the training process stops and the parameters of the intelligent conversion algorithm model are stored. Step 2: Usage phase. Using the intelligent conversion algorithm model parameters obtained in the training phase of Step 1, the input near-infrared image is converted into a visible light image and stored. The generated visible light image is evaluated against the real visible light image, and the performance of the intelligent conversion algorithm model in converting the near-infrared image into a visible light image is judged.

2. The method for near-infrared-visible image conversion as described in claim 1, characterized in that, In step 1.1, frequency domain transformation and spatial domain transformation are used to extract global and local information of all pixels in the input near-infrared image. In the frequency domain transformation, Fourier forward and inverse transforms and Fourier convolutions are used to map the pixels onto the feature space of the frequency domain. When performing spatial transformation, a neural network is used to map pixels onto the feature space of the spatial domain. The depth of the feature space and the number of network parameters are controlled by controlling the depth of the neural network. Finally, by combining the extracted global and local information, a grayscale image is generated.

3. The method for near-infrared-visible image conversion as described in claim 1, characterized in that, In step 1.2, firstly, based on the obtained grayscale image, the brightness channel information in the grayscale image is read; Then, the images are grouped according to their brightness, and the grayscale image coloring task is treated as a classification problem. Based on the color probability distribution corresponding to different brightness levels in the grayscale-visible light image dataset, the color channel information of each pixel is predicted. Finally, the luminance channel information and color channel information are stitched together to obtain the final visible light image; The result is converted to the RGB color space using standard color space conversion functions.

4. The method for near-infrared-visible image conversion as described in claim 1, characterized in that, The grayscale image generation module selects different loss functions based on specific details and different training datasets, including the smoothing absolute bias loss function and the mean squared error loss function.

5. The method for near-infrared-visible image conversion as described in claim 1, characterized in that, The visible light image generation module selects different loss functions based on the specific details of the intelligent conversion algorithm model and the different training datasets, including reconstruction loss function, feature matching loss function, perception loss function, and multi-size discriminator loss function.

6. The method for near-infrared-visible image conversion as described in claim 1 or 2, characterized in that, Frequency domain conversion uses the ResNet network architecture, while spatial domain conversion uses the U-Net network architecture.

7. The method for near-infrared-visible image conversion as described in claim 1, characterized in that, In step 2, the input near-infrared image is converted to RGB format and the size is preset; if the input near-infrared image is not in this format, it is first cropped and saved. Then, based on the intelligent conversion algorithm model structure and parameters saved during the training phase, an intelligent conversion algorithm model is created. The near-infrared image is input into the model for conversion to generate the corresponding grayscale image. Then, the grayscale image is processed by the visible light image generation module to obtain the final predicted visible light image, and the generated grayscale image and visible light image are saved. The visible light image is in RGB image format and its size is consistent with the input near-infrared image.

8. A system for near-infrared-visible image conversion, characterized in that, It includes a real image processing subsystem, a grayscale image generation subsystem, a visible light image generation subsystem, a loss function calculation subsystem, and a generation result evaluation subsystem; The real image processing subsystem includes a real image augmentation module and a real image formatting module. The Real Image Augmentation module is used to manipulate near-infrared-visible images, including random cropping, rotation, and flipping, to augment paired-match near-infrared-visible image datasets; The real image formatting module is used to convert near-infrared-visible images stored in bitmap format into tensor format and normalize them to meet the input data format requirements of the grayscale image generation subsystem. The grayscale image generation subsystem includes a frequency domain transformation module and a spatial domain transformation module, both of which are composed of convolutional layers, pooling layers, activation layers, and cross-layer connection structures. It is used to extract features from near-infrared images layer by layer, and can fully understand the semantic information corresponding to each pixel, providing important information for generating grayscale images. The visible light image generation subsystem includes an encoder and a decoder using a self-attention mechanism; it predicts the corresponding color channel information based on the grayscale brightness channel information obtained from the grayscale image generation subsystem, treating it as a classification problem, predicting the color channel based on the color probability distribution, and finally generating a visible light image; The loss function calculation subsystem includes a grayscale generation loss function calculation module and a visible light generation loss function calculation module. The grayscale generation loss function calculation module calculates the absolute deviation loss based on the real grayscale image and the grayscale image converted by the grayscale image generation subsystem. This loss is backpropagated to update the parameters of each neural network layer in the grayscale image generation subsystem, correcting the brightness information converted by the grayscale image generation subsystem. The visible light generation loss function calculation module calculates the cross-entropy loss based on the real visible light image and the final visible light image obtained by the visible light image generation subsystem. This loss is backpropagated to update the parameters of each neural network layer in the visible light image generation subsystem, and indirectly affects the parameters in the grayscale image generation subsystem through the visible light image generation subsystem, correcting the final visible light image generated by the system. The generation result evaluation subsystem is used to compare the predicted visible light image with the real visible light image and can calculate the objective evaluation index of the generated image. The connections between the above components are as follows: The real image processing subsystem is connected to the grayscale image generation subsystem and the loss function calculation subsystem, and provides the grayscale image generation subsystem with processed near-infrared images that meet the requirements. The grayscale image generation subsystem is connected to the real image processing subsystem, the visible light image generation subsystem, and the loss function calculation subsystem. It provides the visible light image generation subsystem with the predicted brightness channel information in the image, and at the same time receives the loss value returned by the visible light image generation subsystem for training itself. The visible light image generation subsystem is connected to the grayscale image generation subsystem and the loss function calculation subsystem, respectively. It predicts the corresponding color channel information based on the brightness channel information and finally obtains the visible light image. The loss function calculation subsystem is connected to the grayscale image generation subsystem, the visible light image generation subsystem, and the real image processing subsystem, respectively. It obtains real grayscale and real visible light images from the real image processing subsystem, then obtains the grayscale image generated by the grayscale image generation subsystem, compares it with the real grayscale image to calculate the loss value, and then feeds the loss value back to the grayscale image generation subsystem for training. Simultaneously, the loss function calculation subsystem obtains the visible light image generated by the visible light image generation subsystem, compares it with the real visible light image to calculate the loss value, and then feeds the loss value back to the visible light image generation subsystem for training. The result evaluation subsystem is connected to the real image processing subsystem and the visible light image generation subsystem. It obtains real visible light images from the real image processing subsystem and converted visible light images from the visible light image generation subsystem. By comparing and calculating evaluation indicators, it helps users quantitatively understand the quality of the visible light images converted by the system. The working process of the above system is as follows: Step 1: Input the near-infrared-visible light image into the real image processing subsystem, perform data augmentation, normalization and formatting, and store the processed near-infrared-visible light image in memory for later use; Step 2: The grayscale image generation subsystem reads the near-infrared image, processes it through a neural network layer to generate a grayscale image, and calculates a loss function with the real grayscale image. The brightness channel information in the grayscale image is sent to the visible light image generation subsystem. The generated grayscale image is stored for later use. Step 3: The visible light image generation subsystem reads the brightness channel information from the grayscale image, processes it through a neural network layer with a self-attention mechanism to generate the final visible light image, and calculates the loss function with the real visible light image; the generated visible light image is stored for later use. Step 4: In the loss function calculation subsystem, the grayscale image absolute loss function calculation part reads the real grayscale image and the generated grayscale image, compares them to calculate the loss value, and then sends the loss value back to the grayscale image generation subsystem for training; in the visible light image cross-entropy loss function calculation part, the real visible light image and the generated visible light image are read and compared to calculate the loss value; then, the loss value is sent back to the visible light generation subsystem for training. Step 5: Each neural network layer in the visible light image generation subsystem modifies its parameters according to the loss value returned by the subsystem based on the loss function, completes one training iteration, and returns the loss value of the first layer to the grayscale image generation subsystem. Step 6: Each neural network layer in the grayscale image generation subsystem calculates the loss value returned by the subsystem and the network map generation subsystem according to the loss function, modifies the parameters, and completes one training iteration; Step 7: Repeat steps 1 to 6; after each iteration of all training data, send the real visible light image and the converted visible light image to the generation result evaluation subsystem to calculate the index; if the index meets the preset conditions, or the total number of iterations reaches the preset upper limit, stop training and proceed to step 8. Step 8: Perform steps 1 to 3 to convert and generate the final visible light image; Step 9: Send the real visible light image and the converted visible light image to the generation result evaluation subsystem to calculate the final index.

Images