Color night vision imaging method based on infrared imaging

Abstract

The invention discloses a color night vision imaging method based on infrared imaging, and the method comprises the steps: enabling a low-light sensor and a neural network to be embedded into an infrared imaging system, and inputting the contour information extracted through an infrared imaging technology and the weak color information in a dark state into a deep learning neural network; and fitting the image with a real color through a neural network, and finally outputting a target image, so that the night vision device can generate a night vision image with a real color. According to the method, the visibility and authenticity of night vision imaging can be effectively improved, information in various spectral bands can be accurately distinguished, powerful support is provided for follow-up data analysis and decision making, and the method is suitable for multiple fields such as traffic, aviation, medical treatment and security and protection.

Description

Technical Field

[0001] The invention relates to an image processing method, in particular to a color night vision imaging method. Background Art

[0002] At present, night vision imaging technology is widely used in security, transportation, medical and other fields. At night or in low light environments, human vision is significantly reduced, but for some special fields, accurate night vision imaging technology can greatly improve work efficiency and safety.

[0003] Traditional infrared imaging technology achieves night vision imaging by detecting and recording infrared radiation, so the picture is presented in black and white. In order to make up for the above limitations, the existing color night vision technology mainly uses pseudo-color technology, that is, the monochrome image is colored by algorithm to present a color image. The existing algorithms are usually based on the extraction, analysis and mapping of infrared radiation features.

[0004] However, since the data processed by pseudo-color technology does not contain color information itself, pseudo-color images still have limitations, and their reconstruction quality depends on the understanding and operation of the algorithm and operator. In addition, the color range of pseudo-color images is usually very limited, and color accuracy and visibility are also limited. Summary of the invention

[0005] Purpose of the invention: In view of the above-mentioned prior art, a color night vision imaging method based on infrared imaging is proposed, which can effectively improve the visibility and authenticity of night vision imaging.

[0006] Technical solution: A color night vision imaging method based on infrared imaging, comprising: adding a low-light sensor to an infrared night vision device, using the image collected by the infrared sensor as contour information, and the image collected by the low-light sensor as color information, respectively inputting them into a neural network, performing feature map fusion and feature extraction in the neural network, and then fitting the real image to output a target image with real color.

[0007] Furthermore, the model structure of the neural network has two inputs, infrared sensor image and low-light sensor image, with an input size of 640×480×3, which are convolved by 3×3 / 2 convolution kernels with 32 channels and output two sizes of 320×240×32; the second part is a feature map fusion layer, which fuses the two inputs with a size of 320×240×32 and outputs 320×240×64; the third part consists of 8 convolution layers, each with a convolution kernel size of 1. The sizes of the convolution kernels are 3×3 / 2 and 1×1 / 1 alternating in sequence, and the numbers of channels are 64, 128, 128, 256, 256, 512, and 102 respectively, and the final output size is 20×15×1024; the fourth part consists of 10 deconvolution layers, and the sizes of the convolution kernels are 4×4 / 2 and 1×1 / 1 alternating in sequence, and the numbers of channels are 1024, 512, 512, 256, 256, 128, 128, 64, 64, and 3 respectively, and the output size is 640×480×3.

[0008] Furthermore, the training of the neural network includes the following steps: taking groups of pictures for several different scenes respectively as training sets and test sets; each group of pictures includes infrared sensor images and low-light sensor images of scenes taken in a dark room, and images of the same scenes taken by a high-lumen light source; using the weighted average of the MSE function and the improved SSIM function to calculate the loss value; using the training set and the test set to train and test the established neural network, during the training process, updating the model parameters according to the loss value, and using the Adam algorithm to adjust the learning rate.

[0009] Furthermore, the loss function constructed using the MSE function and the improved SSIM function is expressed as: LOSS=SSIM(X,Y)×0.4+MESloss×0.6; wherein MESloss represents the mean square error between the predicted value and the actual value; SSIM(X,Y) represents the similarity between pixel block X and pixel block Y; wherein, after the infrared sensor image and the low-light sensor image are input into the neural network, the fused feature map output by the first and second parts of the model structure is used as the pixel block X; and the images of the same scene taken by the corresponding high-lumen light source are used as the two inputs of the neural network, and the fused feature map output by the first and second parts of the model structure is used as the pixel block Y; the expression of SSIM(X,Y) is: SSIM(X,Y)=[L(X,Y)] a [C(X,Y)] b [S(X,Y)] c ; Where, L(X,Y)=(2μ x μ y +C1) / (μ x 2 +μ y 2+C1), C(X,Y)=(2σ x σ y +C2) / (σ x 2 +σ y 2 +C2), S(X,Y)=(σ xy +C3) / (σ x σ y +C3); where μ x , μ y Respectively represent the average values ​​of pixel block X and pixel block Y; L(X,Y) represents the comparison of the brightness of pixel block X and pixel block Y; C(X,Y) represents the comparison of the contrast of pixel block X and pixel block Y. First, pixel block X and pixel block Y are divided into sub-blocks with e*e pixels each, and the median of the pixel values ​​of the pixels contained in each pixel sub-block is taken as the block pixel value of this pixel sub-block. The standard deviation σ of the block pixel values ​​of the pixel sub-blocks in pixel block X and pixel block Y is calculated respectively. x With σ y ; S(X,Y) represents the structure of comparing pixel block X and pixel block Y. First, pixel block Y is convolved with the Laplace operator and the image pixel values ​​are normalized to [0,1]. Pixels with values ​​greater than 0.5 are extracted, and pixels at the same position in pixel block X are extracted. σ xy Represents the covariance of the pixels extracted from pixel block X and the pixels extracted from pixel block Y; a>0, b>0, c>0 are coefficients for adjusting the relative importance of L(X,Y), C(X,Y), and S(X,Y); C1, C2, and C3 are all constant coefficients used to maintain the stability of L(X,Y), C(X,Y), and S(X,Y).

[0010] Furthermore, a=b=c=1, C1=(k1*L) 2 , C2=(k2*L) 2 , C2=2C3, L is the dynamic range of pixel values, coefficient k1=0.03, coefficient k2=0.04.

[0011] Beneficial effect: The present invention embeds a low-light sensor and a neural network into an infrared imaging system, inputs the contour information extracted by infrared imaging technology and the color information that exists faintly in a dark state into a deep learning neural network, fits it to a real color map through the neural network, and finally outputs the target image, so that the night vision device can produce night vision images with real colors.

[0012] This method can effectively improve the visibility and authenticity of night vision imaging, and accurately distinguish information in various spectral bands, providing strong support for subsequent data analysis and decision-making. It is applicable to multiple fields, such as transportation, aviation, medical care, and security. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is the flow chart of the method; Figure 2 This is a schematic diagram of the model structure of the neural network of this method; Figure 3 This is the principle diagram of night vision imaging of this method; Figure 4 An example for comparing the contrast of two pixel blocks in the improved SSIM function of the present invention; Figure 5 This is a rendering of an embodiment of the present invention. DETAILED DESCRIPTION

[0014] The present invention will be further explained below in conjunction with the accompanying drawings.

[0015] A color night vision imaging method based on infrared imaging includes the steps of establishing an image data set containing rich scenes, establishing a deep learning neural network model, constructing a loss function for model training, and training and testing the model.

[0016] 1. To establish an image dataset containing a variety of scenes, take several sets of pictures as training sets and test sets. Each set of pictures includes infrared sensor images taken in a dark room, low-light sensor images, and images of the same scene taken with a high-lumen light source. In this embodiment, 200 scenes are arranged, and 3 types of photos are taken from 25 different angles for each scene, so that the trained neural network is more universal.

[0017] 2. Establish the backbone model of the neural network. The backbone of the neural network mainly uses 3×3 / 2 convolution kernels and 1×1 / 1 convolution kernels. Each layer of convolution is followed by a ReLU activation function. At the end of the model, 4×4 / 2 and 1×1 / 1 deconvolution kernels are mainly used to restore the feature map to the input size for fitting.

[0018] Specifically, Figure 2 As shown in the figure, the first part of the model structure of the neural network has two inputs: infrared sensor image and low-light sensor image. Since the infrared sensor collects infrared image data of three different band channels, namely short-wave infrared (SWIR), medium-wave infrared (MWIR) and long-wave infrared (LWIR), and the low-light sensor collects signals of three channels of red, green and blue, the image input size is 640×480×3, and the 3×3 / 2 convolution kernel with 32 channels performs convolution respectively and outputs two sizes of 320×240×32.

[0019] The second part is the feature map fusion layer, which fuses the feature maps of two inputs of size 320×240×32 and outputs 320×240×64.

[0020] The third part consists of 8 convolutional layers. The sizes of the convolution kernels are 3×3 / 2 and 1×1 / 1 alternating in sequence. The number of channels is 64, 128, 128, 256, 256, 512, and 102 respectively. The final output size is 20×15×1024.

[0021] The fourth part consists of 10 deconvolution layers. The sizes of the convolution kernels are 4×4 / 2 and 1×1 / 1 alternating in sequence. The number of channels are 1024, 512, 512, 256, 256, 128, 128, 64, 64, and 3 respectively, and the output size is 640×480×3.

[0022] 3. Use the weighted average of the MSE (mean square error) function and the improved SSIM (Structural Similarity Index) function to comprehensively calculate the loss value: LOSS=SSIM(X,Y)×0.4+MESloss×0.6.

[0023] Among them, MESloss represents the mean square error between the predicted value and the actual value of the neural network:

[0024] Where N, H, and W represent the number of channels, height, and width of the image, respectively; y{x,y,z} represents the true pixel value of the xth channel, yth row, and zth column of the image. Represents the corresponding predicted pixel value, which comes from the image of the same scene taken by the high lumen light source corresponding to this image in the dataset; n represents the total pixel value of all channels of the image, that is, n=N*H*W.

[0025] SSIM(X,Y) represents the similarity between pixel block X and pixel block Y. After the infrared sensor image and the low-light sensor image are input into the neural network, the fused feature map output by the first and second parts of the model structure is used as the pixel block X; and the images of the same scene taken with the corresponding high-lumen light source are used as the two inputs of the neural network, and the fused feature map output by the first and second parts of the model structure is used as the pixel block Y.

[0026] The expression of SSIM(X,Y) is: SSIM(X,Y)=[L(X,Y)] a [C(X,Y)] b [S(X,Y)] c ; Where, L(X,Y)=(2μ x μ y +C1) / (μ x 2 +μ y 2 +C1), C(X,Y)=(2σx σ y +C2) / (σ x 2 +σ y 2 +C2), S(X,Y)=(σ xy +C3) / (σ x σ y +C3); where L(X,Y) is used to compare the brightness of pixel block X and pixel block Y, μ x , μ y They represent the average values ​​of pixel block X and pixel block Y respectively; C(X,Y) is used to compare the contrast between pixel block X and pixel block Y; S(X,Y) is used to compare the structure of pixel block X and pixel block Y; a>0, b>0, c>0 are coefficients for adjusting the relative importance of L(X,Y), C(X,Y), and S(X,Y); C1, C2, and C3 are all constant coefficients used to maintain the stability of L(X,Y), C(X,Y), and S(X,Y).

[0027] C(X,Y) is used to compare the contrast of pixel block X and pixel block Y. First, pixel block X and pixel block Y are divided into sub-blocks with e*e pixels each, where the value of e ranges from 4 to 64. The median of the pixel values ​​of each pixel sub-block is taken as the block pixel value of this pixel sub-block. The standard deviation σ of the block pixel values ​​of the pixel sub-blocks in pixel block X and pixel block Y is calculated respectively. x With σ y . The existing mainstream SSIM function directly uses all pixel values ​​in pixel blocks X and Y to calculate their respective standard deviations, which requires a large amount of calculations; at the same time, the night image is dark overall and accompanied by local point highlights (such as car lights, etc.). If the local point highlights are used to participate in the standard deviation calculation of the night image according to the mainstream SSIM function, the overall dark characteristics of the image will be destroyed. The improved SSIM function of the present invention divides the sub-blocks and takes the median of each sub-block to reduce the chances of local point highlights in the standard deviation calculation, thereby reducing the standard deviation level of night images with only small highlighted areas, distinguishing them from significantly high-contrast images with large highlights, and further more accurately confirming the contrast characteristics of night images. Figure 4 As shown in the figure, for a 4*4 pixel matrix, except for a highlighted pixel in the lower right corner, the rest of the pixels are dark as a whole, and the standard deviation value should be at a low level. The image standard deviation calculated by the mainstream SSIM function is 59.441, and the improved SSIM function of the present invention is used to divide the image into 4 blocks and use median sampling to completely eliminate the influence of a single highlight pixel on the image features. The calculated image standard deviation is 25.1661, and the overall standard deviation level is reduced by half.

[0028] S(X,Y) is used to compare the structure of pixel block X and pixel block Y. First, pixel block Y is convolved with the Laplace operator and the image pixel values ​​are normalized to [0,1]. Pixels with values ​​greater than 0.5 are extracted, and pixels at the same position in pixel block X are extracted. xy It represents the covariance of the pixels extracted from the pixel block X and the pixels extracted from the pixel block Y. The existing mainstream SSIM function directly uses all the pixel values ​​in the pixel block X and the pixel block Y to calculate the covariance, which not only compares the structural features, but also analyzes the brightness and contrast features of the pixel blocks. Compared with C(X,Y) and L(X,Y), there is an overlap in content. The present invention only performs covariance analysis on the extracted pixel block contour features to avoid being affected by other features when converging the structural features.

[0029] In this embodiment, set a=b=c=1, C1=(k1*L) 2 , C2=(k2*L) 2 , C2=2C3, L is the dynamic range of pixel values ​​(255 for 8-bit image), k1=0.03, k2=0.04, then: SSIM(X,Y)=(2μ x μ y +C1)(2σ x σ y +C2) / [(μ x 2 +μ y 2 +C1)(σ x 2 +σ y 2 +C2)].

[0030] 4. Introduce the Adam (Adaptive Moment Estimation) algorithm, which is an optimization algorithm for adaptive learning rate that combines RMSProp and momentum. It is used to optimize the gradient of neural network parameters to improve the efficiency and stability of training.

[0031] The learning rate update formula of the Adam algorithm is expressed as: t+1 =w t -α·m t / √(v t +ε), where w t represents the parameter to be optimized; α represents the learning rate, which indicates the step size of updating the weights at each iteration; ε is a small constant used for numerical stability to avoid the denominator being 0; m t and v t They represent the first-order moment estimate (momentum estimate for calculating the gradient) and the second-order moment estimate (momentum estimate for calculating the square of the gradient), respectively, and are specifically expressed as: m t=β 1 ·m t-1 +(1-β 1 ) t , v t =β 2 ·v t-1 +(1-β 2 ) t 2 , where g t represents the current parameter gradient; β 1 represents the exponentially weighted average coefficient of the first-order moment estimate; β 2 represents the exponentially weighted average coefficient of the second-order moment estimate.

[0032] In this embodiment, take β 1 =0.9,β 2 =0.999, ε=1e-8, and the learning rate is adaptively adjusted by calculating the update amount of the weight parameters, so that the learning rate can be adaptively adjusted within different input data ranges, thereby learning from the training data more effectively.

[0033] The established neural network is trained and tested using the prepared training set and test set. During the training process, the model parameters are updated according to the calculated loss value LOSS, the learning rate is adjusted using the Adam algorithm, and the weighted parameters for loss value calculation are adjusted according to the test results.

[0034] A low-light sensor is added to the infrared night vision device, and the image collected by the infrared sensor is used as contour information, and the image collected by the low-light sensor is used as color information, which are respectively input into the trained neural network mentioned above. After feature map fusion and feature extraction are performed in the neural network, the real image is fitted to achieve the output of the target image with real color based on the input infrared image and low-light image.

[0035] like Figure 5 Taking the example shown in the figure, the image data collected by the infrared sensor and the low-light sensor are respectively extracted and fused using the neural network model. After training, a high-quality night vision restoration picture will be obtained.

[0036] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A color night vision imaging method based on infrared imaging, characterized in that: include: A low-light sensor is added to the infrared night vision device, and the image collected by the infrared sensor is used as contour information, and the image collected by the low-light sensor is used as color information, which are respectively input into the neural network. After feature map fusion and feature extraction are performed in the neural network, the real image is fitted to output the target image with real color.

2. The color night vision imaging method based on infrared imaging according to claim 1, characterized in that: The model structure of the neural network has two inputs, infrared sensor image and low-light sensor image, with an input size of 640×480×3. The 3×3 / 2 convolution kernel with 32 channels performs convolution respectively and outputs two sizes of 320×240×32. The second part is the feature map fusion layer, which performs feature map fusion on the two inputs with a size of 320×240×32 and outputs 320×240×64. The third part consists of 8 convolution layers, and the size of each convolution kernel is 3×3 / 2 and 1×1 / 1 are alternated in sequence, and the number of channels is 64, 128, 128, 256, 256, 512, and 102 respectively, and the final output size is 20×15×1024; the fourth part consists of 10 deconvolution layers, and the size of each convolution kernel is 4×4 / 2 and 1×1 / 1 alternating in sequence, and the number of channels is 1024, 512, 512, 256, 256, 128, 128, 64, 64, and 3 respectively, and the output size is 640×480×3.

3. The color night vision imaging method based on infrared imaging according to claim 2, characterized in that: The training of the neural network includes the following steps: taking picture groups for several different scenes respectively as training sets and test sets; each group of pictures includes infrared sensor images and low-light sensor images of scenes taken in a dark room, and images of the same scenes taken by a high-lumen light source; using the weighted average of the MSE function and the improved SSIM function to calculate the loss value; using the training set and the test set to train and test the established neural network, during the training process, updating the model parameters according to the loss value, and using the Adam algorithm to adjust the learning rate.

4. The color night vision imaging method based on infrared imaging according to claim 3, characterized in that: The loss function constructed using the MSE function and the improved SSIM function is expressed as: LOSS=SSIM(X,Y)×0.4+MESloss×0.6; where MESloss represents the mean square error between the predicted value and the actual value; SSIM(X,Y) represents the similarity between pixel block X and pixel block Y; where the infrared sensor image and the low-light sensor image are input into the neural network, the fused feature map output by the first and second parts of the model structure is used as the pixel block X; and the images of the same scene taken by the corresponding high-lumen light source are used as the two inputs of the neural network, and the fused feature map output by the first and second parts of the model structure is used as the pixel block Y; the expression of SSIM(X,Y) is: SSIM(X,Y)=[L(X,Y)] a [C(X,Y)] b [S(X,Y)] c ; Where, L(X,Y)=(2μ x μ y +C1) / (μ x 2 +μ y 2 +C1), C(X,Y)=(2σ x σ y +C2) / (σ x 2 +σ y 2 +C2), S(X,Y)=(σ xy +C3) / (σ x σ y +C3); where μ x , μ y Respectively represent the average values ​​of pixel block X and pixel block Y; L(X,Y) represents the comparison of the brightness of pixel block X and pixel block Y; C(X,Y) represents the comparison of the contrast of pixel block X and pixel block Y. First, pixel block X and pixel block Y are divided into sub-blocks with e*e pixels each, and the median of the pixel values ​​of the pixels contained in each pixel sub-block is taken as the block pixel value of this pixel sub-block. The standard deviation σ of the block pixel values ​​of the pixel sub-blocks in pixel block X and pixel block Y is calculated respectively. x With σ y ; S(X,Y) represents the structure of comparing pixel block X and pixel block Y. First, pixel block Y is convolved with the Laplace operator and the image pixel values ​​are normalized to [0,1]. Pixels with values ​​greater than 0.5 are extracted, and pixels at the same position in pixel block X are extracted. σ xy Represents the covariance of the pixels extracted from pixel block X and the pixels extracted from pixel block Y; a>0, b>0, c>0 are coefficients for adjusting the relative importance of L(X,Y), C(X,Y), and S(X,Y); C1, C2, and C3 are all constant coefficients used to maintain the stability of L(X,Y), C(X,Y), and S(X,Y).

5. The color night vision imaging method based on infrared imaging according to claim 4, characterized in that: a=b=c=1, C1=(k1*L) 2 , C2=(k2*L) 2 , C2=2C3, L is the dynamic range of pixel values, coefficient k1=0.03, coefficient k2=0.04.

Images