Construction method of Forklan neural network for day and night image conversion

Abstract

A construction method of a Forkgan neural network for day and night image conversion comprises the following steps: step 1, establishing a training image data set which comprises a night traffic image data set and a day traffic image data set; step 2, constructing an improved Forklan network model which comprises the three improved modules provided by the invention; and step 3, carrying out network convergence based on loss. The objective of the invention is to solve the technical problems that the pixel spatial distance description degree is poor, high-level feature details are lost, and deconvolution causes a chessboard effect due to complete loss of local information in the existing day and night image conversion, so that the detail information of the image is easy to lose. The construction method of the Forkgan neural network for day and night image conversion is provided.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for constructing a ForkGAN neural network for day-night image conversion. Background Art

[0002] Night vision tasks are one of the many challenges in the field of deep learning. Network models trained on a dataset in a certain field usually face performance degradation in other fields, namely data bias. Taking object detection in both day and night conditions as an example, an object detection model trained on a daytime dataset will generally experience a 30%-50% drop in accuracy when placed on nighttime images. The most direct solution is to collect more labeled data under nighttime conditions and then retrain, but this method is costly, time-consuming, and does not fundamentally solve the problem of data bias. A better approach is domain adaptation, which is to use images from a certain field to automatically synthesize images from another field. In the field of day-night conversion, researchers usually have two strategies: one is the data enhancement method, which manually processes daytime images into nighttime images, and then uses daytime images as labels to train the network. This method requires that daytime and nighttime images appear in pairs in the dataset, which undoubtedly increases the difficulty of data preparation and cannot solve some complex image conversion problems, such as illumination changes in day-night conversion; the other is to use a generative network to generate daytime images based on the domain-invariant features of nighttime images. This method does not require paired datasets and can capture subtle differences in illumination and color between day and night.

[0003] In order to solve the problems of poor description of pixel space distance, loss of high-level feature details and checkerboard effect caused by complete loss of local information due to deconvolution, and bilinear interpolation does not have the ability to learn, which will result in loss of detail information, a method for constructing a ForkGAN neural network for day and night image conversion is proposed. Summary of the invention

[0004] The purpose of the present invention is to solve the technical problems existing in the existing day-night image conversion, such as poor pixel space distance description, loss of high-level feature details, and checkerboard effect caused by complete loss of local information in deconvolution, which leads to the easy loss of detail information of the image. A method for constructing a Forkgan neural network for day-night image conversion is proposed.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] A method for constructing a ForkGAN neural network for day-night image conversion includes the following steps:

[0007] Step 1: Establish a training image dataset, which includes a night traffic image dataset and a day traffic image dataset;

[0008] Step 2: Construct the improved ForkGAN network model as follows Figure 2 As shown, the training data set in step 1 is imported into the network model for training, wherein the improved Forkgan network model includes a multi-scale feature fusion module, a perceptual loss module, and a convolution upsampling module;

[0009] Step 3: Based on the loss, the network converges.

[0010] In step 2, the improved ForkGAN network model is specifically divided into two parts:

[0011] In the first part, we input a night image and output a day image:

[0012] The night image Ix is input to the encoder E x , the output is connected to the input of the multi-scale feature fusion module, and the multi-scale feature fusion module outputs the feature z x ; The outputs of the multi-scale feature fusion module are connected to the reconstruction generator and the transformation generator Input, feature z x Input Reconstruction Generator The convolution upsampling module in the generator restores the size and outputs a night picture Feature z x Input Transformation Generator The convolution upsampling module in the generator restores the size and outputs a daytime picture

[0013] Among them, the conversion generator The output of the second encoder E x Input, transformation generator Output After the second encoder E x Output characteristics The second encoder E x The output connection transformation generator Input, features Input Transformation Generator Output night pictures Based on the perception loss module, the loss is continuously reduced, the network is continuously converged, and the image quality is continuously improved;

[0014] The second part, input daytime images, output nighttime images:

[0015] Daytime Image I y Input encoder E y, the output is connected to the input of the multi-scale feature fusion module, and the multi-scale feature fusion module outputs the feature z y ; The outputs of the multi-scale feature fusion module are connected to the reconstruction generator and the transformation generator Input, feature z y Input Reconstruction Generator The convolution upsampling module in the generator restores the size and outputs a daytime picture Feature z y Input Transformation Generator The convolution upsampling module in the generator restores the size and outputs a night picture

[0016] Among them, the conversion generator The output of the second encoder E y Input, transformation generator Output After the second encoder E y Output characteristics The second encoder E y The output connection transformation generator Input, features Input Transformation Generator Output daytime pictures Based on the perceptual loss module, the loss continues to decrease, the network continues to converge, and the output image quality continues to improve.

[0017] The multi-scale feature fusion module is specifically:

[0018] The first layer is the output of the 1x1 convolutional layer connected to the input of the position attention mechanism module PAM, and the first layer outputs the feature map F1;

[0019] The second layer is the output of the 3x3 convolutional layer with a void rate of 6, which is connected to the input of the position attention mechanism module PAM. The second layer outputs the feature map F2;

[0020] The third layer is the output of the 3x3 convolutional layer with a void rate of 12, which is connected to the input of the position attention mechanism module PAM. The third layer outputs the feature map F3;

[0021] The fourth layer is the output of the 3x3 convolutional layer with a void rate of 18, which is connected to the input of the position attention mechanism module PAM. The fourth layer outputs the feature map F4;

[0022] The fifth layer is the convolutional block attention mechanism module CBAM, and the fifth layer outputs the feature map F5;

[0023] Feature maps F1, F2, F3, F4, and F5 are fused to output feature map F6; feature map F6 is input into the 1x1 convolutional layer output processed feature map F7.

[0024] The position attention mechanism module PAM is specifically:

[0025] The feature map A{C*H*W} is input into three convolutional layers respectively to output three feature maps {B, C, D} with a shape of C*H*W; then the three feature maps {B, C, D} are reshaped into C*N, where C represents the number of channels, H represents the feature map height, W represents the feature map width, and N=H×W represents the number of pixels; the feature maps C and B are transposed and multiplied, and then the attention feature map S{N*N} is obtained through softmax; the feature maps D and S are transposed and multiplied, and the shape of the resulting feature map is reconstructed into C×H×W and then multiplied by the scale factor α, and added to the input feature map A to obtain the final output feature map E.

[0026] The convolutional block attention mechanism module CBAM is specifically:

[0027] The input feature is multiplied by the output of the channel attention mechanism module and the input feature, and the intermediate feature is output; the intermediate feature is multiplied by the output of the spatial attention mechanism module and the result is output;

[0028] Among them, the channel attention mechanism module is: the input feature map passes through two parallel MaxPool layers and AvgPool layers, the feature map is changed from C*H*W to C*1*1, and then passes through the Share MLP module, in which it first compresses the number of channels to 1 / r times the original, where r is the Reduction rate, and then expands to the original number of channels, and obtains two activated results through the ReLU activation function; the two output results are added element by element, and then a sigmoid activation function is used to obtain the output result of the channel attention mechanism, and then the output result is multiplied by the original image to change back to the size of C*H*W; the spatial attention mechanism module is: the output result of the channel attention mechanism is obtained through maximum pooling and average pooling to obtain two 1*H*W feature maps, and then the two feature maps are spliced ​​through the Concat operation, converted into a 1-channel feature map through a 7*7 convolution, and then a sigmoid function is used to obtain the feature map of the spatial attention mechanism, and finally the output result is multiplied by the original image to change back to C*H*W size.

[0029] Figure 6 for Figure 2Schematic diagram of the structure of the convolution upsampling module in , the structure of the convolution upsampling module is: the image of the original image size is input into the encoder output low-resolution feature map of shape, the low-resolution feature map is input into the convolution, batch normalization, and the output of the ReLU activation function is used to rearrange the pixels of the feature map through the PixelShuffle method, and the channel information of the feature map is rearranged to the spatial dimension, thereby realizing upsampling; wherein, the specific principle of the PixelShuffle method for pixel rearrangement of the feature map is: the original image size is H×W×3, and after passing through the encoder, a feature map of shape h×w×c is obtained (where h=H / r, w=W / r), and 3 h×w×r are obtained through the convolution operation 2 Each subgraph consists of h×w 1×1×r 2 These feature maps are reshaped into an r×r×1 shape and superimposed to obtain a H×W×1 feature map. Finally, three H×W×1 feature maps are combined to restore the image size.

[0030] The network convergence is based on the perceptual loss module. The perceptual loss module is: define x to represent the input image, represents the output image, y represents the style image, Represents the VGG network pre-trained on Imagenet; and y in the feature space output style perception loss of the relu1_2 layer of the VGG network and y in the feature space output style perception loss of the relu2_2 layer of the VGG network and y in the feature space output style perception loss of the relu3_3 layer of the VGG network and y in the feature space output style perception loss of the relu4_3 layer of the VGG network and x in the feature space output content-aware loss of the relu3_3 layer of the VGG network

[0031] Let Gram matrix For the input image x in the network The co-activation information of different features of the jth layer is calculated as follows:

[0032]

[0033] In the formula, For H j × j C for each point on the grid j Dimensional features, This means that the inner product between each pair of channels is calculated at each position (h, w) on the feature map, and then summed and normalized. By calculating the Gram matrix of the feature map, the co-activation information of different features in the image can be obtained. Similarly, For the style image y in the network The co-activation information of different features in the jth layer, The converted image In the network The co-activation information of different features in the jth layer.

[0034] Perceptual loss specifically includes style perception loss and content perception loss:

[0035] The style-aware loss is the square of the difference between the output and the Gram matrix of the style image, and its specific calculation formula is:

[0036]

[0037] In the formula, is the style-aware loss, is the Gram matrix of the output image, is the Gram matrix of the style image;

[0038] The content-aware loss is the square of the difference between the Gram matrices of the output and input images, and its specific calculation formula is:

[0039]

[0040] In the formula, is the content-aware loss, is the Gram matrix of the converted image, is the Gram matrix of the input image;

[0041] The perceptual metric loss module calculates the distance between the converted image and the original image in the feature space (such as the co-activation information represented by the Gram matrix), which can ensure that the converted image is consistent with the original image in content and the style can be effectively transferred.

[0042] The specific method for judging the input image based on the loss calculation is as follows:

[0043] The relu2_2 layer based on the VGG loss network calculates the difference between the converted image and the content image in the feature space. The relu1_2, relu2_2, relu3_3 and relu4_3 layers calculate the style reconstruction loss of the converted and style images. The content loss and style loss are weighted and summed in proportion to obtain the perceptual loss. This loss function is used to guide the training of the generator so that the images it generates are close to the target image in both content and style.

[0044] Compared with the prior art, the present invention has the following technical effects:

[0045] 1) The present invention proposes a perceptual metric loss module, a multi-scale feature fusion module, and a convolution upsampling module, which are respectively used to solve the problems of poor pixel space distance description, loss of high-level feature details, and loss of upsampling information in mainstream style transfer models. The day-night conversion effect on the bdd100k dataset exceeds that of most style transfer models. This method effectively improves the quality of generated images and the restoration of foreground objects;

[0046] 2) The present invention can also improve the effects of other visual tasks. For the instance segmentation task of nighttime images, on the bdd100k dataset, compared with other day-night conversion models, the method has the highest improvement effect on the segmentation accuracy mIoU. For the nighttime target detection task, on the ExDark dataset, the method of the present invention has the most obvious improvement on the detection effect of each target.

[0047] 3) The present invention proposes a convolution upsampling module, which enlarges the size of the downsampled feature map through the convolution operation, thereby achieving the purpose of restoring the image. At the same time, it also utilizes the learning ability of the convolution operation to ensure that important information is not lost when the size is enlarged.

[0048] 4) Table 1 shows the quantitative evaluation of the effects of different image conversion methods under the same data set. The method of the present invention has the best FID and SIFT data. Compared with ForkGan, FID is optimized by 20.3 and SIFT is improved by 2.02. It can be shown that the method of the present invention significantly improves the quality of the output image.

[0049] 5) The image segmentation network trained on the daytime dataset will have poor segmentation effect when facing nighttime images. The bdd100k dataset provides a large number of nighttime images and their segmentation color map labels. In order to compare the improvement effect of different conversion methods on the image segmentation task, this paper uses the Deeplab-v3 model pre-trained on the cityscapes dataset to segment 100 nighttime images and nighttime images processed by various methods, and takes the average of the 100 data. The segmentation effect of different image conversion methods is compared as follows: Fig.10 As shown in Table 2, the quantitative indicators are shown in Table 2. FID represents the image quality of the color map (the lower the better), and mIoU is an evaluation indicator of segmentation performance (the higher the better). It can be seen that compared with other methods, the method of the present invention also significantly improves the effect of instance segmentation tasks. Compared with ForkGan, FID is optimized by 3.71 and mIoU is improved by 1.18.

[0050] 6) In order to compare the improvement effect of different conversion methods on the target detection network, the present invention uses the yolov5 network to train 50 epochs on the ExDark dataset after different conversion methods, and compares the changes in mAP. Table 3 shows the AP changes of different categories of different conversion methods on the ExDark dataset. Fig.11 The visual comparison of the target detection effects of different conversion methods is shown. The arrows in the figure represent the positions that need attention, the red arrows represent the positions that are easy to miss, and the green arrows represent the positions that are easy to misdetect. It can be seen from the chart that the method of the present invention also significantly improves the performance of the detection network. Compared with other processing methods, the mAP of the method of the present invention reaches the highest value, which is 3.3% higher than ForkGan. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:

[0052] Figure 1 is a flow chart of the present invention;

[0053] Figure 2 It is a schematic diagram of the overall structure of the Forkgan network model in the present invention;

[0054] Figure 3 for Figure 2 Schematic diagram of the structure of the multi-scale feature fusion module in;

[0055] Figure 4 for Figure 3 Schematic diagram of the mid-position attention mechanism;

[0056] Figure 5 for Figure 3 Schematic diagram of the attention mechanism of the convolutional block;

[0057] Figure 6 for Figure 2 Schematic diagram of the convolutional upsampling module in ;

[0058] Figure 7 for Figure 2 Schematic diagram of the perceptual loss module in ;

[0059] Figure 8 Comparison diagrams of the present invention before and after processing various visual tasks;

[0060] Fig. 9 This is a diagram showing the difference in effect between this method and other day-night conversion methods;

[0061] Fig.10 This is a comparison chart of segmentation effects of different image conversion methods;

[0062] Fig.11A visual comparison of the target detection effects of different conversion methods. DETAILED DESCRIPTION

[0063] A method for constructing a ForkGAN neural network for day-night image conversion, comprising:

[0064] Step 1: Establish a training image dataset, which includes a night traffic image dataset and a day traffic image dataset;

[0065] Step 2: Construct the improved ForkGAN network model as follows Figure 2 As shown, the training data set in step 1 is imported into the network model for training, wherein the improved Forkgan network model includes the three improved modules proposed in the present invention;

[0066] Step 3: Based on the loss, the network converges and the day-night conversion is achieved.

[0067] like Figure 2 As shown, in step 2, the construction of the improved ForkGAN network model mentioned is specifically: the training is divided into the direction from night to day and the direction from day to night. Figure 2 The upper part represents the direction from night to day, and the night image I x Input encoder E x , the output is connected to the input of the multi-scale feature fusion module, and the multi-scale feature fusion module outputs the feature z x The outputs of the multi-scale feature fusion module are connected to the reconstruction generator and the transformation generator Input, feature z x Input Reconstruction Generator Output night pictures Feature z x Input Transformation Generator Output daytime pictures Transformation Generator The output of the second encoder E x Input, transformation generator Output After the second encoder E x Output characteristics The second encoder E x The output connection transformation generator Input, features Input Transformation Generator Output night pictures Figure 2 The lower part represents the direction from day to night, and the daytime image I y Input encoder E y, the output is connected to the input of the multi-scale feature fusion module, and the multi-scale feature fusion module outputs the feature z y The outputs of the multi-scale feature fusion module are connected to the reconstruction generator and the transformation generator Input, feature z y Input Reconstruction Generator Output daytime pictures Feature z y Input Transformation Generator Output night picture Transformation Generator The output of the second encoder E y Input, transformation generator Output After the second encoder E y Output characteristics The second encoder E y The output connection transformation generator Input, features Input Transformation Generator Output daytime pictures

[0068] like Figure 3 As shown, Figure 3 for Figure 2 The schematic diagram of the structure of the multi-scale feature fusion module in FIG. The structure of the multi-scale feature fusion module is as follows: the first layer is the output of the 1x1 convolutional layer connected to the input of the position attention mechanism module, and the first layer outputs the feature map F1;

[0069] The second layer is the output of the 3x3 convolutional layer with a void rate of 6, which is connected to the input of the position attention mechanism module. The second layer outputs the feature map F2;

[0070] The third layer is the output of the 3x3 convolutional layer with a void rate of 12, which is connected to the input of the position attention mechanism module. The third layer outputs the feature map F3;

[0071] The fourth layer is the output of the 3x3 convolutional layer with a void rate of 18, which is connected to the input of the position attention mechanism module. The fourth layer outputs the feature map F4;

[0072] The fifth layer is a convolutional block attention mechanism module, and the fifth layer outputs the feature map F5;

[0073] Feature maps F1, F2, F3, F4, and F5 are fused to output feature map F6; feature map F6 is input into the 1x1 convolutional layer output processed feature map F7.

[0074] Specifically, the dilated convolution is to enhance the details and overall quality of image conversion by analyzing and fusing image features at multiple scales, and to obtain feature maps of receptive fields of different scales using dilated convolutions with different expansion rates.

[0075] The specific calculation formula of the receptive field of the dilated convolution is:

[0076] RF i+1 =RF i +(k`-1)×S i

[0077]

[0078] Where RF i+1 Represents the receptive field of the current layer, RF i represents the receptive field of the previous layer, k` represents the size of the convolution kernel, S i Represents the product of the step lengths of all previous layers. The function of the multi-scale feature fusion module is to process the encoder E x The obtained feature Z makes the obtained feature information richer without losing details. This module mainly solves the problem that when the encoder is extracting features, as the number of network layers increases, the size of the feature map will gradually decrease, resulting in the loss of local feature information, and the low-level features have higher resolution and contain more position and detail information, but because they have passed fewer convolutions, their semantics are lower and there is more noise; high-level features have stronger semantic information, but the resolution is very low and the perception of details is poor.

[0079] like Figure 4 As shown, Figure 4 for Figure 3 Schematic diagram of the structure of the position attention mechanism module (PAM) in the figure. The structure of the position attention mechanism module is as follows: the feature map A (C*H*W) is input into three convolutional layers to output three feature maps {B, C, D} with a shape of C*H*W; then the three feature maps {B, C, D} are reshaped into C*N, where C represents the number of channels, H represents the height of the feature map, W represents the width of the feature map, and N = H×W represents the number of pixels; the feature maps C and B are transposed and multiplied, and then the attention feature map S (N*N) is obtained through softmax; the feature maps D and S are transposed and multiplied, and the shape of the resulting feature map is reconstructed into C×H×W and then multiplied by the scale factor α, and added to the input feature map A to obtain the final output feature map E. The calculation formula is as follows:

[0080]

[0081]

[0082] In the formula, S jiEach element of the S feature map represents the influence of position i on position j, E j represents each element in E, α is the scale factor, D i is an element of D, A j is an element of A;

[0083] like Figure 5 As shown, Figure 5 for Figure 3 The structure diagram of the convolutional block attention mechanism module (CBAM) in the figure is as follows: the input feature is multiplied by the output of the channel attention mechanism module and the input feature is output, and the intermediate feature is output; the intermediate feature is multiplied by the output of the spatial attention mechanism module and the result is output. Among them, the channel attention mechanism module is: the input feature map passes through two parallel MaxPool layers and AvgPool layers, the feature map is changed from C*H*W to C*1*1, and then passes through the Share MLP module, in which it first compresses the number of channels to 1 / r times the original number (r is Reduction, reduction rate), and then expands to the original number of channels, and obtains two activated results through the ReLU activation function. The two output results are added element by element, and then a sigmoid activation function is used to obtain the output result of the channel attention mechanism, and then this output result is multiplied by the original image to return to the size of C*H*W; the spatial attention mechanism module is: the output result of the channel attention mechanism is obtained through maximum pooling and average pooling to obtain two 1*H*W feature maps, and then the two feature maps are spliced ​​through the Concat operation, converted into a 1-channel feature map through a 7*7 convolution, and then a sigmoid function is used to obtain the feature map of the spatial attention mechanism, and finally the output result is multiplied by the original image to return to the size of C*H*W.

[0084] like Figure 6 As shown, Figure 6 for Figure 2 The structure diagram of the convolution upsampling module in the figure is as follows: the original image size is H×W×3, and the encoder outputs a low-resolution feature map of the shape of , the low-resolution feature map is input into the convolution, batch normalization, and the output of the ReLU activation function is used to rearrange the pixels of the feature map through the PixelShuffle method, and the channel information of the feature map is rearranged to the spatial dimension, thereby achieving upsampling. Among them, the specific principle of the PixelShuffle method to rearrange the pixels of the feature map is as follows: the original image size is H×W×3, and after passing through the encoder, a feature map of the shape of h×w×c is obtained (where h=H / r, w=W / r), and 3 h×w×r are obtained through the convolution operation. 2 Each subgraph consists of h×w 1×1×r 2The feature maps are reconstructed into r×r×1 shape and superimposed to obtain H×W×1 feature maps. Finally, the three H×W×1 feature maps are combined to restore the image size. The function of the convolution upsampling module is to generate smoother and more natural image edges than the traditional interpolation method. The convolution upsampling module can reduce the information loss caused by the downsampling process to a certain extent.

[0085] like Figure 7 As shown, Figure 7 for Figure 2 Schematic diagram of the structure of the perceptual loss module in the figure. The structure of the perceptual loss module is as follows: x is defined to represent the input image, represents the output image, y represents the style image, Represents the VGG network pre-trained on Imagenet. and y in the feature space output style perception loss of the relu1_2 layer of the VGG network and y in the feature space output style perception loss of the relu2_2 layer of the VGG network and y in the feature space output style perception loss of the relu3_3 layer of the VGG network and y in the feature space output style perception loss of the relu4_3 layer of the VGG network and x in the feature space output content-aware loss of the relu3_3 layer of the VGG network

[0086] Let Gram matrix For the input image x in the network The co-activation information of different features of the jth layer is calculated as follows:

[0087]

[0088] In the formula, For H j × j C for each point on the grid j Dimensional features, This means that the inner product between each pair of channels is calculated at each position (h, w) on the feature map, and then summed and normalized. By calculating the Gram matrix of the feature map, the co-activation information of different features in the image can be obtained. Similarly, For the style image y in the network The co-activation information of different features in the jth layer, The converted image In the network The co-activation information of different features in the jth layer.

[0089] Perceptual loss specifically includes style perception loss and content perception loss:

[0090] The style-aware loss is the square of the difference between the output and the Gram matrix of the style image, and its specific calculation formula is:

[0091]

[0092] In the formula, is the style-aware loss, is the Gram matrix of the output image, is the Gram matrix of the style image;

[0093] The content-aware loss is the square of the difference between the Gram matrices of the output and input images, and its specific calculation formula is:

[0094]

[0095] In the formula, is the content-aware loss, is the Gram matrix of the converted image, is the Gram matrix of the input image.

[0096] The perceptual metric loss module calculates the distance between the converted image and the original image in the feature space (such as the co-activation information represented by the Gram matrix), which can ensure that the converted image is consistent with the original image in content and the style can be effectively transferred.

[0097] The specific method for judging the input image based on the loss calculation is as follows:

[0098] The relu2_2 layer of the VGG-16 loss network calculates the difference between the converted image and the content image in the feature space. The relu1_2, relu2_2, relu3_3 and relu4_3 layers calculate the style reconstruction loss of the converted and style images. The content loss and style loss are weighted and summed in proportion to obtain the perceptual loss. This loss function is used to guide the training of the generator so that the images it generates are close to the target images in both content and style.

[0099] The training dataset consists of bdd100k and Oxford Robotcars datasets, and is manually divided into two categories: night and day.

[0100] The Forkgan network model uses the Pytorch deep learning framework, and the Adam optimizer is used for training, with a momentum of 0.9 and a weight decay of 10. -5 , batch size is 1, training is done for 200 epochs, of which the first 100 epochs are warm-up phases, and the learning rate is increased from 10 -3Increase to 5×10 -3 , the logarithm of the last 100 epochs decreases to 2.5×10 -3 .

[0101] The training model uses the Bdd100k dataset, which contains 100,000 high-definition videos, each of which is about 40 seconds, 720p, and 30fps. The key frame is sampled at the 10th second of each video to obtain 100,000 images (image size: 1280*720) and annotated. After training, FID and SIFT are used to evaluate the source image and the converted image. FID is an evaluation index of the quality of the generated image, which represents the distance between the generated image and the real image in the multivariate normal distribution. The smaller the data, the better; SIFT represents the matching degree of the feature points of the two images, which can evaluate the similarity of the two images. The larger the data, the better. The effect of the converted images of different methods on the improvement of the segmentation performance is also compared. The evaluation index is mIoU, which represents the evaluation ratio of the intersection and union of various real labels and predicted values. The larger the value, the better the segmentation performance.

[0102] The Oxford RobotCar dataset is a public dataset widely used in autonomous driving research, provided by the Mobile Robotics Group (MRG) of the University of Oxford. It contains urban driving data collected under various weather, time and traffic conditions; the Alderley dataset is a dataset created by researchers at the University of Manchester in the UK for studying day and night scene classification and recognition in computer vision and autonomous driving. The dataset contains images and related labels during day and night periods for training and evaluating the performance of algorithms under different lighting conditions.

[0103] First, the source images are labeled to obtain the training data sets of daytime images and nighttime images. The training data sets are imported into the ForkGAN network model. The encoder gradually converts the images into high-level features through convolution and pooling operations, and performs feature extraction to obtain feature information.

[0104] In order to verify the effectiveness of the method of the present invention and the influence of each module on the conversion effect, a comparative experiment is carried out:

[0105] (1) Comparison of day-night conversion results of different methods on the Bdd100k dataset. Table 1 shows the quantitative evaluation of the effects of different image conversion methods on the same dataset. This method has the best FID and SIFT data. Compared with ForkGan, FID is optimized by 20.3 and SIFT is improved by 2.02. It can be shown that this method does improve the quality of the output image.

[0106] Table 1 Evaluation results of different methods on the day-night conversion task on the bdd100k dataset

[0107]

[0108] (2) Compare the effects of different conversion methods on image segmentation tasks. Use the Deeplab-v3 model pre-trained on the cityscapes dataset to segment 100 night images and night images processed by various methods, and take the average of the 100 data. Fig.10 As shown in Table 2, the quantitative indicators are shown in Table 2. FID represents the image quality of the color map (the lower the better), and mIoU is an evaluation indicator of segmentation performance (the higher the better). It can be seen that compared with other methods, the method of the present invention also significantly improves the effect of instance segmentation tasks. Compared with ForkGan, FID is optimized by 3.71 and mIoU is improved by 1.18.

[0109] Table 2 Quantitative comparison of segmentation performance of different methods on the bdd100k dataset

[0110]

[0111]

[0112] (3) In order to compare the improvement effect of different conversion methods on the target detection network, the yolov5 network was selected to train 50 epochs on the ExDark dataset after being processed by different conversion methods, and the changes in mAP were compared. Table 3 shows the AP changes of different categories of different conversion methods on the ExDark dataset. Fig.11 The visual comparison of the target detection effects of different conversion methods is shown. The arrows in the figure represent the positions that need attention, the red arrows represent the positions that are easy to miss, and the green arrows represent the positions that are easy to misdetect. It can be seen from the chart that the method of the present invention also significantly improves the performance of the detection network. Compared with other processing methods, the mAP of the method of the present invention reaches the highest value, which is 3.3% higher than ForkGan.

[0113] Table 3 Effects of target detection after processing by different methods

[0114]

[0115] Working principle: By proposing a perceptual metric loss module, a multi-scale feature fusion module, and a convolutional upsampling module, they are used to solve the problems of poor pixel space distance description, loss of high-level feature details, and loss of upsampling information in mainstream style transfer models. The day-night conversion effect on the bdd100k dataset exceeds that of most style transfer models. This method effectively improves the quality of generated images and the restoration of foreground targets.

Claims

1. A method for constructing a ForkGAN neural network for day-night image conversion, characterized in that: The following steps are involved: Step 1: Establish a training image dataset, which includes a night traffic image dataset and a day traffic image dataset; Step 2: Construct an improved ForkGAN network model including a multi-scale feature fusion module, a perceptual loss module, and a convolutional upsampling module; Step 3: Based on the loss, the network converges.

2. The method according to claim 1, characterized in that In step 2, the improved ForkGAN network model specifically includes two parts: In the first part, we input a night image and output a day image: Night Image I x Input encoder E x , the output is connected to the input of the multi-scale feature fusion module, and the multi-scale feature fusion module outputs the feature z x ; The outputs of the multi-scale feature fusion module are connected to the reconstruction generator and the transformation generator Input, feature z x Input Reconstruction Generator The convolution upsampling module in the generator restores the size and outputs a night picture Feature z x Input Transformation Generator The convolution upsampling module in the generator restores the size and outputs a daytime picture Among them, the conversion generator The output of the second encoder E x Input, transformation generator Output After the second encoder E x Output characteristics The second encoder E x The output connection transformation generator Input, features Input Transformation Generator Output night pictures Based on the perception loss module, the loss is continuously reduced, the network is continuously converged, and the image quality is continuously improved; The second part, input daytime images, output nighttime images: Daytime Image I y Input encoder E y , the output is connected to the input of the multi-scale feature fusion module, and the multi-scale feature fusion module outputs the feature z y ; The outputs of the multi-scale feature fusion module are connected to the reconstruction generator and the transformation generator Input, feature z y Input Reconstruction Generator The convolution upsampling module in the generator restores the size and outputs a daytime picture Feature z y Input Transformation Generator The convolution upsampling module in the generator restores the size and outputs a night picture Among them, the conversion generator The output of the second encoder E y Input, transformation generator Output After the second encoder E y Output characteristics The second encoder E y The output connection transformation generator Input, features Input Transformation Generator Output daytime pictures Based on the perceptual loss module, the loss continues to decrease, the network continues to converge, and the output image quality continues to improve.

3. The method according to claim 2, characterized in that The multi-scale feature fusion module is specifically: The first layer is the output of the 1x1 convolutional layer connected to the input of the position attention mechanism module PAM, and the first layer outputs the feature map F1; The second layer is the output of the 3x3 convolutional layer with a void rate of 6, which is connected to the input of the position attention mechanism module PAM. The second layer outputs the feature map F2; The third layer is the output of the 3x3 convolutional layer with a void rate of 12, which is connected to the input of the position attention mechanism module PAM. The third layer outputs the feature map F3; The fourth layer is the output of the 3x3 convolutional layer with a void rate of 18, which is connected to the input of the position attention mechanism module PAM. The fourth layer outputs the feature map F4; The fifth layer is the convolutional block attention mechanism module CBAM, and the fifth layer outputs the feature map F5; Feature maps F1, F2, F3, F4, and F5 are fused to output feature map F6; feature map F6 is input into the 1x1 convolutional layer output processed feature map F7.

4. The method according to claim 3, characterized in that The position attention mechanism module PAM is specifically: The feature map A{C*H*W} is input into three convolutional layers respectively to output three feature maps {B, C, D} with a shape of C*H*W; then the three feature maps {B, C, D} are reshaped into C*N, where C represents the number of channels, H represents the feature map height, W represents the feature map width, and N=H×W represents the number of pixels; the feature maps C and B are transposed and multiplied, and then the attention feature map S{N*N} is obtained through softmax; the feature maps D and S are transposed and multiplied, and the shape of the resulting feature map is reconstructed into C×H×W and then multiplied by the scale factor α, and added to the input feature map A to obtain the final output feature map E.

5. The method according to claim 3, characterized in that: The convolutional block attention mechanism module CBAM is specifically: The input feature is multiplied by the output of the channel attention mechanism module and the input feature, and the intermediate feature is output; the intermediate feature is multiplied by the output of the spatial attention mechanism module and the result is output; Among them, the channel attention mechanism module is: the input feature map passes through two parallel MaxPool layers and AvgPool layers, the feature map is changed from C*H*W to C*1*1, and then passes through the Share MLP module, in which it first compresses the number of channels to 1 / r times the original, where r is the Reduction rate, and then expands to the original number of channels, and obtains two activated results through the ReLU activation function; the two output results are added element by element, and then a sigmoid activation function is used to obtain the output result of the channel attention mechanism, and then the output result is multiplied by the original image to change back to the size of C*H*W; the spatial attention mechanism module is: the output result of the channel attention mechanism is obtained through maximum pooling and average pooling to obtain two 1*H*W feature maps, and then the two feature maps are spliced ​​through the Concat operation, converted into a 1-channel feature map through a 7*7 convolution, and then a sigmoid function is used to obtain the feature map of the spatial attention mechanism, and finally the output result is multiplied by the original image to change back to C*H*W size.

6. The method according to claim 2, characterized in that The structural diagram of the convolution upsampling module is as follows: the image of the original image size is input into the low-resolution feature map of the encoder output shape, the low-resolution feature map is input into the convolution, batch normalization, and the output of the ReLU activation function is used to rearrange the pixels of the feature map through the PixelShuffle method, and the channel information of the feature map is rearranged to the spatial dimension, thereby realizing upsampling; wherein, the specific principle of the PixelShuffle method for rearranging the pixels of the feature map is as follows: the original image size is H×W×3, and after passing through the encoder, a feature map of the shape of h×w×c is obtained, and 3 h×w×r shapes are obtained through the convolution operation. 2 Each subgraph consists of h×w 1×1×r 2 These feature maps are reshaped into an r×r×1 shape and superimposed to obtain a H×W×1 feature map. Finally, three H×W×1 feature maps are combined to restore the image size.

7. The method according to claim 1, characterized in that The network convergence is based on the perceptual loss module. The perceptual loss module is: define x to represent the input image, represents the output image, y represents the style image, Represents the VGG network pre-trained on Imagenet; and y in the feature space output style perception loss of the relu1_2 layer of the VGG network and y in the feature space output style perception loss of the relu2_2 layer of the VGG network and y in the feature space output style perception loss of the relu3_3 layer of the VGG network and y in the feature space output style perception loss of the relu4_3 layer of the VGG network and x in the feature space output content-aware loss of the relu3_3 layer of the VGG network Let Gram matrix For the input image x in the network The co-activation information of different features of the jth layer is calculated as follows: In the formula, For H j × j C for each point on the grid j Dimensional features, This means that the inner product between each pair of channels is calculated at each position (h, w) on the feature map, and then summed and normalized. By calculating the Gram matrix of the feature map, the co-activation information of different features in the image can be obtained. Similarly, For the style image y in the network The co-activation information of different features in the jth layer, The converted image In the network The co-activation information of different features in the jth layer.

8. The method according to claim 7, characterized in that The perceptual loss specifically includes style perceptual loss and content perceptual loss: The style-aware loss is the square of the difference between the output and the Gram matrix of the style image, and its specific calculation formula is: In the formula, is the style-aware loss, is the Gram matrix of the output image, is the Gram matrix of the style image; The content-aware loss is the square of the difference between the Gram matrices of the output and input images, and its specific calculation formula is: In the formula, is the content-aware loss, is the Gram matrix of the converted image, is the Gram matrix of the input image; The perceptual metric loss module calculates the distance between the converted image and the original image in the feature space; such as the co-activation information represented by the Gram matrix, which can ensure that the converted image is consistent with the original image in content and the style can be effectively transferred.

9. The method according to claim 7 or 8, characterized in that: The specific method for judging the input image based on the loss calculation is as follows: The relu2_2 layer based on the VGG loss network calculates the difference between the converted image and the content image in the feature space. The relu1_2, relu2_2, relu3_3 and relu4_3 layers calculate the style reconstruction loss of the converted and style images. The content loss and style loss are weighted and summed in proportion to obtain the perceptual loss. This loss function is used to guide the training of the generator so that the images it generates are close to the target images in both content and style.

Images