Night-day image conversion method based on lightweight cyclic generative adversarial network

Abstract

The invention discloses a night-daytime image conversion method based on a lightweight cyclic generative adversarial network, and relates to the field of image conversion and model compression, and the method comprises the following steps: selecting image data from a night data set and a daytime data set, combining a joint loss function, and generating a final network model for converting night and daytime images; a shared attention module is introduced into the generated final network model, and a shared attention cyclic adversarial generative network model used for night-to-day image conversion is constructed; compressing the shared attention cycle adversarial generative network model by introducing a knowledge distillation algorithm to obtain a teacher generator model, and transmitting knowledge to a student generator model through the teacher generator model; and performing up-sampling reduction on the feature map in the student generator model through a decoder to obtain an image after conversion from night to day. The problems of image color distortion, blurred target details and the like are effectively relieved.

Description

Technical Field

[0001] The present invention relates to the field of image conversion and model compression, and in particular to a night-time-daytime image conversion method based on a lightweight cyclic adversarial generation network. Background Art

[0002] With the vigorous development of network technology and the widespread application of multimedia devices, image conversion technology is becoming increasingly popular in various scenarios. This technology aims to convert a source image into an image with different styles. The training set of existing image recognition models generally only has image samples collected during the day, and the generalization to night images is not high. For example, under night conditions, due to the deterioration of light and visibility, the visual information of the image is less, resulting in the image recognition model based on daytime image training. The recognition accuracy of night images is reduced. At the same time, in the complex lighting environment at night, most of the existing image recognition algorithms will be disturbed by light, resulting in a significant decrease in detection accuracy. Therefore, how to achieve night-to-day image conversion, thereby improving the accuracy of image recognition models, is particularly urgent and urgent.

[0003] In the past few decades, the academic community has proposed many methods for night-day image conversion. Traditional image conversion methods can be divided into two categories: image enhancement-based and generative model-based image style transfer methods. Early methods based on image enhancement have attracted much attention due to their relatively simple algorithms. However, these methods operate at the image pixel level, so they show poor generalization ability in different night environments. In addition, these methods often fail to achieve ideal results when dealing with the overall conversion of low-light night images. Another generative image conversion method uses generative models such as generative adversarial networks to learn the image mapping relationship between different fields, thereby realizing the conversion of images between different fields. In this process, the generator and the discriminator are continuously optimized through adversarial training: the generator aims to generate realistic images to confuse the discriminator, while the discriminator is committed to more accurately distinguishing the generated images from the real images. Through this adversarial mechanism, the generator gradually improves the authenticity and quality of its generated images. This type of method shows significant advantages in improving image quality, maintaining semantic consistency, and dealing with complex scenes. Thus, night domain images are converted into day domain images, and image conversion from night to day is realized.

[0004] However, the existing night-to-day image conversion methods based on generative models have the following shortcomings:

[0005] 1) In the existing technology, a recurrent generative adversarial network is generally used for image conversion. However, the model has high complexity and is difficult to deploy on edge devices and mobile devices, which seriously limits the practicality of night image conversion technology.

[0006] 2) The existing image conversion method based on recurrent adversarial generative network has the problem of poor recognition accuracy. The main reason is that the converted image recognition subject is easily disturbed by the background, resulting in a decrease in the quality of the image recognition subject conversion.

[0007] Currently, no effective solution has been proposed for the problems in the related technologies. Summary of the invention

[0008] In response to the problems in the related art, the present invention proposes a night-time-day image conversion method based on a lightweight cyclic adversarial generative network to overcome the above-mentioned technical problems existing in the existing related art.

[0009] To this end, the specific technical solution adopted by the present invention is as follows:

[0010] S1, select image data from the nighttime dataset and the daytime dataset, combine the joint loss function and generate the final network model for converting nighttime and daytime images;

[0011] S2. Introduce the shared attention module into the generated final network model and construct a shared attention recurrent adversarial generation network model for night-to-day image conversion;

[0012] S3. Compress the shared attention recurrent adversarial generative network model by referencing the knowledge distillation algorithm to obtain the teacher generator model, and transfer the knowledge to the student generator model through the teacher generator model;

[0013] S4. The feature map in the student generator model is upsampled and restored through the decoder to obtain the image converted from night to day.

[0014] Furthermore, the method of selecting image data from the nighttime dataset and the daytime dataset, combining the joint loss function and generating a final network model for converting nighttime and daytime images includes the following steps:

[0015] S11, input the nighttime dataset into the generator and calculate the cycle consistency loss of the nighttime domain;

[0016] S12, input the daytime dataset into the generator and calculate the cycle consistency loss of the daytime domain;

[0017] S13. Use the joint loss function to optimize the network model parameters and construct the final network model.

[0018] Furthermore, the joint loss function formula is as follows:

[0019] L(G,F,D X ,D Y )=L GAN (G,D Y,X,Y)+L GAN (F,D X ,Y,X);

[0020] Where F represents the mapping function from night domain X to day domain Y; D x represents the domain discriminator of night domain X; G represents the mapping function from day domain Y to night domain X; D Y represents the domain discriminator of the daytime domain Y; X represents the nighttime domain; Y represents the daytime domain; L cyc (G, F) represents cycle consistency loss; L GAN (G,D Y ,X,Y) represents the adversarial loss of converting from night domain X to day domain Y; L GAN (F,D X ,Y,X) represents the adversarial loss of converting from daytime domain Y to nighttime domain X; L identity (G,F) represents the coloring loss; L(G,F,D X ,D Y ) represents the joint loss function of the generative adversarial network.

[0021] Furthermore, the step of inputting the nighttime dataset into the generator and calculating the cycle consistency loss of the nighttime domain includes the following steps:

[0022] S111, input the sample data in the nighttime data set into the generator to generate a converted daytime domain image, input the generated daytime domain image into the discriminator to obtain the discrimination result, and calculate the adversarial loss of the nighttime domain image conversion;

[0023] S112, input the generated daytime domain image into the generator again, convert it to generate a nighttime domain image, and pass the generated nighttime domain image into the discriminator to obtain the discrimination result, and calculate the cycle consistency loss of the nighttime domain.

[0024] Furthermore, the generator is composed of an encoder, a bottleneck module, and a decoder;

[0025] The encoder extracts features from the input image and converts the features into an abstract feature map;

[0026] The bottleneck module processes the encoded features through residual blocks to increase the depth of the network architecture;

[0027] The decoder decodes the feature map extracted by the bottleneck module to the same spatial dimension as the input image and generates the final output image.

[0028] Furthermore, the method of introducing the shared attention module into the generated final network model and constructing a shared attention recurrent adversarial generative network model for night-time to day-time image conversion includes the following steps:

[0029] S21, setting the shared attention module before the convolution layer of the encoder and after the transposed convolution layer of the decoder;

[0030] S22. Use skip connections between each key vector and introduce a refinement network between the connections. Transpose the attention map shared between the convolutional layers based on the refinement network.

[0031] S23, taking the number of attention maps as a hyperparameter, and setting the number of attention maps to half the number of input channels per layer, to learn the dependency between the hyperparameters;

[0032] S24. Through dependencies, a shared attention recurrent adversarial generative network model is constructed for night-to-day image conversion.

[0033] Furthermore, the calculation of the shared attention weight is given by the following formula:

[0034]

[0035] where σ row represents the application of the softmax function along each row of the matrix Y; σ col represents the application of the softmax function along each column of the matrix Y; d k represents the dimension of the key vector; Q represents the query vector; K represents the key vector; T represents the matrix transpose operation; O represents the attention weight of the shared attention output of the encoder.

[0036] Furthermore, the method of compressing the shared attention recurrent adversarial generative network model by referencing the knowledge distillation algorithm to obtain the teacher generator model, and transferring the knowledge to the student generator model through the teacher generator model includes the following steps:

[0037] S31. The generator in the night conversion model based on the shared attention recurrent adversarial generative network is used as the teacher generator to build a teacher generator model;

[0038] S32, train the generator to map the convolutional layer of the night domain to Y i The convolutional layer of the daytime domain uses a discriminator to distinguish between the generated images of the teacher generator and the real images;

[0039] S33, the teacher generator generates a feature map of a night domain image through an encoder, and decodes the feature map of the night domain image into an image feature map of a day domain through a decoder;

[0040] S34, combining the joint loss function, so that the daytime domain image feature map output by the teacher generator is close to the real daytime domain image;

[0041] S35. Through the knowledge distillation algorithm, the teacher generator is used to guide the student generator to train and build a student generator model;

[0042] S36, the encoder structure of the teacher generator is consistent with that of the student generator, and the decoder structure of the teacher generator corresponds to that of the student network;

[0043] S37, supervising the conversion image generated by the student generator through the online knowledge distillation loss;

[0044] S38. Use online knowledge distillation loss to measure the difference between the generated conversion images of the teacher generator and the generated conversion images of the student generator.

[0045] Furthermore, the lightweight student generator model is obtained after training based on the student generator model, and the feature map in the lightweight student generator model is sampled and restored through a decoder to obtain an image converted from night to day, including the following steps:

[0046] S41, in the process of knowledge distillation, the teacher generator model is learned through the student generator model;

[0047] S42. In the process of learning the student generator model, the discriminator is used to train and optimize the student generator model to obtain a lightweight student generator model, and the joint loss function is optimized and trained to obtain a total loss function;

[0048] S43. Through the decoder in the lightweight student generator model, the daytime domain feature map generated by the student generator is sampled to obtain the image converted from night to day.

[0049] Furthermore, the total loss function and optimization objective are defined as:

[0050] L KD (p t ,p s )=λ SSIM L SSIM +λ feature L feature +λ style L style ;

[0051] Among them, L SSIM represents the structural similarity loss; L feature represents feature loss; L style represents the style loss; λ SSIM Indicates L SSIM The weight hyperparameter coefficient of the loss; λfeature Indicates L feature The weight hyperparameter coefficient of the loss; λ style Indicates L style The weight hyperparameter coefficient of the loss; in addition, λ SSIM , feature , style Three hyperparameter coefficients balance L SSIM , L feature , L style loss.

[0052] The beneficial effects of the present invention are:

[0053] 1. The present invention combines the recurrent generative adversarial network algorithm with the attention mechanism and uses online knowledge distillation to compress the attention generator. It is a night-to-day image conversion method of a lightweight recurrent generative adversarial network model. After the image is converted from night to day using this method, the image has the advantages of prominent subject and clear target details. At the same time, the algorithm effectively alleviates the problems of image color distortion and blurred target details.

[0054] 2. The present invention combines a recurrent adversarial generative network model and a shared attention network, which can be more targeted at the main semantic information in the image, reduce the impact of background information on its night-day conversion, make the converted semantic information more real and effective, and the image recognition model more accurate.

[0055] 3. We designed online knowledge distillation for the shared-attention recurrent adversarial generative network to compress the generator, which effectively avoids the shortcomings of traditional offline knowledge distillation, greatly improves the compression degree of the model, and achieves lightweight model. The compressed student model can maintain a high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0057] Figure 1 is a flowchart of a method for night-time-daytime image conversion based on a lightweight cyclic adversarial generative network according to an embodiment of the present invention;

[0058] Figure 2 is a schematic diagram of a night image conversion process in a night-time-day image conversion method based on a lightweight cyclic adversarial generative network according to an embodiment of the present invention;

[0059] Figure 3is a schematic diagram of a generation network after adding an attention mechanism to a night-time-day image conversion method based on a lightweight cyclic adversarial generation network according to an embodiment of the present invention;

[0060] Figure 4 2. It is a schematic diagram of the structure of a shared attention algorithm in a night-time-daytime image conversion method based on a lightweight cyclic adversarial generative network according to an embodiment of the present invention;

[0061] Figure 5 is a schematic diagram of the structure of a cyclic adversarial generative network in a night-time-day image conversion method based on a lightweight cyclic adversarial generative network according to an embodiment of the present invention;

[0062] Figure 6 2. It is a schematic diagram of the discriminant network structure in the night-time-daytime image conversion method based on a lightweight cyclic adversarial generative network according to an embodiment of the present invention;

[0063] Figure 7 It is a schematic diagram of the online knowledge distillation structure in the night-time-daytime image conversion method based on a lightweight cyclic adversarial generative network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0064] To further illustrate each embodiment, the present invention provides drawings, which are part of the disclosure of the present invention and are mainly used to illustrate the embodiments. They can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, ordinary technicians in this field should be able to understand other possible implementation methods and advantages of the present invention.

[0065] According to an embodiment of the present invention, a night-time-daytime image conversion method based on a lightweight cyclic adversarial generation network is provided.

[0066] The present invention is further described with reference to the accompanying drawings and specific embodiments. Figure 1-Figure 7 As shown, according to one embodiment of the present invention, a night-time-day image conversion method based on a lightweight cyclic adversarial generation network is provided, and the night-time-day image conversion method includes the following steps:

[0067] S1. Select image data from the nighttime dataset and the daytime dataset, combine the joint loss function and generate the final network model for converting nighttime and daytime images.

[0068] Furthermore, the method of selecting image data from the nighttime dataset and the daytime dataset, combining the joint loss function and generating a final network model for converting nighttime and daytime images includes the following steps:

[0069] S11. Input the nighttime dataset into the generator and calculate the cycle consistency loss of the nighttime domain.

[0070] Furthermore, the step of inputting the nighttime dataset into the generator and calculating the cycle consistency loss of the nighttime domain includes the following steps:

[0071] S111, input the sample data in the nighttime data set into the generator to generate a converted daytime domain image, input the generated daytime domain image into the discriminator to obtain the discrimination result, and calculate the adversarial loss of the nighttime domain image conversion;

[0072] S112, input the generated daytime domain image into the generator again, convert it to generate a nighttime domain image, and pass the generated nighttime domain image into the discriminator to obtain the discrimination result, and calculate the cycle consistency loss of the nighttime domain.

[0073] It should be explained that the generator is composed of an encoder, a bottleneck module, and a decoder;

[0074] The encoder extracts features from the input image and converts the features into an abstract feature map;

[0075] The bottleneck module processes the encoded features through residual blocks to increase the depth of the network architecture;

[0076] The decoder decodes the feature map extracted by the bottleneck module to the same spatial dimension as the input image and generates the final output image.

[0077] S12. Input the daytime dataset into the generator and calculate the cycle consistency loss of the daytime domain.

[0078] S13. Use the joint loss function to optimize the network model parameters and construct the final network model.

[0079] Furthermore, the joint loss function formula is as follows:

[0080] L(G,F,D X ,D Y )=L GAN (G,D Y ,X,Y)+L GAN (F,D X ,Y,X);

[0081] Where F represents the mapping function from night domain X to day domain Y; D x represents the domain discriminator of night domain X; G represents the mapping function from day domain Y to night domain X; D Y represents the domain discriminator of the daytime domain Y; X represents the nighttime domain; Y represents the daytime domain; L cyc (G, F) represents cycle consistency loss; L GAN (G,D Y,X,Y) represents the adversarial loss of converting from S night domain X to day domain Y; L GAN (F,D X ,Y,X) represents the adversarial loss of converting from daytime domain Y to nighttime domain X; L identity (G,F) represents the coloring loss; L(G,F,D X ,D Y ) represents the joint loss function of the generative adversarial network.

[0082] It should be explained that the night domain image is input into the generator G to generate the converted day domain image, and the generated day domain image is input into the discriminator to obtain the discrimination result, and the adversarial loss of the night domain image conversion is calculated. Then the image is converted to generate a night domain image, and the cycle consistency loss of the night domain is calculated. Then, the same steps are performed on the day image. Figure 5 As shown, X and Y represent the image domain, where x i and i Represents the X domain image, x i ∈X and y i ∈Y, the goal is to learn the mapping function between X and Y given the training samples, record G:X→Y and F:Y→X, and represent the data distribution as x~p data (x) and y~p data (y), two adversarial discriminators D are introduced X and D Y , where D X Aims to distinguish image x from transformed image F(y); similarly, D Y Aims to distinguish y from G(x);

[0083] The present invention uses a joint loss function to optimize the network model parameters to achieve image conversion. The joint loss function consists of adversarial loss, cycle consistency loss and coloring loss, which are as follows:

[0084] 1) In the adversarial loss, G is the generator, which generates the image G(x), thereby realizing the conversion from image domain X to image domain Y, and D Y is the discriminator that distinguishes G(x) from the real sample y in the image domain Y, then the adversarial loss L GAN (G,D Y ,X,Y) is calculated as follows:

[0085]

[0086] Where G is the generator, generating the image G(x) to achieve image conversion to domain Y, and D Y It is a discriminator that distinguishes image conversion samples G(x) from real samples y, using adversarial training strategies to minimize G and maximize DY The loss function for model parameter optimization G aims to minimize the objective by maximizing the discriminator D, as shown below:

[0087]

[0088] 2) In the cycle consistency loss, for each image x from the image domain X, the result of two image transformations using generators G and F should be similar to the input image x, that is, x→G(x)→F(G(x))≈x, where F(G(X)) represents the image transformed from domain X to domain Y by generator G and then transformed back to domain X by F mapping. Similarly, for each image y from domain Y, G and F should also satisfy y→F(y)→G(F(y))≈y. In order to meet the above constraints, the cycle consistency loss L is used. cyc To reduce the loss of semantic information during image conversion, thereby reducing the domain shift directly between image domains X and Y:

[0089]

[0090] Among them, F(G(x)) is the image converted from the X domain to the Y domain through the generator G, and then converted back to the X domain through the F mapping. The purpose is to calculate the difference between the converted image and the original image. The same is true for G(F(y));

[0091] 3) In the coloring loss, during the image night-day conversion process, the role of the coloring loss is to ensure that the color of certain key objects or areas in the image changes reasonably, that is, when the ambient light changes significantly, the color tone can still retain the characteristics related to the real appearance of the object. This loss term is intended to encourage the generator G to perform reasonable color migration in the process of night to day, so that the color tone in the image can not only reflect the change of illumination, but also retain the inherent color characteristics of the object after the conversion, so that the generated image is more natural and realistic. When the real sample y of the image domain Y is used as the input of the generator G, it is expected that G can be regularized into an approximate identity mapping. Therefore, the coloring loss L identity The calculation is as follows:

[0092]

[0093] Among them, F(y) represents the image converted from the Y domain to the X domain through the generator G. By comparing the converted image with the original image, the difference in the same domain is calculated to ensure that the color in the converted image is reasonably transformed, that is, when the ambient light changes significantly, the color tone can still retain the characteristics related to the real appearance of the object.

[0094] In summary, the joint loss function L(G,F,D X ,D Y) is calculated as follows:

[0095]

[0096] S2. Introduce the shared attention module into the generated final network model and construct a shared attention recurrent adversarial generation network model for night-to-day image conversion.

[0097] Furthermore, the method of introducing the shared attention module into the generated final network model and constructing a shared attention recurrent adversarial generative network model for night-time to day-time image conversion includes the following steps:

[0098] S21, setting the shared attention module before the convolution layer of the encoder and after the transposed convolution layer of the decoder;

[0099] S22. Use skip connections between each key vector and introduce a refinement network between the connections. Transpose the attention map shared between the convolutional layers based on the refinement network.

[0100] S23, taking the number of attention maps as a hyperparameter, and setting the number of attention maps to half the number of input channels per layer, to learn the dependency between the hyperparameters;

[0101] S24. Through dependencies, a shared attention recurrent adversarial generative network model is constructed for night-to-day image conversion.

[0102] Furthermore, the calculation of the shared attention weight is given by the following formula:

[0103]

[0104] where σ row represents the application of the softmax function along each row of the matrix Y; σ col represents the application of the softmax function along each column of the matrix Y; d k represents the dimension of the key vector; Q represents the query vector; K represents the key vector; T represents the matrix transpose operation; O represents the attention weight of the shared attention output of the encoder.

[0105] It should be explained that: in order to improve the effect of night image conversion, the shared attention algorithm is combined with the generator G to construct a night conversion network that combines shared attention and recurrent adversarial generation network. The goal of this architecture is not only to integrate the attention mechanism into the image conversion process, but also to reuse the attention map calculated from the night domain X when reconstructing the image conversion sample in the target domain daytime domain Y, thereby improving the quality of the converted image;

[0106] like Figure 3As shown in the figure, ch represents the number of channels, the first one is the number of input channels, the second one is the number of output channels, K is the convolution kernel size, S is the stride size, and P is the padding size;

[0107] The generator G consists of three structures (encoder, bottleneck, decoder). The main task of the encoder is to extract features from the input image and convert it into an abstract feature map. The encoder contains multiple convolutional layers. Each layer uses a convolution with a stride of 2 to reduce the spatial dimension of the feature map. Instance normalization is applied after each convolutional layer to help stabilize the training process. Each convolutional layer is followed by a ReLU activation function to achieve faster network convergence and introduce nonlinear features. The main task of the bottleneck is to process the encoded features through the residual block to increase the depth of the network architecture. The bottleneck contains 9 residual blocks, each residual block contains two convolutional layers, each using a 3x3 convolution kernel. Instance normalization and ReLU activation functions are applied after each convolutional layer. The main task of the decoder is to decode the feature map extracted by the bottleneck to the same spatial dimension as the input image and generate the final output image. The decoder uses a transposed convolutional layer for upsampling. Instance normalization and ReLU activation functions are applied after each transposed convolutional layer. The last convolutional layer uses the Tanh activation function to limit the output to the range of [-1, 1];

[0108] In the attention mechanism, key, value and query are three core concepts. Query refers to the feature representation of the current input, which is used to perform similarity calculations with the key to determine the information that needs to be paid attention to. Each element of the query usually represents the input of a specific time step or the feature of a specific position. Key is a feature representation related to the input, which is used to match the query and calculate the attention weight (i.e., similarity score) to determine which parts of the input should be paid attention to. Each element of the key usually represents the feature of a specific time step or position. Value is also a feature representation related to the input. It is a feature vector of the image, which is used to multiply the attention weight generated by the key and query to obtain the result feature of the attention.

[0109] The shared attention algorithm is set before the convolutional layer of the encoder and after the transposed convolutional layer of the decoder. Unlike dot-product self-attention, the shared attention algorithm interprets each channel of the Key as a single attention map, rather than corresponding to the attention between two points in the image, which learns to represent the semantic aspects of the image. This approach allows each channel to focus on capturing different features of different channels of the image (such as texture, edge, color), thereby improving the model's ability to understand the diversity of image features. This is because the semantic content of the input and output images of the generator G is highly correlated, so it can be inferred that for the i-th convolutional layer of the encoder and the corresponding iL-th convolutional layer of the decoder, the Key generated by its shared attention algorithm can be reused. Given that the encoder's attention is calculated on real samples, after the decoder is shared and reused, it becomes beneficial to share these attention maps to achieve more consistent and realistic outputs for reconstructed transformed images. The calculation of shared attention weights is given by the following formula:

[0110]

[0111] where σ row represents the application of the softmax function along each row of the matrix Y; σ col represents the application of the softmax function along each column of the matrix Y; d k represents the dimension of the key vector; Q represents the query vector; K represents the key vector; T represents the matrix transpose operation; O represents the attention weight of the shared attention output of the encoder.

[0112] Between each Key vector, a skip connection is used, and a refinement network is introduced between these connections. The network adopts 3×3 deep convolution. Specifically, the skip connection directly connects different layers to share the attention weights of the i-th convolution layer of the encoder to the iL-th convolution layer of the decoder for reuse to help information flow and gradient propagation. The refinement network processes the channels of each Key vector by grouping the input channels and using independent convolution kernels for each group. This method ensures that each channel is refined, thereby improving the quality of the attention map. The shared attention module regards the Key as an attention map shared between the i-th convolution layer of the encoder and the iL-th transposed convolution layer of the decoder;

[0113] The number of attention maps is taken as a hyperparameter. For this, the number of attention maps is set to half of the number of input channels per layer. In addition, the first shared attention algorithm directly learns the dependencies between local features from the input 3-channel image, and the number of output attention maps is fixed to 8;

[0114] Given that it is easier for the network to learn dependencies in local neighborhoods, in the initial training steps, it is desirable to rely more on these local dependencies and wait for long-range dependencies to become more meaningful. To achieve this, a learnable scale parameter is introduced to multiply the output of the attention before adding the input features. Where γ is initialized to zero to initially focus on the easier task of generating good features based on local neighborhoods with convolutions, and then assign higher weights to long-range dependencies learned from more meaningful features. In this way, the final output of the introduced shared attention algorithm is given by:

[0115] y i =γO i +x i ;

[0116] Where O is the output of the encoder’s shared attention algorithm, x is the input feature map, and γ is a learnable scalar initialized to 0. The introduction of learnable γ allows the network to first rely on clues in the local neighborhood, and then gradually learn to assign more weight to non-local evidence.

[0117] S3. Compress the shared attention recurrent adversarial generative network model by referring to the knowledge distillation algorithm to obtain the teacher generator model, and transfer the knowledge to the student generator model through the teacher generator model.

[0118] Furthermore, the method of compressing the shared attention recurrent adversarial generative network model by referencing the knowledge distillation algorithm to obtain the teacher generator model, and transferring the knowledge to the student generator model through the teacher generator model includes the following steps:

[0119] S31. The generator in the night conversion model based on the shared attention recurrent adversarial generative network is used as the teacher generator to build a teacher generator model;

[0120] S32, train the generator to map the convolutional layer of the night domain to Y i The convolutional layer of the daytime domain uses a discriminator to distinguish between the generated images of the teacher generator and the real images;

[0121] S33, the teacher generator generates a feature map of a night domain image through an encoder, and decodes the feature map of the night domain image into an image feature map of a day domain through a decoder;

[0122] S34, combining the joint loss function, making the daytime domain image feature map output by the teacher generator close to the real daytime domain image;

[0123] S35. Through the knowledge distillation algorithm, the teacher generator is used to guide the student generator to train and build a student generator model;

[0124] S36, the encoder structure of the teacher generator is consistent with that of the student generator, and the decoder structure of the teacher generator corresponds to that of the student network;

[0125] S37, supervising the conversion image generated by the student generator through the online knowledge distillation loss;

[0126] S38. Use online knowledge distillation loss to measure the difference between the generated conversion images of the teacher generator and the generated conversion images of the student generator.

[0127] It should be explained that the generator G in S2 is used as the teacher generator G T , G T The purpose is to learn the mapping function that transforms the image of night domain X into the image of day domain Y, and train the generator G T X i Mapping to Y i , use the discriminator D to distinguish G T The generated images and real images of the teacher generator G T The encoder generates a feature map of the night domain image, denoted as F gt , and then the decoder converts F gt Decoded into the daytime domain image G T (x), in order to make G T The output daytime domain image G T (x) is close to the real daytime domain image y, minimizing the following reconstruction loss:

[0128] L Recon (G T )=E x,y [||yG T (x) ‖1];

[0129] Where y is the real daytime domain image, G T (x) is the daytime domain image generated by the generator;

[0130] Therefore, the loss function and optimization objective are defined as:

[0131]

[0132] Where L GAN (G T ,D) is the adversarial loss of S1;

[0133] Teacher GeneratorG T The encoder and student generator G S The encoder structure is consistent with the teacher generator GT The decoder and student network G S Corresponding to the decoder structure of , the teacher generator G is upsampled by a 3×3 transposed convolution layer with a step size of 2. T The generated transformation image is denoted as p t , the student generator G S The generated transformation image is denoted as p s , p t Through the online knowledge distillation loss, p s To supervise the student generator G S Get G T Prior knowledge of features, using online knowledge distillation loss to measure p t and p s The difference between them, the structural similarity loss L SSIM It is a loss function based on perceptual image quality, which is used to evaluate and optimize image quality in image generation and restoration tasks. SSIM The model training is guided by measuring the structural similarity between two images. The formula is as follows:

[0134]

[0135] where μ t , μ s is the estimated mean, is the standard deviation of contrast, σ ts is the covariance of the structural similarity estimate, C1 and C2 are constants to avoid the denominator being zero;

[0136] Feature loss L feature Image quality is evaluated by calculating the distance between two image features in feature space. The goal is to capture high-level image features to ensure that the generated images are similar not only at the pixel level but also in feature statistics. t , and p s have similar feature representations, minimizing the following feature loss:

[0137]

[0138] where φ j (x) is the feature map obtained after the j-th layer network processing of input x, C j ×H j ×W j is φ j The dimension of (x) is calculated using the style loss L style To penalize differences in style features, such as color, texture, shape, etc.:

[0139]

[0140] in It is the Gram matrix activated by the jth layer in the VGG network. This matrix captures the style features of the image by calculating the inner product relationship between the channels in the activation map. The hierarchical structure of the VGG network (i.e., feature extraction from low-level to high-level) can provide an effective way to extract the style of the image in layers. By using the Gram matrix generated by different layers of the VGG network, it can capture multiple levels of style features from subtle textures to overall color layout, and is often used in tasks such as style migration.

[0141] Therefore, the total loss function and optimization objective are defined as:

[0142] L KD (p t ,p s )=λ SSIM L SSIM +λ feature L feature +λ style L style ;

[0143] Among them, L SSIM represents the structural similarity loss; L feature represents feature loss; L style represents the style loss; λ SSIM Indicates L SSIM The weight hyperparameter coefficient of the loss; λ feature Indicates L feature The weight hyperparameter coefficient of the loss; λ style Indicates L style The weight hyperparameter coefficient of the loss; in addition, λ SSIM , feature , style Three hyperparameter coefficients balance L SSIM , L feature , L style loss.

[0144] In the present invention, through the above-mentioned online knowledge distillation, a larger deep neural network is realized, with a large number of parameters of the architect generator G T , guide the student generator G S By training, a smaller student model with fewer parameters is obtained. By learning the teacher generator, it can complete the night-to-day image conversion task with less accuracy loss and less computing resources. In the process of knowledge distillation, the teacher model provides rich knowledge representation, and G is trained and optimized without the participation of the discriminator. S , and obtain a G with better performance and lighter weight S Network model;

[0145] Finally, using the student generator G S To perform night-time-day image conversion, the model is lightweight. Online knowledge distillation is not only lightweight in terms of parameters, but also simplifies the network structure by reducing the number of layers and width. Under the premise of maintaining accuracy, knowledge distillation helps the student model find a balance between performance and complexity, so that it can be efficiently executed in practical applications. The student generator G S It is smaller, has significantly lower inference computing costs, has lower hardware requirements, and has faster inference speed, so it can be deployed on resource-constrained devices (such as mobile devices or edge devices).

[0146] S4. The feature map in the student generator model is upsampled and restored through the decoder to obtain the image converted from night to day.

[0147] Furthermore, the lightweight student generator model is obtained after training based on the student generator model, and the feature map in the lightweight student generator model is sampled and restored through a decoder to obtain an image converted from night to day, including the following steps:

[0148] S41, in the process of knowledge distillation, the teacher generator model is learned through the student generator model;

[0149] S42. In the process of learning the student generator model, the discriminator is used to train and optimize the student generator model to obtain a lightweight student generator model, and the joint loss function is optimized and trained to obtain a total loss function;

[0150] S43. Through the decoder in the lightweight student generator model, the daytime domain feature map generated by the student generator is sampled to obtain the image converted from night to day.

[0151] Furthermore, the total loss function and optimization objective are defined as:

[0152] L KD (p t ,p s )=λ SSIM L SSIM +λ feature L feature +λ style L style ;

[0153] Among them, L SSIM represents the structural similarity loss; L feature represents feature loss; L style represents the style loss; λ SSIM Indicates L SSIM The weight hyperparameter coefficient of the loss; λ feature Indicates Lfeature The weight hyperparameter coefficient of the loss; λ style Indicates L style The weight hyperparameter coefficient of the loss; in addition, λ SSIM , feature , style Three hyperparameter coefficients balance L SSIM , L feature , L style loss.

[0154] It should be explained that the lightweight student generator G trained by S3 S The model uses the transposed convolution operation of the decoder upsampling operation to map the small-sized features to a larger-sized space, while retaining important feature information and gradually restoring the spatial resolution of the image. After each upsampling, the convolution layer is used to further extract features and adjust the number of channels. ReLU is used as the main activation function to introduce nonlinearity and effectively propagate gradients, while improving the processing of image details during the decoding process. In addition, when generating the final RGB image, the decoder usually uses the Tanh activation function. The Tanh function compresses the output to the range of [-1, 1], which is suitable for generating pixel values ​​of natural images. The low-level features in the encoder are combined with the high-level features in the decoder using jump connections to enhance the detail restoration effect. As the upsampling proceeds layer by layer, the resolution of the feature map gradually increases until it is restored to the size of the original image. Finally, the decoder converts the high-resolution feature map into a three-channel RGB image through 1x1 convolution to generate the final daytime image.

[0155] The specific example is as follows: The present invention is trained on the Day2Night training set, which contains 6450 daytime images and 7744 nighttime images. Images in the data set are randomly selected as test images. The CPU model is AMD Ryzen 77735H, the GPU model is GTX4060, and the video memory size is 16GB. The programming language used is Python, version 3.7, and the framework uses Pytorch, version number 1.11.0.

[0156] Six metrics are used to evaluate the algorithms. BRISQUE is defined as a metric that determines how close an image is to a natural scene by evaluating the degree of distortion in the image. The lower the score, the better the image quality. NIQE examines the average features of a clean, undistorted image and compares these features to those of a test image. The closer the match between the features of the test image and the reference clean image, the higher the quality. CEIQ is a quality metric that addresses contrast distortion. The higher the score, the better the image quality. Michelson contrast provides a quantitative measure of the degree of distinction between bright and dark areas. Higher contrast values ​​indicate larger intensity differences, resulting in clearer boundaries and improved visual perception of details in the image. MACs is the total number of multiplication-addition operations in the model. The lower the value, the smaller the model complexity. Parameters is the number of parameters in the network. The lower the value, the smaller the model complexity.

[0157] The method of the present invention is compared with the night-to-day image conversion algorithms of Pix2Pix, recurrent generative adversarial network, GAN-Compression, and OMGD respectively, and the processing results of each algorithm are evaluated using the above six indicators.

[0158] Among them, Pix2Pix is ​​an image conversion model based on generative adversarial networks (GAN). It is used to solve the paired image conversion problem. The training dataset needs to be a paired dataset. Using conditional GAN ​​(cGAN), the generator receives the input image and generates an image with specific attributes. Its training goal is to make the image generated by the generator as close to the real image as possible, while the discriminator strives to distinguish between the two, thereby continuously improving the quality of the generated image.

[0159] The Cyclic Generative Adversarial Network is a GAN-based image conversion model, but unlike Pix2Pix, the Cyclic Generative Adversarial Network does not require paired training data. It is suitable for unpaired image conversion tasks, and the key technology is the "cycle consistency loss", which ensures that the converted image remains consistent by converting it back to the original image after conversion.

[0160] GAN-Compression is a general compression method used to reduce the inference time and computational cost of cGAN. It transfers knowledge from the intermediate representation of the original teacher generator to the layer corresponding to its compressed student generator, and converts unpaired learning to paired learning. Then, it uses the Up-scaled Neural Network Architecture Search (NAS) to automatically find an efficient network with significantly reduced computational cost and parameters. This achieves model compression of the cyclic adversarial generative network and realizes a lightweight model.

[0161] OMGD uses a new online multi-granularity distillation scheme to obtain lightweight GANs, generate high-fidelity images with low computational requirements, and adopts a single-stage online distillation for GAN compression, in which the gradually improved teacher generator helps to improve the student generator without a discriminator. The complementary teacher generator and network layers provide a comprehensive multi-granularity concept to enhance visual fidelity from different dimensions. This achieves the lightweight model of the cyclic adversarial generative network.

[0162] From the comparative analysis in Table 1, it can be seen that the present invention achieves good image conversion effect and lower model complexity from night to day, and achieves a performance balance between image conversion effect and model complexity.

[0163] Table 1 Model comparison

[0164]

[0165] Among them, Pix2Pix is ​​an image conversion model for paired datasets based on Generative Adversarial Networks (GANs), and Cyclic Generative Adversarial Networks is an image conversion model for unpaired image conversion tasks based on Generative Adversarial Networks (GANs); GAN-Compression is a general GAN ​​compression method that uses intermediate layer knowledge distillation and neural network architecture search (NAS) to achieve lightweight Cyclic Generative Adversarial Network models; OMGD is a lightweight GAN obtained through a new online multi-granularity distillation compression method, which provides comprehensive multi-granularity to achieve lightweight Cyclic Generative Adversarial Network models through complementary teacher generators and network layers. Our method places a shared attention algorithm before the convolution of the encoder and after the transposed convolution in the decoder, and uses online knowledge distillation to compress the generator with the shared attention algorithm to achieve lightweight Cyclic Generative Adversarial Network models.

[0166] Table 1 compares the existing Pix2Pix, cyclic adversarial network, GAN-Compression and OMGD methods. The experimental results show that compared with the existing methods, the images generated by the model have achieved better results in contrast distortion image quality evaluation (CEIQ) and busy push quality evaluation (BRISQUE). However, in terms of natural scene image quality evaluation (NIQE) and image brightness and darkness contrast (Michelson contrast) related indicators, the performance is not ideal compared with the proposed method. In addition, compared with the existing methods, the proposed method is more lightweight in terms of model size (MACs) and parameter amount (Parameters), which balances the relationship between the quality of generated images and model size to a certain extent.

[0167] Therefore, the method of the present invention enhances local contrast, effectively solves the image distortion problem during night image transformation, improves the clarity of objects in the image, improves the overall quality of the image, and compresses the student generator with lower model complexity and lighter weight through the online knowledge distillation method. This method can be well applied to edge devices in low-light scenes to achieve night image transformation of edge devices, which is of great significance for improving the performance of detection and recognition models.

[0168] To sum up, with the help of the above-mentioned technical scheme of the present invention, the present invention combines the recurrent adversarial generative network model and the shared attention network, which can be more targeted at the main semantic information in the image, reduce the influence of background information on its night-day conversion, and make the converted semantic information more real and effective, and the image recognition model has higher accuracy; and designs online knowledge distillation to compress the generator of the recurrent adversarial generative network with shared attention, effectively avoiding the shortcomings of traditional offline knowledge distillation, so that the compression degree of the model is greatly improved, and the lightweight of the model is achieved. The compressed student model can maintain a high degree of accuracy; the present invention combines the recurrent adversarial generative network algorithm with the attention mechanism, and uses online knowledge distillation to compress the attention generator, a night-day image conversion method of a lightweight recurrent adversarial generative network model, after using this method to convert the image from night to day, the image has the advantages of prominent subject and clear target details. At the same time, the algorithm effectively alleviates the problems of image color distortion and blurred target details.

[0169] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A night-time-day image conversion method based on a lightweight cyclic adversarial generative network, characterized in that: The night-time-daytime image conversion method comprises the following steps: S1, select image data from the nighttime dataset and the daytime dataset, combine the joint loss function and generate the final network model for converting nighttime and daytime images; S2. Introduce the shared attention module into the generated final network model and construct a shared attention recurrent adversarial generation network model for night-to-day image conversion; S3. Compress the shared attention recurrent adversarial generative network model by referencing the knowledge distillation algorithm to obtain the teacher generator model, and transfer the knowledge to the student generator model through the teacher generator model; S4. The feature map in the student generator model is upsampled and restored through the decoder to obtain the image converted from night to day.

2. The method for night-time-daytime image conversion based on a lightweight cyclic adversarial generative network according to claim 1, characterized in that: The method of selecting image data from the nighttime dataset and the daytime dataset, combining the joint loss function and generating the final network model for converting nighttime and daytime images includes the following steps: S11, input the nighttime dataset into the generator and calculate the cycle consistency loss of the nighttime domain; S12, input the daytime dataset into the generator and calculate the cycle consistency loss of the daytime domain; S13. Use the joint loss function to optimize the network model parameters and construct the final network model.

3. The method for night-time-daytime image conversion based on a lightweight cyclic adversarial generative network according to claim 2, characterized in that: The joint loss function formula is as follows: L(G,F,D X ,D Y )=L GAN (G,D Y ,X,Y)+L GAN (F,D X ,Y,X); Where F represents the mapping function from night domain X to day domain Y; D x represents the domain discriminator of night domain X; G represents the mapping function from day domain Y to night domain X; D Y represents the domain discriminator of the daytime domain Y; X represents the nighttime domain; Y represents the daytime domain; L cyc (G, F) represents cycle consistency loss; L GAN (G,D Y ,X,Y) represents the adversarial loss of converting from night domain X to day domain Y; L GAN (F,D X ,Y,X) represents the adversarial loss of converting from daytime domain Y to nighttime domain X; L identity (G,F) represents the coloring loss; L(G,F,D X ,D Y ) represents the joint loss function of the generative adversarial network.

4. The method for night-time-daytime image conversion based on a lightweight cyclic adversarial generative network according to claim 2, characterized in that: The step of inputting the nighttime dataset into the generator and calculating the cycle consistency loss of the nighttime domain includes the following steps: S111, input the sample data in the nighttime data set into the generator to generate a converted daytime domain image, input the generated daytime domain image into the discriminator to obtain the discrimination result, and calculate the adversarial loss of the nighttime domain image conversion; S112, input the generated daytime domain image into the generator again, convert it to generate a nighttime domain image, and pass the generated nighttime domain image into the discriminator to obtain the discrimination result, and calculate the cycle consistency loss of the nighttime domain.

5. The method for night-time-daytime image conversion based on a lightweight cyclic adversarial generative network according to claim 4, characterized in that: The generator consists of an encoder, a bottleneck module, and a decoder; The encoder extracts features from the input image and converts the features into an abstract feature map; The bottleneck module processes the encoded features through residual blocks to increase the depth of the network architecture; The decoder decodes the feature map extracted by the bottleneck module to the same spatial dimension as the input image and generates the final output image.

6. The method for night-time-daytime image conversion based on a lightweight cyclic adversarial generative network according to claim 5, characterized in that: The method of introducing the shared attention module into the generated final network model and constructing a shared attention recurrent adversarial generation network model for night-time to day-time image conversion includes the following steps: S21, setting the shared attention module before the convolution layer of the encoder and after the transposed convolution layer of the decoder; S22. Use skip connections between each key vector and introduce a refinement network between the connections. Transpose the attention map shared between the convolutional layers based on the refinement network. S23, taking the number of attention maps as a hyperparameter, and setting the number of attention maps to half the number of input channels per layer, to learn the dependency between the hyperparameters; S24. Through dependencies, a shared attention recurrent adversarial generative network model is constructed for night-to-day image conversion.

7. The method for night-time-daytime image conversion based on a lightweight cyclic adversarial generative network according to claim 6, characterized in that: The calculation of the shared attention weight is given by the following formula: where σ row represents the application of the softmax function along each row of the matrix Y; σ col represents the application of the softmax function along each column of the matrix Y; d k represents the dimension of the key vector; Q represents the query vector; K represents the key vector; T represents the matrix transpose operation; O represents the attention weight of the shared attention output of the encoder.

8. The method for night-time-daytime image conversion based on a lightweight cyclic adversarial generative network according to claim 1, characterized in that: The method of compressing the shared attention recurrent adversarial generative network model by referencing the knowledge distillation algorithm to obtain the teacher generator model, and transferring the knowledge to the student generator model through the teacher generator model includes the following steps: S31. The generator in the night conversion model based on the shared attention recurrent adversarial generative network is used as the teacher generator to build a teacher generator model; S32, train the generator to map the convolutional layer of the night domain to Y i The convolutional layer of the daytime domain uses a discriminator to distinguish between the generated images of the teacher generator and the real images; S33, the teacher generator generates a feature map of a night domain image through an encoder, and decodes the feature map of the night domain image into an image feature map of a day domain through a decoder; S34, combining the joint loss function, making the daytime domain image feature map output by the teacher generator close to the real daytime domain image; S35. Through the knowledge distillation algorithm, the teacher generator is used to guide the student generator to train and build a student generator model; S36, the encoder structure of the teacher generator is consistent with that of the student generator, and the decoder structure of the teacher generator corresponds to that of the student network; S37, supervising the conversion image generated by the student generator through the online knowledge distillation loss; S38. Use online knowledge distillation loss to measure the difference between the generated conversion images of the teacher generator and the generated conversion images of the student generator.

9. The method for night-time-daytime image conversion based on a lightweight cyclic adversarial generative network according to claim 1, characterized in that: The method of obtaining a lightweight student generator model after training based on the student generator model, sampling and restoring the feature map in the lightweight student generator model through a decoder to obtain an image converted from night to day includes the following steps: S41, in the process of knowledge distillation, the teacher generator model is learned through the student generator model; S42. In the process of learning the student generator model, the discriminator is used to train and optimize the student generator model to obtain a lightweight student generator model, and the joint loss function is optimized and trained to obtain a total loss function; S43. Through the decoder in the lightweight student generator model, the daytime domain feature map generated by the student generator is sampled to obtain the image converted from night to day.

10. The method for night-time-daytime image conversion based on a lightweight cyclic adversarial generative network according to claim 9, characterized in that: The total loss function and optimization objective are defined as: L KD (p t ,p s )=λ SSIM L SSIM +λ feature L feature +λ style L style ; Among them, L SSIM represents the structural similarity loss; L feature represents feature loss; L style represents the style loss; λ SSIM Indicates L SSIM The weight hyperparameter coefficient of the loss; λ feature Indicates L feature The weight hyperparameter coefficient of the loss; λ style Indicates L style The weight hyperparameter coefficient of the loss; in addition, λ SSIM , feature , style Three hyperparameter coefficients balance L SSIM , L feature , L style loss.

Images