Image translation method, system, medium and device

Abstract

The present invention relates to the field of deep learning image translation, and in particular to an image translation method, system, medium and device. The method comprises: inputting a preprocessed image to be translated into a visible light to infrared image translation network to obtain a translation result and edge information of the image, wherein the visible light to infrared image translation network comprises an image translation module and an image edge information supervision module, wherein the image translation module comprises an attention submodule. Through the present invention, it is possible to introduce a self-attention mechanism in a generator and a discriminator to guide the visible light to infrared image translation network to focus on more discriminative areas in the image, and by introducing an edge information supervision module to constrain the consistency of the edge of the generated image with the real image in the target domain, it is helpful to solve the problem of edge blur and misalignment of the image translated by the existing method.

Description

Technical Field

[0001] The present invention relates to the field of deep learning image translation, and in particular to an image translation method, system, medium and device. Background Art

[0002] Traditional infrared image generation methods are mainly based on infrared image simulation software. Since such methods are based on actual physical models, involve a large number of parameters, and require human intervention in the research and application process, they cannot meet the needs of rapid and large-scale generation of infrared images. With the introduction of deep learning and the increasingly in-depth research of image translation technology in image generation tasks such as image style transfer, image deraining, and image restoration, the generation problem of infrared images can also be solved by image translation technology in deep learning.

[0003] Image translation has been a research hotspot in deep learning in recent years. As the name suggests, the purpose of image translation is to convert an image from the expression of one domain to the expression of another domain. The Generative Adversarial Network (GAN), proposed in 2014, has received great attention and development at home and abroad in recent years due to its outstanding performance in image conversion. The Generative Adversarial Network consists of a generator and a discriminator. Through the maximum-minimum game between the generator and the discriminator, the network can output synthetic data that fits the distribution of real data.

[0004] The original GAN ​​network is difficult to ensure the stability of training, and the quality of generated images is limited. The Cyclegan network proposed later introduced cycle consistency loss on the basis of the original generative adversarial loss to improve the structural stability of the network conversion image. With the significant effect of the attention mechanism on classification tasks, SAGAN and U-GAT-IT networks also introduced attention sub-modules in the image translation process, so that the network pays more attention to the more discriminative areas in the conversion process, thereby improving the generation quality of the network. Although the performance of existing image translation networks is getting better and better, the network still has problems such as edge diffusion, deformation, and instability of generated images. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide an image translation method, system, medium and device.

[0006] The technical solution of the present invention to solve the above technical problem is as follows: a visible light to infrared image translation method, comprising:

[0007] The preprocessed image to be translated is input into a visible light to infrared image translation network to obtain a translation result and edge information of the image. The visible light to infrared image translation network comprises an image translation module and an image edge information supervision module, wherein the image translation module comprises an attention submodule.

[0008] The beneficial effects of the present invention are as follows: by introducing a self-attention mechanism in the generator and the discriminator, the visible light to infrared image translation network is guided to focus on the more discriminative areas in the image, thereby better realizing the translation of the image; by introducing an edge information supervision module to constrain the consistency of the edges of the generated image and the input image, it is helpful to solve the problems of blurred and misaligned edges in the images translated by the existing methods.

[0009] Based on the above technical solution, the present invention can also be improved as follows.

[0010] Further, the preprocessing specifically includes:

[0011] The image to be translated is rotated, scaled, cropped and normalized in sequence.

[0012] Further, the process of obtaining the translation result is specifically as follows:

[0013] Extracting features of the preprocessed image to be translated by a coding submodule in a generator in the image translation module and obtaining a corresponding feature spectrum, wherein the coding submodule is composed of three downsampling blocks and six residual blocks;

[0014] The attention submodule in the generator weights the feature spectrum finally output by the encoding submodule to obtain an attention spectrum;

[0015] The attention spectrum is decoded by a decoding submodule in the generator to obtain the translation result.

[0016] Furthermore, the specific process of obtaining edge information is as follows:

[0017] After the first down-sampling block of the encoding submodule in the generator in the image translation module, an edge head is externally connected, and edge information is predicted for the first feature spectrum obtained after processing the first down-sampling block through the edge head, and the predicted edge information is constrained by the edge loss function.

[0018] Furthermore, the visible light to infrared image translation network is trained by a training sample set, optimized by minimizing a total loss function, and tested by a test sample set.

[0019] Another technical solution of the present invention to solve the above technical problem is as follows: a visible light to infrared image translation system, comprising:

[0020] The processing module is used to input the preprocessed image to be translated into a visible light to infrared image translation network to obtain a translation result and edge information of the image. The visible light to infrared image translation network includes an image translation module and an image edge information supervision module, wherein the image translation module includes an attention submodule.

[0021] The beneficial effects of the present invention are as follows: by introducing a self-attention mechanism in the generator and the discriminator, the visible light to infrared image translation network is guided to focus on the more discriminative areas in the image, thereby better realizing the translation of the image; by introducing an edge information supervision module to constrain the consistency of the edges of the generated image and the input image, it is helpful to solve the problems of blurred and misaligned edges in the images translated by the existing methods.

[0022] Further, the preprocessing specifically includes:

[0023] The image to be translated is rotated, scaled, cropped and normalized in sequence.

[0024] Further, the process of obtaining the translation result is specifically as follows:

[0025] Extracting features of the preprocessed image to be translated by a coding submodule in a generator in the image translation module and obtaining a corresponding feature spectrum, wherein the coding submodule is composed of three downsampling blocks and six residual blocks;

[0026] The attention submodule in the generator weights the feature spectrum finally output by the encoding submodule to obtain an attention spectrum;

[0027] The attention spectrum is decoded by a decoding submodule in the generator to obtain the translation result.

[0028] Furthermore, the specific process of obtaining edge information is as follows:

[0029] After the first down-sampling block of the encoding submodule in the generator in the image translation module, an edge head is externally connected, and edge information is predicted for the first feature spectrum obtained after processing the first down-sampling block through the edge head, and the predicted edge information is constrained by the edge loss function.

[0030] Furthermore, the visible light to infrared image translation network is trained by a training sample set, optimized by minimizing a total loss function, and tested by a test sample set.

[0031] Another technical solution of the present invention to solve the above technical problem is as follows: a storage medium, in which instructions are stored. When a computer reads the instructions, the computer executes a visible light to infrared image translation method as described in any one of the above items.

[0032] The beneficial effects of the present invention are as follows: by introducing a self-attention mechanism in the generator and the discriminator, the visible light to infrared image translation network is guided to focus on the more discriminative areas in the image, thereby better realizing the translation of the image; by introducing an edge information supervision module to constrain the consistency of the edges of the generated image and the input image, it is helpful to solve the problems of blurred and misaligned edges in the images translated by the existing methods.

[0033] Another technical solution of the present invention to solve the above technical problem is as follows: an electronic device comprises the above storage medium and a processor for executing instructions in the above storage medium.

[0034] The beneficial effects of the present invention are as follows: by introducing a self-attention mechanism in the generator and the discriminator, the visible light to infrared image translation network is guided to focus on the more discriminative areas in the image, thereby better realizing the translation of the image; by introducing an edge information supervision module to constrain the consistency of the edges of the generated image and the input image, it is helpful to solve the problems of blurred and misaligned edges in the images translated by the existing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A flowchart provided for an embodiment of a visible light to infrared image translation method of the present invention;

[0036] Figure 2 A structural framework diagram provided for an embodiment of a visible light to infrared image translation system of the present invention;

[0037] Figure 3 A schematic diagram of a model for unidirectional conversion from a source domain (visible light) to a target domain (infrared) provided in an embodiment of a visible light to infrared image translation method of the present invention;

[0038] Figure 4 A schematic diagram of the self-attention mechanism in the attention submodule provided in an embodiment of a visible light to infrared image translation method of the present invention. DETAILED DESCRIPTION

[0039] The principles and features of the present invention are described below. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0040] like Figure 1 As shown, a visible light to infrared image translation method comprises:

[0041] Step 1, preprocessing the image to be translated;

[0042] Step 2: input the preprocessed image to be translated into a visible light to infrared image translation network to obtain a translation result and edge information of the image. The visible light to infrared image translation network includes an image translation module and an image edge information supervision module, wherein the image translation module includes an attention submodule.

[0043] In some possible implementations, a self-attention mechanism is introduced in the generator and the discriminator to guide the visible light to infrared image translation network to focus on more discriminative areas in the image, thereby better achieving image translation; and an edge information supervision module is introduced to constrain the consistency of the edges of the generated image and the input image, which helps to solve the problems of blurred and misaligned edges in the images translated by existing methods.

[0044] It should be noted that the image to be translated is: Since the total loss function introduces a cyclic loss, the system has two translation directions. One is the translation (i.e. conversion) from a visible light image to an infrared image, and the image to be translated is the visible light image; the other direction is the translation from an infrared image to a visible light image, and the image to be translated is the infrared image.

[0045] The specific process of preprocessing the data set is as follows:

[0046] Perform data preprocessing operations such as random horizontal rotation, random scaling, random cropping, and normalization on the translated images;

[0047] To elaborate, S11: randomly flip the original image horizontally;

[0048] S12: scaling the image data obtained in S11 to a size of 286×286;

[0049] S13: Crop the image data obtained in S12 into a size of 256×256.

[0050] The beneficial effect of preprocessing is that by preprocessing the image dataset, similar but not identical samples are generated, thereby expanding the training dataset, suppressing overfitting, and improving the generalization ability of the model.

[0051] For the image translation module:

[0052] like Figure 3 As shown in Figure 1, the image translation module consists of a generator and a discriminator that introduce a self-attention mechanism respectively;

[0053] The generator converts the input image (the preprocessed image to be translated) into a synthetic image that fits the target domain data distribution, and the discriminator distinguishes the true and false of the synthetic image input and the real image of the target domain (the target image that has been preprocessed in the same way), and outputs the discrimination probability. The discrimination probability is used to optimize the training of the visible light to infrared image translation network. The generator module includes the encoding submodule, attention submodule and decoding submodule of the generator, and the discriminator includes the encoding submodule, attention submodule and classification submodule;

[0054] The encoding submodule of the generator extracts features from the input image. The encoding submodule consists of three downsampling blocks and six residual blocks. The input image is passed through the three downsampling blocks to obtain feature spectra F e1 、F e2 、F e3 The channel dimensions corresponding to these three feature spectra are 64, 128, and 256 respectively. In order to increase the network depth, a residual layer consisting of six residual blocks is stacked after the downsampling layer. The residual layer does not change the output feature spectrum F e4 (The preprocessed input image outputs the first feature spectrum under the action of the first downsampling block of the encoding module; the first feature spectrum outputs the second feature spectrum under the action of the second downsampling block; the second feature spectrum outputs the third feature spectrum under the action of the third downsampling block; the third feature spectrum outputs the feature spectrum under the action of the residual layer, that is, the fourth feature spectrum) The channel dimension is still 256;

[0055] The attention submodule of the generator is concerned with the feature spectrum F e4 Importance weighting is performed. First, the feature spectrum F e4 Apply the self-attention mechanism to obtain the self-attention spectrum F sa , the principle of the self-attention mechanism is as follows Figure 4 As shown. Specifically, the size of R C×H×W The characteristic spectrum F e4 After three convolution kernels of size 1×1, we get three vectors Q, K, and V, where , V∈R C×H×W , and C s =C / 4; then the vector Reorganize and transpose to get vector , and the vector , V∈R C×H×W Then the new vector is obtained through adaptive average pooling , V′∈R C×(H′×W′) ; Then multiply the vector Q′ and K′ and perform the softmax operation to obtain the attention weight matrix, and then multiply the obtained attention weight with the vector V′ to obtain the self-attention spectrum F sa ; Then the self-attention spectrum Fsa The global average pooling and maximum pooling operations are used to obtain the global feature spectrum F gap and F gmp , F gap With F gmp At the same time, a fully connected layer is connected to obtain two classification outputs L gap and L gmp , the corresponding parameters of the fully connected layer (the corresponding parameters of the fully connected layer are randomly initialized by the network. Since the parameters are learnable parameters, they will be updated as the network is optimized) are used as the self-attention spectrum F sa The weight of the weight is re-weighted to obtain the weighted characteristic spectrum F′ gap and F′ gmp , F′ gap and F′ gmp Splice them together according to the channel dimension to get the attention spectrum F a ; In addition to the above feature spectrum attention weighting operation, the two classification outputs L gap and L gmp The final classification output is concatenated according to the channel dimension. The purpose of this classification output is to enable the generator to pay more attention to the more discriminative positions in the image.

[0056] The decoding submodule of the generator pays attention to the spectrum F a Decode to get the translated image. The decoding submodule consists of a residual block with adaptive instance / layer normalization and an upsampling layer. Specifically, the attention spectrum F a Send it to the multi-layer perceptron (MLP) to obtain the parameters γ and β in the normalization operation, and let the network adaptively select instance normalization or layer normalization to obtain the normalized feature spectrum F norm , where the purpose of introducing the normalization operation is to enable the network to adaptively learn the corresponding changes in shape, color, texture, etc. during image translation; the feature spectrum F norm After the upsampling layer, the image is reconstructed and finally a translation image with the same resolution as the input image is obtained;

[0057] The encoding submodule of the discriminator extracts features from the translated image output by the generator or the real image in the target domain (the dataset provides pairs of visible light images and infrared images. When the translation direction is from visible light to infrared image, the corresponding real image in the target domain is the infrared image provided by the dataset and preprocessed. Similarly, when the translation direction is from infrared image to visible light image, the corresponding real image in the target domain is the visible light image provided by the dataset and preprocessed). The encoding submodule of the discriminator only contains a downsampling layer, which is composed of four downsampling blocks. The image input to the discriminator is passed through the four downsampling blocks to obtain the feature spectrum F D1 、F D2 、F D3、F D4 ,The channel dimensions corresponding to these four feature spectra are 64, 128, 256, and 512 respectively;

[0058] The attention submodule of the discriminator pays attention to the feature spectrum F D4 Importance weighting is performed. First, the feature spectrum F D4 Apply the self-attention mechanism to obtain the self-attention spectrum F Dsa , followed by the self-attention spectrum F Dsa The global average pooling and maximum pooling operations are used to obtain the global feature spectrum F Dgap and F Dgmp , F Dgap With F Dgmp After the fully connected layer, two classification outputs L are obtained Dgap and L Dgmp , the corresponding parameters of the fully connected layer are used as the self-attention spectrum F Dsa The weight of the weight is re-weighted to obtain the weighted characteristic spectrum F′ Dgap and F′ Dgmp , F′ Dgap and F′ Dgmp Splice them together according to the channel dimension to get the attention spectrum F Da ; In addition to the above feature spectrum attention weighting operation, the two classification outputs L Dgap and L Dgmp The final classification output is concatenated according to the channel dimension. The purpose of this classification output is to enable the discriminator network to pay more attention to the more discriminative positions in the image.

[0059] The classification submodule of the discriminator focuses on the spectrum F Da The channel dimension is reduced to 1, and the true or false probability of the discriminator input image is obtained (after the channel is reduced to 1, the corresponding true or false probability value is obtained).

[0060] The beneficial effect of the translation module is that when the generator performs image conversion and the discriminator performs image discrimination, the network can focus on areas that are more discriminative for image translation, thereby improving the performance of the generator and the discriminator, making the quality of the translated image better and closer to the target image.

[0061] For edge information supervision module:

[0062] In order to make the edges of the translated image consistent with those of the input image (i.e. aligned), an edge information supervision module is introduced based on the translation network;

[0063] The first downsampling block of the generator encoding module is followed by an edge prediction head, whose purpose is to predict the feature spectrum F e1Corresponding edge information; the supervisory information corresponding to the predicted edge information is the edge information of the corresponding image in the target domain (since the total loss function introduces cyclic loss, the system has two translation directions. One is the translation (i.e. conversion) of visible light images to infrared images, and its corresponding target domain is the infrared domain; the other direction is the translation of infrared images to visible light images, and its corresponding target domain is the visible light domain). The supervisory information is obtained by applying the Sobel operator to the target domain image.

[0064] The beneficial effect of the edge information supervision module is that it helps guide the system to focus on edge consistency during the translation process while the network is translating, thereby making the contour edges of the translated image clearer, regular, and consistent, improving the overall translation quality of the image.

[0065] For training the visible to infrared image translation network:

[0066] First, define the total loss function of the network. The total loss function of the network is a joint function composed of generation adversarial loss, cycle loss, consistency loss, classification loss and edge loss. The adversarial loss makes the distribution of the generated image constantly fit the distribution of the corresponding output domain. The cycle consistency loss can improve the structural stability of the network conversion image. The consistency loss can increase the convergence speed. The BCE classification loss can guide the network to better distinguish the source domain (visible light) image and the target domain (infrared) image, and the edge loss uses the Dice loss to guide the edge information of the generated image to be consistent with the edge information of the target image. The total loss function of the network is:

[0067] L=p1L gan +p2L cyc +p3L iden +p4L class +p5L edge (1)

[0068] Among them, p1~p5 are the corresponding generated adversarial losses L gan , circulation loss L cyc , consistency loss L iden , classification loss L class And the edge information loss L edge The proportional coefficients are all hyperparameters. The optimization of the network includes both the conversion from the source domain to the target domain and the conversion from the target domain to the source domain. The corresponding loss terms of formula (1) when converting from the source domain to the target domain are expressed as follows:

[0069]

[0070]

[0071]

[0072]

[0073]

[0074]

[0075] Where x is the source domain X se Or target domain X tt The image of, se→tt means the translation direction is from source domain to target domain, E is the expectation. η in formula (5) and formula (6) se (x) and They are the generator G se→tt Attention module and discriminator D tt The final classification output of the attention module is Represents the predicted edge The i-th pixel of Indicates the corresponding edge information of the real image in the target domain The ith pixel of , pixels represents the total number of pixels, and σ is a Laplace smoothing term with a value of 1. Similarly, the expression formulas for each loss term corresponding to the conversion from the target domain to the source domain are similar to the above formulas. The process is as follows. (After network optimization, an optimal model will be obtained, and the optimal model will be used as the final model for testing the test sample set).

[0076] When converting the target domain to the source domain, the corresponding loss terms of formula (1) are expressed as follows:

[0077]

[0078]

[0079]

[0080]

[0081]

[0082] Where x is the source domain X se Or target domain X tt tt→se indicates the translation direction is from the target domain to the source domain. η in formula (11) and formula (12) tt (x) and They are the generator G se→tt Attention module and discriminator D se The final classification output of the attention module is Represents the predicted edge The i-th pixel of Indicates the corresponding edge information of the real image in the source domain The i-th pixel of , pixels represents the total number of pixels, and σ is a Laplace smoothing term with a value of 1.

[0083] The beneficial effect is that the distribution of the generated image is constantly fitted to the distribution of the corresponding output domain (source domain or target domain) through adversarial loss, the introduction of cycle consistency loss is used to improve the structural stability of the network conversion image, the introduction of consistency loss can increase the convergence speed, the introduction of BCE classification loss can guide the network to better distinguish the source domain image from the target domain image, and the edge loss uses Dice loss to guide the edge information of the generated image to be consistent with the edge information of the target image.

[0084] Then, a training sample set (the public data set LLVIP downloaded from the Internet includes a training sample set and a test sample set) is used to optimize the visible light to infrared image translation network according to the total loss function through automatic differentiation technology, using a stochastic gradient descent and back propagation algorithm (the visible light to infrared image translation network is optimized by minimizing the total loss function);

[0085] The test sample set is used to test the visible light to infrared image translation network based on the training set weights (the public data set LLVIP downloaded from the Internet includes a training sample set and a test sample set. The test process is to send the test sample set to the generator G in the optimized visible light to infrared image translation network. se→tt , that is, the generator G after the test sample set is optimized se→tt Get the output translation image).

[0086] Using the automatic differentiation technology provided in Pytorch, the network is optimized by minimizing the loss terms in the total loss function using the stochastic gradient descent and back-propagation algorithms. The training weights are used for testing using the test sample set.

[0087] The beneficial effect is to optimize the network parameters through stochastic gradient descent, back propagation algorithm and loss function; and then use the test set to test on the basis of the training set weights.

[0088] The image translation module with the self-attention mechanism is composed of a generator and a discriminator that each introduces the self-attention mechanism. Since the image translation system has two translation directions, namely, the translation direction from the source domain (visible light) to the target domain (infrared) and the translation direction from the target domain (infrared) to the source domain (visible light), the system has two generators in corresponding directions and two discriminators in corresponding directions. The encoding module of the generator extracts features from the input image, and the self-attention mechanism introduced enables the system to pay more attention to the areas with large changes during the translation process. The decoding module that introduces adaptive instance-layer normalization guides the system to adaptively learn the changes in shape, color, texture, etc. during the translation process while performing image reconstruction.

[0089] The image edge information supervision module introduces an edge head in the encoding module of the generator to predict edge information. It guides the system to focus on edge consistency during the translation process by performing loss supervision on the edge information of the image corresponding to the output domain extracted by the Sobel operator, thereby making the contour edge of the translated image clearer and more regular.

[0090] The loss function module is used to define the total loss function of the network consisting of generative adversarial loss, cycle loss, consistency loss, classification loss, and edge loss;

[0091] The training and testing module uses the training sample set to optimize the visible light to infrared image translation network, and uses the test sample set to test the visible light to infrared image translation network.

[0092] A memory, a processor and a program stored in the memory. When the processor executes the program, the steps of the visible light to infrared image translation method are implemented.

[0093] The embodiment of the present invention tests the proposed algorithm on the public dataset LLVIP. The LLVIP dataset is a visible light-infrared image pairing dataset for low-light vision. The dataset contains 33,672 images, that is, a total of 16,836 visible light-infrared image pairs, most of which are taken in very dark scenes, and all images are strictly aligned in time and space.

[0094] The proposed method is implemented using NVIDIA TITAN V GPU and open source machine learning library Pytorch. During each iteration of training, the initial learning rate is set to 10 -4 , the batch size is set to 16, the number of iterations is 1,000,000, and the optimal model parameters are saved after training.

[0095] Embodiment 1, combination Figure 1 and Figure 3To illustrate, S1: perform data preprocessing operations such as random horizontal rotation, scaling, cropping, and normalization on the translated image (due to the introduction of cyclic loss in the objective function, the system has two translation directions. One is the translation (i.e., conversion) from visible light image to infrared image, and the corresponding input image is the visible light image; the other direction is the translation from infrared image to visible light image, and the corresponding input image is the infrared image.);

[0096] S2: Construct an image translation module. The construction of the image translation module includes the construction of a generator and a discriminator, wherein the generator converts the preprocessed input image described in step S1 (converting the preprocessed input image (i.e., the image to be translated) to the generator, i.e., the input image is converted into a translation image under the action of the encoding module, attention module, and decoding module of the generator) into a translation image that fits the target domain data distribution, and the discriminator discriminates the true or false of the translation image (the translation image obtained by the generator) or the real image of the target domain (i.e., the preprocessed target image described in S1) input therein (the translation image or the real image of the target domain is converted into a translation image in the encoding module of the discriminator network). Under the action of the block, attention module and classification module, a probability value is obtained to judge whether the image is real or synthetic (i.e. fake). A probability value of 1 indicates that the image is real, and a probability value of 0 indicates that the image is synthetic (fake) data. True and false discrimination is the discrimination between the translated image and the corresponding real image in the target domain. When the translation direction is visible light to infrared image, the true and false discrimination is the discrimination between the infrared image generated by the generator and the real infrared image preprocessed as described in S1; when the translation direction is infrared image to visible light image, the true and false discrimination is the discrimination between the generated visible light image and the real visible light image preprocessed as described in S1), and the discrimination probability is output. (The discrimination probability is the generative adversarial loss L in the loss function. gan ) The generator includes an encoding module, an attention module and a decoding module of the generator, and the discriminator includes an encoding module, an attention module and a classification module;

[0097] S3: Construct an image edge information supervision module, connect the output feature information of the first downsampling layer of the generator encoding module to an edge head (the edge head is a network composed of two convolutional layers, a batch normalization layer, an activation layer and an upsampling layer) to predict edge information (the output feature information of the first downsampling layer is sent to the edge head, and the edge head outputs the predicted edge information, that is, the function of the edge head is to generate edge information), and its corresponding supervision information (which refers to the edge information prediction result. The supervision information can be understood as the target information) is the edge information of the real image in the target domain extracted by the Sobel operator; (This step is indeed not applied to the discrimination probability of step 2. The purpose of this step is to constrain the generator to reduce the loss of edge information during the encoding process. This step ultimately uses edge loss to make the predicted edge map close to the edge map corresponding to the real image in the target domain, thereby reducing the loss of edge information).

[0098] When the generator generates the translated image, it predicts the edge information at the same time; the entire visible light to infrared image translation network constrains the generator to retain the edge information in the encoding process (feature extraction) through the predicted edge information, that is, the output feature of the first downsampling layer can obtain the edge information through the edge head, which requires that the output feature itself retains the edge information, so the predicted edge information plays a role of constraint; the entire visible light to infrared image translation network is optimized by minimizing the total loss function in S4. Each optimization of the network means that the parameters of the network will be updated, and the updated network will obtain a new output (translation image). As the number of optimizations increases, the translation image will fit the target domain data distribution more and more, that is, the quality of the translation image will increase. Finally, an optimal model (the highest quality translation image) is selected as the final network model, and the input image is sent to the final network model to obtain the final translation image.

[0099] The predicted edge information is obtained through the edge head, and the corresponding image in the target domain is processed by the Solbay operator (when the direction is from visible light to infrared, the corresponding image in the target domain is the infrared image provided by the data set and preprocessed) to obtain the supervision information. The subsequent use of edge loss requires the predicted edge information and its corresponding supervision information. The role of edge loss is to make the predicted edge information infinitely close to its corresponding supervision information.

[0100] The edge information and target information are characteristic spectra, and their visualization effect is an edge map.

[0101] The predicted edge information needs to be given a supervision information (ground-truth often mentioned in deep learning) so that the edge head knows what the output information should look like. Without supervision information, the output of the edge head is meaningless and the network does not know what edge map should be output. Edge loss requires the use of predicted edge information and its corresponding supervision information.

[0102] S4: Define the total loss function of the network (the total loss function of the visible light to infrared image translation network, the entire visible light to infrared image translation network includes the image translation module described in S2 and the image edge information supervision module described in S3). The total loss function of the network is a joint function composed of generative adversarial loss, cycle loss, consistency loss, classification loss and edge loss;

[0103] Except for the edge loss used in the edge information supervision module, other losses are used in the image translation module, and the input of each loss can be seen through each loss formula, such as D tt (G se→tt (x)) represents the translation image G se→tt (x) is sent to the discriminator D tt The obtained discrimination probability.

[0104] S5: Optimize the visible light to infrared image translation network (the visible light to infrared image translation network is composed of the image translation module described in step S2 and the image edge information supervision module described in step S3) using a training sample set (the public data set LLVIP downloaded from the Internet includes a training sample set and a test sample set); test the visible light to infrared image translation network using a test sample set (the public data set LLVIP downloaded from the Internet includes a training sample set and a test sample set).

[0105] (The training sample set is sent to the visible light to infrared image translation network for training, and the network is optimized according to the total loss function described in S4. The connection relationship of the whole system is to first perform the image preprocessing operation described in S1, and then send the preprocessed input image obtained in step S1 to the visible light to infrared image translation network for image translation, and the visible light to infrared image translation network includes the image translation module described in S2 and the image edge information supervision module described in S3. After the entire visible light to infrared image translation network composed of the image translation module described in S2 and the image edge information supervision module described in S3 is constructed, the total loss function of the entire network is defined as described in S4. The total loss function is the joint function of several loss functions described in S4. The process of neural network training or optimization is the process of minimizing the loss function. The smaller the loss function value, the closer the corresponding predicted result and the true result value. Next, as described in S5, the training sample set is sent to the visible light to infrared image translation network for training, and the network is optimized according to the total loss function described in S4; after the network is optimized, the test sample set is sent to the optimized network model for testing.).

[0106] Preferably, in any of the above embodiments, the preprocessing specifically includes:

[0107] The image to be translated is rotated, scaled, cropped and normalized in sequence.

[0108] Preferably, in any of the above embodiments, the process of obtaining the translation result is specifically:

[0109] Extracting features of the preprocessed image to be translated by a coding submodule in a generator in the image translation module and obtaining a corresponding feature spectrum, wherein the coding submodule is composed of three downsampling blocks and six residual blocks;

[0110] The attention submodule in the generator weights the feature spectrum finally output by the encoding submodule to obtain an attention spectrum;

[0111] The attention spectrum is decoded by a decoding submodule in the generator to obtain the translation result.

[0112] Preferably, in any of the above embodiments, the specific process of obtaining the edge information is:

[0113] After the first down-sampling block of the encoding submodule in the generator in the image translation module, an edge head is externally connected, and edge information is predicted for the first feature spectrum obtained after processing the first down-sampling block through the edge head, and the predicted edge information is constrained by the edge loss function.

[0114] Preferably, in any of the above embodiments, the visible light to infrared image translation network is trained by a training sample set, optimized by minimizing a total loss function, and tested by a test sample set.

[0115] like Figure 2 As shown, a visible light to infrared image translation system comprises:

[0116] The acquisition module 100 is used to preprocess the image to be translated to obtain the preprocessed image to be translated;

[0117] The processing module 200 is used to input the preprocessed image to be translated into a visible light to infrared image translation network to obtain a translation result and edge information of the image. The visible light to infrared image translation network includes an image translation module and an image edge information supervision module, wherein the image translation module includes an attention submodule.

[0118] The beneficial effects of the present invention are as follows: by introducing a self-attention mechanism in the generator and the discriminator, the visible light to infrared image translation network is guided to focus on the more discriminative areas in the image, thereby better realizing the translation of the image; by introducing an edge information supervision module to constrain the consistency of the edges of the generated image and the input image, it is helpful to solve the problems of blurred and misaligned edges in the images translated by the existing methods.

[0119] Preferably, in any of the above embodiments, the acquisition module 100 specifically includes:

[0120] The image to be translated is rotated, scaled, cropped and normalized in sequence to obtain the pre-processed image to be translated.

[0121] Preferably, in any of the above embodiments, the process of obtaining the translation result is specifically:

[0122] Extracting features of the preprocessed image to be translated by a coding submodule in a generator in the image translation module and obtaining a corresponding feature spectrum, wherein the coding submodule is composed of three downsampling blocks and six residual blocks;

[0123] The attention submodule in the generator weights the feature spectrum finally output by the encoding submodule to obtain an attention spectrum;

[0124] The attention spectrum is decoded by a decoding submodule in the generator to obtain the translation result.

[0125] Preferably, in any of the above embodiments, the specific process of obtaining the edge information is:

[0126] After the first down-sampling block of the encoding submodule in the generator in the image translation module, an edge head is externally connected, and edge information is predicted for the first feature spectrum obtained after processing the first down-sampling block through the edge head, and the predicted edge information is constrained by the edge loss function.

[0127] Preferably, in any of the above embodiments, the visible light to infrared image translation network is trained by a training sample set, optimized by minimizing a total loss function, and tested by a test sample set.

[0128] Another technical solution of the present invention to solve the above technical problem is as follows: a storage medium, in which instructions are stored. When a computer reads the instructions, the computer executes a visible light to infrared image translation method as described in any one of the above items.

[0129] In some possible implementations, a self-attention mechanism is introduced in the generator and the discriminator to guide the visible light to infrared image translation network to focus on more discriminative areas in the image, thereby better achieving image translation; and an edge information supervision module is introduced to constrain the consistency of the edges of the generated image and the input image, which helps to solve the problems of blurred and misaligned edges in the images translated by existing methods.

[0130] Another technical solution of the present invention to solve the above technical problem is as follows: an electronic device comprises the above storage medium and a processor for executing instructions in the above storage medium.

[0131] In some possible implementations, a self-attention mechanism is introduced in the generator and the discriminator to guide the visible light to infrared image translation network to focus on more discriminative areas in the image, thereby better achieving image translation; and an edge information supervision module is introduced to constrain the consistency of the edges of the generated image and the input image, which helps to solve the problems of blurred and misaligned edges in the images translated by existing methods.

[0132] The reader should understand that in the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0133] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the method embodiments described above are only illustrative, for example, the division of steps is only a logical function division, and there may be other division methods in actual implementation, such as multiple steps can be combined or integrated into another step, or some features can be ignored or not executed.

[0134] If the above method is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0135] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A visible light to infrared image translation method, characterized in that: include: Step 1, preprocessing the image to be translated to obtain a preprocessed image to be translated; Step 2, inputting the preprocessed image to be translated into a visible light to infrared image translation network to obtain a translation result and edge information of the image, wherein the visible light to infrared image translation network comprises an image translation module and an image edge information supervision module, wherein the image translation module comprises an attention submodule; The process of obtaining the translation result is specifically as follows: Extracting features of the preprocessed image to be translated by a coding submodule in a generator in the image translation module and obtaining a corresponding feature spectrum, wherein the coding submodule is composed of three downsampling blocks and six residual blocks; The attention submodule in the generator weights the feature spectrum finally output by the encoding submodule to obtain an attention spectrum; Decoding the attention spectrum through a decoding submodule in the generator to obtain the translation result; The specific process of obtaining edge information is as follows: After the first down-sampling block of the encoding submodule in the generator in the image translation module, an edge head is externally connected, edge information is predicted for the first feature spectrum obtained after processing the first down-sampling block through the edge head, and the predicted edge information is constrained through the edge loss function; The total loss function of the network is a joint function composed of generation adversarial loss, cycle loss, consistency loss, classification loss and edge loss. The adversarial loss makes the distribution of the generated image continuously fit the distribution of the corresponding output domain. The cycle consistency loss can improve the structural stability of the network conversion image. The consistency loss allows small errors between the generated image and the target image and improves the convergence speed. The classification loss can guide the network to distinguish the source domain image from the target domain image, where the source domain image refers to the visible light image and the target domain image refers to the infrared image. The edge loss uses the Dice loss to guide the edge information of the generated image to be consistent with the edge information of the target image. The total loss function of the network is: L=p1L gan +p2L cyc +p3L iden +p4L class +p5L edge (1) Among them, p1~p5 are the corresponding generated adversarial losses L gan , circulation loss L cyc , consistency loss L iden , classification loss L class And the edge information loss L edge The proportional coefficients are all hyperparameters. The optimization of the network includes both the conversion from the source domain to the target domain and the conversion from the target domain to the source domain. When the source domain is converted to the target domain, the corresponding loss terms of formula (1) are expressed as follows: Where x is the source domain X se Or target domain X tt , se→tt indicates the translation direction is from the source domain to the target domain, E is the expectation, and η in formula (5) and formula (6) se (x) and They are the generator G se→tt Attention module and discriminator D tt The final classification output of the attention module is Represents the predicted edge The i-th pixel of Indicates the corresponding edge information of the real image in the target domain The i-th pixel of , pixels represents the total number of pixels, σ is a Laplace smoothing term with a value of 1, and an optimal model will be obtained after network optimization. The optimal model is used as the final model for testing the test sample set.

2. A visible light to infrared image translation method according to claim 1, characterized in that: The step 1 specifically includes: The image to be translated is rotated, scaled, cropped and normalized in sequence to obtain the pre-processed image to be translated.

3. The visible light to infrared image translation method according to claim 1, characterized in that: The visible light to infrared image translation network is trained by a training sample set, optimized by minimizing a total loss function, and tested by a test sample set.

4. A visible light to infrared image translation system, characterized in that: include: An acquisition module is used to preprocess the image to be translated to obtain the preprocessed image to be translated; A processing module, used for inputting the preprocessed image to be translated into a visible light to infrared image translation network to obtain a translation result and edge information of the image, wherein the visible light to infrared image translation network comprises an image translation module and an image edge information supervision module, wherein the image translation module comprises an attention submodule; The process of obtaining the translation result is specifically as follows: Extracting features of the preprocessed image to be translated by a coding submodule in a generator in the image translation module and obtaining a corresponding feature spectrum, wherein the coding submodule is composed of three downsampling blocks and six residual blocks; The attention submodule in the generator weights the feature spectrum finally output by the encoding submodule to obtain an attention spectrum; Decoding the attention spectrum through a decoding submodule in the generator to obtain the translation result; The specific process of obtaining edge information is as follows: After the first down-sampling block of the encoding submodule in the generator in the image translation module, an edge head is externally connected, edge information is predicted for the first feature spectrum obtained after processing the first down-sampling block through the edge head, and the predicted edge information is constrained through the edge loss function; The total loss function of the network is a joint function composed of generation adversarial loss, cycle loss, consistency loss, classification loss and edge loss. The adversarial loss makes the distribution of the generated image continuously fit the distribution of the corresponding output domain. The cycle consistency loss can improve the structural stability of the network conversion image. The consistency loss allows small errors between the generated image and the target image and improves the convergence speed. The classification loss can guide the network to distinguish the source domain image from the target domain image, where the source domain image refers to the visible light image and the target domain image refers to the infrared image. The edge loss uses the Dice loss to guide the edge information of the generated image to be consistent with the edge information of the target image. The total loss function of the network is: L=p1L gan +p2L cyc +p3L iden +p4L class +p5L edge (1) Among them, p1~p5 are the corresponding generated adversarial losses L gan , circulation loss L cyc , consistency loss L iden , classification loss L class And the edge information loss L edge The proportional coefficients are all hyperparameters. The optimization of the network includes both the conversion from the source domain to the target domain and the conversion from the target domain to the source domain. When the source domain is converted to the target domain, the corresponding loss terms of formula (1) are expressed as follows: Where x is the source domain X se Or target domain X tt , se→tt indicates the translation direction is from the source domain to the target domain, E is the expectation, and η in formula (5) and formula (6) se (x) and They are the generator G se→tt Attention module and discriminator D tt The final classification output of the attention module is Represents the predicted edge The i-th pixel of Indicates the corresponding edge information of the real image in the target domain The i-th pixel of , pixels represents the total number of pixels, σ is a Laplace smoothing term with a value of 1, and an optimal model will be obtained after network optimization. The optimal model is used as the final model for testing the test sample set.

5. A visible light to infrared image translation system according to claim 4, characterized in that: The acquisition module specifically includes: The image to be translated is rotated, scaled, cropped and normalized in sequence to obtain the pre-processed image to be translated.

6. A medium, characterized in that The medium stores instructions, and when a computer reads the instructions, the computer executes a visible light to infrared image translation method as claimed in any one of claims 1 to 3.

7. A device, characterized in that: The invention comprises the storage medium as claimed in claim 6 and a processor for executing instructions in the storage medium.

Images