Image conversion method for decomposing multi-vision task based on comparative learning

Abstract

The invention discloses an image conversion method for decomposing a multi-vision task based on comparative learning, and the method comprises the steps: inputting a to-be-processed image into an image conversion model, and obtaining a needed target image; wherein the image conversion model adopts a generator network F (.), and network parameters of the image conversion model are determined according to the following steps: step A, constructing a sample set containing a source image and a target image; step B, training the generator network F (.) based on a pre-training network and a feature selection layer; the thought of a triple loss function is used, the distance between positive and negative samples and a generated sample is restrained to optimize the generator network, after the triple loss function is obtained through calculation, the loss gradient is passed back through back propagation, parameters of the generator network are updated, gradient accumulation is conducted, and finally the weight of the generator network is adjusted through an optimization algorithm. According to the image conversion method, the perception quality can be improved, and the style characteristics of the image can be changed while the image content is kept in the image conversion process.

Description

Technical Field

[0001] The present invention belongs to the technical field of image conversion and image generation in computer vision, and relates to the contrastive learning direction in deep learning, and specifically to an image conversion method based on contrastive learning decomposition of multiple visual tasks. Background Art

[0002] With the development of deep learning, some recent studies have introduced perceptual learning to deal with the problem of image conversion, such as “ESRGAN: Enhanced super-resolution generative adversarial networks” by Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chengchang Lei, published in Proceedings of the European Conference on Computer Vision (ECCV) (2018). The main idea is to use the pre-trained classification network Extract high-dimensional features as generated images and the target image The representation of , and then measure the distance between the two as the loss function, the formula is: (1) Compared to distance metrics such as MAE or MSE used in earlier works, perceptual distance is measured in feature space rather than pixel space, which is considered more compact and more relevant to human perception. In addition, Roey Mechrez, Itamar Talmi, and Lihi Zelnik-Manor proposed the contextual loss in "The contextual loss for image transformation with nonaligned data", Proceedings of the European Conference on Computer Vision (ECCV) (2018): (2) in, It is a feature and The similarity between them is usually calculated using the normalized cosine distance, where N represents the total number of samples used for regularization in the contrastive learning process, the index i represents the operation on the i-th sample among the N samples, and j represents the j-th feature in all feature dimensions of a certain sample. compared to, is calculated in the feature context space. When the training image and It should be more robust when misaligned. In addition to the aforementioned articles, “Deep learning for single image super-resolution: A brief review” by Wenming Yang, Xuechen Zhang, Yapeng Tian, ​​Wei Wang, Jinghao Xue, and Qingmin Liao, published in IEEE Transactions on Multimedia (Vol. 21, No. 12, 2019), analyzes the impact of these methods in more detail.

[0003] Generally speaking, the features extracted from the pre-trained network can be viewed as the input image and The probability distribution on and . It can be proved that the use or minimize and The distance between them is similar to minimizing the Kullback-Leibler (KL) divergence between the two distributions. and , as shown in formula (3): (3) because and From the pre-trained network The features extracted from the classification network (usually trained on a large-scale classification dataset) can be used to and Assuming that the mapping from pixel space to semantic space is obtained by the pre-trained network The learned natural image manifold, where Represents the Kullback-Leibler divergence, which is used to measure the information loss between natural image manifolds. Images generated using these perceptual learning methods can be more realistic. However, it should be noted that these two loss functions usually play a relatively small role in the final loss function of the entire network training, even though they may be important for human perception. Summary of the invention

[0004] The purpose of the present invention is to provide an image conversion method based on contrastive learning decomposition of multiple visual tasks, which can improve the perceived quality and change the style characteristics of the image while maintaining the image content during the image conversion process.

[0005] In order to achieve the above object, the solution of the present invention is: An image conversion method based on contrastive learning decomposition of multiple visual tasks, in which the image to be processed is input into the image conversion model to obtain the desired target image; Among them, the image conversion model adopts a generator network F (·), determine its network parameters according to the following steps: Step A, constructing a sample set including a source image and a target image; wherein the source image and the target image have similar contents but different styles; Step B, generator network F (·) The following methods are used for training; Step B1: randomly obtain source images from the sample set as training samples X and the target image Y ; Step B2: source image X Conducting the Generator Network F (·) forward propagation to obtain the generated image; for the source image X Performing distortion processing to obtain a distorted image; Step B3: input the target image, distorted image and generated image into the pre-trained network respectively Ψ (·), get the perceptual features h n , h p , h a , represent negative samples, positive samples and generated samples respectively; Step B4, using feature selection layer Φ (·) The perceptual characteristics h n , h p , h a Further select specific features e n , e p , e a , respectively represent negative sample features, positive sample features, and generated sample features; Step B5, based on the characteristics e n , ep , e a , respectively calculating the characteristic distances between the generated sample and the positive sample and the negative sample; Step B6: Use the Adam optimizer to optimize the generator network F (·) and feature selection layer Φ (·) updating the parameters, determining the adjustment direction of the parameters according to the feature distance, and the adjustment goal is to minimize the distance between the generated sample and the positive sample, and maximize the distance between the generated sample and the negative sample; Step B7, repeat steps B1 to B6 until the set termination condition is reached to obtain the trained generator network F (·); Specifically, steps B5 to B6 use the idea of ​​a triplet loss function to optimize the generator network by constraining the distance between positive and negative samples and generated samples; First, the distances between the generated samples and the positive samples and negative samples are calculated respectively, and measured by Euclidean distance; The distance between the generated sample and the positive sample is calculated according to the following formula:

[0006] The distance between the generated sample and the negative sample is calculated according to the following formula:

[0007] Based on the distance, the triplet loss function is calculated: Ltriplet =max(0, d ( e a , e p )- d ( e a , e n )- α ) in, α is the boundary value; In the above formula, d ( e a , e p ) represents the distance between the generated sample and the positive sample. The optimization goal is to minimize the distance so that the characteristics of the generated sample are closer to the positive sample. d ( e a , e n) represents the distance between the generated sample and the negative sample. The optimization goal is to maximize the distance so that the features of the generated sample are clearly distinguishable from the negative sample. It is necessary to ensure that the distance between the generated sample and the positive sample is d ( e a , e p ) is smaller than the distance between the generated sample and the negative sample d ( e a , e n ) plus the boundary value α ,Right now d ( e a , e p )< d ( e a , e n )+ α ,if d ( e a , e p ) has been d ( e a , e n )+ α If the value is small, then the loss is 0, that is, the model has achieved the ideal distinction effect on this pair of samples; otherwise, the loss value is the difference between the two, that is, d ( e a , e p )-[ d ( e a , e n )+ α ], and through optimization, the model shortens the distance between the generated samples and the positive samples, and pushes the distance between the generated samples and the negative samples away; After calculating the triplet loss function, the loss gradient is propagated back through back propagation to update the parameters of the generator network; the gradient will be gradually accumulated based on the calculated triplet loss, and finally the weight of the generator network will be adjusted through the optimization algorithm, so that the characteristics of the generated samples are closer to the characteristics of the positive samples, and away from the characteristics of the negative samples.

[0008] The generator network above F(·) includes an input layer, a fully connected layer, four deconvolution layers, and an output layer, which are arranged in sequence from input to output. The fully connected layer expands the feature map of the input layer into a tensor with a shape of 4×4×1024. After the first deconvolution layer, its output size is 8×8×512. The second deconvolution layer adjusts the feature map to 16×16×256, and then the third deconvolution layer outputs a feature map of 32×32×128. The fourth deconvolution layer outputs a feature map of 64×64×64. Finally, the output layer outputs a feature map of 128×128×3, and the output value is normalized to the interval [-1,1] by the Tanh activation function.

[0009] In the above step A, an original image is obtained from the Art by Ai-neural style transfer dataset, and the original image is preprocessed and then put into a sample set.

[0010] In the above step B, in each training process, the training samples are processed in batches.

[0011] In the above step B2, the source image X Perform distortion processing, including, X Perform color distortion and / or blur distortion.

[0012] Before the above step B3, the target image, the distorted image and the generated image are randomly cropped and then input into the pre-trained network. Ψ (·).

[0013] In the above step B3, the pre-trained network Ψ (·) The discriminator is used. The processing process is as follows: the input image is set to 128×128×3. First, the first convolution layer with a 4×4 convolution kernel, a stride of 2, and a padding of 1 increases the number of channels to 64, and the output feature map size is 64×64×64; the second convolution layer uses a 4×4 convolution kernel, a stride of 2, and a padding of 1, which increases the number of channels to 128, and the output feature map size is 32×32×128; then, the number of channels in the third convolution layer is increased to 256, and the output size is 16×16×256; the number of channels in the fourth convolution layer is increased to 512, and the output is 8×8×512; the fifth convolution layer expands the number of channels to 1024, and the output feature map size is 4×4×1024; at the same time, Leaky ReLU is attached after each convolution layer. activation function; then, the discriminator uses a global average pooling layer to compress the 4×4×1024 feature map into a 1×1×1024 vector; finally, the vector is input to the fully connected layer, outputs a scalar value, and passes through the Sigmoid activation function to normalize the result to the [0,1] interval.

[0014] Image transformation aims to convert an image from one situation / scene to another, for example, from low resolution to high resolution, from low light to normal light, etc. Some recent deep learning-based methods have achieved remarkable results, such as “Perceptual losses for real-time style transfer and superresolution” by Justin Johnson, Alexandre Alahi, and Li Fei-Fei, European conference on computer vision (2016), and “Image super-resolution using deep convolutional networks” by Chao Dong, Chengchang Lei, Kaiming He, and Xiaoou Tang, TPAMI (IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)) (2015). These works have improved in converting image content, but the visual quality of the converted images is often not perfect, especially in terms of human perception. Generally speaking, the pursuit of realistic results by human visual perception is a core concern of image translation tasks. Previous perceptual learning methods have been empirically effective for such tasks, but they usually rely on pre-trained classification networks to provide features that are not necessarily optimal in terms of visual perception of image transformations.

[0015] Based on this, the proposed method can separate the perception-related dimensions from the representation, and activate the perception of relevant dimensions and suppress irrelevant dimensions by using triplet loss, which can further disentangle the original representation features to obtain better perceptual quality. Experiments on multi-task image conversion tasks have proved that the proposed framework is superior to existing methods using pre-trained networks and empirically designed losses in terms of human visual perception.

[0016] After adopting the above scheme, the present invention mainly includes the following three major contents: 1) Online contrastive learning module,The superiority of perceptual learning mainly comes from the pre-trained classification network it uses.

[0017] 2) A fine-tuned feature selection module that combines features from different channels by introducing a feature selection layer and learns to activate the features of each channel using the feature selection layer.

[0018] 3) Task-oriented separation module, which introduces task-oriented separation, so that each perceptual factor can be implicitly separated from the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a process flow diagram of the present invention; Among them, A represents the generated image, P represents the positive sample image, and N represents the negative sample image; Figure 2 Image comparison diagram obtained by combining context loss with L1 of different weights; Among them, the Low-resolution Raw Input on the left represents the low-resolution raw input; Figure 3 This is a comparison chart of the transformation results from winter to summer and from summer to winter; Figure 4 A comparison chart of conversion results in RAW domain image super-resolution; Among them, Bicubic: Bicubic interpolation (an interpolation method used for image scaling) ESRGAN: ESRGAN (Super-Resolution Generative Adversarial Network, a neural network model for improving image resolution) EDSR (L1): EDSR (L1 loss) (Enhanced Deep Super-Resolution Network, trained using L1 loss) EDSR (CX): EDSR (Contextual Loss) (Enhanced Deep Super-Resolution Network, trained using contextual loss) EDSR (CX + Color): EDSR (Context Loss + Color Distortion) (Enhanced Deep Super Resolution Network, trained using context loss and color distortion of the input image) Ours: This invention. DETAILED DESCRIPTION

[0020] The technical solutions and beneficial effects of the present invention will be described in detail below with reference to the accompanying drawings.

[0021] The present invention provides an image conversion method based on contrastive learning decomposition of multiple visual tasks. The core purpose is to convert an image of a given style into an image with a target style, for example, converting a winter image into a summer image, aiming to change its style characteristics while maintaining the image content.

[0022] The implementation of the present invention includes: inputting the image to be processed into the image conversion model to obtain the desired target image; wherein the image conversion model adopts a generator network F (·),include: Input layer: The generator network takes a random noise vector (usually a z vector with dimension 100) as input.

[0023] Fully connected layer: input vector z First, it passes through a fully connected layer and is expanded to a tensor of shape 4×4×1024.

[0024] Deconvolution layer 1: The 4×4×1024 tensor is fed into the first deconvolution layer, the number of channels is reduced to 512, and the output size is 8×8×512 (stride 2, padding 1). The LeakyReLU activation function is then applied.

[0025] Deconvolution layer 2: Through the second deconvolution layer, the feature map size is expanded to 16×16 and the number of channels is reduced to 256 (stride 2, padding 1). LeakyReLU activation is still used.

[0026] Deconvolution layer 3: Continue to use deconvolution with a stride of 2 and a padding of 1, expand the feature map size to 32×32, and reduce the number of channels to 128. Use LeakyReLU activation.

[0027] Deconvolution layer 4: expands the feature map to 64×64 with 64 channels (stride 2, padding 1), and applies LeakyReLU activation.

[0028] Output layer: The last layer of deconvolution expands the feature map to 128×128 (stride 1, padding 0), with 3 channels (corresponding to RGB channels). This layer uses the Tanh activation function to normalize the output value to the [-1,1] interval.

[0029] The image conversion model needs to be trained in advance to determine its network parameters. The training process is as follows: Step A, constructing a sample set; In this embodiment, the sample set uses the Art by Ai-Neural Style Transfer dataset, which is a Kaggle open source dataset that contains diverse images generated by the neural style transfer algorithm. When constructing the sample set, appropriate source images and target images are selected, and these images are preprocessed to adapt to the input requirements of the network model. In this embodiment, the dataset contains 10,200 images with an image size of 400x300 and a jpg format.

[0030] The source image and the target image have similar contents but different styles. For example, in this embodiment, the training goal is to convert a summer image into a winter image. The source image and the target image should show similar scenes or objects, but have obvious differences in seasonal characteristics.

[0031] In order to facilitate the processing of the image conversion model, the original image needs to be preprocessed to meet the input conditions of the image conversion model before being put into the sample set. The preprocessing content includes converting the image data from the original pixel value range to a range that the neural network can better process, adjusting the image size, changing the color space, and applying data enhancement techniques such as random rotation, horizontal flipping, color transformation, etc. to improve the generalization ability of the image conversion model; the preprocessing steps include: Step a, convert the image data from the original pixel value range (usually 0 to 255) to a range that is easier for the neural network to handle (such as 0 to 1 or -1 to 1); Step b, adjust the image size to meet the input requirements of the image conversion model and increase the diversity of the data by cropping; Step c, convert the RGB image to other color space HSV using image processing libraries (such as OpenCV) when dealing with tasks involving color changes; Step d, data augmentation increases the generalization ability of the image conversion model by simulating different shooting conditions; For example: Random rotation: Randomly rotate the image within a certain range (such as -10 degrees to 10 degrees).

[0032] Horizontal flip: Flip the image horizontally with a certain probability (such as 50%).

[0033] Color transformation: such as adjusting brightness, contrast or saturation, can be achieved using PIL or OpenCV libraries.

[0034] Step e: batch processing is performed both during preprocessing and when feeding into the neural network to improve training efficiency and enable the network to learn multiple samples simultaneously; Step f: feature normalization to maintain consistency in data distribution across all layers of the network and accelerate the training process.

[0035] Step B, generator network F (·) The following methods are used for training; 1.1 Random Sampling In each training, a source image is randomly selected from the sample set. X and the target image Y In this embodiment, each training is based on the size of N Based on the batch (mini-batch), they are represented as { x k}and{ y k}.

[0036] 1.2 Generator forward propagation Generator network on the source image F The forward propagation of (·) yields the generated image.

[0037] 1.3 Distortion Processing and Random Cropping The source image is distorted to obtain a distorted image. In this embodiment, the source image may be distorted by color distortion and blur distortion. The color distortion refers to different degrees of color distortion on the image, which may include changes in color saturation, brightness, contrast, etc. The blur distortion refers to different degrees of blurring effects on the image, which is usually used to simulate camera defocus or motion blur. The target image, distorted image, and generated image are randomly cropped to obtain the cropped images.

[0038] 1.4 Perceptual Feature Extraction Feed the cropped image into the pre-trained network Ψ (·), extract their perceptual features and get h n , h p , h a , represent negative samples, positive samples and generated samples respectively.

[0039] In this embodiment, the pre-trained network Ψ(·) The discriminator used in RealisticGAN can be used. Its structure includes 5 convolutional layers, pooling layers, and fully connected layers arranged in sequence from input to output. The processing process is as follows: the input image is set to 128×128×3. First, the first convolutional layer with a 4×4 convolution kernel, a stride of 2, and a padding of 1 increases the number of channels to 64, and the output feature map size is 64×64×64. Nonlinearity is introduced through the Leaky ReLU activation function. Next, the second convolution layer uses a 4×4 convolution kernel, a stride of 2, and a padding of 1, increasing the number of channels to 128, and the output feature map size is 32×32×128. Batch normalization and Leaky ReLU are also added. ReLU activation; subsequently, the settings of the third convolution layer remain similar, but the number of channels increases to 256, and the output size is 16×16×256; the number of channels in the fourth convolution layer is further increased to 512, and the output is 8×8×512; the fifth convolution layer expands the number of channels to 1024, and the output feature map size is 4×4×1024; at the same time, batch normalization and leaky ReLU activation functions are attached after each convolution layer; then, the discriminator uses a global average pooling layer to compress the 4×4×1024 feature map into a 1×1×1024 vector to achieve global aggregation of features; finally, the vector is input to the fully connected layer, outputting a scalar value, and through the Sigmoid activation function, the result is normalized to the [0,1] interval to represent the probability that the input image is real or generated.

[0040] Send the target image into the pre-trained network to obtain the corresponding perceptual features h n ; Send the distorted image into the pre-trained network to obtain the corresponding perceptual features h p ; Send the generated image into the pre-trained network to obtain the corresponding perceptual features h a .

[0041] 1.5 Feature Selection Through the feature selection layer Φ (·), the extracted perceptual features h n , h p , h a Further select specific features e n , e p , e a , respectively represent the negative sample features, positive sample features, and generated sample features.

[0042] In this embodiment, the feature selection layer Φ(·) It consists of two convolutional layers with a kernel size of 1×1 and a Leaky ReLU activation layer in between.

[0043] 1.6 Loss Calculation and Gradient Accumulation Calculate the loss value. The idea of ​​the triplet loss function is used here to optimize the generator network by constraining the distance between positive and negative samples and generated samples.

[0044] In this embodiment, the distances between the generated sample and the positive sample and the negative sample are first calculated and measured by Euclidean distance; The distance between the generated sample and the positive sample is calculated according to the following formula:

[0045] The distance between the generated sample and the negative sample is calculated according to the following formula:

[0046] Based on the distance, the triplet loss function is calculated: Ltriplet =max(0, d ( e a , e p )- d ( e a , e n )- α ) in, α is the boundary value.

[0047] The core idea of ​​the triplet loss function is to make the distance between the generated sample and the positive sample as small as possible, and the distance between the generated sample and the negative sample as large as possible. In the above formula, d ( e a , e p ) represents the distance between the generated sample and the positive sample. The optimization goal is to make the distance as small as possible so that the characteristics of the generated sample are closer to the positive sample. d ( e a , e n ) represents the distance between the generated sample and the negative sample. The optimization goal is to make this distance as large as possible so that the features of the generated sample can be clearly distinguished from the negative sample. It is necessary to ensure that the distance between the generated sample and the positive sample d ( e a , e p) is smaller than the distance between the generated sample and the negative sample d ( e a , e n ) plus the boundary value α ,Right now d ( e a , e p )< d ( e a , e n )+ α ,if d ( e a , e p ) has been d ( e a , e n )+ α If the value is small, then the loss is 0, that is, the model has achieved the ideal distinction effect on this pair of samples; otherwise, the loss value is the difference between the two, that is, d ( e a , e p )-[ d ( e a , e n )+ α ], and through optimization, the model shortens the distance between the generated samples and the positive samples, and pushes the distance between the generated samples and the negative samples.

[0048] After calculating the triplet loss function, the loss gradient is propagated back through back propagation to update the parameters of the generator network. The gradient is gradually accumulated based on the calculated triplet loss, and finally the weight of the generator network is adjusted through an optimization algorithm (such as Adam) so that the characteristics of the generated samples are closer to the characteristics of the positive samples and away from the characteristics of the negative samples.

[0049] 1.7 Update the generator and feature selection layer After completing the set gradient accumulation, use the Adam optimizer to optimize the generator network F (·) and feature selection layer Φ (·) Perform parameter update. In this embodiment, the adjustment direction of the parameters is determined according to the feature distance in the triplet loss. The adjustment goal is to minimize the distance between the generated sample and the positive sample and maximize the distance between the generated sample and the negative sample.

[0050] In this embodiment, training samples are extracted in batches, and then each pair of samples is trained separately, so as to ensure that the mapping relationship between each source image and the corresponding target image is maintained, rather than the usual mixed processing between batches. x k , y k ) samples and perform gradient accumulation or parameter update after a mini-batch is completed, so as to preserve the independence between sample pairs while ensuring randomness and data diversity.

[0051] 1.8 Repeat the training steps Repeat the above training steps until the set termination conditions are reached, such as reaching the target loss value or number of training times.

[0052] 1.9 Final model output Finally, output the trained generator network F (·), and discard the pre-trained network Ψ (·) and feature selection layer Φ (·) because they are only used to calculate the loss during training and do not participate in generating the final image.

[0053] The present invention summarizes the operation flow of the overall method in Table 1 as shown below.

[0054] Table 1 Decomposition perception learning algorithm framework proposed in this invention

[0055] We first conducted several experiments and their corresponding analysis, combining the context loss with the L1 loss with different weights, and observed that using the context loss directly can lead to unexpected artifacts in the generated images, such as color shift, moiré artifacts, blurring, etc. These results can be found in Figure 2 This embodiment uses L1 loss to measure the pixel-level difference between the generated image and the target image, ensuring that the generated image is similar to the target image in terms of overall structure or pixel distribution in the image conversion task. Increasing the weight of L1 loss will reduce artifacts, but will cause the generated image to be blurred. In addition, Figure 1 In , the present invention visualizes the reduced dimensionality representation of three images from two scenes and two seasons, which are mapped using a pre-trained network. It can be noticed that the distance between the two images from the same scene but different seasons is smaller than the distance between the two images from different scenes but the same season, which means that the pre-trained features are not suitable for the distances between different seasons before and after learning. In order to achieve seasonal migration, the implementation process of the present invention needs to increase the distance between images of different seasons and decrease the distance between images of the same season.

[0056] In other words, the present invention considers that the features pre-trained for the classification task contain information that is irrelevant to other image conversion tasks during the implementation of the analysis process. The perceptual loss designed based on these features, although it can capture more perceptual information than simple pixel loss, also introduces a lot of irrelevant information, which can mislead the training of the image conversion network. Therefore, in the present invention, task-oriented instance triple samples are used to fine-tune the pre-trained network so that the traditional pre-trained representation can be decomposed into a set of task-related dimensions and a set of task-irrelevant dimensions to better represent the relationship between the result and the target image. Specifically, the present invention proposes a feature selection layer while freezing the pre-training to retain the natural image statistics in the pre-training and reduce irrelevant features. In addition, the present invention constructs task-oriented triple samples during the fine-tuning process, which makes the feature selection layer more sensitive to task-related statistics.

[0057] In this embodiment, a pre-trained network can be used. Ψ Extract high-dimensional features to obtain loss function. The pre-trained network Ψ It is a deep convolutional network that consists of a series of convolutional layers, each of which aims to extract image features and gradually reduce the spatial size of the image. First, there is a convolutional layer that processes the input image. The convolution kernel size of this layer is 4×4 and the stride is 2, which means that it halves the height and width of the image while extracting important visual features. After the first layer, an activation function is used to introduce nonlinearity, which helps capture complex patterns in the image. This activation function is LeakyReLU, which has the characteristic of providing small gradients for negative inputs and avoiding activation values ​​being completely zero. The middle part of the network consists of multiple convolutional layers, which are designed to further halve the image size while doubling the number of channels. Each convolutional layer also uses a 4x4 kernel and a stride of 2, so that the spatial size of the image is reduced by half after each layer. After each convolutional layer, if the model settings allow, a batch normalization layer is added to standardize the feature map, which helps the training stability and convergence speed of the network. The final layer of the network is a convolutional layer that transforms the deep feature map into a single prediction output. This layer no longer reduces the spatial size, but directly outputs a value that represents the probability that the input image is a real image. At the end of the model, there is a Sigmoid activation function that compresses the output of the final layer to a probability value between 0 and 1. This value represents the model's judgment on whether the input image is real or generated.

[0058] Generator Network FIt starts with a set of random numbers that are used as seeds for generating images. These random numbers are transformed into an initial spatial feature map through a deconvolution layer (also called a transposed convolution layer). This layer has no bias, uses a 4x4 filter, stride 1, and no padding. It is followed by an activation layer, LeakyReLU, to introduce nonlinearity. If spectral normalization is not configured and the network is not specifically designed without batch normalization, there is a batch normalization layer after the activation. The size of the initial feature map is the size of the generator network hidden layer multiplied by a multiple of 4x4. These layers are repeated many times to gradually increase the spatial size of the feature map until it approaches the size of the final image. This is achieved by alternating deconvolution layers (or a combination of upsampling and convolution layers) with activation function layers. After each intermediate layer, if spectral normalization is not configured and the network is configured to use batch normalization, a batch normalization layer is added. As the network goes deeper, the depth of the feature map is halved, while the width and height increase. Finally, the network uses another deconvolution layer or a combination of upsampling and convolution layers to convert the final feature map to the size of a color image. The number of filters in this layer is the same as the number of color channels of the output image. The final activation layer is a Tanh function that clamps the output values ​​between -1 and 1.

[0059] In this embodiment, the loss function is calculated by back propagation algorithm to the feature selection layer Φ The gradients of the parameters and perform gradient descent or other optimization algorithms to update those parameters.

[0060] Experimental Section In this section, the present invention first conducts three different experiments to verify the performance of the introduced method and analyze the relationship between perceptual quality. In addition, the present invention explores the relationship between perceptual quality and different settings. The three experiments were conducted on seasonal image conversion, RAW low-light image illumination, and RAW image super-resolution, respectively. Before training the model of the present invention, the present invention applied data augmentation to the above dataset, including random flipping, random rotation, and random cropping. During training, the present invention uses the Adam optimizer to update the network using parameters (β1=0.9, β2=0.999). The entire training process uses a learning rate of 1e-4 and a mini-batch size of 1. In addition, all experiments were performed using the PyTorch framework of 8*2080Ti.

[0061] 1. Seasonal image conversion Seasonal image transfer is one of the most representative tasks in unpaired image translation. During the learning process, the transformation network is trained on a set of winter images and a set of summer images without corresponding relationships, while GAN-based networks are usually used to learn cyclic relationships, i.e., winter → summer → winter. In order to improve the quality of the generated images, perceptual learning is applied to maximize the preservation of the original image X and the transformed image X. The perceived similarity between .

[0062] In this embodiment, perceptual loss is used to improve the perceptual quality of the generated image. Perceptual loss is a loss function based on high-level features. Unlike the L1 or L2 loss at the pixel level, perceptual loss does not directly compare the pixel values ​​of the generated image and the target image, but compares their high-level features extracted from a pre-trained convolutional neural network (usually a VGG network). This can better capture the semantic information of the image and make the generated image more consistent with the perception of the human visual system.

[0063] This paper uses MUNIT (Multimodal Unsupervised) proposed by Huang Xin, Liu Mingyu and Belongie S in the Proceedings of the European conference on computer vision (ECCV) (2018) as a baseline for method comparison. More specifically, for online contrastive learning, the instance triplet samples between the source image and the generated image are constructed as ,in f c is a random cropping operation, and the triplet is used as<Anchor,Positive,Negative> , where Anchor refers to the generated sample, Positive refers to the positive sample, and Negative refers to the negative sample. With such a setting, online contrastive learning aims to maximize the distance in the feature space that is only related to the season and minimize the distance in the feature space that is not related to the season. In other words, the representation network is optimized to pay attention to more scenes that most affect human recognition perception in winter and summer. Since there are recognizable differences between winter images and summer images, the present invention only updates the perception network after every 100 iterations of the generator network, and does not use feature selection and task-oriented enhancement. The generated results are as follows Figure 3 As shown. Figure 3In the left part of the figure, the present invention shows the results of the transformation from winter to summer and from summer to winter. It can be clearly seen that applying perception can generate brighter results, but there is no obvious effect in the scene change. However, after combining perception with the online contrastive learning introduced by the present invention, the generated results look like there are more green plants and less white snow in the scene. The difference in results also exists in the figure showing the summer → winter results. Figure 3 Therefore, it can be concluded that the proposed online contrastive learning is beneficial for preserving the perceptual similarities between images even if they belong to different seasons.

[0064] 2. Low-light image enhancement based on RAW domain In the field of low-light enhancement, RAW domain images have received much attention in recent works. Compared with previously used RGB images, RAW images can provide more information with less information loss because they are generated directly from CMOS sensors. However, these images have significant differences from normal images acceptable to humans in terms of visualization, such as color space shift, optical noise, optical distortion, etc. Therefore, improving the perceptual quality of generated normal light images is of vital practical significance.

[0065] Here, the present invention adopts SMD proposed in “Image super-resolution using deep convolutional networks” by Dong Chao, Lei Chengchang, He Kaiming and Tang Xiaoou, published in TPAMI (IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)) (2015) as the baseline for method comparison, and compares with EnlightenGAN, which can be used in conjunction with Figure 4As shown in Figure 2, this is the most advanced method in low-light image enhancement. SMD uses the features extracted from different layers of the pre-trained VGG19 network to calculate the distance between the illuminated image and the target image in the extracted feature space. The final loss of the training is a combination of the VGG19 loss and the L1 loss. However, the results generated using this loss function often have an uneven lighting appearance. In order to improve the visual quality of the generated results, the present invention constructs task-oriented instance triple samples to fine-tune the pre-trained VGG19 network through online contrastive learning. In Table 2, the present invention gradually presents each component of the method. The evaluation indicators include pixel-based PSNR (peak signal-to-noise ratio), perceptually related MS-SSIM (visual structure similarity), and LPIPS (deep learning network feature similarity). For convenience, the feature selection layer is used to indicate whether the feature selection used as fine-tuning is used. It is obvious that the framework proposed by the present invention can not only achieve state-of-the-art performance in terms of perception, but also generate results with the best visual quality and more realistic details, even compared with hand-crafted texture losses.

[0066] Table 2 Quantitative comparison in low-light enhancement datasets

[0067] The English meanings are as follows: Methods: Backbone: backbone network Loss: Loss Traditional: Traditional method EnlightenGAN: A Generative Adversarial Network Model for Improving Low-Light Image Quality SMD: A low-light enhancement model based on deep learning Adversarial: Adversarial Loss VGG: Pre-trained VGG network for extracting perceptual features VGG+Texture Loss: VGG + texture loss, using VGG perceptual loss plus texture loss FS: Feature Selection Layer Task: Blur: Blur (refers to a specific blur process) 3. Image super-resolution based on RAW domain In “Perceptual losses for real-time style transfer andsuperresolution” by Justin Johnson, Alexandre Alahi, and Li Fei-Fei, European conference on computer vision (2016), it was demonstrated that there are significant differences between real-world super-resolution problems and simulated super-resolution, especially in the way low-resolution images are degraded. However, even though training on their proposed super-resolution dataset can improve results, it also raises two issues that cannot be ignored, especially when applied to perceptual learning. The first issue is the misalignment between low-resolution images and the corresponding high-resolution images during the acquisition process. The second issue is the difference in color space between RAW images and RGB images. Both will severely affect the final result and make pixel-based loss functions and context-dependent loss functions not work well. In Figure 2 In this paper, the current trade-off solution is presented, which adjusts the weight between L1 loss and context loss to obtain a balance between color and clarity.

[0068] Table 3 Quantitative performance comparison between different super-resolution methods and the method proposed in this invention

[0069] The English meanings are as follows: RGB: Red, Green, Blue color space, refers to ordinary color images RGB + Bi-cubic: RGB + bicubic interpolation, where bicubic interpolation is a commonly used image interpolation method for image enlargement or reduction RGB + ESRGAN: Use the ESRGAN model to process RGB images, where ESRGAN is a super-resolution generative adversarial network EDSR: Enhanced Deep Super-Resolution Network, a convolutional neural network for image super-resolution Contextual: Contextual Loss (used to capture contextual information of the image) Contextual+Color: Contextual loss + color distortion (Color Distortion) Color: Color enhancement task To solve the long-standing weight adjustment problem, the present invention applies online contrastive learning to the representation network used in context. It should be noted that even though the context loss calculates the distance in the feature context space instead of the Euclidean space, the online contrastive learning proposed in the present invention is robust enough because the triplet loss is relative to<Anchor, Positive> and<Anchor, Negative> The difference between the samples is not the difference between the samples. In Table 3, the quantitative performance comparison between the baseline method and the proposed method is shown. Here, random color jittering is applied to high-resolution images as negative samples, which makes the results with context more colorful. The qualitative visual results are also shown in Figure 4 As shown in Figure 2, it is easy to conclude that the proposed method can achieve the best trade-off between clarity and color, even without using carefully designed weights. Note that due to the obvious misalignment of the images used for verification, pixel-based indicators such as PSNR have limited reference value.

[0070] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The solutions in the embodiments of the present invention may be implemented in various computer languages, for example, object-oriented programming language Java and interpreted scripting language JavaScript, etc.

[0071] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0072] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0073] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0074] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0075] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. An image translation method based on contrastive learning to decompose multiple visual tasks, Features: Input the image to be processed into the image conversion model to obtain the desired target image; Among them, the image conversion model adopts a generator network F (·), determine its network parameters according to the following steps: Step A, constructing a sample set including a source image and a target image; wherein the source image and the target image have similar contents but different styles; Step B, generator network F (·) The following methods are used for training; Step B1: randomly obtain source images from the sample set as training samples X and the target image Y ; Step B2: source image X Conducting the Generator Network F (·) forward propagation to obtain the generated image; for the source image X Performing distortion processing to obtain a distorted image; Step B3: input the target image, distorted image and generated image into the pre-trained network respectively Ψ (·), get the perceptual features h n , h p , h a , represent negative samples, positive samples and generated samples respectively; Step B4, using feature selection layer Φ (·) The perceptual characteristics h n , h p , h a Further select specific features e n , e p , e a , respectively represent negative sample features, positive sample features, and generated sample features; Step B5, based on the characteristics e n , e p , e a , respectively calculating the characteristic distances between the generated sample and the positive sample and the negative sample; Step B6: Use the Adam optimizer to optimize the generator network F (·) and feature selection layer Φ (·) updating the parameters, determining the adjustment direction of the parameters according to the feature distance, and the adjustment goal is to minimize the distance between the generated sample and the positive sample, and maximize the distance between the generated sample and the negative sample; Step B7, repeat steps B1 to B6 until the set termination condition is reached to obtain the trained generator network F (·); Specifically, steps B5 to B6 use the idea of ​​a triplet loss function to optimize the generator network by constraining the distance between positive and negative samples and generated samples; First, the distances between the generated samples and the positive samples and negative samples are calculated respectively, and measured by Euclidean distance; The distance between the generated sample and the positive sample is calculated according to the following formula: The distance between the generated sample and the negative sample is calculated according to the following formula: Based on the distance, the triplet loss function is calculated: Ltriplet =max(0, d ( e a , e p )- d ( e a , e n )- α ) in, α is the boundary value; In the above formula, d ( e a , e p ) represents the distance between the generated sample and the positive sample. The optimization goal is to minimize the distance so that the characteristics of the generated sample are closer to the positive sample. d ( e a , e n ) represents the distance between the generated sample and the negative sample. The optimization goal is to maximize the distance so that the features of the generated sample are clearly distinguishable from the negative sample. It is necessary to ensure that the distance between the generated sample and the positive sample is d ( e a , e p ) is smaller than the distance between the generated sample and the negative sample d ( e a , e n ) plus the boundary value α ,Right now d ( e a , e p )< d ( e a , e n )+ α ,if d ( e a , e p ) has been d ( e a , e n )+ α If the value is small, then the loss is 0, that is, the model has achieved the ideal distinction effect on this pair of samples; otherwise, the loss value is the difference between the two, that is, d ( e a , e p )-[ d ( e a , e n )+ α ], and through optimization, the model shortens the distance between the generated samples and the positive samples, and pushes the distance between the generated samples and the negative samples away; After calculating the triplet loss function, the loss gradient is propagated back through back propagation to update the parameters of the generator network; the gradient will be gradually accumulated based on the calculated triplet loss, and finally the weight of the generator network will be adjusted through the optimization algorithm, so that the characteristics of the generated samples are closer to the characteristics of the positive samples, and away from the characteristics of the negative samples.

2. The image conversion method based on contrastive learning decomposition of multiple visual tasks as claimed in claim 1, Features: The generator network F (·) includes an input layer, a fully connected layer, four deconvolution layers, and an output layer, which are arranged in sequence from input to output. The fully connected layer expands the feature map of the input layer into a tensor with a shape of 4×4×1024. After the first deconvolution layer, its output size is 8×8×512. The second deconvolution layer adjusts the feature map to 16×16×256, and then the third deconvolution layer outputs a feature map of 32×32×128. The fourth deconvolution layer outputs a feature map of 64×64×64. Finally, the output layer outputs a feature map of 128×128×3, and the output value is normalized to the interval [-1,1] by the Tanh activation function.

3. The image conversion method based on contrastive learning decomposition of multiple visual tasks as claimed in claim 1, Features: In the step A, an original image is obtained from the Art by Ai-neural style transfer dataset, and the original image is preprocessed and then put into a sample set.

4. The image conversion method based on contrastive learning decomposition of multiple visual tasks as claimed in claim 1, Features: In step B, in each training process, the training samples are processed in batches.

5. The image conversion method based on contrastive learning decomposition of multiple visual tasks as claimed in claim 1, Features: In step B2, the source image X Distortion processing, Including, for the source image X Perform color distortion and / or blur distortion.

6. The image conversion method based on contrastive learning decomposition of multiple visual tasks as claimed in claim 1, Features: Before step B3, the target image, the distorted image and the generated image are randomly cropped and then input into the pre-trained network. Ψ (·).

7. The image conversion method based on contrastive learning decomposition of multiple visual tasks as claimed in claim 1, Features: In step B3, the pre-trained network Ψ (·) The discriminator is used. The processing process is as follows: the input image is set to 128×128×3. First, the first convolution layer with a 4×4 convolution kernel, a stride of 2, and a padding of 1 increases the number of channels to 64, and the output feature map size is 64×64×64; the second convolution layer uses a 4×4 convolution kernel, a stride of 2, and a padding of 1, which increases the number of channels to 128, and the output feature map size is 32×32×128; then, the number of channels in the third convolution layer is increased to 256, and the output size is 16×16×256; the number of channels in the fourth convolution layer is increased to 512, and the output is 8×8×512; the fifth convolution layer expands the number of channels to 1024, and the output feature map size is 4×4×1024; at the same time, Leaky ReLU is attached after each convolution layer. activation function; then, the discriminator uses a global average pooling layer to compress the 4×4×1024 feature map into a 1×1×1024 vector; finally, the vector is input to the fully connected layer, outputs a scalar value, and passes through the Sigmoid activation function to normalize the result to the [0,1] interval.

Images