Quality-adaptive image cross-domain mapping method

Abstract

The invention discloses a quality self-adaptive image cross-domain mapping method. The method comprises the following steps: 1, carrying out quality self-adaptive image cross-domain mapping; the method comprises the following steps of: 1, acquiring data of a target domain image s and a source domain image c, and preprocessing; 2, designing a generative adversarial network model, and inserting a quality adaptive convolution module into a GLD generator network by the adversarial network model on the basis of a network structure of an unsupervised image translation model; 3, inputting the target domain image s and the source domain image c preprocessed in the step 1 into the generative adversarial network model in the step 2 for training; and step 4, mapping from a source domain image to a target domain image by using the generative adversarial network model trained in the step 3. After an unsupervised image translation model is modified, an image translation result better than that of an original model is achieved.

Description

Technical Field

[0001] The invention belongs to the technical field of image translation, and in particular relates to a quality-adaptive image cross-domain mapping method. Background Art

[0002] The image translation task is to translate an image from the source domain to the target domain, preserving the intrinsic content features and changing the external style features. Due to its innovation and diversity in the field of image processing and computer vision, it has a wide range of application backgrounds in academia and real life.

[0003] No-reference image quality assessment is an objective quality assessment method that does not require any original image information, but only relies on the information of the image to be evaluated to perform quality assessment. Traditional no-reference quality assessment methods are based on manually extracted spatial, frequency and other feature information, while in recent years, the use of deep neural networks and visual-language contrastive learning has made great progress in no-reference image quality assessment.

[0004] In traditional convolutional networks, fixed convolution kernels are usually used for convolution operations, which limits the model's ability to extract features of different scales to a certain extent. The multi-scale attention mechanism uses convolution kernels of different sizes to extract features of different scales, which can better capture different information of the input data and thus improve the performance of the model.

[0005] In practical applications, advanced deep neural networks usually perform well when processing high-quality images, but their performance drops sharply when processing low-quality images. Usually, people tend to use data augmentation operations to alleviate this problem, but using too much perturbed data may seriously degrade the performance of the model on the original images. Therefore, it is crucial to design and implement a method that enables existing generative models to effectively process images of various qualities.

[0006] The patent application with application publication number CN116229229A and titled "Multi-domain image fusion method and system based on deep learning" discloses a layer-consistent dynamic convolution for implementing image translation tasks. The shortcoming of this method is that it ignores the problem that existing image translation tasks have poor performance on low-quality images, and does not consider the quality attributes of the input image itself during the generation process.

[0007] Although the existing image translation technology has made significant progress, it still faces some key challenges. One of the most important problems is that the performance of the model will drop sharply when faced with severely degraded low-quality images, such as blur, noise, etc. When faced with the above problems, common solutions are mostly data-driven methods, such as using data augmentation technology to increase data diversity or introducing larger-scale low-quality data sets, but the above methods are prone to defects such as limited model generalization ability and high data collection costs. Summary of the invention

[0008] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a quality-adaptive image cross-domain mapping method. The method is inspired by the visual space overlap hypothesis, and uses an adaptive method to integrate the image quality into the image translation process through a quality-adaptive dynamic convolution module to cope with the situation in real scenes where the original image quality that users want to change the style is uneven. After modifying the unsupervised image translation model (GLD), better image translation results are achieved than the original model.

[0009] In order to achieve the above object, the technical solution adopted by the present invention is:

[0010] A quality-adaptive image cross-domain mapping method comprises the following steps:

[0011] Step 1: Obtain the data of the target domain image s and the source domain image c and perform preprocessing;

[0012] Step 2: Design a generative adversarial network model. The generative adversarial network model is based on the network structure of the unsupervised image translation model (GLD) and inserts the quality adaptive convolution module into the generator G of the unsupervised image translation model. A middle;

[0013] Step 3: Use the target domain image s and source domain image c preprocessed in step 1 to input into the generative adversarial network model in step 2 for training;

[0014] Step 4: Use the generative adversarial network model trained in step 3 to map the source domain image to the target domain image.

[0015] The preprocessing of step 1 includes: modifying the image to be of the same size and performing data enhancement operation on the image;

[0016] The cross-domain mapping method refers to mapping the image from the source domain to the target domain. The domain is represented by uppercase letters, and the image in the domain is represented by lowercase letters: target domain-S, target domain image-s; source domain-C, source domain image-c.

[0017] The training data all come from existing public datasets, the source domain data comes from the COCO dataset, and the target domain data comes from the Anime Colorization dataset.

[0018] The quality adaptive convolution module includes a multi-scale multi-core convolution block and a quality adaptive routing network. The multi-scale multi-core convolution block captures receptive field information of different scales through a plurality of convolution layers with different convolution kernel sizes. The quality adaptive routing network obtains the weight parameters of the receptive field information of different scales through a quality evaluation network and a multi-layer perceptron, and dynamically selects different receptive field information.

[0019] The step 2 is specifically as follows:

[0020] Step (1): The generator G of the unsupervised image translation model A The Encoder-Decoder structure is adopted, and the quality adaptive dynamic convolution module is inserted between the first convolution layer and the second convolution layer of the Encoder. The receptive field information of different scales is obtained through the multi-scale multi-core convolution block, and the weights of the receptive field information of different scales are obtained through the quality adaptive routing network, thereby obtaining the weighted information of the receptive fields of different scales.

[0021] Step (2): Adopt the training strategy of the CycleGAN to optimize the network parameters and generate the result G A (c) GeomLoss and CLIPLoss are used to perform geometric consistency loss between the generated image and the source image c to ensure the content consistency between the generated result and the input image.

[0022] The multi-scale multi-core convolution block consists of n sub-modules, each of which contains m convolutional layers with convolution kernels of different sizes;

[0023] Each submodule is used to obtain the receptive field information of different scales of the input image features of this group; different submodules are used to process different grouping feature information, and each submodule is independent, that is, the parameters are independent;

[0024] Different convolutional layers in the same submodule are also independent of each other. The convolution kernels of different convolutional layers have different sizes, so the receptive field sizes of extracted features are also different. Each convolutional layer processes the feature information input to this group independently. Using multiple convolutional layers can obtain information under different receptive fields.

[0025] The quality adaptive routing module consists of a no-reference quality evaluation network and a multi-layer perceptron MLP network; the no-reference quality evaluation network is used to obtain the quality features of the current image, and the multi-layer perceptron MLP network is used to map the quality features into weight parameters of different receptive field information;

[0026] The weight parameters of the receptive field information of different scales obtained by the quality adaptive routing module are used to weight and combine the receptive field information of different scales obtained by the multi-scale multi-core convolution block, and finally the image feature information that adapts to the current image quality is obtained.

[0027] The multi-scale multi-core convolution block is used to obtain information of different receptive fields; the quality adaptive routing network is used to obtain the weight parameters of the above different receptive field information. Then the two are multiplied to obtain the final overall information.

[0028] The steps of the quality adaptive dynamic convolution module in step (1) are as follows:

[0029] s1: multi-scale multi-kernel convolution block:

[0030] A multi-scale multi-core convolution block is used to obtain a multi-scale receptive field. The multi-scale multi-core convolution block consists of n sub-modules, each of which contains m convolution layers with convolution kernels of different sizes. The steps to obtain a multi-scale receptive field are as follows:

[0031] First, G of GLD A The first layer of the Encoder convolutional network extracts the features of the source domain image c F∈R H×W×C Divide into n groups in the channel dimension;

[0032]

[0033] Then a multi-scale convolution block is designed, which consists of m different convolutional layers, and the convolution kernel size of each convolutional layer is K. j The setup looks like this:

[0034]

[0035] Feature grouping F of source domain images i After the convolution layer in the multi-scale convolution module, the feature information obtained in the current receptive field is Z i,j , the process is as follows, where i represents F i The i-th group in the, j represents K j The jth convolutional layer in ;

[0036]

[0037] s2: Quality Adaptive Routing Network:

[0038] The quality adaptive routing network consists of a no-reference quality evaluation network And a multi-layer perceptron MLP network;

[0039] The specific steps are as follows:

[0040] Step 1): Extract quality features of the source domain image c:

[0041] Input the source domain image c into the no-reference quality assessment network Extract the quality feature vector f of the source domain image Q :

[0042]

[0043] Step 2) Get weight information

[0044] The quality feature vector f of each image described in step 1) Q After a multi-layer perceptron MLP network, the corresponding weight w is obtained i,j , indicating that the jth convolutional layer in the multi-scale convolutional module corresponds to F i The weight factor of is as follows:

[0045] w i,j =Softmax(MLP(f Q )),i∈{1,...,n};

[0046] s3: Feature fusion:

[0047] The intermediate result Z after convolution obtained according to s2 i,j and the weight w obtained by s3 i,j , multiply the two together to obtain multiple continuous receptive field information H that matches the current image quality i i, represents the i-th group of feature information of the source domain image c. Finally, H i , after splicing, we get the complete feature information H of the final source domain image c;

[0048]

[0049] H = concat(H i ),i∈{1,...,n};

[0050] H is the final image feature information extracted from the source domain image after passing through the quality adaptive convolution module.

[0051] The generator in the generative adversarial network model described in step (2) includes a generator G A , generator G B ;

[0052] Generator G A , generator G B Both use the Encoder-Decoder structure containing the Res-Net structure, where the target domain generator G AIt is used to generate images from the source domain C to the target domain S, that is, input the source domain image c and obtain the representation G of c in the target domain A (c), source domain generator G B Generate an image from the target domain S to the source domain C, that is, input the target domain image s and obtain the representation G of s in the source domain B (s);

[0053] The discriminator in the generative adversarial network model described in step (2) includes the discriminator D A , Discriminator D B , discriminator D A , Discriminator D B Both use Patch-Based structure, where the target domain discriminator D A Used to identify whether the image belongs to the target domain S, the source domain discriminator D B Used to identify whether the image belongs to the source domain C;

[0054] Among them G A With D A Constitute a confrontational relationship, G B With D B constitute an adversarial relationship.

[0055] The calculation of GeomLoss in step (2) is as follows:

[0056] L geom =||G geom (I(G A (c)))-F(c)||

[0057] Among them G A (c) represents the generated result after the source domain image c is input into the target domain generator, I(G A(c) ) indicates that the result G will be generated A (c) Image features extracted by inputting into the Inception v3 network, G geom (I(G A (c))) represents the input of image features into the pre-trained depth map prediction network G geom The generated result depth map is obtained; F(c) represents the content depth map obtained by inputting the source domain image c into the depth map prediction network F;

[0058] The calculation of CLIPLoss is as follows:

[0059] L CLIP =||CLIP(G A (c))-CLIP(c)||

[0060] Among them, CLIP (G A(c)) and CLIP(c) respectively represent the use of the generated result G A (c) The generated result and the semantic information of the source domain image obtained by inputting the source domain image c into the CLIP model.

[0061] The training process of step 3 is as follows:

[0062] (1) Input the source domain image c into the target domain generator G A Get the target domain generation result G A (c), and then generate the result G A (c) Input source domain generator G B Get the source domain results

[0063] (2) G A (c) Input to the target discriminator D A Get the target domain identification result D A (G A (c)), Input to the source domain discriminator D B Get the source domain identification result

[0064] (3) Calculating the loss in step (2) and optimizing network parameters;

[0065] (4) Repeat the above steps (1)–(3) until the network achieves the expected effect and save the training weight file.

[0066] The step 4 is specifically as follows:

[0067] (1) Load the training weight file obtained in step 3 into the generator G A middle.

[0068] (2) Input the source domain image c to be mapped across domains into the target domain generator G A In the above example, we can obtain the mapping result of c in the target domain.

[0069] Beneficial effects of the present invention:

[0070] This paper proposes a quality-adaptive dynamic convolution module, which aims to overcome the problem that the performance of existing deep neural networks drops sharply when facing low-quality image inputs, while ensuring the generalization of the model and reducing data costs. Through multi-scale multi-core convolution blocks and quality-adaptive routing mechanisms, the relationship between the input image quality and the generated image quality is constructed to dynamically adjust the image generation process.

[0071] (1) The present invention proposes a quality-adaptive dynamic convolution module, which calculates the quality features of each input image and dynamically selects image features under different receptive fields, thereby significantly and robustly improving the recognition accuracy and generation quality of the image translation model for degraded images.

[0072] (2) The present invention not only improves the performance of the model for degraded images, but also enhances the robustness of the overall image generation model to uneven data distribution. The quality adaptive convolution module enables the model to adaptively obtain image information under different receptive fields based on the quality of the input content image, ensuring that the model can obtain effective image feature information when facing high-definition images and low-quality images. The model has good generation effects on both high-quality and low-quality images, ensuring the generalization of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 It is a schematic diagram of image quality feature extraction of the present invention.

[0074] Figure 2 Dynamic convolution module model framework for quality adaptation.

[0075] Figure 3 Schematic diagram of the location of the quality adaptive dynamic convolution module in the GLD model. DETAILED DESCRIPTION

[0076] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0077] like Figure 1 , Figure 2 As shown, a quality adaptive image cross-domain mapping method comprises the following steps:

[0078] Step 1: Obtain the data of the target domain image s and the source domain image c and perform preprocessing;

[0079] Step 2: Design a generative adversarial network model. The adversarial network model is based on the network structure of the unsupervised image translation model (GLD), and the quality adaptive convolution module is inserted into the GLD generator network;

[0080] Step 3: Use the target domain image s and source domain image c preprocessed in step 1 to input into the generative adversarial network model in step 2 for training;

[0081] Step 4: Use the generative adversarial network model trained in step 3 to map the source domain image to the target domain image.

[0082] The preprocessing of step 1 includes: modifying the image to have a consistent size and performing data enhancement operations on the image.

[0083] The operation steps of step 2 are:

[0084] Step (1): The quality adaptive dynamic convolution module is located in G A In the Encoder, the target generator G of GLD A The Encoder-Decoder structure containing the Res-Net structure is adopted. The Res-Net structure is located in G A The dashed box in the encoder is located at the GLD original G A Between the first and second convolution layers of the encoder, that is, after the first convolution layer added at the encoder to convert the image from 3 channels to 64 channels, the receptive field information of different scales is obtained through the multi-scale multi-core convolution block, and the weights of the receptive field information of different scales are obtained through the quality adaptive routing network, and then the weighted information of the receptive fields of different scales is obtained to ensure that effective and rich image feature information is obtained under the current image quality, which helps the model to adaptively adjust according to the image quality in the encoding stage and better represent the image information;

[0085] Step (2): Adopt the training strategy of the CycleGAN to optimize the network parameters and generate the result G A (c) Geometric consistency loss GeomLoss and semantic loss CLIPLoss are also performed between the generated result and the source domain image c to ensure the content consistency between the generated result and the input image.

[0086] The quality adaptive convolution module includes a multi-scale multi-core convolution block and a quality adaptive routing network. The multi-scale multi-core convolution block captures receptive field information of different scales through multiple convolution layers with different convolution kernel sizes. The quality adaptive routing network obtains weight parameters of receptive field information of different scales through a quality evaluation network and a multi-layer perceptron, and then dynamically selects different receptive field information.

[0087] The multi-scale multi-core convolution block consists of n sub-modules, each of which contains m convolutional layers with convolution kernels of different sizes;

[0088] The multi-scale multi-core convolution block is used to obtain receptive field information of different scales, and each submodule is used to obtain receptive field information of different scales of the input image features of this group.

[0089] The quality adaptive routing module consists of a no-reference quality evaluation network and a multi-layer perceptron MLP network;

[0090] The quality adaptive routing module is used to obtain weight parameters of receptive field information of different scales under the current quality, the reference-free quality evaluation network is used to obtain the quality features of the current image, and the multi-layer perceptron MLP network is used to map the quality features into weight parameters of different receptive field information.

[0091] The weight parameters of the receptive field information of different scales obtained by the quality adaptive routing module are used to weight and combine the receptive field information of different scales obtained by the multi-scale multi-core convolution block, and finally the image feature information that adapts to the current image quality is obtained.

[0092] The steps of the quality adaptive dynamic convolution module in step (1) are as follows:

[0093] s1: multi-scale multi-kernel convolution block:

[0094] A multi-scale multi-core convolution block is used to obtain a multi-scale receptive field. The multi-scale multi-core convolution block consists of n sub-modules, each of which contains m convolution layers with convolution kernels of different sizes. The steps to obtain a multi-scale receptive field are as follows:

[0095] First, G of GLD A The first layer of the Encoder convolutional network extracts the features of the source domain image c F∈R H×W×C Divide into n groups in the channel dimension;

[0096]

[0097] Then a multi-scale convolution block is designed, which consists of m different convolutional layers, and the convolution kernel size of each convolutional layer is K. j The setup looks like this:

[0098]

[0099] Feature grouping F of source domain images i After the convolution layer in the multi-scale convolution module, the feature information obtained in the current receptive field is Z i,j , the process is as follows, where i represents F i The i-th group in the, j represents K j The jth convolutional layer in .

[0100]

[0101] s2: Quality Adaptive Routing Network:

[0102] The quality adaptive routing network consists of a no-reference quality evaluation network And a multi-layer perceptron MLP network;

[0103] The specific steps are as follows:

[0104] Step 1): Extract quality features of the source domain image c:

[0105] Input the source domain image c into the no-reference quality assessment network Extract the quality feature vector f of the source domain imageQ :

[0106]

[0107] Step 2) Get weight information

[0108] The quality feature vector f of each image described in step 1) Q After a multi-layer perceptron MLP network, the corresponding weight w is obtained i,j , indicating that the jth convolutional layer in the multi-scale convolutional module corresponds to F i The weight factor of is as follows:

[0109] w i,j =Softmax(MLP(f Q )),i∈{1,...,n},j∈{1,...,m};

[0110] Through the above operations, the image fusion network obtains the weight information of image features under different receptive fields based on the quality characteristics of the image, and then obtains the weighted information under different receptive fields to meet the extraction of image features of different qualities;

[0111] s3: Feature fusion:

[0112] The intermediate result Z after convolution obtained according to s2 i,j and the weight w obtained by s3 i,j , multiply the two together to obtain multiple continuous receptive field information H that matches the current image quality i i represents the i-th group of feature information of the source domain image c. Finally, H i After splicing, the complete feature information H of the final source domain image c is obtained;

[0113]

[0114] H = concat(H i ),i∈{1,...,n};

[0115] H is the final image feature information extracted from the source domain image after passing through the quality adaptive convolution module.

[0116] In the above steps, through the quality adaptive convolution module, the image fusion network can obtain the weighted feature information of the input image under different receptive fields based on the quality of the input image, realize the dynamic adjustment of the acquired image information based on image quality adaptation, and ensure that rich and accurate image information can be extracted under different qualities.

[0117] The generator in the generative adversarial network model described in step (2) includes a generator G A , generator GB ;

[0118] Generator G A , generator G B Both use the Encoder-Decoder structure containing the Res-Net structure, where the target domain generator G A It is used to generate images from the source domain C to the target domain S, that is, input the source image c and obtain the representation G of c in the target domain A (c), source domain generator G B Generate an image from the target domain S to the source domain C, that is, input the target domain image s and obtain the representation G of s in the source domain B (s);

[0119] The discriminator in the generative adversarial network model described in step (2) includes the discriminator D A , Discriminator D B , discriminator D A , Discriminator D B Both use Patch-Based structure, where the target domain discriminator D A Used to identify whether the image belongs to the target domain S, the source domain discriminator D B Used to identify whether the image belongs to the source domain C;

[0120] Among them G A With D A Constitute a confrontational relationship, G B With D B constitute an adversarial relationship;

[0121] The calculation of GeomLoss in step (2) is as follows:

[0122] L geom =||G geom (I(G A (c)))-F(c)||

[0123] Among them G A (c) represents the generated result after the source domain image c is input into the target domain generator, I(G A(c) ) indicates that the result G will be generated A (c) Image features extracted by inputting into the Inception v3 network, G geom (I(G A (c))) represents the input of image features into the pre-trained depth map prediction network G geom The generated result depth map is obtained; F(c) represents the content depth map obtained by inputting the source domain image c into the depth map prediction network F.

[0124] The calculation of CLIPLoss is as follows:

[0125] L CLIP =||CLIP(G A (c))-CLIP(c)||

[0126] Among them, CLIP (G A (c)) and CLIP(c) respectively represent the use of the generated result C A (c) The generated result and the semantic information of the source domain image obtained by inputting the source domain image c into the CLIP model.

[0127] The generated image is closer to the source domain image in terms of geometric structure and semantic information, thereby ensuring the content consistency between the generated result and the input image.

[0128] The training process of step 3 is as follows:

[0129] (1) Input the source domain image c into the target domain generator G A Get the target domain generation result G A (c), and then generate the result G A (c) Input source domain generator G B Get the source domain generation results

[0130] (2) G A (c) Input to the target domain discriminator D A Get the target domain identification result D A (G A (c)), Input to the source domain discriminator D B Get the source domain identification result

[0131] (3) Calculating the loss in step (2) and optimizing network parameters;

[0132] (4) Repeat the above steps (1)–(3) until the network achieves the expected effect and save the training weight file.

[0133] The step 4 is specifically as follows:

[0134] (1) Load the training weight file obtained in step 3 into the generator G A middle.

[0135] (2) Input the source domain image c to be mapped across domains into the target domain generator G A In the above example, we can obtain the mapping result of c in the target domain.

[0136] Experimental setup

[0137] Dataset: In order to verify the effectiveness of the quality-adaptive dynamic convolution module, experiments were conducted on image translation tasks, and the designed datasets include: COCO dataset and Anime Colorization dataset. The COCO dataset is a large, rich object detection, segmentation and captioning dataset, with images including 91 categories of targets, 328,000 images and 2,500,000 labels; the Anime Colorization dataset is an anime image coloring dataset, which contains a wealth of color anime images and black and white line-drawn anime images.

[0138] Implementation details:

[0139] The model block diagram of the present invention is as follows Figure 2 :

[0140] The present invention conducts experiments on the unsupervised image translation model GLD. GLD consists of two generators and a discriminator, combined with CLIP, to achieve an effective bridge between image features and text semantics. By training the network to map from image features to depth images, the model is able to capture key geometric details and semantic levels in the scene while keeping the outline of the object clear.

[0141] Specific insertion position combination Figure 3 , located in G A The dashed box in the encoder, that is, after the first convolutional layer added to the encoder to convert the image from 3 channels to 64 channels, helps the model to adaptively adjust according to the image quality during the encoding stage and better represent the image information.

[0142] In the specific training, 10,000 images from COCO were randomly selected as source domain images, 14,224 images from Anime were randomly selected as target domain images, and 1,000 COCO images and 3,545 Anime images were selected as the test set. Adam was used for optimization with a learning rate of 2e-4, and a maximum of 40 epochs were trained with a batch size of 1. The number of channel groups n mentioned in step 2 above was set to 4, the number of convolutional layers in each group was 4, and a 3×3 dilated convolution kernel was used to achieve multi-scale receptive fields. The specific receptive field size was achieved by adjusting the dilation rate.

[0143] Experimental evaluation indicators

[0144] This experiment uses commonly used image generation evaluation indicators to measure the quality of generated images, including FID, SIFID and CPBD.

[0145] FID (Frechet Inception Distance) is an indicator for evaluating the quality of images generated by generative models (such as GAN). It measures the similarity between the generated images and the real images by comparing their distributions. The smaller the FID value, the higher the quality of the generated image and the closer it is to the distribution of the real image. The calculation formula of FID is as follows:

[0146]

[0147] Among them, μ r , μ g The feature mean of the real image and the generated image; Σ r ,Σ g is the feature covariance matrix of the real image and the generated image, Tr is the trace of the matrix, which represents the sum of the diagonal elements of the matrix; represents the geometric mean of the covariance matrix.

[0148] SIFID (Single Image Frechet Inception Distance) is a variant of FID, specifically designed for single image generation quality assessment, especially for assessing the similarity between a generated image and a single real reference image. It is often used in scenarios such as Image-to-Image Translation tasks, such as Style Transfer or super-resolution reconstruction.

[0149]

[0150] Among them, μ ref ,Σ ref The mean and covariance matrix of the features of a single reference image, μ gen ,Σ gen The mean and covariance matrix of the single generated image features, and the rest of the representations are the same as the above FID.

[0151] CPBD is a gradient-based no-reference clarity assessment method. Its core idea is to use the gradient changes in the image to quantify the clarity. Rapid changes in gradients are usually related to the clarity of the image edges. Therefore, the CPBD indicator reflects the clarity of the image by statistically calculating the cumulative probability of effective edge information. The value of CPBD is between [0,1][0,1][0,1]. The higher the value, the clearer the image, and the lower the value, the blurrier the image. The calculation formula of CPBD is as follows:

[0152]

[0153] Where N represents the number of all valid edge pixels in the image; ω irepresents the weight of the i-th edge pixel, which is usually related to the edge significance or gradient amplitude; G i represents the gradient magnitude of an edge pixel; P(G)P(G)P(G) is the probability density function of the gradient magnitude, which represents the distribution of the gradient.

[0154] Experimental Results

[0155] Visual Effects:

[0156] In the experiment of converting images to line draft style using the above GLD model, the first column HQ Content represents the original high-definition source domain image; the second column LQ Content represents the blurred source domain image after degradation; the third column HQ Result is the standard output obtained by inputting the high-definition source domain image into the model weights provided by the official, which is used as a benchmark for visual effect comparison; the fourth column Official indicates that after inputting the blurred source domain image into the official weight model, it can be seen that the output quality is greatly reduced and the lines are blurred; the fifth column Data Aug indicates that after using the commonly used data enhancement method, the model is retrained and it is found that the output effect is improved; the sixth column Ours indicates that by adding the present invention to the model position described above, the output result is more significantly improved and the details are excellent.

[0157] Quantitative indicators:

[0158] In order to verify the effectiveness of the quality-adaptive dynamic convolution module proposed in this method, this experiment conducted three settings on the baseline model GLD, namely the official baseline model, setting data augmentation operations on the baseline model, and setting the quality-adaptive dynamic convolution module proposed in this method on the baseline model, and compared the test results on the above experimental indicators FID, SIFID, and CPBD. The detailed test indicators for the above test set are shown in the table

[0159] Table 1 Performance evaluation of the quality-adaptive dynamic convolution module on the GLD model

[0160]

[0161]

[0162] The results show that the present invention can achieve effective performance improvement on various baseline models and can achieve competitive results on SIFID, CPBD, and FID indicators.

Claims

1. A quality-adaptive image cross-domain mapping method, characterized in that: The steps include: Step 1: Obtain the data of the target domain image s and the source domain image c and perform preprocessing; Step 2: Design a generative adversarial network model. The generative adversarial network model is based on the network structure of the unsupervised image translation model (GLD) and inserts the quality adaptive convolution module into the generator G of the unsupervised image translation model. A middle; Step 3: Use the target domain image s and source domain image c preprocessed in step 1 to input into the generative adversarial network model in step 2 for training; Step 4: Use the generative adversarial network model trained in step 3 to map the source domain image to the target domain image.

2. The quality-adaptive image cross-domain mapping method according to claim 1, characterized in that: The preprocessing of step 1 includes: modifying the image to be of the same size and performing data enhancement operations on the image; the data is obtained in such a way that the training data are all from existing public data sets, the source domain data are from the COCO data set, and the target domain data are from the AnimeColorization data set.

3. The quality-adaptive image cross-domain mapping method according to claim 1, characterized in that: The quality adaptive convolution module includes a multi-scale multi-core convolution block and a quality adaptive routing network. The multi-scale multi-core convolution block captures receptive field information of different scales through a plurality of convolution layers with different convolution kernel sizes. The quality adaptive routing network obtains the weight parameters of the receptive field information of different scales through a quality evaluation network and a multi-layer perceptron, and dynamically selects different receptive field information.

4. The quality-adaptive image cross-domain mapping method according to claim 3, characterized in that: The step 2 is specifically as follows: Step (1): The generator G of the unsupervised image translation model A The Encoder-Decoder structure is adopted, and the quality adaptive dynamic convolution module is inserted between the first convolution layer and the second convolution layer of the Encoder. The receptive field information of different scales is obtained through the multi-scale multi-core convolution block, and the weights of the receptive field information of different scales are obtained through the quality adaptive routing network, thereby obtaining the weighted information of the receptive fields of different scales. Step (2): Adopt the training strategy of the CycleGAN to optimize the network parameters and generate the result G A (c) GeomLoss and CLIPLoss are used to perform geometric consistency loss between the generated image and the source image c to ensure the content consistency between the generated result and the input image.

5. The quality-adaptive image cross-domain mapping method according to claim 4, characterized in that: The quality adaptive convolution module includes a multi-scale multi-core convolution block and a quality adaptive routing network, wherein the multi-scale multi-core convolution block consists of n sub-modules, each of which contains m convolution layers with convolution kernels of different sizes; Each submodule is used to obtain the receptive field information of different scales of the input image features of this group; The quality adaptive routing network consists of a no-reference quality evaluation network and a multi-layer perceptron MLP network; The no-reference quality assessment network is used to obtain the quality features of the current image, and the multi-layer perceptron MLP network is used to map the quality features into weight parameters of different receptive field information; The weight parameters of the receptive field information of different scales obtained by the quality adaptive routing module are used to weight and combine the receptive field information of different scales obtained by the multi-scale multi-core convolution block, and finally the image feature information that adapts to the current image quality is obtained.

6. The quality-adaptive image cross-domain mapping method according to claim 4, characterized in that: The steps of the quality adaptive dynamic convolution module in step (1) are as follows: s1: multi-scale multi-kernel convolution block: A multi-scale multi-core convolution block is used to obtain a multi-scale receptive field. The multi-scale multi-core convolution block consists of n sub-modules, each of which contains m convolution layers with convolution kernels of different sizes. The steps to obtain a multi-scale receptive field are as follows: First, G of GLD A The first layer of the Encoder convolutional network extracts the features of the source domain image c F∈R H×W×C Divide into n groups in the channel dimension; Then a multi-scale convolution block is designed, which consists of m different convolutional layers, and the convolution kernel size of each convolutional layer is K. j The setup looks like this: Feature grouping F of source domain images i After the convolution layer in the multi-scale convolution module, the feature information obtained in the current receptive field is Z i,j , the process is as follows, where i represents F i The i-th group in the, j represents K j The jth convolutional layer in ; s2: Quality Adaptive Routing Network: The quality adaptive routing network consists of a no-reference quality evaluation network And a multi-layer perceptron MLP network; The specific steps are as follows: Step 1): Extract quality features of the source domain image c: Input the source domain image c into the no-reference quality assessment network Extract the quality feature vector f of the source domain image Q : Step 2) Get weight information The quality feature vector f of each image described in step 1) Q After a multi-layer perceptron MLP network, the corresponding weight w is obtained i,j , indicating that the jth convolutional layer in the multi-scale convolutional module corresponds to F i The weight factor of is as follows: w i,j =Softmax(MLP(f Q )),i∈{1,...,n},j∈{1,...,m}; s3: Feature fusion: The intermediate result Z after convolution obtained according to s2 i,j and the weight w obtained by s3 i,j , multiply the two together to obtain multiple continuous receptive field information H that matches the current image quality i i, represents the i-th group of feature information of the source domain image c. Finally, H i , after splicing, we get the complete feature information H of the final source domain image c; H=concat(H i ),i∈{1,...,n}; H is the final image feature information extracted from the source domain image after passing through the quality adaptive convolution module.

7. The quality-adaptive image cross-domain mapping method according to claim 4, characterized in that: The generator in the generative adversarial network model described in step (2) includes a generator G A , generator G B ; Generator G A , generator G B Both use the Encoder-Decoder structure containing the Res-Net structure, in which the style generator G A It is used to generate images from the source domain C to the target domain S, that is, input the source domain image c and obtain the representation G of c in the target domain A (c), source domain generator G B Generate an image from the target domain S to the source domain C, that is, input the target domain image s and obtain the representation G of s in the source domain B (s); The discriminator in the generative adversarial network model described in step (2) includes the discriminator D A , Discriminator D B , discriminator D A , Discriminator D B Both use Patch-Based structure, where the target discriminator D A Used to identify whether the image belongs to the target domain S, the source domain discriminator D B Used to identify whether the image belongs to the source domain C; Among them G A With D A Constitute a confrontational relationship, G B With D B constitute an adversarial relationship.

8. The quality-adaptive image cross-domain mapping method according to claim 7, characterized in that: The calculation of GeomLoss in step (2) is as follows: L geom =‖G geom (I(G A (c)))-F(c)‖ Among them G A (c) represents the generated result after the source domain image c is input into the target generator, I(G A(c) ) indicates that the result G will be generated A (c) Image features extracted by inputting into the Inception v3 network, G geom (I(G A (c))) represents the input of image features into the pre-trained depth map prediction network G geom The generated result depth map is obtained; F(c) represents the content depth map obtained by inputting the source domain image c into the depth map prediction network F; The calculation of CLIPLoss is as follows: L CLIP =‖CLIP(G A (c))-CLIP(c)‖ Among them, CLIP (G A (c)) and CLIP(c) respectively represent the use of the generated result G A (c) The generated result and the semantic information of the source domain image obtained by inputting the source domain image c into the CLIP model.

9. The quality-adaptive image cross-domain mapping method according to claim 8, characterized in that: The training process of step 3 is as follows: (1) Input the source domain image c into the generator G A Get the target domain generation result G A (c), and then generate the result G A (c) Input generator G B Get the source domain results (2) G A (c) Input to the discriminator D A Get the target domain identification result D A (G A (c)), Input to the discriminator D B Get the source domain identification result (3) Calculating the loss in step (2) and optimizing network parameters; (4) Repeat the above steps (1)–(3) until the network achieves the expected effect and save the training weight file.

10. The quality-adaptive image cross-domain mapping method according to claim 9, characterized in that: The step 4 is specifically as follows: (1) Load the training weight file obtained in step 3 into the generator G A middle. (2) Input the source domain image c to be cross-domain mapped into the generator G A In the above example, we can obtain the mapping result of c in the target domain.

Images