A quality-adaptive image cross-domain mapping method

Abstract

The present invention discloses a quality-adaptive cross-domain image mapping method, comprising the following steps: Step 1: obtaining and preprocessing data of a target domain image s and a source domain image c; Step 2: designing a generative adversarial network model, which is based on the network structure of an unsupervised image translation model and incorporates a quality-adaptive convolutional module into the GLD generator network; Step 3: using the target domain image s and source domain image c preprocessed in Step 1 as inputs to the generative adversarial network model trained in Step 2; Step 4: using the generative adversarial network model trained in Step 3 to map the source domain image to the target domain image. The present invention achieves better image translation results than the original model by modifying the unsupervised image translation model.

Description

Technical Field

[0001] The present invention belongs to the technical field of image translation, and in particular relates to a quality-adaptive image cross-domain mapping method. Background Art

[0002] Image translation involves translating an image from a source domain to a target domain, preserving the intrinsic content while modifying the extrinsic style. Due to its innovative and diverse applications in image processing and computer vision, it has broad applications in both academia and real-world applications.

[0003] No-reference image quality assessment is an objective quality assessment method that relies solely on the image itself, without requiring any original image information. Traditional no-reference quality assessment methods rely on manually extracted spatial and frequency features. However, in recent years, the use of deep neural networks and visual-language contrastive learning has enabled significant advancements in no-reference image quality assessment.

[0004] In traditional convolutional networks, fixed convolution kernels are typically used for convolution operations, which to some extent limits the model's ability to extract features of different scales. The multi-scale attention mechanism, on the other hand, uses convolution kernels of different sizes to extract features of different scales, which can better capture the different information of the input data, thereby improving model performance.

[0005] In practical applications, advanced deep neural networks typically perform well on high-quality images, but their performance degrades dramatically when processing low-quality images. Data augmentation is often used to alleviate this problem, but using too much perturbed data can severely degrade the model's performance on original images. Therefore, it is crucial to design and implement a method that enables existing generative models to effectively handle images of varying quality.

[0006] The patent application with application publication number CN116229229A, titled "Multi-domain image fusion method and system based on deep learning", discloses a layer-consistent dynamic convolution for image translation tasks. The shortcoming of this method is that it ignores the problem that existing image translation tasks perform poorly on low-quality images and does not consider the quality attributes of the input image itself during the generation process.

[0007] While existing image translation technology has made significant progress, it still faces several key challenges. One of the most significant issues is that model performance can plummet when faced with severely degraded, low-quality images, such as blur and noise. Common solutions to this problem are data-driven approaches, such as using data augmentation techniques to increase data diversity or introducing larger, low-quality datasets. However, these approaches are prone to limitations, such as limited model generalization and high data acquisition costs. Summary of the Invention

[0008] In order to overcome the shortcomings of the above-mentioned existing technologies, the purpose of the present invention is to provide a quality-adaptive image cross-domain mapping method. This method is inspired by the visual space overlap hypothesis. Through a quality-adaptive dynamic convolution module, an adaptive method is used to integrate image quality into the image translation process. This is to cope with the situation in real scenes where the quality of the original images that users want to style transform varies. After modifying the unsupervised image translation model (GLD), better image translation results are achieved than the original model.

[0009] In order to achieve the above object, the technical solution adopted by the present invention is:

[0010] A quality-adaptive image cross-domain mapping method comprises the following steps:

[0011] Step 1: Obtain the data of the target domain image s and the source domain image c and perform preprocessing;

[0012] Step 2: Design a generative adversarial network model. The generative adversarial network model is based on the network structure of the unsupervised image translation model (GLD) and inserts the quality adaptive convolution module into the generator G of the unsupervised image translation model. A middle;

[0013] Step 3: Use the target domain image s and source domain image c preprocessed in step 1 to input into the generative adversarial network model in step 2 for training;

[0014] Step 4: Use the generative adversarial network model trained in step 3 to map the source domain image to the target domain image.

[0015] The preprocessing of step 1 includes: modifying the image to be of the same size and performing data enhancement operations on the image;

[0016] The cross-domain mapping method refers to mapping an image from a source domain to a target domain. The domain is represented by an uppercase letter, and the image in the domain is represented by a lowercase letter: target domain-S, target domain image-s; source domain-C, source domain image-c.

[0017] The training data are all from existing public datasets, the source domain data is from the COCO dataset, and the target domain data is from the Anime Colorization dataset.

[0018] The quality adaptive convolution module includes a multi-scale multi-core convolution block and a quality adaptive routing network. The multi-scale multi-core convolution block captures receptive field information of different scales through multiple convolution layers with different convolution kernel sizes. The quality adaptive routing network obtains the weight parameters of the receptive field information of different scales through the quality evaluation network and the multi-layer perceptron, and dynamically selects different receptive field information.

[0019] The step 2 is specifically as follows:

[0020] Step (1): Generator G of the unsupervised image translation model A The encoder-decoder structure is adopted. The quality-adaptive dynamic convolution module is inserted between the first and second convolution layers of the encoder. The receptive field information of different scales is obtained through the multi-scale multi-core convolution block. The weights of the receptive field information of different scales are obtained through the quality-adaptive routing network, and then the weighted information of the receptive fields of different scales is obtained.

[0021] Step (2): Adopt the training strategy of CycleGAN to optimize the network parameters and generate the result G A (c) Perform geometric consistency loss GeomLoss and semantic loss CLIPLoss between the source domain image c to ensure the content consistency between the generated result and the input image.

[0022] The multi-scale multi-core convolution block consists of n sub-modules, each of which contains m convolution layers with convolution kernels of different sizes;

[0023] Each submodule is used to obtain receptive field information of different scales of the input image features of this group; different submodules are used to process different group feature information, and each submodule is independent, that is, the parameters are independent;

[0024] Different convolutional layers within the same submodule are independent of each other. The convolution kernels of different convolutional layers vary in size, resulting in different receptive fields for extracted features. Each convolutional layer independently processes the input feature information. Using multiple convolutional layers allows us to capture information with different receptive fields.

[0025] The quality adaptive routing module consists of a no-reference quality assessment network and a multi-layer perceptron (MLP) network. The no-reference quality assessment network is used to obtain the quality features of the current image, and the multi-layer perceptron (MLP) network is used to map the quality features into weight parameters of different receptive field information.

[0026] The weight parameters of the receptive field information of different scales obtained by the quality adaptive routing module are used to weight and combine the receptive field information of different scales obtained by the multi-scale multi-kernel convolution block, and finally obtain image feature information that adapts to the current image quality.

[0027] The multi-scale multi-kernel convolution block is used to obtain information from different receptive fields, while the quality adaptive routing network is used to obtain the weight parameters of the above different receptive field information. The two are then multiplied to obtain the final overall information.

[0028] The steps of the quality adaptive dynamic convolution module in step (1) are as follows:

[0029] s1: multi-scale multi-kernel convolution block:

[0030] A multi-scale multi-core convolution block is used to obtain a multi-scale receptive field. The multi-scale multi-core convolution block consists of n submodules, each of which contains m convolution layers with convolution kernels of different sizes. The steps to obtain a multi-scale receptive field are as follows:

[0031] First, the G of GLD A The first convolutional network layer of the Encoder extracts the features F∈R of the source domain image c H×W×C Divide into n groups in the channel dimension;

[0032]

[0033] Then a multi-scale convolution block is designed, which consists of m different convolutional layers, and the convolution kernel size of each convolutional layer is K. j The setup looks like this:

[0034]

[0035] Feature grouping F of source domain images i After the convolution layer in the multi-scale convolution module, the feature information obtained in the current receptive field is Z i,j , the process is as follows, where i represents F i The i-th group in, j represents K j The j-th convolutional layer in ;

[0036]

[0037] s2: Quality Adaptive Routing Network:

[0038] The quality adaptive routing network consists of a no-reference quality evaluation network And a multi-layer perceptron MLP network;

[0039] The specific steps are as follows:

[0040] Step 1): Extract quality features of the source domain image c:

[0041] Input the source domain image c into the no-reference quality assessment network Extract the quality feature vector f of the source domain image Q :

[0042]

[0043] Step 2) Get weight information

[0044] The quality feature vector f of each image described in step 1) Q After a multi-layer perceptron MLP network, the corresponding weight w is obtained i,j , indicating that the j-th convolution layer in the multi-scale convolution module corresponds to F i The weight factor of , the corresponding expression is as follows:

[0045] w i,j =Softmax(MLP(f Q )),i∈{1,...,n};

[0046] s3: Feature fusion:

[0047] The intermediate result Z after convolution obtained according to s2 i,j and the weight w obtained by s3 i,j , multiply the two together to obtain multiple continuous receptive field information H that matches the current image quality i i, represents the i-th group of feature information of the source domain image c. Finally, H i , after splicing, we get the complete feature information H of the final source domain image c;

[0048]

[0049] H=concat(H i ),i∈{1,...,n};

[0050] H is the final image feature information extracted after the source domain image passes through the quality adaptive convolution module.

[0051] The generator in the generative adversarial network model described in step (2) includes the generator G A , generator G B ;

[0052] Generator G A , generator G B Both use the Encoder-Decoder structure containing the Res-Net structure, where the target domain generator G AIt is used to generate an image from the source domain C to the target domain S, that is, input the source domain image c and obtain the representation G of c in the target domain A (c), Source domain generator G B Generate the image from the target domain S to the source domain C, that is, input the target domain image s and obtain the representation G of s in the source domain B (s);

[0053] The discriminator in the generative adversarial network model described in step (2) includes the discriminator D A , Discriminator D B , discriminator D A , Discriminator D B Both adopt Patch-Based structure, where the target domain discriminator D A Used to identify whether the image belongs to the target domain S, the source domain discriminator D B Used to identify whether the image belongs to the source domain C;

[0054] Among them G A With D A Constitute an adversarial relationship, G B With D B constitute an adversarial relationship.

[0055] The calculation of GeomLoss in step (2) is as follows:

[0056] L geom =||G geom (I(G A (c)))-F(c)||

[0057] Among them G A (c) represents the generated result after the source domain image c is input into the target domain generator, I(G A(c) ) indicates that the result G will be generated A (c) Image features extracted from the input Inception v3 network, G geom (I(G A (c))) represents the input of image features into the pre-trained depth map prediction network G geom The generated depth map is obtained; F(c) represents the content depth map obtained by inputting the source domain image c into the depth map prediction network F;

[0058] The calculation of CLIPLoss is as follows:

[0059] L CLIP =||CLIP(G A (c))-CLIP(c)||

[0060] Among them, CLIP(G A(c)) and CLIP(c) respectively represent the use of the generated result G A (c) The generated result and semantic information of the source domain image obtained by inputting the source domain image c into the CLIP model.

[0061] The training process of step 3 is as follows:

[0062] (1) Input the source domain image c into the target domain generator G A Get the target domain generation result G A (c), and then generate the result G A (c) Input source domain generator G B Get the source domain results

[0063] (2) G A (c) Input to target discriminator D A Get the target domain identification result D A (G A (c)), Input to the source domain discriminator D B Get the source domain identification result

[0064] (3) Calculating the loss in step (2) and optimizing network parameters;

[0065] (4) Repeat the above steps (1)–(3) until the network achieves the expected effect and save the training weight file.

[0066] The step 4 is specifically as follows:

[0067] (1) Load the training weight file obtained in step 3 into the generator G A middle.

[0068] (2) Input the source domain image c to be cross-domain mapped into the target domain generator G A In the example, we can obtain the mapping result of c in the target domain.

[0069] Beneficial effects of the present invention:

[0070] This paper proposes a quality-adaptive dynamic convolution module, aiming to overcome the sharp performance degradation of existing deep neural networks when faced with low-quality image inputs, while ensuring model generalization and reducing data costs. By using multi-scale multi-kernel convolution blocks and a quality-adaptive routing mechanism, the relationship between input image quality and generated image quality is established to dynamically adjust the image generation process.

[0071] (1) This paper proposes a quality-adaptive dynamic convolution module, which calculates the quality features of each input image and dynamically selects image features under different receptive fields, significantly and robustly improving the recognition accuracy and generation quality of the image translation model for degraded images.

[0072] (2) The present invention not only improves the performance of the model for degraded images, but also enhances the robustness of the overall image generation model to uneven data distribution. The quality-adaptive convolution module enables the model to adaptively obtain image information under different receptive fields based on the quality of the input content image, ensuring that the model can obtain effective image feature information when facing high-definition images and low-quality images. The model has good generation effects on both high-quality and low-quality images, ensuring the generalization of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 Schematic diagram of image quality feature extraction according to the present invention.

[0074] Figure 2 A dynamic convolution module model framework for quality adaptation.

[0075] Figure 3 Schematic diagram of the location of the quality-adaptive dynamic convolution module in the GLD model. DETAILED DESCRIPTION

[0076] The present invention will be described in further detail below with reference to the accompanying drawings.

[0077] like Figure 1 、 Figure 2 As shown, a quality-adaptive image cross-domain mapping method includes the following steps:

[0078] Step 1: Obtain the data of the target domain image s and the source domain image c and perform preprocessing;

[0079] Step 2: Design a generative adversarial network model. The adversarial network model is based on the network structure of the unsupervised image translation model (GLD), and the quality adaptive convolution module is inserted into the GLD generator network.

[0080] Step 3: Use the target domain image s and source domain image c preprocessed in step 1 to input into the generative adversarial network model in step 2 for training;

[0081] Step 4: Use the generative adversarial network model trained in step 3 to map the source domain image to the target domain image.

[0082] The preprocessing of step 1 includes: modifying the image to have a consistent size and performing data enhancement operations on the image.

[0083] The operation steps of step 2 are:

[0084] Step (1): The quality adaptive dynamic convolution module is located in G A In the Encoder, the target generator G of GLD A The Encoder-Decoder structure including the Res-Net structure is adopted. The Res-Net structure is located in G A The dotted box in the Encoder is located at the GLD original G A Between the first and second convolution layers of the encoder, that is, after the first convolution layer added to the encoder to convert the image from 3 channels to 64 channels, the multi-scale multi-core convolution block obtains the receptive field information of different scales, and the quality adaptive routing network obtains the weights of the receptive field information of different scales, and then obtains the weighted information of the receptive field of different scales to ensure that effective and rich image feature information is obtained under the current image quality, helping the model to adaptively adjust according to the image quality during the encoding stage and better represent the image information;

[0085] Step (2): Adopt the training strategy of CycleGAN to optimize the network parameters and generate the result G A (c) Geometric consistency loss GeomLoss and semantic loss CLIPLoss are also performed between the source domain image c to ensure the content consistency between the generated result and the input image.

[0086] The quality-adaptive convolution module includes a multi-scale multi-core convolution block and a quality-adaptive routing network. The multi-scale multi-core convolution block captures receptive field information of different scales through multiple convolution layers with different convolution kernel sizes. The quality-adaptive routing network obtains the weight parameters of receptive field information of different scales through a quality evaluation network and a multi-layer perceptron, and then dynamically selects different receptive field information.

[0087] The multi-scale multi-core convolution block consists of n sub-modules, each of which contains m convolution layers with convolution kernels of different sizes;

[0088] The multi-scale multi-kernel convolution block is used to obtain receptive field information of different scales, and each submodule is used to obtain receptive field information of different scales of the input image features of this group.

[0089] The quality adaptive routing module consists of a no-reference quality evaluation network and a multi-layer perceptron (MLP) network;

[0090] The quality adaptive routing module is used to obtain weight parameters of receptive field information of different scales under the current quality, the reference-free quality evaluation network is used to obtain the quality features of the current image, and the multi-layer perceptron MLP network is used to map the quality features into weight parameters of different receptive field information.

[0091] The weight parameters of the receptive field information of different scales obtained by the quality adaptive routing module are used to weight and combine the receptive field information of different scales obtained by the multi-scale multi-kernel convolution block, and finally obtain image feature information that adapts to the current image quality.

[0092] The steps of the quality adaptive dynamic convolution module in step (1) are as follows:

[0093] s1: multi-scale multi-kernel convolution block:

[0094] A multi-scale multi-core convolution block is used to obtain a multi-scale receptive field. The multi-scale multi-core convolution block consists of n submodules, each of which contains m convolution layers with convolution kernels of different sizes. The steps to obtain a multi-scale receptive field are as follows:

[0095] First, the G of GLD A The first convolutional network layer of the Encoder extracts the features F∈R of the source domain image c H×W×C Divide into n groups in the channel dimension;

[0096]

[0097] Then a multi-scale convolution block is designed, which consists of m different convolutional layers, and the convolution kernel size of each convolutional layer is K. j The setup looks like this:

[0098]

[0099] Feature grouping F of source domain images i After the convolution layer in the multi-scale convolution module, the feature information obtained in the current receptive field is Z i,j , the process is as follows, where i represents F i The i-th group in, j represents K j The jth convolutional layer in .

[0100]

[0101] s2: Quality Adaptive Routing Network:

[0102] The quality adaptive routing network consists of a no-reference quality evaluation network And a multi-layer perceptron MLP network;

[0103] The specific steps are as follows:

[0104] Step 1): Extract quality features of the source domain image c:

[0105] Input the source domain image c into the no-reference quality assessment network Extract the quality feature vector f of the source domain imageQ :

[0106]

[0107] Step 2) Get weight information

[0108] The quality feature vector f of each image described in step 1) Q After a multi-layer perceptron MLP network, the corresponding weight w is obtained i,j , indicating that the j-th convolution layer in the multi-scale convolution module corresponds to F i The weight factor of , the corresponding expression is as follows:

[0109] w i,j =Softmax(MLP(f Q )),i∈{1,...,n},j∈{1,...,m};

[0110] Through the above operations, the image fusion network obtains the weight information of image features under different receptive fields based on the image quality characteristics, and then obtains the weighted information under different receptive fields to meet the extraction of image features with different qualities;

[0111] s3: Feature fusion:

[0112] The intermediate result Z after convolution obtained according to s2 i,j and the weight w obtained by s3 i,j , multiply the two to get multiple continuous receptive field information H that matches the current image quality i i represents the i-th group of feature information of the source domain image c. Finally, H i After splicing, the complete feature information H of the final source domain image c is obtained;

[0113]

[0114] H=concat(H i ),i∈{1,...,n};

[0115] H is the final image feature information extracted after the source domain image passes through the quality adaptive convolution module.

[0116] In the above steps, through the quality adaptive convolution module, the image fusion network can obtain the weighted feature information of the input image under different receptive fields based on the quality of the input image, realize the dynamic adjustment of the acquired image information based on image quality adaptation, and ensure that rich and accurate image information can be extracted under different qualities.

[0117] The generator in the generative adversarial network model described in step (2) includes the generator G A , generator GB ;

[0118] Generator G A , generator G B Both use the Encoder-Decoder structure containing the Res-Net structure, where the target domain generator G A It is used to generate an image from the source domain C to the target domain S, that is, input the source image c and obtain the representation G of c in the target domain A (c), Source domain generator G B Generate the image from the target domain S to the source domain C, that is, input the target domain image s and obtain the representation G of s in the source domain B (s);

[0119] The discriminator in the generative adversarial network model described in step (2) includes the discriminator D A , Discriminator D B , discriminator D A , Discriminator D B Both adopt Patch-Based structure, where the target domain discriminator D A Used to identify whether the image belongs to the target domain S, the source domain discriminator D B Used to identify whether the image belongs to the source domain C;

[0120] Among them G A With D A Constitute an adversarial relationship, G B With D B constitute an adversarial relationship;

[0121] The calculation of GeomLoss in step (2) is as follows:

[0122] L geom =||G geom (I(G A (c)))-F(c)||

[0123] Among them G A (c) represents the generated result after the source domain image c is input into the target domain generator, I(G A(c) ) indicates that the result G will be generated A (c) Image features extracted from the input Inception v3 network, G geom (I(G A (c))) represents the input of image features into the pre-trained depth map prediction network G geom The generated result depth map is obtained; F(c) represents the content depth map obtained by inputting the source domain image c into the depth map prediction network F.

[0124] The calculation of CLIPLoss is as follows:

[0125] L CLIP =||CLIP(G A (c))-CLIP(c)||

[0126] Among them, CLIP(G A (c)) and CLIP(c) respectively represent the use of the generated result C A (c) The generated result and semantic information of the source domain image obtained by inputting the source domain image c into the CLIP model.

[0127] The generated image is closer to the source domain image in terms of geometric structure and semantic information, thereby ensuring the content consistency between the generated result and the input image.

[0128] The training process of step 3 is as follows:

[0129] (1) Input the source domain image c into the target domain generator G A Get the target domain generation result G A (c), and then generate the result G A (c) Input source domain generator G B Get the source domain generation results

[0130] (2) G A (c) Input to the target domain discriminator D A Get the target domain identification result D A (G A (c)), Input to the source domain discriminator D B Get the source domain identification result

[0131] (3) Calculating the loss in step (2) and optimizing network parameters;

[0132] (4) Repeat the above steps (1)–(3) until the network achieves the expected effect and save the training weight file.

[0133] The step 4 is specifically as follows:

[0134] (1) Load the training weight file obtained in step 3 into the generator G A middle.

[0135] (2) Input the source domain image c to be cross-domain mapped into the target domain generator G A In the example, we can obtain the mapping result of c in the target domain.

[0136] Experimental setup

[0137] Datasets: To verify the effectiveness of the quality-adaptive dynamic convolution module, experiments were conducted on image translation tasks using the COCO dataset and the Anime Colorization dataset. The COCO dataset is a large, rich dataset for object detection, segmentation, and captioning, encompassing 91 categories of objects, 328,000 images, and 2.5 million labels. The Anime Colorization dataset, a dataset for anime colorization, contains both color and black-and-white line drawings of anime.

[0138] Implementation details:

[0139] The model block diagram of the present invention is as follows Figure 2 :

[0140] This paper experiments with the unsupervised image translation model GLD. GLD, consisting of two generators and a discriminator, combines CLIP to effectively bridge image features with text semantics. By training the network to map image features to depth images, the model is able to capture key geometric details and semantic layers in a scene while maintaining clear object outlines.

[0141] Specific insertion position combination Figure 3 , located in G A The dotted box in the encoder is added after the first convolutional layer that converts the image from 3 channels to 64 channels. This helps the model to adaptively adjust according to the image quality during the encoding stage and better represent the image information.

[0142] In training, 10,000 images from COCO were randomly selected as source images, 14,224 images from Anime were randomly selected as target images, and 1,000 COCO images and 3,545 Anime images were selected as the test set. Adam was used for optimization with a learning rate of 2e-4 and a batch size of 1 for a maximum of 40 epochs. The number of channel groups n mentioned in step 2 above was set to 4, with 4 convolutional layers per group. A 3×3 dilated convolution kernel was used to achieve multi-scale receptive fields, and the specific receptive field size was achieved by adjusting the dilation rate.

[0143] Experimental evaluation indicators

[0144] This experiment uses commonly used image generation evaluation metrics to measure the quality of generated images, including FID, SIFID, and CPBD.

[0145] FID (Frechet Inception Distance) is a metric used to evaluate the quality of images generated by generative models (such as GANs). It measures the similarity between the generated image and the real image by comparing their distributions. The smaller the FID value, the higher the quality of the generated image and the closer it is to the real image distribution. The calculation formula for FID is as follows:

[0146]

[0147] Among them, μ r , μ g The feature mean of the real image and the generated image; Σ r ,Σ g is the feature covariance matrix of the real image and the generated image, Tr is the trace of the matrix, which represents the sum of the diagonal elements of the matrix; represents the geometric mean of the covariance matrix.

[0148] SIFID (Single Image Frechet Inception Distance) is a variant of FID, specifically designed for single image generation quality assessment, particularly for evaluating the similarity between a generated image and a single real reference image. It is commonly used in scenarios such as Image-to-Image Translation tasks, such as style transfer or super-resolution reconstruction.

[0149]

[0150] Among them, μ ref ,Σ ref The mean and covariance matrix of the features of a single reference image, μ gen ,Σ gen The mean and covariance matrix of the single generated image features are the same as the FID above.

[0151] CPBD is a gradient-based, no-reference clarity assessment method. Its core concept is to quantify image clarity using gradient changes in an image. Rapid gradient changes are often associated with image edge clarity. Therefore, the CPBD metric reflects image clarity by statistically analyzing the cumulative probability of valid edge information. CPBD values ​​range from [0,1][0,1][0,1], with higher values ​​indicating clearer images and lower values ​​indicating blurrier images. The CPBD calculation formula is as follows:

[0152]

[0153] Where N represents the number of all valid edge pixels in the image; ω irepresents the weight of the i-th edge pixel, which is usually related to the edge significance or gradient amplitude; G i represents the gradient magnitude of an edge pixel; P(G)P(G)P(G) is the probability density function of the gradient magnitude, which represents the distribution of the gradient.

[0154] Experimental results

[0155] Visual Effects:

[0156] In the experiment of converting images to line draft style using the above-mentioned GLD model, the first column HQ Content represents the original high-definition source domain image; the second column LQ Content represents the blurred source domain image after degradation; the third column HQ Result is the standard output obtained by inputting the high-definition source domain image into the officially provided model weights, which serves as the benchmark for visual effect comparison; the fourth column Official indicates that after inputting the blurred source domain image into the official weight model, it can be seen that the output quality is greatly reduced and the lines are blurred; the fifth column Data Aug indicates that after using the commonly used data enhancement method, the model is retrained and it is found that the output effect is improved; the sixth column Ours indicates that by adding the present invention to the position of the model described above, the output result is more significantly improved, and the details are excellent.

[0157] Quantitative indicators:

[0158] To verify the effectiveness of the quality-adaptive dynamic convolution module proposed in this method, this experiment conducted three settings on the baseline model GLD, namely the official baseline model, setting data augmentation operations on the baseline model, and setting the quality-adaptive dynamic convolution module proposed in this method on the baseline model. The test results were compared on the above experimental indicators FID, SIFID, and CPBD. The detailed test indicators for the above test set are shown in the table.

[0159] Table 1 Performance evaluation of the quality-adaptive dynamic convolution module on the GLD model

[0160]

[0161]

[0162] The results show that the present invention can achieve effective performance improvement on various baseline models and achieve competitive results in SIFID, CPBD, and FID indicators.

Claims

1. A quality-adaptive image cross-domain mapping method, characterized in that: The following steps are included: Step 1: Obtain the data of the target domain image s and the source domain image c and perform preprocessing; Step 2: Design a generative adversarial network model. The generative adversarial network model is based on the network structure of the unsupervised image translation model and inserts the quality adaptive convolution module into the generator G of the unsupervised image translation model. A middle; Step 3: Use the target domain image s and source domain image c preprocessed in step 1 to input into the generative adversarial network model in step 2 for training; Step 4: Use the generative adversarial network model trained in step 3 to map the source domain image to the target domain image; The quality-adaptive convolution module includes a multi-scale multi-core convolution block and a quality-adaptive routing network. The multi-scale multi-core convolution block consists of n sub-modules, each of which contains m convolution layers with convolution kernels of different sizes. Each submodule is used to obtain the receptive field information of different scales of the input image features of this group; The quality adaptive routing network consists of a no-reference quality evaluation network and a multi-layer perceptron (MLP) network; The no-reference quality assessment network is used to obtain the quality features of the current image, and the multi-layer perceptron (MLP) network is used to map the quality features into weight parameters of different receptive field information; The weight parameters of the receptive field information of different scales obtained by the quality adaptive routing network are used to weight and combine the receptive field information of different scales obtained by the multi-scale multi-kernel convolution block, and finally obtain image feature information that adapts to the current image quality; The step 2 is specifically as follows: Step (1): Generator G of the unsupervised image translation model A The encoder-decoder structure is adopted. The quality adaptive convolution module is inserted between the first and second convolution layers of the encoder. The receptive field information of different scales is obtained through the multi-scale multi-core convolution block. The weights of the receptive field information of different scales are obtained through the quality adaptive routing network, and then the weighted information of the receptive fields of different scales is obtained. Step (2): Adopt the training strategy of CycleGAN to optimize the network parameters and generate the result G A (c) Perform geometric consistency loss GeomLoss and semantic loss CLIPLoss between the source domain image c to ensure the content consistency between the generated result and the input image.

2. The quality-adaptive image cross-domain mapping method according to claim 1, characterized in that: The preprocessing of step 1 includes: modifying the image to be of the same size and performing data enhancement operations on the image; The training data are all from existing public datasets, the source domain data is from the COCO dataset, and the target domain data is from the AnimeColorization dataset.

3. The quality-adaptive image cross-domain mapping method according to claim 1, characterized in that: The steps of the quality adaptive convolution module in step (1) are as follows: s1: multi-scale multi-kernel convolution block: A multi-scale multi-core convolution block is used to obtain a multi-scale receptive field. The multi-scale multi-core convolution block consists of n sub-modules, each of which contains m convolution layers with convolution kernels of different sizes. The steps to obtain a multi-scale receptive field are as follows: First, the G of the unsupervised image translation model is A The first convolutional network layer of the Encoder extracts the features F∈R of the source domain image c H×W×C Divide into n groups in the channel dimension; Then a multi-scale convolution block is designed, which consists of m different convolutional layers, and the convolution kernel width of each convolutional layer is K. j The setup looks like this: Feature grouping F of source domain images i After the convolution layer in the multi-scale convolution block, the feature information obtained in the current receptive field is Z i,j , the process is as follows; s2: Quality Adaptive Routing Network: The quality adaptive routing network consists of a no-reference quality evaluation network And a multi-layer perceptron MLP network; The specific steps are as follows: Step 1): Extract quality features of the source domain image c: Input the source domain image c into the no-reference quality assessment network Extract the quality feature vector f of the source domain image Q : Step 2) Get weight information The quality feature vector f of each image in step 1) Q After a multi-layer perceptron MLP network, the corresponding weight w is obtained i,j , the corresponding expression is as follows: w i,j =Softmax(MLP(f Q )),i∈{1,...,n},j∈{1,...,m}; s3: Feature fusion: The intermediate result Z after convolution obtained according to s1 i,j and the weight w obtained by s2 i,j , multiply the two together to obtain multiple continuous receptive field information H that matches the current image quality i , H i Represents the i-th group of feature information of the source domain image c, and finally H i After splicing, the complete feature information H of the final source domain image c is obtained; H=concat(H i ),i∈{1,...,n}; H is the final image feature information extracted after the source domain image passes through the quality adaptive convolution module.

4. The quality-adaptive image cross-domain mapping method according to claim 1, characterized in that: The generator in the generative adversarial network model described in step (2) includes the generator G A , generator G B ; Generator G A , generator G B Both use the Encoder-Decoder structure containing the Res-Net structure, where the target generator G A It is used to generate an image from the source domain C to the target domain S, that is, input the source domain image c and obtain the representation G of c in the target domain A (c), Source domain generator G B Generate the image from the target domain S to the source domain C, that is, input the target domain image s and obtain the representation G of s in the source domain B (s); The discriminator in the generative adversarial network described in step (2) includes the discriminator D A , Discriminator D B , discriminator D A , Discriminator D B Both adopt Patch-Based structure, where the target discriminator D A Used to identify whether the image belongs to the target domain S, the source domain discriminator D B Used to identify whether the image belongs to the source domain C; Among them G A With D A Constitute an adversarial relationship, G B With D B constitute an adversarial relationship.

5. The quality-adaptive image cross-domain mapping method according to claim 4, characterized in that: The calculation of GeomLoss in step (2) is as follows: L geom =||G geom (I(G A (c)))-F(c)|| Among them G A (c) represents the generated result after the source domain image c is input to the target generator, I(G A (c)) indicates that the result G will be generated A (c) Image features extracted from the input Inception v3 network, G geom (I(G A (c))) represents the input of image features into the pre-trained depth map prediction network G geom The generated depth map is obtained; F(c) represents the content depth map obtained by inputting the source domain image c into the depth map prediction network F; The calculation of CLIPLoss is as follows: L CLIP =||CLIP(G A (c))-CLIP(c)|| Among them, CLIP(G A (c)) and CLIP(c) respectively represent the use of the generated result G A (c) The generated result and semantic information of the source domain image obtained by inputting the source domain image c into the CLIP model.

6. The quality-adaptive image cross-domain mapping method according to claim 5, characterized in that: The training process of step 3 is as follows: (3.1) Input the source domain image c into the generator G A Get the target domain generation result G A (c), and then generate the result G A (c) Input generator G B Get the source domain results (3.2) G A (c) Input to the discriminator D A Get the target domain identification result D A (G A (c)), Input to the discriminator D B Get the source domain identification result (3.3) Calculate the loss in step (3.2) and optimize the network parameters; (3.4) Repeat the above (3.1)–(3.3) until the network achieves the expected effect and save the training weight file.

7. The quality-adaptive image cross-domain mapping method according to claim 6, characterized in that: The step 4 is specifically as follows: (1) Load the training weight file obtained in step 3 into the generator G A middle; (2) Input the source domain image c to be cross-domain mapped into the generator G A In the example, we can obtain the mapping result of c in the target domain.

Images