Adaptive texture enhancement method based on global consistency and WGAN-GP

Abstract

The invention discloses an adaptive texture enhancement method based on global consistency and WGAN-GP, comprising: grayscale transformation, normalization and expansion of an original texture image; constructing a two-stage texture enhancement network, wherein the first stage roughly predicts the grayscale image, and the second stage refines the texture details to obtain a texture enhanced image; introducing a block that captures long-term dependencies to fuse global information into local operations; constructing a loss function, calculating an evaluation index of the texture enhancement network according to a result of the loss function, and judging whether the texture enhancement network is excellent according to the evaluation index; the method designed by the invention improves the consistency, naturalness and adaptability of the texture enhancement effect by considering global information and using the WGAN-GP method, and the method overcomes the limitations of traditional texture enhancement methods and has broad application potential in practical applications.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an adaptive texture enhancement method based on global consistency and WGAN-GP. Background Art

[0002] Texture has important application value in the field of image processing. Texture can provide additional information, enrich image content, and make images more visually attractive and expressive. In order to improve the texture details and quality in images, researchers have proposed various texture enhancement methods. Texture enhancement is an important task that aims to improve the texture details and quality in images, making images clearer, more natural, and with good visual effects.

[0003] At present, most of the texture enhancement methods based on deep learning use generative adversarial networks (GANs). GANs are an unsupervised learning method that does not require pre-labeled training data. Instead, they are trained by maximizing the adversarial objective function between the generator and the discriminator. The generator is responsible for generating realistic data samples, while the discriminator is used to evaluate the difference between the generated samples and the real samples. By iteratively training the generator and the discriminator, GAN can gradually improve the generator's ability to generate realistic samples.

[0004] In the prior art, there are methods of training convolutional neural networks to extract and enhance local texture features in images. The network structure uses multiple layers of convolution and pooling layers to capture local features of different scales and levels. By learning training data on a large scale, the network can adaptively learn and enhance local textures in images. However, this texture enhancement method often only focuses on local texture features in the image and ignores global context information, which may cause the enhanced image to have incoherent or unnatural textures, especially where there are repeated textures or texture edges.

[0005] There are also methods in the prior art that use GAN for underwater image enhancement. The network performance can be improved by combining multi-scale dense blocks (MSDB) with residual learning, and multiple loss functions can produce visually pleasing enhancement effects. However, this method lacks adaptability to different types of textures, which makes them unable to effectively process different types of textures, limiting their wide application. In addition, the GAN method also has problems such as gradient disappearance and model collapse during the training process.

[0006] The current texture enhancement methods still have defects. The present invention aims to design an adaptive texture enhancement method based on global consistency and WGAN-GP. By considering global information and using the WGAN-GP method, the consistency, naturalness and adaptability of the texture enhancement effect are improved. This method overcomes the limitations of traditional texture enhancement methods and has broad application potential in practical applications. Summary of the invention

[0007] The purpose of the present invention is to provide an adaptive texture enhancement method based on global consistency and WGAN-GP to improve the coherence, naturalness and adaptability of texture enhancement effect.

[0008] To achieve the above functions, the present invention designs an adaptive texture enhancement method based on global consistency and WGAN-GP, executes the following steps S1 to S5, constructs and trains a texture enhancement network, and completes the enhancement of the input texture image:

[0009] Step S1: Acquire the original texture image, convert the original texture image into a grayscale image, scale the pixels to the range of [0,1] after Min-Max normalization, and then use the traditional method to expand the number to obtain a preset number of grayscale images;

[0010] Step S2: construct a two-stage texture enhancement network. The first stage roughly predicts the grayscale image based on convolution, and the second stage refines the texture details of the grayscale image based on the WGAN-GP network. The texture enhancement network takes the grayscale image obtained in step S1 as input and outputs a texture enhanced image.

[0011] Step S3: Add a block that captures long-term dependencies to the texture enhancement network, fuse global information into local operations, form a detailed attention map for each pixel at the target position, and weight the prediction of the pixel at the current position according to these detailed attention maps, and use the global discriminator and the local discriminator to compare the texture enhanced image with the original texture image;

[0012] Step S4: training the texture enhancement network, constructing a loss function based on MSE distance, TV loss, and MS-SSIM loss, and calculating the evaluation index of the texture enhancement network according to the result of the loss function;

[0013] Step S5: If the performance of the texture enhancement network reaches the evaluation index, it is considered excellent and the training is completed. Otherwise, it is considered not excellent and the weight coefficient of the loss function needs to be readjusted and the texture enhancement network is retrained. Whether the texture enhancement network is excellent is judged according to the evaluation index. If it is, a successful texture enhancement network is obtained. Otherwise, the weight coefficient is updated.

[0014] As a preferred technical solution of the present invention: the method for expanding the number in step S1 is to rotate each grayscale image by 90°, 180°, and 270° respectively, and then flip them one by one to increase the number of grayscale images by 8 times to obtain training samples for the input texture enhancement network.

[0015] As a preferred technical solution of the present invention: the two-stage texture enhancement network described in step S2, in the first stage, the downsampling layer convolution and pooling are performed to capture global features, multiple intermediate convolution layers are used to process feature maps, and the upsampling layer transposed convolution or nearest neighbor interpolation is performed to improve spatial resolution to obtain a roughly predicted texture enhanced image; the function equation of the second stage WGAN-GP network is as follows:

[0016]

[0017] Among them, L is the function output result, E represents the corresponding expected value, S r is a real sample, P g is the generator generating distribution, D(S r ) is the discriminator for S r The output, S f is a false sample, P r is the real data distribution, D(S f ) is the discriminator for S f The output of , λ is the regularization coefficient, are samples from the true distribution and the generated distribution, Yes S r The gradient of ||·||2 is The norm of .

[0018] As a preferred technical solution of the present invention: in step S3, two NL blocks are added to the first stage of the texture enhancement network. The NL block calculates the attention map of the similarity between the target position pixel and all other position pixels, and then weights the importance according to the size of the similarity. The specific calculation of the NL block is as follows:

[0019]

[0020] Among them, y i represents the calculation result of the NL block at the target pixel i, C(x) represents the normalization factor, x represents the feature map of texture enhancement image processing, and j represents x i All possible indices associated with a position, f is a scalar function of positions, x i is the characteristic value of pixel i at the target position, x j is the feature value of the pixel j at the possible associated position, g is the position input signal function, and the input features are weighted. g(x j ) is used to calculate x j Function of

[0021] Two GC blocks are added to the second stage of the texture enhancement network to form a discriminator. The specific calculation of the GC block is as follows:

[0022]

[0023] Among them, Z i represents the calculation result of the GC block at the target pixel i, F(.,.) represents the fusion function, x represents the feature map of texture enhancement image processing, i represents the target pixel, δ(·) represents the bottleneck transformation module, N p represents the global space, j represents x i All possible indices associated with a position, α j Represents the weight of the global attention pool, which is used to model x in the context j Weighted, x j is the feature value of the pixel j that may be associated with the position.

[0024] As a preferred technical solution of the present invention: the training of the texture enhancement network in step S4 is divided into two training steps, the first training step only trains the texture enhancement network, and the second training step trains the texture enhancement network and the adversarial network; in the first training step, the loss function selects MSE loss and TV loss to jointly constrain the accuracy of the result, and the second training step involves the texture enhancement network and the adversarial network, the loss function of the texture enhancement network uses l1 distance and MS-SSIM loss, and the loss function of the adversarial network uses Wasserstein distance with gradient penalty as the specification; the loss function used in the first training step is:

[0025] loss_cn1=λ1×l2+(1-λ1)×l TV

[0026] Among them, loss_cn1 represents the value of the loss of the first training step, and λ1 is used to distribute the two loss functions l2 and l of loss_cn1. TV The weight coefficient is a random number between [0, 1], l2 is the MSE loss, l TV It is TV loss;

[0027] In the second training step, the loss function of the texture enhancement network is calculated as:

[0028] loss_cn2=λ2×l1+(1-λ2)×l MS-SSIM

[0029] Among them, loss_cn2 is the value of the loss of the second training step, and λ2 is used to distribute the two loss functions l1 and l2 of loss_cn2. MS-SSIM The weight coefficient is a random number between [0, 1], l1 is the L1 norm loss function, l MS-SSIM It is the multi-scale structural similarity.

[0030] As a preferred technical solution of the present invention: in step S4, the Adam optimizer is used to train the texture enhancement network.

[0031] As a preferred technical solution of the present invention: the evaluation index in step S5 includes SSIM and PSNR.

[0032] Beneficial effects: Compared with the prior art, the advantages of the present invention include:

[0033] 1. A new global context modeling model based on non-local operations is proposed, which can better capture the global information in the image and effectively use the global context to guide the texture enhancement process;

[0034] 2. WGAN-GP is applied to texture enhancement, which has a more stable training process than the original GAN ​​model and adopts multiple forms of loss functions to obtain faster convergence. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 4 is a flowchart of an adaptive texture enhancement method based on global consistency and WGAN-GP provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0037] Reference Figure 1 The adaptive texture enhancement method based on global consistency and WGAN-GP provided in the embodiment of the present invention performs the following steps S1 to S5 to construct and train a texture enhancement network to enhance the input texture image:

[0038] Step S1: Acquire the original texture image, perform grayscale transformation on the original texture image, convert it into a grayscale image, perform Min-Max normalization on each pixel in the grayscale image, scale the pixel value to the range of [0,1], and then use the traditional method to expand the number to obtain a preset number of grayscale images;

[0039] The WGAN-GP network input data is usually required to be in the range of [-1, 1]. Min-Max normalization is usually used to scale the pixel values ​​to the range of [0, 1]. After normalization, the pixel values ​​of the image will be between 0 and 1, which is convenient for network training and data processing. The normalization formula is:

[0040]

[0041] Among them, y represents the normalized value, X represents any value in the data set, and X min and Xmax are the minimum and maximum values ​​in the data set, respectively. Considering the small sample size of the original texture data set, it is divided into a training set and a test set according to 8:2. In addition, due to the small amount of original data, the number of grayscale images is expanded. The method of quantity expansion is to rotate each grayscale image by 90°, 180°, and 270° respectively, and then flip them one by one to increase the number of grayscale images by 8 times, and obtain the training samples for the input texture enhancement network.

[0042] Step S2: construct a two-stage texture enhancement network. The first stage roughly predicts the grayscale image based on convolution, and the second stage refines the texture details of the grayscale image based on the WGAN-GP network. The texture enhancement network takes the grayscale image obtained in step S1 as input and outputs a texture enhanced image.

[0043] The two-stage texture enhancement network described in step S2 uses a convolutional neural network (CNN) to generate a rough grayscale image in the first stage. This network can be composed of multiple convolutional layers, pooling layers, and fully connected layers. Its goal is to extract features from the input image and generate a grayscale image of the same size as the input image, which will be used as the input of the second stage; in the model of the first stage, two Conv2D layers are used, each with 64 and 128 convolution kernels, the size of these convolution kernels is (3,3), and the ReLU activation function is used. The number and size of the convolution kernels can be adjusted according to the complexity of the task; the maximum pooling layer uses two MaxPooling2D layers, each using a pooling window of (2,2) to reduce the size of the feature map; two fully connected layers are added, each with 256 neurons and an output of the same size as the input image, and the size of these layers is 128. The purpose is to convert the convolution feature map into a grayscale image; based on the rough prediction of the first stage, the convolution layer added in the second stage can refine the texture details and enhance the local features to obtain a refined texture-enhanced image. In the second stage, the WGAN-GP function is introduced for the training of the texture enhancement model in the second stage. The wasserstein distance of the WGAN-GP network can replace the traditional GAN ​​loss, such as binary cross entropy, which will encourage stable and meaningful updates during training, generate new texture images, and amplify the original data size, balance the data size of different types of textures, and improve the robustness of the texture recognition model; the function equation of the second stage WGAN-GP network is as follows:

[0044]

[0045] Among them, L is the function output result, E represents the corresponding expected value, S r is a real sample, P g is the generator generating distribution, D(S r ) is the discriminator for Sr The output, S f is a false sample, P r is the real data distribution, D(S f ) is the discriminator for S f The output of , λ is the regularization coefficient, are samples from the true distribution and the generated distribution, Yes S r The gradient of ||·||2 is The WGAN-GP network adds a regular term based on WGAN to avoid mode collapse.

[0046] Step S3: Add a block that captures long-term dependencies to the texture enhancement network, fuse global information into local operations, form a detailed attention map for each pixel at the target position, and weight the prediction of the pixel at the current position according to these detailed attention maps, and use the global discriminator and the local discriminator to compare the texture enhanced image with the original texture image;

[0047] In step S3, two NL (modules for capturing long-range dependency features) blocks are added to the first stage of the texture enhancement network to enable global information viewing. When predicting the current position pixel, the NL block calculates the attention map of the similarity between the target position pixel and all other position pixels, and then weights the importance according to the size of the similarity. The specific calculation of the NL block is as follows:

[0048]

[0049] Among them, y i represents the calculation result of the NL block at the target pixel i, C(x) represents the normalization factor, x represents the feature map of texture enhancement image processing, and j represents x i All possible indices associated with a position, f is a scalar function of positions, x i is the characteristic value of pixel i at the target position, x j is the feature value of the pixel j at the possible associated position, g is the position input signal function, and the input features are weighted. g(x j ) is used to calculate x j Function of

[0050] This structure allows the insertion of NL blocks in any pre-trained model without affecting the original layout. The present invention adds two NL blocks in the first stage of the texture enhancement network, making the calculation more intensive; through the NL block, after visualizing the attention map of each pixel, most of these maps are similar, which will lead to computational waste and inefficient network optimization. The GC block (global context block) only needs to calculate the attention map once. Two GC blocks are added to the second stage of the texture enhancement network to form a discriminator. The specific calculation of the GC block is as follows:

[0051]

[0052] Among them, Z i represents the calculation result of the GC block at the target pixel i, F(.,.) represents the fusion function, which is used to merge the GC features into the features of each position, x represents the feature map of texture enhancement image processing, i represents the target pixel, δ(·) represents the bottleneck transformation module, and N p represents the global space, j represents x i All possible indices associated with a position, α j Represents the weight of the global attention pool, which is used to model x in the context j Weighted, x j It is the feature value of the pixel j that may be associated with the position, which greatly reduces the number of parameters. In addition, two GC blocks are added to the texture enhancement network to implement the local discriminator of the second phase. Compared with the method mentioned in NL, the number of parameters is significantly reduced and the performance is reduced. This non-local filtering method for capturing global information achieves lightweight improvements and avoids useless waste of calculations. The enhanced image is compared with the initial image through such a discriminator.

[0053] Step S4: training the texture enhancement network, constructing a loss function based on MSE distance, TV loss, and MS-SSIM loss, and presetting and calculating the evaluation index of the texture enhancement network according to the result of the loss function;

[0054] The training of the texture enhancement network in step S4 is divided into two training steps. The first training step only trains the texture enhancement network, and the second training step trains the texture enhancement network and the adversarial network. In the first training step, the loss function selects MSE loss and TV loss to jointly constrain the accuracy of the result. The second training step involves the texture enhancement network and the adversarial network, and the loss function of the texture enhancement network adopts Distance and MS-SSIM (multi-scale structural similarity) loss. The loss function of the adversarial network uses Wasserstein distance with gradient penalty (WGAN-GP) as the norm; the loss function used in the first training step is:

[0055] loss_cn1=λ1×l2+(1-λ1)×l TV

[0056] Among them, loss_cn1 represents the value of the loss of the first training step, and λ1 is used to distribute the two loss functions l2 and l of loss_cn1. TV The weight coefficient is a random number between [0, 1], l2 is the MSE loss, l TV is the TV loss, a commonly used regularization term used with other criteria to constrain noise;

[0057] In the second training step, the loss function of the texture enhancement network is calculated as:

[0058] loss_cn2=λ2×l1+(1-λ2)×l MS-SSIM

[0059] Among them, loss_cn2 is the value of the loss of the second training step, and λ2 is used to distribute the two loss functions l1 and l2 of loss_cn2. MS-SSIM The weight coefficient is a random number between [0, 1], l1 is the L1 norm loss function, l MS-SSIM is the multi-scale structural similarity; l1 can retain the brightness and color information of the texture image, l MS-SSIM Taking the features of texture and structural details and combining the two as evaluation metrics for texture enhancement can make the results closer to people's visual expectations than other results.

[0060] In step S4, the texture enhancement network is trained using the Adam optimizer for 250,000 times to generate new high-quality texture images.

[0061] Step S5: According to the evaluation index, determine whether the texture enhancement network is excellent. If the performance of the texture enhancement network reaches the evaluation index, it is considered excellent and the training is completed to obtain a trained texture enhancement network. Otherwise, it is considered not excellent, and the weight coefficient of the loss function is adjusted to train the texture enhancement network until the preset number of training times is reached.

[0062] The results of texture enhancement are evaluated using evaluation indicators, such as SSIM, PSN, etc. SSIM is an indicator used to compare the similarity between two images. It takes into account brightness, contrast and structural information. When evaluating, the SSIM value between the enhanced image and the original image can be calculated. The value range of SSIM is between -1 and 1. The closer to 1, the more similar the images are. PSNR is an indicator used to measure image quality. It calculates the peak signal-to-noise ratio between the original image and the enhanced image to evaluate the quality of image reconstruction. The higher the PSNR value, the better the image quality. Compare with the original texture image to obtain an evaluation score. According to the evaluation score, adjust the hyperparameters of the WGAN-GP network, such as the learning rate and batch size, to optimize the convergence and performance of the network. Repeat training, evaluation and tuning until the expected texture enhancement effect is achieved or the number of training times reaches the set upper limit. During the tuning process, the trained WGAN-GP network model is saved regularly for backtracking and comparison when needed.

[0063] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. An adaptive texture enhancement method based on global consistency and WGAN-GP, characterized in that: Execute the following steps S1 to S5 to build and train a texture enhancement network to enhance the input texture image: Step S1: Acquire the original texture image, convert the original texture image into a grayscale image, scale the pixels to the range of [0,1] after Min-Max normalization, and then use the traditional method to expand the number to obtain a preset number of grayscale images; Step S2: construct a two-stage texture enhancement network. The first stage roughly predicts the grayscale image based on convolution, and the second stage refines the texture details of the grayscale image based on the WGAN-GP network. The texture enhancement network takes the grayscale image obtained in step S1 as input and outputs a texture enhanced image. Step S3: Add a block that captures long-term dependencies to the texture enhancement network, fuse global information into local operations, form a detailed attention map for each pixel at the target position, and weight the prediction of the pixel at the current position according to these detailed attention maps, and use the global discriminator and the local discriminator to compare the texture enhanced image with the original texture image; In step S3, two NL blocks are added to the first stage of the texture enhancement network. The NL block calculates the attention map of the similarity between the target position pixel and all other position pixels, and then weights the importance according to the size of the similarity. The specific calculation of the NL block is as follows: Among them, y i represents the calculation result of the NL block at the target pixel i, C(x) represents the normalization factor, x represents the feature map of texture enhancement image processing, and j represents x i All possible indices associated with a position, f is a scalar function of positions, x i is the characteristic value of pixel i at the target position, x j is the feature value of the pixel j at the possible associated position, g is the position input signal function, and the input features are weighted. g(x j ) is used to calculate x j Function of Two GC blocks are added to the second stage of the texture enhancement network to form a discriminator. The specific calculation of the GC block is as follows: Among them, Z i represents the calculation result of the GC block at the target pixel i, F(.,.) represents the fusion function, x represents the feature map of texture enhancement image processing, i represents the target pixel, δ(·) represents the bottleneck transformation module, N p represents the global space, j represents x i All possible indices associated with a position, α j Represents the weight of the global attention pool, which is used to model x in the context j Weighted, x j is the feature value of pixel j that may be associated with the position; Step S4: training the texture enhancement network, constructing a loss function based on MSE distance, TV loss, and MS-SSIM loss, and calculating the evaluation index of the texture enhancement network according to the result of the loss function; The training of the texture enhancement network in step S4 is divided into two training steps. The first training step only trains the texture enhancement network, and the second training step trains the texture enhancement network and the adversarial network. In the first training step, the loss function selects MSE loss and TV loss to jointly constrain the accuracy of the result. The second training step involves the texture enhancement network and the adversarial network, and the loss function of the texture enhancement network adopts Distance and MS-SSIM loss,The loss function of the adversarial network uses the Wasserstein distance with gradient penalty as the norm; the loss function used in the first training step is: Among them, loss_cn1 represents the value of the loss of the first training step, and λ1 is used to distribute the two loss functions of loss_cn1 and The weight coefficient is a random number between [0,1]. is the MSE loss, It is TV loss; In the second training step, the loss function of the texture enhancement network is calculated as: Among them, loss_cn2 is the value of the loss of the second training step, and λ2 is used to distribute the two loss functions of loss_cn2 and The weight coefficient is a random number between [0,1]. is the L1 norm loss function, is the multi-scale structural similarity; Step S5: If the performance of the texture enhancement network reaches the evaluation index, it is considered excellent and the training is completed. Otherwise, it is considered not excellent and the weight coefficient of the loss function needs to be readjusted and the texture enhancement network is retrained. Whether the texture enhancement network is excellent is judged according to the evaluation index. If it is, a successful texture enhancement network is obtained. Otherwise, the weight coefficient is updated.

2. The adaptive texture enhancement method based on global consistency and WGAN-GP according to claim 1, characterized in that: The method for expanding the number in step S1 is to rotate each grayscale image by 90°, 180°, and 270° respectively, and then flip them one by one, so as to increase the number of grayscale images by 8 times and obtain training samples for the input texture enhancement network.

3. The adaptive texture enhancement method based on global consistency and WGAN-GP according to claim 1, characterized in that: The two-stage texture enhancement network described in step S2, in the first stage, the downsampling layer convolution and pooling are used to capture global features, multiple intermediate convolution layers are used to process feature maps, and the upsampling layer transposed convolution or nearest neighbor interpolation is used to improve the spatial resolution to obtain a roughly predicted texture enhanced image; the function equation of the second stage WGAN-GP network is as follows: Among them, L is the function output result, E represents the corresponding expected value, S r is a real sample, P g is the generator generating distribution, D(S r ) is the discriminator for S r The output, S f is a false sample, P r is the real data distribution, D(S f ) is the discriminator for S f The output of , λ is the regularization coefficient, are samples from the true distribution and the generated distribution, Yes S r The gradient of , ∥·∥2 is The norm of .

4. The adaptive texture enhancement method based on global consistency and WGAN-GP according to claim 1, characterized in that: In step S4, the Adam optimizer is used to train the texture enhancement network.

5. The adaptive texture enhancement method based on global consistency and WGAN-GP according to claim 1, characterized in that: The evaluation indicators in step S5 include SSIM and PSNR.

Images