Non-local mean optimization bicubic interpolation and improved generative adversarial network combined method for brocade pattern repair

Abstract

The invention discloses a non-local mean optimization bicubic interpolation and improved generative adversarial network combined method for brocade pattern restoration, which can be used for super-resolution processing of cultural relic images, improvement of image definition, reduction of noise and enhancement of detail information so as to facilitate subsequent high-precision vectorization modeling and digital protection. Comprising the following steps of: 1) classifying brocade pattern images to be processed by adopting a brocade pattern preservation state four-level classification method; (2) carrying out image super-resolution processing on the brocade pattern images meeting the first level and the second level of the brocade pattern storage state by adopting a bicubic interpolation algorithm of non-local mean value optimization; 3) further improving the image resolution by adopting an improved generative adversarial network model for the brocade pattern image processed by the bicubic interpolation algorithm of non-local mean optimization, and 4) performing vectorization modeling on the processed brocade pattern image by adopting a vector drawing tool after the step 3).

Description

Technical Field

[0001] The present invention relates to the technical field of image processing. Specifically, it is a combined method of non-local mean optimized bicubic interpolation and improved generative adversarial network for brocade pattern restoration. Background Art

[0002] The pattern design of ancient Chinese brocade contains rich cultural heritage and artistic value, which has important inspiration for modern design. However, affected by factors such as decay, mildew, insect damage, and fading, brocade cultural relics are prone to irreversible damage during the process of circulation. Among them, a large number of brocade cultural relics currently only exist in images, and it is impossible to verify their physical objects. In order to protect the existing brocade cultural relics patterns, researchers have conducted in-depth research from multiple angles by combining modern digital and artificial intelligence technologies. According to the research theme, it can be divided into two categories: pattern restoration (digital restoration technology of patterns) and pattern modeling (modeling technology of patterns). Among them, the digital restoration technology of patterns mainly proposes solutions for the damage problem on the surface of fabric patterns; the pattern modeling technology provides solutions for the digital modeling of brocade patterns from the perspectives of preprocessing and vectorization modeling of patterns. However, due to the limitations of the original imaging equipment and imaging environment, the original brocade cultural relic images currently used for pattern vectorization modeling generally have problems such as poor imaging quality and low resolution. This seriously hinders the effective information extraction and vectorization modeling of the original brocade patterns.

[0003] Super-resolution reconstruction of low-resolution images is of great significance for the subsequent vectorization modeling of brocade patterns. Common image super-resolution processing techniques include three categories: interpolation-based methods, reconstruction-based methods, and learning-based methods. The super-resolution algorithm based on reconstruction (reconstruction-based method) is based on the image degradation model. By analyzing the image degradation process and using prior knowledge to constrain the reconstruction process, the high-resolution image can be restored. Prior knowledge is usually added in the form of regularization to help reduce noise or other unnatural artifact phenomena. The advantage of this type of method is that it can better preserve the details of the image, but it depends on a specific prior model. For some complex application scenarios, the corresponding image degradation model is usually difficult to obtain. The super-resolution algorithm based on learning (learning-based method) uses a large number of low-resolution and high-resolution image pairs for training to establish the mapping relationship between the two. This method relies on deep learning or machine learning models and can convert low-resolution images into high-resolution images through the learned experience. Compared with the interpolation method and the reconstruction method, the learning-based method has significant advantages in processing complex image details. Especially with the help of deep neural networks, it can capture richer texture and edge information and generate high-quality images. However, this type of method requires a large amount of data and computing resources, and the training process of the model is relatively time-consuming. At the same time, this type of method requires a large amount of data of a specific category for training and learning, making it difficult to perform in the case of data scarcity. In addition, since new information is added during the process of image super-resolution processing based on deep learning methods, this information may not match the details in the original image, and these errors may be transmitted to the subsequent image vector modeling process.

[0004] Interpolation-based super-resolution algorithms (interpolation-based methods) have a wider range of applications in practice because they do not rely on a large amount of training data and accurate prior models. Common interpolation algorithms include nearest-neighbor interpolation, bilinear interpolation, and bicubic interpolation. These algorithms estimate and fill in the missing pixel information in the image by using basis functions or interpolation kernels, thereby improving the resolution of the image. Among them, the bicubic interpolation method is the most classic. This method is simple to calculate and has a relatively fast speed, and is suitable for application scenarios with low requirements for image resolution. However, when dealing with complex details and textures, it is easy to cause image blurring or distortion. The essential reason is that the interpolation algorithm is a local algorithm that performs weighted summation on the neighborhood pixel values of the pixel points to be interpolated. Relevant research shows that the periodic information of the image is of great significance for maintaining image details. Different from local algorithms, non-local algorithms that utilize the periodic redundancy information of the entire image can select similar pixel neighborhood blocks within a larger range in the image to calculate the current pixel, thereby obtaining better image processing effects. Similar to the classic Criminisi algorithm in the field of image inpainting, it fills in the missing or damaged parts by finding similar image blocks in the image. The non-local means filter (NLM) proposed by Buades is an important non-local algorithm in the field of image processing. It calculates the similarity of image blocks instead of the similarity of individual pixels, enhancing the robustness to noise. At the same time, NLM utilizes the information of a larger search box or even the entire image and uses the periodicity of the image to reconstruct the current pixel, thereby effectively maintaining image details. Due to its excellent detail retention ability, NLM has a milestone significance in image processing research and has promoted the non-local improvement of many classic local algorithms.

[0005] In summary, traditional methods have poor adaptability in high-precision demand scenarios for super-resolution processing of ancient brocade patterns, resulting in serious distortion. In addition, each single image super-resolution processing algorithm has its own limitations. Therefore, developing a comprehensive image super-resolution processing technology that combines multiple image processing methods and considering the integration and optimization of different technical solutions has important application value for improving the quality and efficiency of the digital protection of brocade cultural relics. Summary of the Invention

[0006] The purpose of the present invention is to design and provide a combined method of non-local means optimized bicubic interpolation and improved generative adversarial network for brocade pattern restoration, which can be used for super-resolution processing of cultural relic images, improve image clarity, reduce noise, and enhance detail information for subsequent high-precision vectorization modeling and digital protection.

[0007] The present invention is realized through the following technical solutions: A combined method of non-local means optimized bicubic interpolation and improved generative adversarial network for brocade pattern restoration, including the following steps:

[0008] 1) Classify the brocade pattern image to be processed using a four - level classification method for preserving the state of brocade patterns;

[0009] 2) For brocade pattern images that meet the first - level and second - level preservation states of brocade patterns, perform image super - resolution processing using a bicubic interpolation algorithm optimized by non - local means; the bicubic interpolation algorithm optimized by non - local means integrates the global features of the original image into the interpolation calculation through weight adjustment to ensure that detailed information can be more accurately restored in high - resolution image generation and image enhancement tasks.

[0010] 3) For the brocade pattern image processed by the bicubic interpolation algorithm optimized by non - local means, further improve the image resolution using an improved generative adversarial network model; the improved generative adversarial network model optimizes the traditional generative adversarial network model by deepening the network layer, modifying the residual network structure and parameters.

[0011] 4) After step 3), perform vectorization modeling on the processed brocade pattern image using a vector drawing tool.

[0012] To better implement a combined method of non - local means - optimized bicubic interpolation and improved generative adversarial network for brocade pattern restoration described in the present invention, the following setting method is particularly adopted: In step 1), the brocade pattern image is specifically classified into: brocade pattern images of the first level with complete and clearly distinguishable patterns; brocade pattern images of the second level that are basically clearly distinguishable or have minor holes and complete patterns; brocade pattern images of the third level with incomplete pattern repeats but still able to be inferred based on the existing content; brocade pattern images of the fourth level with severe damage and difficult to infer the pattern repeat.

[0013] To better implement a combined method of non - local means - optimized bicubic interpolation and improved generative adversarial network for brocade pattern restoration described in the present invention, the following setting method is particularly adopted: In step 2), when the bicubic interpolation algorithm optimized by non - local means performs image super - resolution processing, it is achieved through the following formula:

[0014]

[0015] In the formula, I is the original image; I n is a pixel point in the original image I, whose abscissa and ordinate are x and y respectively, and the pixel value is I n (x, y); I′ is the image after bicubic interpolation; I′ m is the target pixel point in the image I′ after bicubic interpolation, whose abscissa and ordinate are x′ and y′ respectively, and the coordinate values after mapping its abscissa and ordinate to the original image I are respectively and The target pixel point I' m has a pixel value of I' m (x', y'); I f is the image after non - local means optimization; I″ m is the image I after non - local means optimization f and the target pixel point I″ in it m has a final pixel value of I″ m (x', y').

[0016] Furthermore, to better implement a combined method of non - local means optimized bicubic interpolation and improved generative adversarial network for brocade pattern restoration described in the present invention, the following setting method is specifically adopted: In step 3), the improved generative adversarial network model includes a generator network and a discriminator network; the generator network includes a residual network, a convolutional layer connected after the residual network, a sub - pixel convolutional layer connected after the convolutional layer, a convolutional layer and a Tanh activation layer connected after the sub - pixel convolutional layer; the discriminator network is composed of multiple layers of convolution, LeakyReLU, fully - connected layers and Sigmoid activation functions connected together.

[0017] Furthermore, to better implement a combined method of non - local means optimized bicubic interpolation and improved generative adversarial network for brocade pattern restoration described in the present invention, the following setting method is specifically adopted: The residual network includes 24 residual blocks, the convolutional layer of each residual block uses 64 3*3 convolutional kernels and has a stride of 1, and each residual block uses the ReLU activation function for non - linear mapping; the convolutional layer connected to the residual network performs convolution, mean - value summation through skip connections; the sub - pixel convolutional layer is provided with two for completing 4 - fold image upsampling.

[0018] Furthermore, to better implement a combined method of non - local means optimized bicubic interpolation and improved generative adversarial network for brocade pattern restoration described in the present invention, the following setting method is specifically adopted: In the improved generative adversarial network model, the objective function is:

[0019]

[0020] In the formula, x is the real high - resolution image; z is the real low - resolution image; E is the mathematical expectation of the real data; P data(x) is the probability distribution of the real high - resolution image; P Z(z) is the probability distribution of the real low - resolution image; D(x) is the probability that the discriminator network judges whether the real high - resolution image is real; G(z) is the high - resolution image generated by the real low - resolution image through the generator network; D(G(z)) is the probability that the discriminator network judges whether the high - resolution image generated by the generator network is real.

[0021] To better implement the combined method of non - local mean optimized bicubic interpolation and improved generative adversarial network for brocade pattern restoration described in the present invention, the following setting method is specifically adopted: In the improved generative adversarial network model, the loss function of the generator network consists of perceptual loss and adversarial loss; among them, the perceptual loss function can be expressed as:

[0022]

[0023] In the formula, W i,j and H i,j are the width and height of each feature map in the VGG network; φ i,j (I HR ) is the feature map extracted by the VGG network from the real high - resolution image; is the feature map extracted by the VGG network from the generated fake high - resolution image;

[0024] The adversarial loss can be expressed as:

[0025]

[0026] In the formula, D θD [G θG (I LR )] is the probability that the generated image G θG (I LR ) is the original high - resolution image I HR , and D θD is the discriminator network structure constructed through the parameter θ D ; N is the number of training samples.

[0027] In the actual operation process, it is found that although the generator network not guided by the mean square error (MSE) can well retain the high - frequency part in the image, the generated artifacts affect the image effect. Therefore, the loss function of the final generator network can be expressed as:

[0028]

[0029] In the formula, λ1 and λ2 are the weights corresponding to the adversarial loss and the MSE loss.

[0030] To better implement the combined method of non - local mean optimized bicubic interpolation and improved generative adversarial network for brocade pattern restoration described in the present invention, the following setting method is specifically adopted: The loss function of the discriminator network in the improved generative adversarial network model can be expressed as:

[0031]

[0032] In the formula, l D is the loss function of the discriminator network; is the fake high-resolution image R generated by the generation network according to the input low-resolution image R1 s the cross entropy between the discrimination result obtained in the discrimination network and 0; is the cross entropy between the result of the discrimination network judging the real high-resolution image R h and 1.

[0033] Furthermore, to better implement the combined method of non-local mean optimized bicubic interpolation and improved generative adversarial network for brocade pattern restoration described in the present invention, the following setting method is specifically adopted: the vector drawing tool is the Adobe Illustrator or CorelDraw vector drawing tool.

[0034] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0035] By combining the advantages of non-local mean optimized bicubic interpolation and improved generative adversarial network, the restoration effect of ancient brocade patterns can be made more delicate, with both high efficiency and fineness.

[0036] The method of the present invention has a good preprocessing effect on the low resolution, unclear and noisy original brocade pattern images.

[0037] In the present invention, by introducing non-local mean filtering into the traditional bicubic interpolation algorithm and optimizing the interpolation weights, the high-frequency details and texture information of the image can be effectively retained. At the same time, since the non-local mean optimized bicubic interpolation algorithm is not as efficient as the deep learning method for the restoration of complex image details, the image result calculated by the non-local mean optimized bicubic interpolation algorithm is further transmitted to the improved generative adversarial network model. By combining the advantages of these two technologies, the restoration effect of ancient Shu brocade patterns can be made more delicate, with both high efficiency and fineness.

[0038] The method proposed in the present invention helps to realize the vectorization restoration modeling and preservation of ancient Shu brocade patterns. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is the flowchart of the method described in the present invention.

[0040] Figure 2 is the structure diagram of the traditional generative adversarial network model.

[0041] Figure 3 is the structure diagram of the generation network in the improved generative adversarial network model of the present invention.

[0042] Figure 4 is the structure diagram of the discrimination network in the improved generative adversarial network model of the present invention.

[0043] Figure 5 These are the super-resolution effect diagrams of brocade patterns optimized by using the bicubic interpolation method, the non-local means optimized bicubic interpolation and the combined method of improved generative adversarial network respectively. Specific implementation manners

[0044] The present invention will be further described in detail below with reference to embodiments, but the implementation manners of the present invention are not limited thereto.

[0045] To make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0046] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings. They are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as a limitation of the present invention.

[0047] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, "a plurality" means two or more unless otherwise specifically defined.

[0048] In the present invention, unless otherwise clearly specified or limited, the terms "installed", "connected", "connected to", "fixed", etc. shall be understood in a broad sense. For example, it may be a fixed connection, a detachable connection, or integrated; it may be a mechanical connection or an electrical connection; it may be directly connected or indirectly connected through an intermediate medium, and it may be the communication inside two components or the interaction relationship between two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0049] In the present invention, unless otherwise clearly specified or limited, the first feature being "above" or "below" the second feature may include the direct contact between the first and second features, or may include the situation where the first and second features are not in direct contact but in contact through additional features therebetween. Moreover, the first feature being "above", "over" and "on top of" the second feature includes that the first feature is directly above and obliquely above the second feature, or merely means that the horizontal height of the first feature is higher than that of the second feature. The first feature being "below", "beneath" and "underneath" the second feature includes that the first feature is directly below and obliquely below the second feature, or merely means that the horizontal height of the first feature is lower than that of the second feature.

[0050] Glossary:

[0051] MSE: Abbreviation for mean-square error, the mean square error, which is a measure reflecting the degree of difference between the estimator and the estimated quantity.

[0052] Feature Map: The Feature Map is an output matrix obtained by the action of the input data through a convolution kernel (filter) in a convolutional layer or a pooling layer, and is used to represent the local features of the input data.

[0053] Stride: Stride (the step size) refers to the step length of the convolution kernel sliding on the input tensor (such as an image), usually represented by an integer. The choice of Stride will affect the size change of the Feature Map, and thus affect the performance of the generation network and the discriminant network.

[0054] LeakyReLU: (Leaky Rectified Linear Unit, the leaky ReLU) is an improved version of ReLU (Rectified Linear Unit, the rectified linear unit), which allows a small output when the input is negative to avoid the "neuron death" problem.

[0055] SRGAN: SRGAN (Super-Resolution Generative Adversarial Network) is a super-resolution (SR) method based on the generative adversarial network (GAN).

[0056] Example 1:

[0057] A combined method of non-local mean optimized bicubic interpolation and improved generative adversarial network for brocade pattern restoration, which can be used for super-resolution processing of cultural relic images, improve image clarity, reduce noise, and enhance detail information for subsequent high-precision vectorization modeling and digital protection, including the following steps:

[0058] 1) Classify the brocade pattern images to be processed using a four-level classification method for the preservation status of brocade patterns.

[0059] 2) For brocade pattern images that meet the first and second levels of the preservation status of brocade patterns, use the non-local mean optimized bicubic interpolation algorithm for image super-resolution processing; the non-local mean optimized bicubic interpolation algorithm integrates the global features of the original image into the interpolation calculation through weight adjustment to ensure that detail information can be more accurately restored in high-resolution image generation and image enhancement tasks.

[0060] 3) For the brocade pattern images processed by the non-local mean optimized bicubic interpolation algorithm, use an improved generative adversarial network model to further improve the image resolution; the improved generative adversarial network model optimizes the traditional generative adversarial network model by deepening the network hierarchy, modifying the residual network structure and parameters.

[0061] 4) After step 3), use a vector drawing tool to perform vectorization modeling on the processed brocade pattern images.

[0062] Example 2:

[0063] This embodiment is further optimized on the basis of the above embodiment. The same parts as the foregoing technical solutions will not be described herein again. To better implement a combined method of non-local mean optimized bicubic interpolation and improved generative adversarial network for brocade pattern restoration of the present invention, the following setting method is particularly adopted: In step 1), the brocade pattern images are specifically classified into: brocade pattern images of the first level with complete and clearly distinguishable repeats; brocade pattern images of the second level that are basically clearly distinguishable or have minor holes and have complete repeats; brocade pattern images of the third level with incomplete repeat cycles but still able to be inferred based on the existing content; and brocade pattern images of the fourth level with severe damage and difficult to infer the repeat cycle.

[0064] Example 3:

[0065] This embodiment is further optimized on the basis of any of the above embodiments. The same parts as the foregoing technical solutions will not be described herein again. To better implement a combined method of non-local mean optimized bicubic interpolation and improved generative adversarial network for brocade pattern restoration according to the present invention, the following setting method is particularly adopted: In step 2), when the non-local mean optimized bicubic interpolation algorithm performs super-resolution processing on an image, it is implemented by the following formula:

[0066]

[0067] In the formula, I is the original image; I n is a pixel point in the original image I, whose abscissa and ordinate are x and y respectively, and the pixel value is I n (x, y); I' is the image after bicubic interpolation; I' m is the target pixel point in the image I' after bicubic interpolation, whose abscissa and ordinate are x' and y' respectively, and the coordinate values after mapping its abscissa and ordinate to the original image I are and The pixel value of the target pixel point I' m is I' m (x', y'); I f is the image after non-local mean optimization; I'' m is the target pixel point I'' in the image I f after non-local mean optimization, and its final pixel value is I'' m (x', y'). m (x', y').

[0068] Example 4:

[0069] This embodiment is further optimized on the basis of any of the above embodiments. The same parts as the foregoing technical solutions will not be described herein again. To better implement a combined method of non-local mean optimized bicubic interpolation and improved generative adversarial network for brocade pattern restoration according to the present invention, the following setting method is particularly adopted: In step 3), the improved generative adversarial network model includes a generator network and a discriminator network; the generator network includes a residual network, a convolutional layer connected after the residual network, a sub-pixel convolutional layer connected after the convolutional layer, a convolutional layer and a Tanh activation layer connected after the sub-pixel convolutional layer; the discriminator network is connected by multiple layers of convolution, LeakyReLU, fully connected layer and Sigmoid activation function.

[0070] Example 5:

[0071] This embodiment is a further optimization based on any of the above embodiments. The same parts as the foregoing technical solutions will not be described herein again. To better implement a combined method of non-local mean optimized bicubic interpolation and improved generative adversarial network for brocade pattern restoration according to the present invention, the following setting method is particularly adopted: The residual network includes 24 residual blocks. The convolutional layer of each residual block uses 64 3*3 convolutional kernels with a stride of 1, and each residual block uses the ReLU activation function for non-linear mapping; The convolutional layer connected to the residual network performs convolution to calculate the mean and sum through skip connections; The sub-pixel convolutional layer is provided with two for completing 4-fold image upsampling.

[0072] Embodiment 6:

[0073] This embodiment is a further optimization based on any of the above embodiments. The same parts as the foregoing technical solutions will not be described herein again. To better implement a combined method of non-local mean optimized bicubic interpolation and improved generative adversarial network for brocade pattern restoration according to the present invention, the following setting method is particularly adopted: In the improved generative adversarial network model, the objective function is:

[0074]

[0075] In the formula, x is the real high-resolution image; z is the real low-resolution image; E is the mathematical expectation of the real data; P data(x) is the probability distribution of the real high-resolution image; P Z(z) is the probability distribution of the real low-resolution image; D(x) is the probability that the discriminative network judges whether the real high-resolution image is real; G(z) is the high-resolution image generated by the real low-resolution image through the generative network; D(G(z)) is the probability that the discriminative network judges whether the high-resolution image generated by the generative network is real.

[0076] Embodiment 7:

[0077] This embodiment is a further optimization based on any of the above embodiments. The same parts as the foregoing technical solutions will not be described herein again. To better implement a combined method of non-local mean optimized bicubic interpolation and improved generative adversarial network for brocade pattern restoration according to the present invention, the following setting method is particularly adopted: In the improved generative adversarial network model, the loss function of the generative network consists of perceptual loss and adversarial loss; Among them, the perceptual loss function can be expressed as:

[0078]

[0079] In the formula, W i,j and H i,j are the width and height of each feature map in the VGG network; φ i,j(I HR ) is the feature map extracted from the real high-resolution image through the VGG network; is the feature map extracted from the generated fake high-resolution image through the VGG network;

[0080] The adversarial loss can be expressed as:

[0081]

[0082] In the formula, D θD [G θG (I LR )] is the probability that the generated image G θG (I LR ) is the original high-resolution image I HR , and D θD is the discriminative network structure constructed by the parameter θ D ; N is the number of training samples.

[0083] In the actual operation process, it is found that the generation network not guided by the mean squared error (MSE) can well retain the high-frequency part in the image, but the generated artifacts affect the image effect. Therefore, the loss function of the final generation network can be expressed as:

[0084]

[0085] In the formula, λ1 and λ2 are the weights corresponding to the adversarial loss and the MSE loss.

[0086] Example 8:

[0087] This example is further optimized on the basis of any of the above examples. The same parts as the foregoing technical solutions will not be repeated here. Further, to better implement the non-local mean optimized bicubic interpolation and improved generative adversarial network joint method for brocade pattern restoration described in the present invention, the following setting method is particularly adopted: The loss function of the discriminative network in the improved generative adversarial network model can be expressed as:

[0088]

[0089] In the formula, l D is the loss function of the discriminative network; is the cross entropy of 0 and the discriminative result obtained in the discriminative network for the fake high-resolution image R s generated by the generation network according to the input low-resolution picture R1; is the cross entropy of 1 and the result of the discriminative network judging the real high-resolution picture R h .

[0090] Example 9:

[0091] This embodiment is further optimized on the basis of any of the above embodiments. The same parts as the foregoing technical solutions will not be described herein again. To better implement the combined method of non-local mean optimized bicubic interpolation and improved generative adversarial network for brocade pattern restoration of the present invention, the following setting method is particularly adopted: The vector drawing tool is Adobe Illustrator or CorelDraw vector drawing tool.

[0092] Embodiment 10:

[0093] A combined method of non-local mean optimized bicubic interpolation and improved generative adversarial network for super-resolution restoration of brocade patterns, combined with Figures 1 to 5 As shown, it includes the following steps:

[0094] S1: Classify the preprocessed pattern image (the brocade pattern image to be processed) using the four-level classification method for the preservation state of brocade patterns.

[0095] S2: Select the patterns in the first and second levels from the original brocade pattern image classified in step S1, and perform image super-resolution processing using the non-local mean optimized bicubic interpolation algorithm.

[0096] S3: Further process the brocade pattern image processed in step S2 using an improved generative adversarial network model to improve the image resolution.

[0097] S4: After step S3, use a vector drawing tool to perform vectorization modeling on the processed brocade pattern image.

[0098] The four-level classification method for the preservation state of brocade patterns specifically classifies the brocade pattern image into four levels. The first level is the brocade pattern image with a complete and clearly distinguishable pattern repeat; the second level is the brocade pattern image that is basically clearly distinguishable or has minor holes and a complete pattern repeat; the third level is the brocade pattern image with an incomplete pattern repeat cycle and still able to be inferred based on the existing content; the fourth level is the brocade pattern image with serious damage and difficult to infer the pattern repeat cycle.

[0099] The basic calculation formula for non-local mean filtering is:

[0100]

[0101] In the formula: NL[v](i) represents the filtering result at position i, that is, the pixel value after non-local mean filtering. v(j) represents the original pixel value at position j. N i and N j respectively represent the image blocks centered on pixels i and j, that is, the pixel sets of the local neighborhood window. represents the image block N i and Nj The similarity metric value between them is used to measure the similarity between image patches. The parameter h is a smoothing parameter that controls the attenuation degree of the weight and is usually called the filtering parameter. If h is small, the weight will be more concentrated on the pixels that are very similar to the central pixel patch. If h is large, the weight will be more dispersed, taking into account the influence of more distant pixels. C(i) is a normalization constant representing the sum of all weights, which is used to ensure that the pixel value after filtering is still within the effective gray value range.

[0102] To quantitatively calculate the similarity between the gray value vectors v(N i ) and v(N j ), the Euclidean distance with two-dimensional Gaussian weighting is adopted here There are two reasons for this: First, when performing similarity comparison, the gray value vectors are from the original image, and the gray values are affected by noise. The Euclidean distance calculated directly using the noisy gray values is difficult to accurately reflect the similarity between pixels. Through Gaussian weighting, it is equivalent to performing Gaussian smoothing on the sub-images within the similarity window, which can effectively suppress noise and thus improve the accuracy of the Euclidean distance calculation. Second, usually, the closer the spatial positions of pixels are, the stronger the correlation. Therefore, the pixels closer to the central pixel should have a higher weight in the similarity comparison.

[0103] Equation 1 can also be written in the following form:

[0104]

[0105] In the formula, ω(i,j) is the weight between image patches N i and N j . The weight {ω(i,j)} j depends on the similarity between pixel i and pixel j and satisfies the following conditions: 0 ≤ ω(i,j) ≤ 1, and ∑ j ω(i,j) = 1.

[0106] To solve the problem of image detail loss or blurring caused by the traditional bicubic interpolation algorithm relying only on local information of adjacent pixels, the present invention proposes to optimize the bicubic interpolation algorithm with non-local means and proposes a non-local means optimized bicubic interpolation algorithm for image super-resolution processing. By adjusting the weight, the global features of the original image are incorporated into the interpolation calculation to ensure that in the tasks of high-resolution image generation and image enhancement, the detail information can be more accurately restored.

[0107] Redefine a pixel point in the original image I as I n , whose abscissa and ordinate are x and y respectively, and the pixel value is I n (x, y). The target pixel point I' in the image I' after bicubic interpolation m, where the abscissa and ordinate are \(x'\) and \(y'\) respectively, and the coordinate values after mapping its abscissa and ordinate to the original image are and The pixel value of the target pixel point \(I'\) m is \(I'\) m (x', y'). The image after non - local mean optimization is \(I\) f The target pixel point \(I''\) in m , and its final pixel value is \(I''\) m (x', y').

[0108] The pixel value \(I'\) after bicubic interpolation m (x', y') is calculated by the formula:

[0109]

[0110] The weight calculation formula after non - local mean filtering optimization is:

[0111]

[0112] In the formula, \(N\) m , \(N\) z respectively represent the pixel blocks centered on the pixel points \(I'\) m and \(I'\) z . The Euclidean distance with two - dimensional Gaussian weighting is used to define the similarity metric value between pixel blocks. As shown in the following formula:

[0113]

[0114] In the formula, \(K=\{(k1, k2)||k1|\leq n, |k2|\leq n\}\), \(n\) is the radius of the neighborhood window, and

[0115]

[0116] In the formula, \(\alpha\) is the standard deviation, which controls the width of the Gaussian function. \(1 / 2\pi\alpha\) 2 is the normalization constant to ensure that the integral of the Gaussian function is 1. \(\exp[-(k1 2 + k2 2 ) / 2\alpha 2 is the core part of the Gaussian function, which defines a bell - shaped curve centered at the origin and with a width controlled by \(\alpha\). \(k1\) and \(k2\) are the components of the two - dimensional coordinates, representing positions in space. The Gaussian function reaches its maximum value at \(k1 = 0,k2 = 0\) and decays rapidly as \(k1\) and \(k2\) increase.

[0117] Therefore, the expression of the non - local mean optimized bicubic interpolation (NL - Bicubic) algorithm (model) is:

[0118]

[0119] By definition, the value of ω(I′ m ,I′ z ) decreases as the Gaussian weighted Euclidean distance between image patches increases. Therefore, the larger the value of ω(I′ m ,I′ z ), the higher the similarity between image patches and the greater the impact on pixel reconstruction.

[0120] Although the pattern image processed by the bicubic interpolation algorithm optimized by non-local means can improve the image blurring or distortion problems caused by the traditional bicubic interpolation algorithm when dealing with complex details and textures, the obtained image cannot achieve a high-quality super-resolution restoration effect. Currently, digital image processing technologies based on deep learning techniques have been widely applied to image degradation problems and achieved good results. The present invention further combines an improved generative adversarial network model on the basis of the image results obtained by the above method to achieve further super-resolution restoration of the image.

[0121] Based on the traditional generative adversarial network model, the present invention optimizes the traditional generative adversarial network model by deepening the network hierarchy, modifying the residual network structure and parameters. The structure of the traditional generative adversarial network model is as shown in the appendix Figure 2 . Among them, the input low-resolution image (z ∼ p(z)) is generated and output as a high-resolution image through the generative network G, and the discriminative network D judges the authenticity of the generated high-resolution image. Through the configuration of the loss function feedback and the optimization algorithm, the weights and offsets of the generative network are adjusted to improve the ability of the generative network G to generate high-resolution pictures. At the same time, the discriminative network D is input with high-resolution images to continuously improve its discriminative ability. During the adversarial training iteration process, the parameters in the network are continuously updated based on the training set data, and finally a high-performance improved generative adversarial network model is constructed.

[0122] In order to reduce the loss of details in the original image during the super-resolution process, the generative network of the improved generative adversarial network model of the present invention adopts 24 residual blocks, and at the same time removes the batch normalization layer (BN layer) to save memory and improve network performance. The final generative network is as shown in the appendix Figure 3 . The features of the low-resolution picture are extracted through a convolutional network. The convolutional layer uses 64 3×3 convolutional kernels with a stride of 1, and the Relu activation function is used for non-linear mapping. Subsequently, it passes through 24 residual blocks. After the residual network, there is a convolutional layer, and the convolution is averaged and summed through skip connections. Then, two sub-pixel convolutional layers are connected to complete 4-fold image upsampling. Finally, an image super-resolution reconstruction is completed through a convolutional layer and a tanh activation layer.

[0123] In the discriminative network, the convolution kernel size, the number of Feature Maps, and the Stride size have an important impact on the discriminative quality of the network. Therefore, in order to increase the receptive field, the discriminative network of the improved generative adversarial network model uses a 4×4 convolution kernel for convolution operations. In addition, operations such as multi-layer convolution, LeakyReLU, fully connected layer, and Sigmoid activation function are used to output the discriminative result. Specifically, as shown in Appendix Figure 4 As shown, the first six layers of the discriminative network structure use a 4×4 convolution kernel for image feature extraction. In order to increase the extracted feature information and the number of Feature Maps, the 7th to 9th layer networks use a 1×1 convolution layer to reduce the input channels and convolution kernel parameters to achieve dimensionality reduction, while reducing the computational amount. Finally, the discriminative result is output through the Flatten-Layer, fully connected layer (FC), and Sigmoid activation function. The discriminative result is used to analyze the gap between the generated data and the real data, and then the network parameter values are optimized through the backpropagation function to continuously improve the discriminative ability of the model until the discriminative network can no longer distinguish the high-resolution images generated by the generative network from the real high-resolution images, and the training is completed.

[0124] The traditional generative adversarial network model optimizes the discriminative network and the generative network based on the max-min game idea. When optimizing the discriminative network, the generative network is fixed. It is required that when the real high-resolution picture x is input, the result of the generative network is as large as possible, and for the generated fake sample picture G(z), the discriminative result is as small as possible, that is, D(G(z)) is as small as possible. Since the optimization objectives of the first term and the second term are contradictory, the second term is changed to 1 - D(G(z)). When optimizing the generative network, it has nothing to do with the real image samples, so there is no need to consider it. At this time, there are only fake samples generated by the generative network, but the generative network G hopes that the discriminative result of the fake samples takes a higher value, so D(G(z)) takes a larger value. However, in order to unify it into the form of 1 - D(G(z)), 1 - D(G(z)) is minimized, and the two optimization models are combined into the final objective function, as shown in Equation 12:

[0125]

[0126] In the formula, x is the real high-resolution image; z is the real low-resolution image; E is the mathematical expectation of the real data; P data(x) is the probability distribution of the real high-resolution image; P Z(z) is the probability distribution of the real low-resolution image; D(x) is the probability that the discriminative network judges whether the real high-resolution image is real; G(z) is the high-resolution image generated by the real low-resolution image through the generative network; D(G(z)) is the probability that the discriminative network judges whether the high-resolution image generated by the generative network is real.

[0127] The loss function judges the performance of the network by weighing the gap between the generated image and the original high-resolution image. During the backpropagation process, the loss function continuously optimizes the network by modifying the weight parameters.

[0128] The loss function of the generator network in SRGAN consists of perceptual loss and adversarial loss. The perceptual loss function can be expressed as:

[0129]

[0130] In the formula, W i,j and H i,j are the width and height of each feature map in the VGG network; φ i,j (I HR ) is the feature map extracted by the VGG network from the real high-resolution image; is the feature map extracted by the VGG network from the generated fake high-resolution image.

[0131] The adversarial loss can be expressed as:

[0132]

[0133] In the formula, D θD [G θG (I LR )] is the probability that the generated image G θG (I LR ) is the original high-resolution image I HR ; D θD is the discriminator network constructed by parameters θ D ; N is the number of training samples. It is found during the actual operation that the generator network not guided by the mean square error (MSE) can well retain the high-frequency part in the image, but the generated artifacts affect the image effect. Therefore, the loss function of the final generator network can be expressed as:

[0134]

[0135] In the formula, λ1 and λ2 are the weights corresponding to the adversarial loss and the MSE loss.

[0136] The loss function of the discriminator network in SRGAN can be expressed as:

[0137]

[0138] In the formula, l D is the loss function of the discriminator network. is the cross-entropy between 0 and the discrimination result obtained by the fake high-resolution image R s generated by the generator network according to the input low-resolution picture R1 in the discriminator network; For the discriminative network to judge the result of the cross-entropy between the real high-resolution image R h and 1.

[0139] The brocade pattern image optimized by the improved generative adversarial network model is as shown in the appendix Figure 5 as follows.

[0140] The above are only the preferred embodiments of the present invention, and do not impose any formal restrictions on the present invention. Any simple modifications or equivalent changes made to the above embodiments based on the technical essence of the present invention all fall within the protection scope of the present invention.

Claims

1. A joint method of non - local mean optimized bicubic interpolation and improved generative adversarial network for brocade pattern restoration, characterized in that: It includes the following steps: 1) Classify the to-be-processed brocade pattern image using the four-level classification method for the preservation status of brocade patterns; 2) For the brocade pattern images that meet the first-level and second-level preservation status of brocade patterns, use the bicubic interpolation algorithm optimized by non-local means for image super-resolution processing; 3) For the brocade pattern images processed by the bicubic interpolation algorithm optimized by non-local means, use an improved generative adversarial network model to further improve the image resolution; 4) After step 3), use a vector drawing tool to perform vectorization modeling on the processed brocade pattern image.

2. A joint method of non - local mean - optimized bicubic interpolation and improved generative adversarial network for brocade pattern restoration according to claim 1, characterized in that: In step 1), the brocade pattern image is specifically classified into: the first-level brocade pattern image with a complete and clearly distinguishable motif repeat; the second-level brocade pattern image that is basically clearly distinguishable or has minor holes and a complete motif repeat; the third-level brocade pattern image with an incomplete motif repeat cycle but still able to be inferred based on the existing content; the fourth-level brocade pattern image with severe damage and difficult to infer the motif repeat cycle.

3. A combined method of non - local mean optimized bicubic interpolation and improved generative adversarial network for brocade pattern restoration according to claim 1, characterized in that: In step 2), when the bicubic interpolation algorithm optimized by non-local means performs image super-resolution processing, it is achieved through the following formula: Where, I is the original image; I n is a pixel point in the original image I, whose abscissa and ordinate are x and y respectively, and the pixel value is I n (x, y); I' is the image after bicubic interpolation; I' m is the target pixel point in the image I' after bicubic interpolation, whose abscissa and ordinate are x' and y' respectively, and the coordinate values after mapping its abscissa and ordinate to the original image I are respectively and The pixel value of the target pixel point I' m is I' m (x', y'); I f is the image after non - local means optimization; I'' m is the target pixel point I'' in the image I f after non - local means optimization, and its final pixel value is I'' m (x', y'). m (x', y').

4. A combined method of non - local mean - optimized bicubic interpolation and improved generative adversarial network for brocade pattern restoration according to claim 1 or 2 or 3, characterized in that: In step 3), the improved generative adversarial network model includes a generator network and a discriminator network; the generator network includes a residual network, a convolutional layer connected after the residual network, a sub-pixel convolutional layer connected after the convolutional layer, a convolutional layer and a Tanh activation layer connected after the sub-pixel convolutional layer; the discriminator network is connected by multiple layers of convolution, LeakyReLU, fully connected layers, and a Sigmoid activation function.

5. A combined method of non - local mean optimized bicubic interpolation and improved generative adversarial network for brocade pattern restoration according to claim 4, characterized in that: The residual network includes 24 residual blocks. The convolutional layer of each residual block uses 64 3*3 convolutional kernels with a stride of 1, and each residual block uses a ReLU activation function for non-linear mapping; the convolutional layer connected to the residual network performs convolution to calculate the mean sum through skip connections; the sub-pixel convolutional layer is provided with two for completing 4-fold image upsampling.

6. A joint method of non - local mean - optimized bicubic interpolation and improved generative adversarial network for brocade pattern restoration according to claim 4, characterized in that: In the improved generative adversarial network model, the objective function is: Where x is the true high-resolution image; z is the true low-resolution image; E is the mathematical expectation of the true data; P data(x) is the probability distribution of the true high-resolution image; P Z(z) is the probability distribution of the true low-resolution image; D(x) is the probability that the discriminative network judges whether the true high-resolution image is real; G(z) is the high-resolution image generated by the true low-resolution image through the generative network; D(G(z)) is the probability that the discriminative network judges whether the high-resolution image generated by the generative network is real.

7. A combined method of non - local mean optimized bicubic interpolation and improved generative adversarial network for brocade pattern restoration according to claim 4, characterized in that: In the improved generative adversarial network model, the loss function of the generator network consists of a perceptual loss and an adversarial loss; where the perceptual loss function is expressed as: Where, W i,j and H i,j are the width and height of each feature map in the VGG network; φ i,j (I HR ) is the feature map extracted from the real high-resolution image through the VGG network; is the feature map extracted from the generated fake high-resolution image through the VGG network; The adversarial loss is expressed as: where D θD [G θG (I LR )] is the generated image G θG (I LR ) is the original high - resolution image I HR 's probability, and D θD is the discriminative network constructed with parameters θ D ; N is the number of training samples; The final loss function of the generator network is expressed as: In the formula, λ1 and λ2 are the weights corresponding to the adversarial loss and the MSE loss.

8. A combined method of non - local mean optimized bicubic interpolation and improved generative adversarial network for brocade pattern restoration according to claim 4, characterized in that: In the improved generative adversarial network model, the loss function of the discriminator network is expressed as: where l D is the loss function of the discriminative network; is the fake high-resolution image R generated by the generation network according to the input low-resolution image R1 s which is the cross-entropy between the discrimination result obtained in the discriminative network and 0; is the cross-entropy between the result of the discriminative network judging the real high-resolution image R h and 1.

9. A combined method of non - local mean optimized bicubic interpolation and improved generative adversarial network for brocade pattern restoration according to claim 1 or 2 or 3 or 5 or 6 or 7 or 8, characterized in that: The vector drawing tool is the Adobe Illustrator or CorelDraw vector drawing tool.

Images