A method for automatic generation of target style images based on texture constraints

Abstract

The present invention discloses a method for automatically generating a target style image based on texture constraints, constructs a target style image generation network to generate a target style image, and by introducing a texture constraint network into the target style image generation network, can extract and learn high-level features of the target style image with higher quality, making the generated style image clearer and rich in detail depth. At the same time, when converting a content image into a target style image, the present invention also performs network training on converting the target style image into a content image, and by minimizing the difference between the generated content image and the real content image, further enhances the migration performance of the target style image generation network, and improves the effect of the target style image generation network in converting the content image into the target style image; in addition, the present invention can quickly and massively generate embroidery art style images, meeting people's personalized needs for embroidery art works.

Description

Technical Field

[0001] The invention belongs to the field of computer technology, and in particular relates to a method for automatically generating a target style image based on texture constraints. Background Art

[0002] With the continuous development of computer graphics in recent years, many scholars at home and abroad have adopted image stylization algorithms to simulate various artistic styles. This is a computer image processing technology that retains common image content and applies it to a certain artistic style image. There are two main methods: non-realistic rendering and neural style transfer. Image stylization is popular because it can convert various artistic styles. This technology has great potential in the application of image and video fields, and it is likely to subvert the production method of works of art in the future. The existing image style transfer algorithms have achieved good conversion effects on some images, but in many cases, the results of style transfer are still difficult to meet the requirements. There are still many problems with style transfer algorithms. For example, the style transfer process takes too long and the time cost is too high; the style feature extraction of images with small color changes is incomplete, resulting in unclear style of the converted image; different objects in a picture will be confused during style conversion, so that the generated image will be obviously distorted; the content feature extraction of the image is not thorough, resulting in obvious distortion of the generated image content. Taking embroidery style as an example, although some scholars have proposed style transfer algorithms for embroidery artworks, the embroidery images generated by existing methods have shortcomings such as single style, rough features, and chaotic colors. The scope of application is limited and does not meet the actual needs of embroidery industrial manufacturing. Summary of the invention

[0003] The purpose of the present invention is to provide a method for automatically generating a target style image based on texture constraints.

[0004] The present invention provides a method for automatically generating a target style image based on texture constraints, which comprises the following steps:

[0005] Step 1: Create a data set including content images and style images; the style of the style image is consistent with the target style;

[0006] Step 2: Construct a target style image generation network; the target style image generation network includes a generator G and a discriminator D Y , generator F, discriminator D X and texture constraint network; the generator G is used to obtain a synthetic style image based on the input real content image and real style image; the content of the synthetic style image is consistent with the content image; the style of the synthetic style image is consistent with the style image; the discriminator D YIt is used to distinguish the synthetic style images generated by the generator G from the real style images; the structure of the generator G is the same as that of the generator F; the discriminator D X and the discriminator D Y The structure of is the same as that of ; the generator F is used to obtain a synthetic content image according to the synthetic style image and the real content image generated by the generator G; the content of the synthetic content image is consistent with the content of the synthetic style image; the style of the synthetic content image is consistent with the style of the real content image; the discriminator D X Used to distinguish true from false between synthetic content images and real content images;

[0007] The texture constraint network obtains high-level semantic features by performing multi-level feature extraction and fusion on the input style image; the high-level semantic features are used as texture features corresponding to the style image; the texture features output by the texture constraint network are fused with the image generated by the generator G;

[0008] Step 3: Use the content images in the dataset as real samples in the X domain and the style images as real samples in the Y domain to train the target style image generation network. During the training process, input the real samples in the X domain and the real samples in the Y domain into the generator G one by one to generate new samples in the Y domain, and use the discriminator D Y Distinguish the generated Y domain new samples from the real samples in the Y domain; generate the generated Y domain new samples and the real samples in the X domain through the generator F to generate the X domain new samples, and use the discriminator D X Distinguish the generated X-domain newly added samples from the real X-domain samples; update the target style image generation network through the loss function to obtain the optimal target style image generation network;

[0009] Step 4: By inputting the target content image into the target style image generation network, the image generated by the generator G is fused with the texture features input by the texture constraint network to obtain an output image with content consistent with the target content image and with the target style.

[0010] Preferably, the texture constraint network comprises a recognition module, a synthesis module, a feature extraction module, a convolution processing module and a fully connected layer connected in sequence; the recognition module comprises multiple convolution layers connected in sequence, the convolution kernels in the multiple convolution layers are of equal size, the number of channels increases in sequence, and the convolution layers with different numbers of channels are connected through a maximum pooling layer to extract multi-level features in the target style image and obtain a feature image; the synthesis module comprises four convolution layers, the first three convolution layers have the same structure, and a batch normalization layer and a ReLu activation function are connected after each convolution layer; the last convolution layer is connected after a Tanh activation function; the synthesis module is integrated with the recognition module to extract The multi-level features of the image are used to obtain the synthetic feature image; the feature extraction module includes multiple convolutional layers, the sizes of the convolution kernels in the multiple convolutional layers are equal, the number of channels increases successively, and the convolutional layers with different numbers of channels are connected through the maximum pooling layer to further extract the high-level semantic features in the synthetic feature image; the convolution processing module includes two convolutional layers, the first convolutional layer is connected to the ReLu activation function; the second convolutional layer is connected to the batch normalization layer and the ReLu activation function; the convolution processing module selects and simplifies the features extracted by the feature extraction module; the features extracted by the convolution processing module are integrated through the fully connected layer, and the texture features are obtained by calculating the Hinge loss function.

[0011] Preferably, in step 4, the loss function L constructed total The expression is:

[0012] L total =L GAN (G,D Y ,X,Y)+L GAN (F,D X ,Y,X)+aL cyc (G,F)+bL identity (G,F)+cL textkure (G)

[0013] Among them, L GAN (G,D Y ,X,Y) is the adversarial loss function of the style image generated in the generator G; L GAN (F,D X ,Y,X) is the adversarial loss function of the content image generated by the generator F; L cyc (G, F) is the reconstruction loss function; L identity (G,F) is the identity loss function; L textkure (G) is the texture loss function; a is the cycle consistency loss weight; b is the identity loss weight; c is the texture loss weight;

[0014] Adversarial loss function L GAN (G,D Y,X,Y) is:

[0015] L GAN (G,D Y ,X,Y)=E y～Pdata(y) [logD Y (y)]+E x～Pdata(x) [log(1-D Y (G(x))]

[0016] Among them, x is the real content image; E x～Pdata(x) is the mathematical expectation of the real content image x; y is the real style image; E y～Pdata(y) is the mathematical expectation of the true style image y; D Y (y) is the discriminator D Y The discrimination result of the real style image y; G(x) is the style image generated by the generator G for the real content image x; D Y (G(x)) is the discriminator D Y The discriminant result of the generated style image G(x);

[0017] Adversarial loss function L GAN (F,D X ,Y,X) is:

[0018] L GAN (F,D X ,Y,X)=E x～Pdata(x) [logD X (x)]+E y～Pdata(y) [log(1-D X (F(y))]

[0019] Among them, D X (x) is the discriminator D X The discrimination result of the real content image x; F(y) is the content image generated by the generator F for the real style image y; D X (F(y)) is the discriminator D X The judgment result of the generated content image F(y);

[0020] Reconstruction loss function L cyc The expression of (G,F) is:

[0021] L cyc (G,F)=E x～Pdata(x) [||F(G(x))-x|| 1 ]+E y～Pdata(y) [||G(F(x))-y|| 1 ]

[0022] Where F(G(x)) is the reconstructed image generated by generator F for G(x);

[0023] Identity loss function L identity The expression of (G,F) is:

[0024] L identity (G,F)=E y～Pdata(y) [||(G(y)-y|| 1 ]+E x～Pdata(x) [||(F(x)-x|| 1 ]

[0025] Texture loss function L textkure The expression of (G) is:

[0026]

[0027] Where D, H, W are the depth, width and height of the feature map respectively; Φ is the feature extraction function.

[0028] Preferably, the target style image adopts an embroidery style image.

[0029] Preferably, in the step 2, the generator G includes an encoder and a decoder; the encoder includes a content encoder and a style encoder; the content encoder is used to extract content features of the image based on the input image, including image shape, edges and structure; the style encoder is used to extract style features of the image based on the input image, including texture, strokes, color and lines; the decoder is used to generate an image based on the extracted content features and style features.

[0030] Preferably, the content encoder comprises a reflection filling layer, a convolution processing layer, two downsampling layers and six residual blocks connected in sequence; the reflection filling layer performs reflection filling on the input image to retain the edge information of the image; the convolution processing layer comprises a convolution layer, a BN layer and a ReLU activation function connected in sequence; the two downsampling layers each comprise a convolution layer, an IN normalization layer and a ReLU activation function; the two downsampling layers are used to reduce the size of the feature map and improve the network's ability to perceive global information; the six residual blocks each comprise two convolution layers, a BN normalization layer and a ReLU activation function, and finally output the content code;

[0031] The style encoder includes six convolution processing layers, a pooling layer and a fully connected layer connected in sequence; the six convolution processing layers are used to extract the style features of the image, and they all include a convolution layer, a BN layer and a ReLu activation function; the pooling layer is used to reduce the dimension of the feature map; the fully connected layer is used to integrate the features extracted by the convolution layer and the pooling layer, and output the style code.

[0032] Preferably, the decoder comprises a multilayer perceptron, an adaptive instance normalization layer, four residual blocks, two upsampling layers, a reflection layer, a convolution layer and a Tanh activation function connected in sequence; the style code is input into the multilayer perceptron to obtain the parameters of the adaptive instance normalization layer; the content code is processed using the four residual blocks introducing adaptive instance normalization; and the two upsampling layers both include a deconvolution layer, a BN normalization layer and a ReLU activation function.

[0033] Preferably, the discriminator comprises multiple convolution processing layers connected in sequence; except for the last convolution processing layer, the structures of the remaining convolution processing layers are the same, and all include a convolution layer, an IN layer and a Leaky ReLU activation function; the last convolution processing layer includes a convolution layer and a Sigmoid activation function; the discriminator's discrimination method is: divide the image into multiple blocks, and input each block into the discriminator for true or false discrimination.

[0034] In a second aspect, the present invention provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the memory stores the computer program; and the processor executes the above-mentioned method for automatically generating a target style image.

[0035] In a third aspect, the present invention provides a readable storage medium storing a computer program; when the computer program is executed by a processor, it is used to implement the above-mentioned method for automatically generating a target style image.

[0036] The present invention has the following beneficial effects:

[0037] 1. The present invention constructs a target style image generation network to generate a target style image, and introduces a texture constraint network into the target style image generation network, which can extract and learn high-level features of the target style image with higher quality, making the generated style image clearer and rich in detail depth.

[0038] 2. When training the target style image generation network to convert the content image into the target style image, the present invention also trains the conversion of the target style image into the content image. By minimizing the difference between the generated content image and the real content image, the style transfer performance of the target style image generation network is further enhanced, thereby improving the effect of the target style image generation network in converting the content image into the target style image.

[0039] 3. The present invention uses embroidery style images as target style images, so as to quickly and massively generate embroidery art style images, shorten the process flow, reduce costs, and meet people's personalized needs for embroidery artworks. At the same time, by generating embroidery artworks through the technology in the present invention, more artistic treasures can be preserved, and existing embroidery cultural relics can be used to generate new embroidery works, which is of great significance to the protection and inheritance of embroidery culture. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is the overall flow chart of the present invention.

[0041] Figure 2 Schematic diagram of the target style image generation network constructed in the present invention.

[0042] Figure 3 FIG. 4 is a schematic diagram of the network architecture of the content encoder in the present invention.

[0043] Figure 4 Schematic diagram of the network architecture of the style encoder in the present invention.

[0044] Figure 5 Schematic diagram of the network architecture of the decoder in the present invention.

[0045] Figure 6 Schematic diagram of the texture constraint network architecture in the present invention.

[0046] Figure 7 Schematic diagram of the network architecture of the discriminator in the present invention.

[0047] Figure 8 The figure is a schematic diagram of an embroidery image obtained by the Fabric 3D scanner of the present invention.

[0048] Fig. 9 Optimize loss gradient descent graphs for various optimizers in this invention.

[0049] Fig.10 It is a schematic diagram of the convergence of total loss changes in the present invention.

[0050] Fig.11 Schematic diagram of the comparison results of the ablation experiment in the present invention; wherein, (a) is the input image; (b) is the generation result of the network without texture constraint; (c) is the generation result with texture constraint.

[0051] Fig.12 Schematic diagram for comparing the experimental results of the model of the present invention with those of Embroidery GAN and Cycle GAN; wherein, (a) is the input image, (b) is the generation result of Embroidery GAN; (c) is the generation result of Embroidery GAN; and (d) is the generation result of the model of the present invention.

[0052] Fig.13 for Fig.12 Detail comparison diagram of sample one.

[0053] Fig.14 for Fig.12 Detail comparison diagram of sample 2.

[0054] Fig.15 for Fig.12 Detail comparison diagram of sample three. DETAILED DESCRIPTION

[0055] The present invention will be further described below in conjunction with the accompanying drawings.

[0056] like Figure 1 As shown, a method for automatically generating a target style image based on texture constraints includes the following steps:

[0057] Step 1: Create a data set including content images and style images; the style of the style image is consistent with the target style; the image styles include embroidery style, painting style, calligraphy style and clothing style; standardize all images in the data set to ensure the stability and repeatability of the results.

[0058] Step 2: Build a target style image generation network

[0059] like Figure 2 As shown, the target style image generation network includes a generator G and a discriminator D Y , generator F, discriminator D X and texture constraint network. The structure of generator G is the same as that of generator F; discriminator D Y and the discriminator D X The structure of is the same as that of ; the generator G is used to obtain a synthetic style image based on the input real content image and the real style image; the content of the synthetic style image is consistent with the content image, and the style of the synthetic style image is consistent with the style image; the discriminator D Y The generator F is used to distinguish the synthetic style image generated by the generator G from the real style image; the generator F is used to obtain the synthetic content image according to the synthetic style image generated by the generator G and the real content image; the content of the synthetic content image is consistent with the content of the synthetic style image (that is, consistent with the content of the real content image), and the style of the synthetic content image is consistent with the style of the real content image; the discriminator D X Used to distinguish the true and false of synthetic content images and real content images.

[0060] 2-1. Build the generator

[0061] The generator includes an encoder and a decoder; the encoder includes a content encoder and a style encoder; the content encoder is used to extract the content features of the image according to the input image, including image shape, edge and structure. The style encoder is used to extract the style features of the image according to the input image, including texture, color, strokes and lines.

[0062] 2-1-1. Build the encoder

[0063] like Figure 3 As shown in the figure, the content encoder includes a reflection padding layer, a convolution processing layer, two downsampling layers and six residual blocks connected in sequence. The reflection padding layer performs reflection padding (ReflectionPadding2d) on the input image to retain the edge information of the image and prevent information loss during the convolution operation. The padding size of the reflection padding is (3,3,3,3). The image size of the input reflection padding layer is 3×256×256, and the image size after the reflection padding layer is 3×262×262. The convolution processing layer includes a convolution layer, a BN layer (batch normalization layer) and a ReLU activation function connected in sequence; the number of convolution kernels in the convolution layer is 64, the size is 7×7, and the step size is 1, which is used to learn the low-level features of the content image; the BN layer is used to speed up the training process and improve the stability of the model; the image after the BN layer is output is activated with the ReLU function. The image size of the output image of the convolution processing layer is 64×256×256.

[0064] Both downsampling layers include convolutional layers, IN normalization layers, and ReLU activation functions. The size of the convolution kernel in the convolutional layer is 3×3, the step size is 2, and the padding is 1. The two downsampling layers are used to reduce the size of the feature map and improve the network's ability to perceive global information by doubling the feature channels, that is, to increase the receptive field. The image sizes after processing by the two downsampling layers are 128×128×128 and 256×64×64 respectively.

[0065] Each of the six residual blocks includes two convolutional layers, a BN normalization layer, and a ReLU activation function; the number of filters in the convolutional layer is 256, and some convolution kernels are randomly ignored during training. The residual block can promote gradient flow through jump connections, learn more complex content features, and thus improve the quality of the generated image. After six residual blocks, the output content code is 256×64×64 in size.

[0066] like Figure 4As shown in the figure, the style encoder includes six convolutional processing layers, a pooling layer and a fully connected layer connected in sequence. The six convolutional processing layers are used to extract the style features of the image, which include convolutional layers, BN layers and ReLu activation functions; the size, stride and padding of the convolution kernel in the six convolutional layers are (7,1,1), (3,2,1), (3,1,1), (3,2,1), (3,1,1), (3,1,1), and the size of the output feature map is 128×128×64, 64×64×128, 64×64×128, 32×32×256, 32×32×256, and 32×32×256. With the increase of the number of convolutional layers, the extracted features gradually change from low-level features such as lines and textures to high-level abstract features such as image semantic information. The pooling layer (max pooling) is used to reduce the dimension of the feature map, reduce the amount of calculation and the risk of overfitting, and improve the robustness and translation invariance of the model; the pooling window of the pooling layer is 2×2, the stride is

[22] , and the padding is

[0000] . The size of the feature map before pooling is 32×32×256, and the size of the feature map after pooling is 16×16×256. The fully connected layer is used to integrate the features extracted by the convolution layer and the pooling layer to output the style code.

[0067] 2-1-2. Constructing a decoder

[0068] The role of the decoder is to reconstruct the image from the content encoding and style encoding of the input image. It can gradually reduce and increase the feature map size and use the residual block to learn richer features, so that the generator can generate higher quality output images. In addition, the application of techniques such as normalization layers and reflection filling can make the training more stable.

[0069] like Figure 5 As shown in Figure 1, the decoder includes a multi-layer perceptron (MLP), an adaptive instance normalization layer (AdalN), four residual blocks, two upsampling layers, a reflection layer, a convolutional layer, and a Tanh activation function connected in sequence. The style code obtained by the encoder is input into the multi-layer perceptron to obtain the parameters of the AdalN layer:

[0070]

[0071] Where z is the activation generated by the last convolutional layer in the style encoder; γ and β are the parameters generated by the MLP from the style encoding; μ and σ are the mean and standard deviation of each channel, respectively.

[0072] Content coding is performed using four residual blocks with adaptive instance normalization (AdalN), and the output image size of the residual blocks is 256×64×64.

[0073] Each upsampling layer (convolution kernel size is 3×3, stride is 2, padding is 1) includes a deconvolution layer, a BN normalization layer, and a ReLU activation function. The deconvolution layer can increase the size of the feature map and restore the details of the generated image; the sizes of the feature images output by the two upsampling layers are 128×128×128 and 64×256×256 respectively.

[0074] The reflection layer is used to reflect and fill the feature map output by the upsampling layer with Reflection Pad 2d (3333), and the size of the output image is 64×262×262 (64 is the number of channels). The output of the reflection layer is convolved with a convolution layer (convolution kernel 7×7, step size 1), and a feature image with a size of 3×256×256 is output. The feature image output by the convolution layer is mapped to the range of [-1,1] through the Tanh activation function.

[0075] 2-2. Build a texture constraint network

[0076] like Figure 6 As shown, in order to extract and learn the high-level features of the target style image with higher quality and improve the semantic depth, needle details, brushstroke changes and other embroidery textures of the generated image, a texture constraint network is added to the target style image generation network.

[0077] The texture constraint network includes a recognition module, a synthesis module, a feature extraction module, a convolution processing module and a fully connected layer connected in sequence. The recognition module is used to extract high-level semantic features; the recognition module includes nine convolution layers; the first and second convolution layers have the same structure (the convolution kernel is 3×3 and the number of channels is 64); the third and fourth convolution layers have the same structure (the convolution kernel is 3×3 and the number of channels is 128); the fifth, sixth, seventh and eighth convolution layers have the same structure (the convolution kernel is 3×3 and the number of channels is 256); the convolution kernel of the ninth convolution layer is 3×3 and the number of channels is 512; the convolution layers of different structures are connected with a maximum pooling layer (maxpool). The synthesis module integrates the multi-level features extracted by the recognition module to obtain a synthetic feature image; the synthesis module includes four convolutional layers, the first three convolutional layers have the same structure (convolution kernel is 3×3, step size is 1, padding is 1, number of channels is 512), and each convolutional layer is connected to a batch normalization layer (BN) and a ReLu activation function; the last convolutional layer (convolution kernel is 4×4, step size is 2, number of channels is 128) is connected to a Tanh activation function. The feature extraction module is used to further extract high-level semantic features in the synthetic feature image; the feature extraction module includes five convolutional layers, and the structures of the five convolutional layers are the same as the structures of the first five convolutional layers in the recognition module. The convolution processing module is used to select and simplify the features extracted by the feature extraction module; the convolution processing module includes two convolution layers, the first convolution layer (convolution kernel is 3×3, step size is 1, padding is 1, number of channels is 512) is connected to the ReLu activation function; the second convolution layer (convolution kernel is 4×4, step size is 2, number of channels is 128) is connected to the batch normalization layer (BN) and the ReLu activation function; the features extracted by the convolution processing module are integrated through the fully connected layer, and the Hinge loss function is calculated as the texture feature.

[0078] 2-3. Constructing the Discriminator

[0079] The discriminator is used to distinguish the true and false of synthetic images from real images, and the target style image generation network is optimized with the help of the idea of ​​adversarial network training. The discriminator includes five convolution processing layers connected in sequence; the first four convolution processing layers have the same structure, including a convolution layer (convolution kernel is 4×4, stride is 2), IN layer (instance normalization) and a Leaky ReLU layer. Each convolution processing layer will reduce the size of the input image by half and double the number of channels. The size of the output image of the fourth convolution processing layer is 16×16 and the number of channels is 512. The fifth convolution processing layer includes a convolution layer (convolution kernel is 4×4, stride is 2) and a Sigmoid layer; the size of the output image of the Sigmoid layer is 8×8 and the number of channels is 1. Cut an image into N×N patches (blocks), use the discriminator to distinguish the true and false of each patch, and take the average of the results of all patches of an image as the final discriminator output. When N=1, it is equivalent to operating pixel by pixel; when N=256, it is equivalent to operating on an image (the image size is 256 pixels).

[0080] Step 3: Training the model

[0081] The content images in the dataset are used as real samples in the X domain, and the style images are used as real samples in the Y domain to train the target style image generation network. During the training process, the real samples in the X domain and the real samples in the Y domain are input into the generator G one by one to generate new samples in the Y domain, and the discriminator D is used to generate the new samples in the Y domain. Y Distinguish the generated Y domain new samples from the real samples in the Y domain; generate the generated Y domain new samples and the real samples in the X domain through the generator F to generate the X domain new samples, and use the discriminator D X Distinguish the generated X-domain newly added samples from the real X-domain samples; update the target style image generation network through the loss function to obtain the optimal target style image generation network; during the training process, the specific process of constructing the loss function is as follows:

[0082] (1) Adversarial Loss Function

[0083] The target style image generation network simultaneously learns the mapping of generator G and generator F. Generator G is used to convert images in domain X to images in target domain Y, realizing the conversion from domain X to domain Y. G is used to represent the mapping from domain X to domain Y, and discriminator D is used to identify the target style image. Y Determine whether the image generated by generator G is a real image; this is a generative adversarial process, and the adversarial loss function L of the generated style image in generator G GAN (G,D Y ,X,Y) is:

[0084] L GAN (G,D Y ,X,Y)=Ey～Pdata(y) [logD Y (y)]+E x～Pdata(x) [log(1-D Y (G(x))]

[0085] Among them, x is the real content image; E x～Pdata(x) is the mathematical expectation of the real content image x; y is the real style image; E y～Pdata(y) is the mathematical expectation of the true style image y; D Y (y) is the discriminator D Y The discrimination result of the real style image y; G(x) is the style image generated by the generator G for the real content image x; D Y (G(x)) is the discriminator D Y The discriminant result of the generated style image G(x).

[0086] The generator F is used to convert images in the Y domain into images in the target X domain. The training process needs to convert images in the Y domain into images in the X domain, and the mapping from Y to X is set to F. The discriminator D X Determine whether it is real data. This is another generation adversarial process. The adversarial loss function L of the generated content image in the generator F GAN (F,D X ,Y,X) is:

[0087] L GAN (F,D X ,Y,X)=E x～Pdata(x) [logD X (x)]+E y～Pdata(y) [log(1-D X (F(y))]

[0088] Among them, D X (x) is the discriminator D X The discrimination result of the real content image x; F(y) is the content image generated by the generator F for the real style image y; D X (F(y)) is the discriminator D X The judgment result of the generated content image F(y).

[0089] In order to minimize the maximum value of the gap (i.e., loss) between the synthetic images generated by the two generators and the real images, the final objective function G * and F * As shown below:

[0090]

[0091] (2) Reconstruction loss function

[0092] Construct reconstruction loss function (cycle consistency loss) L cyc The expression of (G,F) is:

[0093] L cyc (G,F)=E x～Pdata(x) [||F(G(x))-x|| 1 ]+E y～Pdata(y) [||G(F(x))-y|| 1 ]

[0094] Where F(G(x)) is the reconstructed image generated by generator F on G(x); ||·|| 1 For L 1 Norm symbol.

[0095] (3) Identity loss function

[0096] Generator G is used to generate images of style y, so providing y to generator G should still generate y. Only in this way can it be proved that generator G can generate style y. Therefore, G(y) should be as close to y as possible. If this loss is not added, the generator may change the hue of the image, resulting in a change in the overall color; construct the identity loss function L identity The expression of (G,F) is:

[0097] L identity (G,F)=E y～Pdata(y) [||(G(y)-y|| 1 ]+E x～Pdata(x) [||(F(x)-x|| 1 ]

[0098] (4) Texture loss function

[0099] Texture loss is the loss generated in the process of fusing the deep texture features extracted by the texture constraint network with the image generated by the generator G. The loss of the activation function connected after the second convolution layer in the convolution processing module is taken as the texture loss. The texture loss function L textkure The expression of (G) is:

[0100]

[0101] Where D, H, W are the depth, width and height of the feature map respectively; Φ is the feature extraction function.

[0102] (5) Total loss function

[0103] According to the above loss function, the total loss of the target style image generation network is constructed. The total loss function L total The expression is:

[0104] Ltotal =L GAN (G,D Y ,X,Y)+L GAN (F,D X ,Y,X)+aL cyc (G,F)+bL identity (G,F)+cL textkure (G)

[0105] Among them, a is the cycle consistency loss weight; b is the identity loss weight; c is the texture loss weight.

[0106] Example

[0107] A method for automatically generating a target style image based on texture constraints, using an embroidery style image as the target style image, comprises the following steps:

[0108] Step 1: Build a dataset

[0109] like Figure 8 As shown in the figure, the real embroidery is scanned by Fabric 3D scanner (3D fabric scanner) to obtain embroidery style images; a sample data set is created with content images and embroidery style images (style images); the sample data set is preprocessed, and the corresponding training set and test set are determined; all images in the training set and test set are standardized and uniformly cropped into 256 pixels × 256 pixels to ensure the stability and repeatability of the results.

[0110] Step 2: Training the model

[0111] like Fig. 9 As shown in the figure, different optimizers are used to train the target style image generation network through the training set. The optimizer can optimize the parameters when training the network, and can lead each parameter to a suitable value during gradient descent, so that the updated parameters minimize the target loss function. Fig. 9 It can be seen that the Adam optimizer converges faster, so the Adam optimizer is selected for iterative automatic parameter adjustment.

[0112] In this embodiment, when the image is cut into patches of size 50×50, the segmentation effect is the best and the discrimination result of the discriminator is the most accurate.

[0113] The optimization result is that when the loss weights and parameters are as shown in Table 1, the network generation result reaches the best, so it is set as the training parameter.

[0114] Table 1 Definition of training parameters

[0115]

[0116] The adversarial loss, cycle consistency loss, identity loss and texture loss are used to jointly complete the target style image generation network training. During the training process, the visualization tool Tensorboard provided by Tensorflow is used to monitor the total loss changes in real time, such as Fig.10 As shown in the figure, after 160k iterations of network training, the loss function maintains a stable jitter and enters a convergence state, which means that the network training has reached the optimal state.

[0117] Step 3: Model Evaluation

[0118] 3-1. Ablation experiment

[0119] The style image is input into the target style image generation network with and without the texture constraint network, and the output results are as follows: Fig.11 By comparison, it can be seen that when the texture constraint algorithm is not added, the generated image has serious blurring problems and does not show the texture details of the embroidery; while the algorithm with texture constraint effectively controls the edge of the image, and the generated simulation effect is clearer and rich in detail depth.

[0120] 3-2. Qualitative analysis

[0121] The results of experiments on five samples were compared using the proposed model and the existing models Embroidery GAN and Cycle GAN. The generated results are shown in Figure 2. Fig.12 As shown. The experimental results of different models are magnified for detailed comparison; the detailed comparison of sample 1 is as follows. Fig.13 As shown, from Fig.13 It can be seen that the model of the present invention obviously retains more and more complete embroidery and stitch information than Embroidery GAN; in the white part of the panda's face, Embroidery GAN lacks texture details, while Cycle GAN produces extra strokes on the panda's chest, and the embroidery details of the ink on the arm are irregular and messy. Fig.14 As shown, from Fig.14 It can be seen that Embroidery GAN does not retain the main part of the content image well, lacking the details of the embroidery stitches; the mountain and the background in the generated image of Cycle GAN are fused together, and the original appearance cannot be seen. There are also errors in the conversion of the Chinese character "Jiangnan". However, the model of the present invention has no errors in the generation of characters and logos, and the embroidery texture generation effect is more delicate and realistic. Fig.15 As shown, from Fig.15It can be seen that the embroidery stitches generated by the model of the present invention have a reasonable direction, and the generated image is more three-dimensional and vivid; the image generated by Embroidery GAN is very flat, lacks embroidery stitch depth, and the texture floats on the surface; the gingerbread man face generated by Cycle GAN is unreasonable, and there are many errors in the stitch direction. The embroidery generation effect of the model of the present invention is better than the previous two in terms of accuracy and authenticity.

[0122] In summary, the model of the present invention can generate embroidery images with clearer textures and more natural stitches than Embroidery-GAN; compared with Cycle-GAN, it effectively improves the mismatching problem in the embroidery generation process. That is, the target style image generation network with texture constraints proposed in the present invention can effectively learn the deep texture features of embroidery and generate more realistic, detailed and layered digital embroidery works.

[0123] Quantitative analysis

[0124] Four evaluation indicators, PSNR (peak signal-to-noise ratio), MSE (mean square error), SSIM (structural similarity) and LPIPS (image perceptual similarity), are used to analyze the generation quality of the migration image, that is, the test time and occupied memory are used to evaluate the model efficiency.

[0125] 3-3-1.PSNR is an important indicator for measuring image quality, and is often used in image compression, super-resolution image reconstruction, and other fields. PSNR is achieved through MSE (mean square error), and the smaller the MSE, the larger the PSNR, and the better the image quality. The expression of PSNR is:

[0126]

[0127] Among them, MAX I The maximum value of the color of each sampling point in the image.

[0128] 3-3-2. MSE is the mean square error between the estimated value (predicted value) and the actual value (basic fact). It is often used to indicate the similarity of pixel values ​​at the same position of two images. It can capture the changes in noise, brightness, contrast, hue and saturation in the image. The smaller the value, the better the result. Its expression is:

[0129]

[0130] Where m and n are the length and width of the image respectively; I(i,j) and K(i,j) are the pixel positions of the real image and the generated image; i = 0, 1, ..., m-1; j = 0, 1, ..., n-1.

[0131] 3-3-3.SSIM is an indicator to measure the similarity between two pictures. There is a strong correlation between adjacent pixels in natural images, and this correlation expresses the structural information of the objects in the picture. The human visual system has a strong ability to understand and extract image information, so structural similarity is a very important consideration when measuring image quality, and SSIM can capture three key features of the image: brightness, contrast, and structure.

[0132] Brightness μ(x) is the mean value of all pixel values ​​in the image, and its expression is:

[0133]

[0134] Among them, x i is the num-th pixel value of the input image x; N is the total number of pixel values ​​in the input image x.

[0135] Contrast σ(x) is the standard deviation (square root of variance) of all pixel values ​​in the image and is expressed as:

[0136]

[0137] The structure s(x) is determined by brightness and contrast, and its expression is:

[0138]

[0139] The expression for obtaining SSIM based on the three key features of the image is:

[0140] SSIM(x,y)=[l(x,y] δ ·[c(x,y] ε ·[s(x,y] ζ

[0141] Among them, x is the input image; y is the output image; l(x,y) is the contrast comparison function; c(x,y) is the brightness comparison function; s(x,y) is the structure comparison function; δ, ε, ζ are the relative importance of each indicator, δ>0, ε>0, ζ>0.

[0142] 3-3-4. The LPIPS model uses two images as input and outputs a perceptual similarity score between the two images. The higher the perceptual similarity score, the higher the similarity between the two compared images. It is implemented through a deep neural network.

[0143] These four indicators are used to quantitatively compare the model of the present invention with Embroidery GAN and Cycle GAN, and the results are shown in Table 2.

[0144] Table 2 Comparison of the model of the present invention, Cycle GAN and Embroidery GAN

[0145]

[0146] As can be seen from Table 2, the model of the present invention performs better in PSNR (the larger the better), SSIM (the larger the better) and LPIPS (the larger the better), and the test time is shorter. Cycle GAN occupies the least memory. Combining the five indicators, the image generation quality and model efficiency of the present invention are the best.

[0147] Step 4: By inputting the target content image into the target style image generation network, the image generated by the generator G is merged with the texture features obtained by the texture constraint network to obtain an output image whose content is consistent with the target content image and has an embroidery style.

Claims

1. A method for automatically generating a target style image based on texture constraints, characterized in that: The following steps are involved: Step 1: Create a data set including content images and style images; the style of the style image is consistent with the target style; Step 2: Construct a target style image generation network; the target style image generation network includes a generator G and a discriminator D Y , generator F, discriminator D X and texture constraint networks; The generator G is used to obtain a synthetic style image based on the input real content image and real style image; the content of the synthetic style image is consistent with the content image; the style of the synthetic style image is consistent with the style image; the discriminator D Y It is used to distinguish the synthetic style images generated by the generator G from the real style images; the structure of the generator G is the same as that of the generator F; the discriminator D X and the discriminator D Y The structure of is the same as that of ; the generator F is used to obtain a synthetic content image according to the synthetic style image and the real content image generated by the generator G; the content of the synthetic content image is consistent with the content of the synthetic style image; the style of the synthetic content image is consistent with the style of the real content image; the discriminator D X Used to distinguish the true from the false between synthetic content images and real content images; The texture constraint network obtains high-level semantic features by performing multi-level feature extraction and fusion on the input style image; high-level The semantic features are used as texture features corresponding to the style image; the texture features output by the texture constraint network are fused with the style image generated by the generator G; Step 3: Use the content images in the dataset as real samples in the X domain and the style images as real samples in the Y domain to train the target style image generation network; During the training process, the real samples from domain X and domain Y are input into the generator G one by one to generate new samples from domain Y, and the discriminator D is used to generate the new samples from domain Y. Y Distinguish the generated Y domain new samples from the real samples in the Y domain; generate the generated Y domain new samples and the real samples in the X domain through the generator F to generate the X domain new samples, and use the discriminator D X Distinguish the generated X-domain newly added samples from the real X-domain samples; update the target style image generation network through the loss function to obtain the optimal target style image generation network; Step 4: By inputting the target content image into the target style image generation network, the image generated by the generator G is fused with the texture features input by the texture constraint network to obtain an output image with content consistent with the target content image and with the target style.

2. The method for automatically generating a target style image based on texture constraints according to claim 1, characterized in that: The texture constraint network includes a recognition module, a synthesis module, a feature extraction module, a convolution processing module and a fully connected layer connected in sequence; the recognition module includes multiple convolution layers connected in sequence, the sizes of the convolution kernels in the multiple convolution layers are equal, the number of channels increases in sequence, and the convolution layers with different numbers of channels are connected through a maximum pooling layer to extract multi-level features in the target style image and obtain a feature image; the synthesis module includes four convolution layers, the structures of the first three convolution layers are the same, and a batch normalization layer and a ReLu activation function are connected after each convolution layer; the last convolution layer is connected after a Tanh activation function; the multiple layers extracted by the recognition module are integrated through the synthesis module The feature extraction module includes multiple convolution layers, the sizes of convolution kernels in the multiple convolution layers are equal, the number of channels increases successively, and the convolution layers with different numbers of channels are connected through the maximum pooling layer to further extract the high-level semantic features in the synthetic feature image; the convolution processing module includes two convolution layers, the first convolution layer is connected to the ReLu activation function; the second convolution layer is connected to the batch normalization layer and the ReLu activation function; the convolution processing module selects and simplifies the features extracted by the feature extraction module; the features extracted by the convolution processing module are integrated through the fully connected layer, and the texture features are obtained by calculating the Hinge loss function.

3. The method for automatically generating a target style image based on texture constraints according to claim 1, characterized in that: In step 4, the loss function L constructed total The expression is: L total =L GAN (G,D Y ,X,Y)+L GAN (F,D X ,Y,X)+aL cyc (G,F)+bL identity (G,F)+cL textkure (G) Among them, L GAN (G,D Y ,X,Y) is the adversarial loss function of the style image generated in the generator G; L GAN (F,D X ,Y,X) is the adversarial loss function of the content image generated in the generator F; L cyc (G, F) is the reconstruction loss function; L identity (G,F) is the identity loss function; L textkure (G) is the texture loss function; a is the cycle consistency loss weight; b is the identity loss weight; c is the texture loss weight; Adversarial loss function L GAN (G,D Y ,X,Y) is: L GAN (G,D Y ,X,Y)=E y～Pdata(y) [logD Y (y)]+E x～Pdata(x) [log(1-D Y (G(x))] Among them, x is the real content image; E x～Pdata(x) is the mathematical expectation of the real content image x; y is the real style image; E y～Pdata(y) is the mathematical expectation of the true style image y; D Y (y) is the discriminator D Y The discrimination result of the real style image y; G(x) is the image generated by the generator G for the real content image x; D Y (G(x)) is the discriminator D x The judgment result of the generated image G(x); Adversarial loss function L GAN (F,D X ,Y,X) is: L GAN (F,D X ,Y,X)=E x～Pdata(x) [logD X (x)]+E y～Pdata(y) [log(1-D X (F(y))] Among them, D X (x) is the discriminator D X The discrimination result of the real content image x; F(y) is the image generated by the generator F for the real style image y; D X (F(y)) is the discriminator D X The judgment result of the generated image F(y); Reconstruction loss function L cyc The expression of (G,F) is: L cyc (G,F)=E x～Pdata(x) [||F(G(x))-x||1]+E y～Pdata(y) [||G(F(x))-y||1] Where F(G(x)) is the reconstructed image generated by generator F for G(x); Identity loss function L identity The expression of (G,F) is: L identity (G,F)=E y～Pdata(y) [||(G(y)-y||1]+E x～Pdata(x) [||(F(x)-x||1] Texture loss function L textkure The expression of (G) is: Where D, H, W are the depth, width and height of the feature map respectively; Φ is the feature extraction function.

4. The method for automatically generating a target style image based on texture constraints according to claim 1, characterized in that: The target style image adopts an embroidery style image.

5. The method for automatically generating a target style image based on texture constraints according to claim 1, characterized in that: In the step 2, the generator G includes an encoder and a decoder; the encoder includes a content encoder and a style encoder; The content encoder is used to extract the content features of the image based on the input image, including image shape, edge, color and structure; The style encoder is used to extract the style features of the image, including texture, strokes, and lines, based on the input image; The decoder is used to generate an image based on the extracted content features and style features.

6. The method for automatically generating a target style image based on texture constraints according to claim 5, characterized in that: The content encoder comprises a reflection filling layer, a convolution processing layer, two downsampling layers and six residual blocks connected in sequence; the reflection filling layer performs reflection filling on the input image to retain the edge information of the image; the convolution processing layer comprises a convolution layer, a BN layer and a ReLU activation function connected in sequence; the two downsampling layers each comprise a convolution layer, an IN normalization layer and a ReLU activation function; the two downsampling layers are used to reduce the size of the feature map and improve the network's ability to perceive global information; the six residual blocks each comprise two convolution layers, a BN normalization layer and a ReLU activation function, and finally output the content code; The style encoder includes six convolution processing layers, a pooling layer and a fully connected layer connected in sequence; the six convolution processing layers are used to extract the style features of the image, and they all include a convolution layer, a BN layer and a ReLu activation function; the pooling layer is used to reduce the dimension of the feature map; the fully connected layer is used to integrate the features extracted by the convolution layer and the pooling layer, and output the style code.

7. The method for automatically generating a target style image based on texture constraints according to claim 5, characterized in that: The decoder includes a multi-layer perceptron, an adaptive instance normalization layer, four residual blocks, two upsampling layers, a reflection layer, a convolution layer and a Tanh activation function connected in sequence; the style code is input into the multi-layer perceptron to obtain the parameters of the adaptive instance normalization layer; Content encoding is processed using four residual blocks that introduce adaptive instance normalization; both upsampling layers include deconvolution layers, BN normalization layers, and ReLU activation functions.

8. The method for automatically generating a target style image based on texture constraints according to claim 1, characterized in that: The discriminator includes multiple convolution processing layers connected in sequence; except for the last convolution processing layer, the structures of the other convolution processing layers are the same, and all include a convolution layer, an IN layer and a Leaky ReLU activation function; the last convolution processing layer includes a convolution layer and a Sigmoid activation function; the discriminator's discrimination method is: divide the image into multiple blocks, and input each block into the discriminator for true and false discrimination.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The memory stores a computer program; the processor executes the method for automatically generating a target style image based on texture constraints as described in claim 1.

10. A readable storage medium storing a computer program; characterized in that: When the computer program is executed by a processor, it is used to implement the method for automatically generating a target style image based on texture constraints as claimed in claim 1.

Images