Method and system for generating Thangka floral leaf element image based on freehand sketch

Abstract

The invention relates to the technical field related to neural networks, in particular to a thangka floral leaf element image generation method and system based on a freehand sketch, and the method comprises the steps: obtaining a paired image data set of thangka floral leaf elements; and inputting the freehand sketch picture set of the thangka floral leaf element standard data set into a preset thangka floral leaf element generation network based on a freehand sketch, and outputting a thangka image matched with the thangka floral leaf element standard data set. It can be understood that according to the technical scheme provided by the invention, diversified Thangka floral leaf element images matched with Thangka painting characteristics can be output through the preset model, the specific composition mode and color rendition requirements of the Thangka floral leaf element images are met, intelligent design and visual creation tasks aiming at floral leaf elements in Thangka are achieved, and the intellectualization of the Thangka floral leaf elements in Thangka is achieved. Therefore, modern deduction and innovative inheritance of the traditional art form are realized.

Description

Technical Field

[0001] The present application relates to the technical field related to neural networks, and in particular to a method and system for generating images of thangka flower and leaf elements based on hand-drawn sketches. Background Art

[0002] Benefiting from the continuous innovation and optimization of deep learning algorithms, image generation technology is undergoing rapid iteration and development, and has gradually become a highly anticipated core topic in computer vision research. Not only has it deepened the understanding and construction capabilities of complex visual expressions at the theoretical level, but it has also shown a trend of diversification and deepening in practical applications, and has widely penetrated into various industries. Art generation methods driven by artificial intelligence have spawned countless novel, unique and creative works, inspired unprecedented sources of inspiration, and opened up new horizons for artistic creation.

[0003] Regong art originated from Tongren City, Huangnan Tibetan Autonomous Prefecture, Qinghai Province, China. As an important artistic carrier of Tibetan Buddhism, it is an intangible cultural heritage of my country and even the world. It covers a wide range of subjects, including religion, history, and all aspects of the lives of the people in Tibetan areas. It is an indispensable and important branch of Chinese painting art. As a shining pearl of Regong art, Thangka is famous for its rigorous composition and balanced and full layout in artistic expression. It especially highlights the fusion of fine brushwork and line drawing techniques. Painters use their own unique brushes, supplemented by precious mineral pigments, to jointly create the unparalleled artistic appeal and cultural value of Regong art.

[0004] In the process of Regong thangka painting, the main deity usually occupies the largest proportion in the center of the picture, surrounded by a large number of plants and flowers. This layout not only highlights the theme of the picture, but also makes the floral elements an important means to set off the main deity and create an atmosphere. The floral elements are vivid in shape, bright in color, smooth in lines, and highly decorative. Many flowers and plants are given specific symbolic meanings, such as the lotus symbolizing purity and elegance, and the peony symbolizing wealth and auspiciousness.

[0005] Unfortunately, the traditional craftsmanship of Rekongtse Thangka faces the challenges of inheritance and protection. The long painting period of Rekongtse artworks, the scarce stock, the sluggish industry, and the limited amount of resource data make it difficult for many precious Thangka works to be widely disseminated and studied in depth. The existing high-quality Thangka resource data sets are scarce and precious. Therefore, digital protection has become a top priority. Recording, storing and displaying Thangkas through digital means can not only effectively protect this cultural heritage, but also promote the global sharing of its artistic value. In this context, the use of modern technology to generate Thangka content will not only help us better understand and inherit this unique art form, but also be an important practice for the rescue and protection of human cultural heritage in the context of globalization.

[0006] In recent years, generative models have made significant progress in the field of image generation, from GAN (Generative Adversarial Network) to Diffusion Model (Diffusion Model), the technology has been continuously upgraded. GAN achieves high-quality image generation by building adversarial learning between the generator and the discriminator; while the Diffusion Model generates more delicate and realistic images by gradually adding noise and then reverse denoising. However, when these technologies are directly applied to thangka content generation in real life, there are many challenges: how to maintain the unique artistic style of thangka, how to effectively integrate its traditional painting techniques with complex color matching, and how to ensure the originality of the generated content while fully respecting its cultural connotations are all key issues that need to be solved.

[0007] Moreover, since the non-Buddhist elements of thangka belong to a specific field, the data set is extremely small, usually only a few hundred to a few thousand, which limits the deep learning feature extraction and style regularity learning in this field. The thangka sketches drawn by professional painters are accurate and expressive, while the sketches drawn by ordinary people are relatively simple and casual, which increases the difficulty of the model learning the diversity of sketches. At the same time, the image generation task based on sketches requires paired image and sketch data, but there are few public paired data sets. For sketches, there are problems such as difficulty in obtaining and insufficient precision, which leads to the current generation methods in the prior art being difficult to meet its unique composition and color restoration requirements when processing Rekong art data sets. Summary of the invention

[0008] In view of this, the purpose of the present invention is to provide a method and system for generating thangka flower and leaf element images based on hand-drawn sketches, so as to solve the problem that the current generation methods in the prior art are difficult to meet the unique composition and color restoration requirements when processing Rekong art data sets.

[0009] According to a first aspect of an embodiment of the present invention, a method for generating a thangka flower and leaf element image based on a hand-drawn sketch is provided, comprising:

[0010] A paired image dataset of thangka flower and leaf elements is obtained; wherein the paired image dataset of thangka flower and leaf elements comprises: a standard dataset of thangka flower and leaf elements, and a hand-drawn sketch set of the standard dataset of thangka flower and leaf elements; wherein the hand-drawn sketch set of the standard dataset of thangka flower and leaf elements is a hand-drawn sketch of flower elements corresponding to the standard dataset of thangka flower and leaf elements;

[0011] The hand-drawn sketch set of the thangka flower and leaf element standard data set is input into a preset thangka flower and leaf element generation network based on hand-drawn sketches, and thangka flower and leaf element images that are adapted to the characteristics of thangka painting and have diversity are output.

[0012] Furthermore, the collection process of the hand-drawn sketch set of the thangka flower and leaf element standard data set includes:

[0013] The preset PhotoSketch model or edge detection operator algorithm is used to extract the outline sketch of the flower and leaf elements in the thangka flower and leaf element standard data set, so as to obtain the hand-drawn sketch set of the thangka flower and leaf element standard data set;

[0014] The edge detection operator algorithms include: Sobel, Laplacian, Canny, and XDoG algorithms.

[0015] Furthermore, the hand-drawn sketch set of the thangka flower and leaf element standard data set is input into a preset thangka flower and leaf element generation network based on hand-drawn sketches, and thangka flower and leaf element images with diversity that are compatible with the thangka painting characteristics and in the standard data set of thangka flower and leaf elements are output, including:

[0016] Inputting a hand-drawn sketch set of the thangka floral and leaf element standard data set, extracting features from the hand-drawn sketch set of the thangka floral and leaf element standard data set through a preset encoder network, and outputting a latent code of a GAN latent code space;

[0017] Inputting the latent code into a preset decoder to obtain a first processing result;

[0018] The color information in the first processing result is corrected using a preset color correction module, a semi-interactive manual control mechanism is introduced, the color deviation in the image is corrected, and a thangka image that is compatible with the standard data set of the thangka flower and leaf elements is output.

[0019] Furthermore, the input of the hand-drawn sketch set of the thangka floral and leaf element standard data set, extracting features from the hand-drawn sketch set of the thangka floral and leaf element standard data set through a preset encoder network, and outputting a latent code of the GAN latent code space includes:

[0020] The FPN structure in the PSP encoder is used to realize multi-scale feature extraction of the hand-drawn sketch set of the thangka flower and leaf element standard data set, and the preset 24-layer ResNet-IRSE50 network structure is optimized to a 16-layer ResNet34 to obtain the first processed data;

[0021] For the first processed data, a preset full-dimensional dynamic convolution is introduced to enhance the feature extraction capability, and a latent code discriminator is added. Adversarial training is used to increase the diversity of generated images, and the latent code of the GAN latent code space is output.

[0022] Furthermore, the preset decoder includes:

[0023] The generator network in StyleGAN2-ADA.

[0024] Furthermore, the calculation formula of the preset full-dimensional dynamic convolution includes:

[0025]

[0026] Among them, α fi , α ci and α si Respectively represent filter attention, channel attention, and spatial attention; X represents the input feature tensor, whose shape is (batch_size, in_planes, height, width), and the return value Y represents the output feature tensor after being enhanced and convolved by the attention mechanism; represents kernel attention; SUM(S i ) indicates S i Aggregation is performed and used as convolution parameters; S i Represents the preset comprehensive weight; R represents the reshaped input feature tensor, and the output y is the convolution operation result of the filter attention, R and the aggregated comprehensive weight.

[0027] Furthermore, the preset thangka flower and leaf element generation network based on hand-drawn sketches also includes:

[0028] The mean square error loss function, as shown in formula (2), is used to measure the difference between the model prediction value and the actual value;

[0029]

[0030] Where n is the number of samples, is the model's predicted value for the i-th sample, y i is the target value of the model for the i-th sample;

[0031] The LPIPS loss function, as shown in formula (3), is used to evaluate the perceptual similarity between images;

[0032]

[0033] H l ,W l ,C l Represent the height, width and number of channels of the feature map of the lth layer respectively, where is the feature extraction function of the pre-trained CNN at layer l; ||·||2 represents the L2 norm, which is used to normalize the features to ensure that the comparison is in the direction rather than the magnitude of the similarity.

[0034] Furthermore, the preset thangka flower and leaf element generation network based on hand-drawn sketches also includes:

[0035] The latent code discriminator loss function, as shown in formula (4), is used to optimize the discriminator to better distinguish between real samples and generated samples;

[0036] L D =-[ylog(D(x))+(1-y)log(1-D(x))] (4)

[0037] Where y is the label of the sample, for real data, y = 1, for generated data, y = 0, and D(x) is the probability that the discriminator predicts that sample x is real data.

[0038] Furthermore, the preset thangka flower and leaf element generation network based on hand-drawn sketches also includes:

[0039] The regularization loss function, as shown in Equation (5), is used to improve the perceptual quality and editability of the generated images;

[0040]

[0041] Where latent represents the input latent variable and latent_avg represents the optional average latent variable.

[0042] According to a second aspect of an embodiment of the present invention, a system for generating a thangka floral and leaf element image based on a hand-drawn sketch is provided, which is applied to any of the above methods for generating a thangka floral and leaf element image based on a hand-drawn sketch, and the method comprises:

[0043] An acquisition module is used to obtain a paired image dataset of thangka flower and leaf elements; wherein the paired image dataset of thangka flower and leaf elements includes: a standard dataset of thangka flower and leaf elements, and a hand-drawn sketch set of the standard dataset of thangka flower and leaf elements; wherein the hand-drawn sketch set of the standard dataset of thangka flower and leaf elements is a hand-drawn sketch of flower elements corresponding to the standard dataset of thangka flower and leaf elements;

[0044] The processing module is used to input the hand-drawn sketch set of the thangka flower and leaf element standard data set into a preset thangka flower and leaf element generation network based on hand-drawn sketches, and output thangka flower and leaf element images that are adapted to the characteristics of thangka painting and have diversity.

[0045] The technical solution provided by the embodiments of the present invention may have the following beneficial effects:

[0046] It can be understood that the technical solution provided by the present invention obtains a paired image data set of thangka flower and leaf elements; wherein the paired image data set of thangka flower and leaf elements includes: a standard data set of thangka flower and leaf elements, a hand-drawn sketch set of the standard data set of thangka flower and leaf elements; wherein the hand-drawn sketch set of the standard data set of thangka flower and leaf elements is a hand-drawn sketch of flower elements corresponding to the standard data set of thangka flower and leaf elements; thereafter, the hand-drawn sketch set of the standard data set of thangka flower and leaf elements is input into a preset thangka flower and leaf element generation network based on hand-drawn sketches, and a thangka flower and leaf element image with diversity that is adapted to the characteristics of thangka painting is output. It can be understood that the technical solution provided by the present invention can output a thangka flower and leaf element image with a higher degree of thangka restoration corresponding to the standard data set of thangka flower and leaf elements through a preset model, meet its unique composition method and color restoration requirements, and realize the intelligent design and visual creation tasks for the flower and leaf elements in thangka, thereby realizing the modern interpretation and innovative inheritance of this traditional art form.

[0047] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0049] Figure 1 It is a schematic diagram of the steps of a method for generating a thangka flower and leaf element image based on a hand-drawn sketch according to an exemplary embodiment;

[0050] Figure 2 is a schematic diagram of a thangka flower and leaf element data set according to an exemplary embodiment;

[0051] Figure 3 is a schematic diagram of edge detection operator extraction results according to an exemplary embodiment;

[0052] Figure 4 is a diagram showing a structure of an AROD-PSP model according to an exemplary embodiment;

[0053] Figure 5 is a diagram of an encoder network structure according to an exemplary embodiment;

[0054] Figure 6 is a progressive style block structure diagram according to an exemplary embodiment;

[0055] Figure 7 is a diagram showing a discriminator network structure according to an exemplary embodiment;

[0056] Figure 8 is a visualization experiment result of image generation of a thangka flower and leaf element dataset according to an exemplary embodiment;

[0057] Fig. 9 is a schematic diagram of the system composition for generating thangka flower and leaf element images based on hand-drawn sketches according to an exemplary embodiment;

[0058] Fig.10 is a flow chart of a color correction module according to an exemplary embodiment. DETAILED DESCRIPTION

[0059] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0060] Embodiment 1

[0061] See also Figure 1 , Figure 1 The figure is a schematic diagram of steps of a method for generating a thangka flower and leaf element image based on a hand-drawn sketch according to an exemplary embodiment, the method comprising:

[0062] S1. Obtain a paired image dataset of thangka flower and leaf elements; wherein the paired image dataset of thangka flower and leaf elements comprises: a standard dataset of thangka flower and leaf elements, and a hand-drawn sketch set of the standard dataset of thangka flower and leaf elements; wherein the hand-drawn sketch set of the standard dataset of thangka flower and leaf elements is a hand-drawn sketch of flower elements corresponding to the standard dataset of thangka flower and leaf elements;

[0063] S2. Input the hand-drawn sketch set of the thangka flower and leaf element standard data set into the preset thangka flower and leaf element generation network based on hand-drawn sketches, and output thangka flower and leaf element images that are adapted to the characteristics of thangka painting and have diversity.

[0064] In one embodiment, as described in step S1, a paired image dataset of thangka flower and leaf elements is obtained, including: the thangka datasets used in this application are all collected by the research team of relevant researchers in Tongren City, Huangnan Tibetan Autonomous Prefecture, Qinghai Province, the source of Regong art. In view of the potential interference effect of background elements in the training stage, this study starts from the rich thangka material library collected and selects image samples with excellent image quality. The image is carefully extracted with Photoshop image processing software, and a standard dataset specifically for flower and leaf elements is constructed. This dataset is stored in PNG format and does not contain background. During the experiment, we preprocessed the dataset, converted it into JPG format, and uniformly set the background color to white (RGB value is 255,255,255). The size of each picture is 256×256 pixels, and a total of 1036 images are included.

[0065] In one embodiment, the collection process of the hand-drawn sketch set of the thangka flower and leaf element standard data set includes:

[0066] A series of edge detection operators, including Sobel, Laplacian, Canny, XDoG algorithms and hand-drawn sketch generation models, were used to extract line contours from the constructed flower and leaf element dataset. After comparison and screening, we selected the PhotoSketch

[28] model, which is closer to human hand-drawing, to generate hand-drawn sketches of corresponding flower elements and obtain a paired image dataset of "flower and leaf hand-drawn sketch-flower and leaf". The extraction results are shown in Figure 2. Figure 3 shown.

[0067] When training with paired training sets, such as PixtoPix and other models, the generated images are often severely constrained by the sketch, and the boundaries are often fixed. When the information provided by the sketch is low, the generated image will be severely distorted. Therefore, we tend to use hand-drawn sketches rather than line drawings as experimental data. Since the number of samples of the basic data set constructed in this application is relatively limited, in order to improve the model training effect and generalization ability, we simply flipped the original data set and expanded it from 1036 to 2072.

[0068] In one embodiment, as described in step S2, the hand-drawn sketch set of the thangka flower and leaf element standard data set is input into a preset thangka flower and leaf element generation network based on hand-drawn sketches, and thangka flower and leaf element images that are adapted to the thangka painting characteristics and have diversity in the standard data set of the thangka flower and leaf elements are output, including:

[0069] Inputting a hand-drawn sketch set of the thangka floral and leaf element standard data set, extracting features from the hand-drawn sketch set of the thangka floral and leaf element standard data set through a preset encoder network, and outputting a latent code of a GAN latent code space;

[0070] Inputting the latent code into a preset decoder to obtain a first processing result;

[0071] The color information in the first processing result is corrected using a preset color correction module, and a semi-interactive manual control mechanism is introduced to correct the color deviation in the image, so as to output a thangka flower and leaf element image that is compatible with the characteristics of thangka painting and has diversity.

[0072] More specifically, a hand-drawn sketch set of the thangka flower and leaf element standard data set is input, features in the hand-drawn sketch set of the thangka flower and leaf element standard data set are extracted through a preset encoder network, and a latent code of the GAN latent code space is output, including:

[0073] The FPN structure in the PSP encoder is used to realize multi-scale feature extraction of the hand-drawn sketch set of the thangka flower and leaf element standard data set, and the preset 24-layer ResNet-IRSE50 network structure is optimized to a 16-layer ResNet34 to obtain the first processed data;

[0074] For the first processed data, a preset full-dimensional dynamic convolution is introduced to enhance the feature extraction capability, and a latent code discriminator is added. Adversarial training is used to increase the diversity of generated images, and the latent code of the GAN latent code space is output.

[0075] It should be noted that the optimized network structure is a 16-layer ResNet34 instead of the original 24-layer ResNet-IRSE50 network structure, and full-dimensional dynamic convolution is introduced to enhance the feature extraction capability of the image. By enhancing the feature extraction capability of the image, the generated thangka flower and leaf elements are made diverse, making the encoded image more vivid and lively while retaining more details and information.

[0076] In the specific implementation, this application generates flower and leaf elements in Regong art based on the StyleGAN2-ADA pre-training model. The overall network model structure is shown in the figure below. Figure 4 shown.

[0077] It should be noted that Figure 4 PhotoSketch in , which represents a pre-trained model for extracting hand-drawn sketches;

[0078] E: latent code encoder; D: pre-trained decoder; DW: latent code discriminator;

[0079] L2, Llpips, LD, respectively represent the loss functions, which are l2 loss, lpips loss and latent code discriminator loss;

[0080] Fake Flower: image generated by the decoder;

[0081] Sketching Flower: Sketch result extracted by PhotoSketch.

[0082] First, the StyleGAN2-ADA model was pre-trained to obtain the corresponding pre-trained model. Then, a hand-drawn sketch dataset was generated through the PhotoSketch model and combined with flower and leaf elements to form a paired dataset. This model uses the generator network in StyleGAN2-ADA as a decoder and designs a corresponding encoder so that the input sketch can be converted into a latent code in the StyleGAN latent code space through the encoder. Finally, these latent codes are used to edit the generated image results.

[0083] See also Fig.10 ,At the end of the framework, we inserted a color correction module.,By inputting the color information based on expert guidance into this module,,we effectively corrected the color distortion in the generation process, thus achieving better visual,effects.

[0084] For specific implementation, please refer to Figure 5 , Encoder part, this paper is based on the PSP network framework, and adopts the FPN structure in the PSP encoder to realize multi-scale feature extraction. The input of the encoder network is a hand-drawn sketch, and the output is the latent code of the StyleGan2-ada latent space. In view of the low complexity of the data set, the original 24-layer ResNet-IRSE50 network structure in the network appears slightly redundant. Therefore, this application optimizes the network structure to a 16-layer ResNet34 without significantly reducing the feature extraction capability, thereby reducing the computational overhead of the model. After preliminary convolution processing, the input enters the ResNet network composed of four residual blocks. Among them, the 2nd, 3rd, and 4th modules generate large, medium, and small feature maps, respectively, representing the information expression of the image at different scales. Subsequently, these feature maps are input into the progressive style extraction block, and the feature maps of each scale generate different numbers of latent codes to form a high-dimensional vector of 14×512 dimensions. During the image generation process, these latent codes are injected into the generator of StyleGAN2-ADA, gradually affecting the generated images, making them closer to the content of the input sketch while maintaining the diversity and detail richness of the generated images, thereby improving the flexibility and expressiveness of image generation. Figure 5 Conv in the above code represents the convolutional layer; BN is the batch normalization layer, which normalizes the data;

[0085] Relu linear rectification layer, using prelu activation function;

[0086] resnet34, layer1, etc., represent the four residual blocks of resnet34;

[0087] Gradualstyleblock represents a progressive style extraction block. Figure 6 Abbreviation of .

[0088] In the encoder network, in order to enhance the adaptability of features, this application uses full-dimensional dynamic convolution (ODConv) to replace some ordinary convolution layers in the progressive style block, see Figure 6 ODConv introduces a multi-dimensional attention mechanism to achieve parallel learning of four types of attention for convolution kernels in all four spatial dimensions. This mechanism applies finely differentiated attention to the corresponding convolution kernels layer by layer, significantly enhancing the feature extraction capability of convolutional neural networks (CNNs) in basic convolution operations. The main idea is:

[0089] y=(α w1 α f1 α c1 α s1 ·S1+…

[0090] +α wn α fn α cn α sn ·S n )*x(1)

[0091] Among them, α wi , α fi , α ci and α si Calculated by the multi-head attention module, α wi is the convolution kernel S i The attention scalar, α fi , α ci and α si According to S i The spatial dimension of the kernel space and the input and output channel dimension are calculated. It represents filter attention, channel attention, and spatial attention. In this application, the calculation formula for the full-dimensional dynamic convolution we use is:

[0092]

[0093] It should be noted that Figure 6 Conv2d in the above code represents a convolutional layer; ODconv2d: a full-dimensional dynamic convolutional layer;

[0094] Leakyrelu: Linear rectifier layer, using Leakyrelu activation function.

[0095] In one embodiment, see Figure 7 Referring to the idea of ​​approximating W space optimization in E4E (encoder4editing), this paper adds a latent discriminator to distinguish the real sample latent from W space from the latent generated by encoder E. In the discriminator module, we use the classic multi-layer perceptron (MLP) structure, with the input as the real latent vector and the latent vector generated by the latent mapping encoder, and the output as the predicted true or false value. The loss of the discriminator consists of the loss of the generated sample and the real sample, while the discriminant loss of the generator comes from the output of the generated sample.

[0096] It should be noted that Figure 7 Linear: fully connected layer; Leakyrelu: linear rectification layer, using Leakyrelu activation function.

[0097] During the specific implementation, we noticed that the colors of the pictures generated by the network model showed brightness changes, which were different from the colors in the real thangka dataset. Therefore, we used the color correction module to correct the colors of the generated images to make them more consistent with the colors of the real thangka images.

[0098] In the art of thangka painting, basic colors are collectively referred to as "primary colors", including white, red, blue, yellow and other colors. Based on this, this paper selects 5 colors for clustering, quantizes the spatial colors in the RGB space, and allocates them to 16×16×16 histograms. Through the K-means clustering algorithm, the main colors in the input image are extracted, and a set of color palettes with 5 colors is generated. In addition, this paper clusters the flowers according to their classification information in the data set to preliminarily generate predefined color prior information. On this basis, with the guidance of Regong art painters, the colors are fine-tuned to achieve the desired color effect, thereby forming complete predefined color prior information. Users can input the color information guided by experts into the palette to change the color information of the image in a semi-interactive way, so that it is more in line with the color of the real thangka image or more in line with the aesthetic needs of the user.

[0099] Furthermore, the loss function used in the preset thangka flower and leaf element generation network based on hand-drawn sketches is as follows:

[0100] L2 loss: also known as mean squared error (MSE) loss, is a loss function widely used in machine learning and deep learning. It is used to measure the difference between the model's predicted value and the actual value, thereby guiding the optimization direction of the model. The formula is as follows:

[0101]

[0102] Where n is the number of samples, is the model's predicted value for the i-th sample. i is the target value of the model for the i-th sample.

[0103] LPIPS loss: A perceptual similarity measurement method based on deep learning that measures the similarity between two images. LPIPS loss uses a pre-trained deep network (VGG16 is used in this article) to extract image features and calculate the distance between these features to evaluate the perceptual similarity between images. Compared with the traditional L2 loss, LPIPS pays more attention to the structural content and perceptual quality of the image. For input images x1 and x2, their LPIPS loss can be expressed as:

[0104]

[0105] H l ,W l ,C l Represent the height, width and number of channels of the feature map of the lth layer respectively, where is the feature extraction function of the pre-trained CNN at layer l. ||·||2 represents the L2 norm, which is used to normalize the features to ensure that the comparison is in the direction rather than the magnitude of the similarity.

[0106] Latent code discriminator loss:

[0107] L D =-[ylog(D(x))+(1-y)log(1-D(x))] (5)

[0108] Where y is the label of the sample, for real data, y = 1, for generated data, y = 0, and D(x) is the probability that the discriminator predicts that sample x is real data.

[0109] In this application, our loss is set as follows:

[0110]

[0111] The loss function L of the discriminator D It consists of the loss of real samples and the loss of generated samples. The average of the negative and positive values ​​output by the discriminator is calculated through the softplus activation function. The total loss is the sum of these two parts of the loss, which is used to optimize the discriminator to better distinguish between real samples and generated samples.

[0112] Regularization loss: In order to ensure that the generated style code remains close to the W space, this paper uses the WNorm regularization loss, which calculates the loss based on the input latent variables and the optional average latent variables, thereby improving the perceptual quality and editability of the generated image. Its formula can be expressed as:

[0113]

[0114] The total loss is calculated as follows:

[0115] L=α1L2+α2L LPIPS +α3L D +α4L W (8)

[0116] In the specific implementation, this application aims at the problem that the generation process of thangka flower and leaf elements may not be consistent with its unique composition method, and optimizes it based on the PSP framework, and proposes a semi-interactive sketch conditional generation model. This model achieves the generation effect that conforms to the composition method and color of thangka flower and leaf elements. In the comparative experiment, the network model proposed in this paper achieved the best results of two indicators and the second best result of one indicator in the quantitative analysis, and obtained the highest score in the qualitative analysis. On the whole, the network model proposed in this paper has preliminarily realized semi-interactive guidance through generated sketches in the task of generating images, successfully achieved a good balance between generation quality and generation diversity, and is in a leading position in the same type of tasks.

[0117] The specific experiments are as follows:

[0118] 1. Experimental environment and hyperparameters

[0119] This experiment was run in the environment of NVIDIA GeForce RTX3090-24G, Python3.7, PyTorch1.8, and CUDA11.0. In the experiment, Batchsize was set to 8, learning rate was set to 0.0001, and training Setp was set to 500000. The loss function weights were set to lpips_lambda=0.8, l2_lambda=1, w_discriminator_lambda=0.1, and w_norm_lambda=0.005.

[0120] 2. Evaluation indicators

[0121] In order to measure the diversity and image quality of generated images, this application uses FID and KID as evaluation indicators. The FID score extracts feature vectors based on the Inception-V3 model and calculates the Fréchet distance between the mean and covariance of the feature vectors between the generated image and the real image. The smaller the value, the higher the similarity between the generated image and the real image, and the better the generation performance of the model. Its calculation formula is

[0122]

[0123] where μ r , μg represents the mean of the feature vectors of the real image and the generated image, and ∑ r and∑ g Represent the covariance matrix of the eigenvectors of the generated image and the real image respectively.

[0124] The KID value measures the difference between the two groups of samples by calculating the maximum mean difference (MMD) square of the generated image and the real image in the Inception feature representation. The smaller the KID value, the closer the generated image is to the real image in the feature space, that is, the better the effect of the generated model. The formula can be expressed as:

[0125] KID=MMD(f real ,f fake ) 2 (10)

[0126] MMD is usually used in conjunction with the kernel method to map the original data into a high-dimensional feature space through the kernel function.

[0127] 3. Experimental results and analysis

[0128] 3.1 Quantitative analysis

[0129] As shown in Table 1, AROD_PSP achieved the best result in the quantitative analysis results in terms of FID value, while MSE and KID values ​​achieved the second best results. At the same time, the experimental results show that E4E and Restyle_PSP, which achieved good quantitative results, have significantly poorer visual performance in the generated images. Their results have a strong correspondence with the hand-drawn sketch. When the hand-drawn sketch is partially missing, the generated image is also partially missing. In the E4E network, the encoder obtains a single style code W and a set of offsets Δ1…Δ n-1 , the final latent variable is the W vector replicated N times plus the corresponding Δ i This method is encouraged to approach W space, which makes it more editable, but it is also affected by inaccurate hand-drawn sketches, which makes the generated image lose its internal hierarchy and obtain higher distortion, which does not conform to the hierarchical characteristics of thangka flower and leaf elements. Restyl e adopts residual training, which uses multiple steps in inferring latent codes. The encoder is in W + The residual latent code Δ in the prediction space t , and then add the predicted residual to the previous latent code W t , to obtain the updated latent code prediction W t+1This method makes the outer contour and color of the generated image closer to the original image, but the internal texture is still affected by the initial latent code, making it difficult to achieve a more ideal generation effect; comparing PSP and AROD-PSP, both have achieved relatively good visual effects. From a quantitative analysis, all indicators of AORD-PSP are higher than PSP. Compared with the PSP model, the quantitative indicators, LPIPS value is reduced by 0.0298, the score is increased by 8.09%, the MSE value is reduced by 0.0116, an increase of 7.18%, the FID score is reduced by 16.1265, an increase of 24.4%, and the KID score is reduced by 0.01984, an increase of 37.94%.

[0130] Table 1

[0131]

[0132] Note: The bold ones are the best results, the underlined ones are the second best results, and ↓ means the lower the index, the better.

[0133] In order to more comprehensively evaluate the perceptual quality of an image, it is usually quantified by comparing the difference between the real distribution and the generated distribution between images. However, the better the quantitative indicators are, the more accurate they are in reflecting the aesthetic performance of the image, and these indicators are easily disturbed by factors such as distortion. In view of this, we introduce qualitative analysis to better capture the correlation between image quality and human subjective perception.

[0134] Thangka paintings are delicate, with smooth lines and bright colors. They have high requirements for the use of colors and composition design. However, computers are relatively weak in the evaluation and analysis of visual senses. Based on this, this paper adopts a qualitative evaluation method to conduct an in-depth analysis of Thangka paintings. The qualitative evaluation of this paper is mainly carried out from the following perspectives:

[0135] 1. Invite professional painters to review the generated images from the perspective of artistic creation, focusing on aspects such as color and lines.

[0136] 2. Invite a general user group to conduct the evaluation. This group includes 40 participants, including 8 students majoring in Regong art, 22 ordinary students, 10 art lovers, and 10 ordinary users.

[0137] These two types of users rated the generated flower and leaf element images based on their own professional knowledge and personal preferences. The evaluation content covers two aspects: visual composition (such as color, light and shadow, brightness, shape, etc.) and technical quality (such as clarity, completeness of details, etc.), with the highest score being 5 points. The final score is the sum of the scores of the two aspects. The specific scoring results are shown in Table 2.

[0138] Table 2

[0139]

[0140] Note: The ones in bold are the best results.

[0141] From the qualitative analysis results, the AROD-PSP and PSP models ranked first and second respectively, while E4E and Restyle_PSP ranked lower in the qualitative analysis. This result is consistent with the judgment of the visual evaluation, further verifying the superiority of the AROD-PSP model in generation quality. However, in terms of generating details, such as the lines of the pedicel and petals, and the brightness of the image, the AROD-PSP model performed better.

[0142] Based on the results of comprehensive quantitative and qualitative analysis, the AROD-PSP model basically achieves a good balance between generation quality and generation diversity under the guidance of hand-drawn sketches, and achieves a relatively ideal generation effect.

[0143] By comparing the FID and KID results, compared with the PSP model, AROD-PSP shows the best quantitative results in all three data sets, especially in the architectural painting data set, showing higher diversity, and the generated image color is closer to the original image. However, due to the complex and relatively abstract geometric structure of the architectural painting data set, there are certain differences between the FID and KID values ​​and the original generated data in StyleGAN2-ADA. In the anime face data set, the experimental results of the two models are similar, but AROD-PSP shows a more superior effect in the processing of details such as the character's neck and hair color. The experimental results show that AROD-PSP is not only suitable for the floral element data set, but can also achieve good generation effects on other data sets.

[0144] In the specific implementation, this application edits the latent space of GAN, combines hand-drawn sketches as a guide, and semi-interactively generates the floral elements in Thangka, which can better capture and restore the unique charm and rich details of Thangka art. An improved PSP model, AROD-PSP, is proposed. In the encoder, a 16-layer network is used to replace the original 24-layer network, and a suitable feature map extraction method is selected; full-dimensional dynamic convolution is introduced to enhance the feature extraction capability; a latent code discriminator is added, and adversarial training is used to increase the diversity of generated images; for the problem of brightness change of the generated image compared to the original image, a color correction module is used to correct the color to make it closer to the original image color. The intelligent design and visual creation tasks for the floral elements in Thangka are realized, thereby realizing the modern interpretation and innovative inheritance of this traditional art form. The experiment was carried out on the Thangka floral element dataset, the anime avatar dataset, and the Regong art architectural painting dataset, and the results proved the robustness of the model. In the quantitative analysis, compared with the basic model, the FID value was reduced by 16.1265, the score was increased by 24.4%, and the KID score was reduced by 0.01984, the score was increased by 37.94%; in the qualitative analysis, the highest evaluation was achieved. Experiments have proved that the method proposed in this application has great application value in the field of specific image generation.

[0145] See also Fig. 9 , Fig. 9 1 is a schematic diagram of a system for generating a thangka flower and leaf element image based on a hand-drawn sketch according to an exemplary embodiment, wherein the system comprises:

[0146] The acquisition module 91 is used to obtain a paired image data set of thangka flower and leaf elements; wherein the paired image data set of thangka flower and leaf elements includes: a standard data set of thangka flower and leaf elements, and a hand-drawn sketch set of the standard data set of thangka flower and leaf elements; wherein the hand-drawn sketch set of the standard data set of thangka flower and leaf elements is a hand-drawn sketch of flower elements corresponding to the standard data set of thangka flower and leaf elements;

[0147] The processing module 92 is used to input the hand-drawn sketch set of the thangka flower and leaf element standard data set into a preset thangka flower and leaf element generation network based on hand-drawn sketches, and output a thangka image that is compatible with the thangka flower and leaf element standard data set.

[0148] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.

[0149] It should be noted that, in the description of the present invention, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "plurality" refers to at least two.

[0150] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0151] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0152] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0153] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0154] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0155] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0156] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for generating a thangka flower and leaf element image based on a hand-drawn sketch, characterized in that: The method comprises: A paired image dataset of thangka flower and leaf elements is obtained; wherein the paired image dataset of thangka flower and leaf elements comprises: a standard dataset of thangka flower and leaf elements, and a hand-drawn sketch set of the standard dataset of thangka flower and leaf elements; wherein the hand-drawn sketch set of the standard dataset of thangka flower and leaf elements is a hand-drawn sketch of flower elements corresponding to the standard dataset of thangka flower and leaf elements; The hand-drawn sketch set of the thangka flower and leaf element standard data set is input into a preset thangka flower and leaf element generation network based on hand-drawn sketches, and thangka flower and leaf element images that are adapted to the characteristics of thangka painting and have diversity are output.

2. The method according to claim 1, characterized in that The collection process of the hand-drawn sketch set of the thangka flower and leaf element standard data set includes: Extracting the outline sketch of the flower and leaf elements in the thangka flower and leaf element standard data set by using a preset PhotoSketch model or a preset edge detection operator algorithm to obtain a hand-drawn sketch set of the thangka flower and leaf element standard data set; The edge detection operator algorithms include: Sobel, Laplacian, Canny, and XDoG algorithms.

3. The method according to claim 1, characterized in that The method of inputting the hand-drawn sketch set of the thangka flower and leaf element standard data set into a preset thangka flower and leaf element generation network based on hand-drawn sketches, and outputting thangka flower and leaf element images that are compatible with the thangka painting characteristics and have diversity in the thangka flower and leaf element standard data set includes: Inputting a hand-drawn sketch set of the thangka floral and leaf element standard data set, extracting features from the hand-drawn sketch set of the thangka floral and leaf element standard data set through a preset encoder network, and outputting a latent code of a GAN latent code space; Inputting the latent code into a preset decoder to obtain a first processing result; The color information in the first processing result is corrected using a preset color correction module, and a semi-interactive manual control mechanism is introduced to correct the color deviation in the image, so as to output a thangka flower and leaf element image that is compatible with the characteristics of thangka painting and has diversity.

4. The method according to claim 3, characterized in that The method of inputting the hand-drawn sketch set of the thangka floral and leaf element standard data set, extracting features from the hand-drawn sketch set of the thangka floral and leaf element standard data set through a preset encoder network, and outputting a latent code of a GAN latent code space includes: The FPN structure in the PSP encoder is used to realize multi-scale feature extraction of the hand-drawn sketch set of the thangka flower and leaf element standard data set, and the preset 24-layer ResNet-IRSE50 network structure is optimized to a 16-layer ResNet34 to obtain the first processed data; For the first processed data, a preset full-dimensional dynamic convolution is introduced to enhance the feature extraction capability, and a latent code discriminator is added. Adversarial training is used to increase the diversity of generated images, and the latent code of the GAN latent code space is output.

5. The method according to claim 2, characterized in that: The preset decoder includes: The generator network in StyleGAN2-ADA.

6. The method according to claim 4, characterized in that The calculation formula of the preset full-dimensional dynamic convolution includes: Among them, α fi , α ci and α si Respectively represent filter attention, channel attention, and spatial attention; X represents the input feature tensor, whose shape is (batch_size, in_planes, height, width), and the return value Y represents the output feature tensor after being enhanced and convolved by the attention mechanism; represents kernel attention; SUM(S i ) indicates S i Aggregation is performed and used as convolution parameters; S i Represents the preset comprehensive weight; R represents the reshaped input feature tensor, and the output y is the convolution operation result of the filter attention, R and the aggregated comprehensive weight.

7. The method according to claim 1, characterized in that The preset thangka flower and leaf element generation network based on hand-drawn sketches also includes: The mean square error loss function, as shown in formula (2), is used to measure the difference between the model prediction value and the actual value; Where n is the number of samples, is the model's predicted value for the i-th sample, y i is the target value of the model for the i-th sample; The LPIPS loss function, as shown in formula (3), is used to evaluate the perceptual similarity between images; H l ,W l ,C l Represent the height, width and number of channels of the feature map of the lth layer respectively, where is the feature extraction function of the pre-trained CNN at layer l; ||·||2 represents the L2 norm, which is used to normalize the features to ensure that the comparison is in the direction rather than the magnitude of the similarity.

8. The method according to claim 1, characterized in that: The preset thangka flower and leaf element generation network based on hand-drawn sketches also includes: The latent code discriminator loss function, as shown in formula (4), is used to optimize the discriminator to better distinguish between real samples and generated samples; L D =-[ylog(D(x))+(1-y)log(1-D(x))] (4) Where y is the label of the sample, for real data, y = 1, for generated data, y = 0, and D(x) is the probability that the discriminator predicts that sample x is real data.

9. The method according to claim 1, characterized in that: The preset thangka flower and leaf element generation network based on hand-drawn sketches also includes: The regularization loss function, as shown in Equation (5), is used to improve the perceptual quality and editability of the generated images; Where latent represents the input latent variable and latent_avg represents the optional average latent variable.

10. A system for generating thangka flower and leaf element images based on hand-drawn sketches, characterized in that: A method for generating a thangka flower and leaf element image based on a hand-drawn sketch as described in any one of claims 1 to 9, the method comprising: An acquisition module is used to obtain a paired image dataset of thangka flower and leaf elements; wherein the paired image dataset of thangka flower and leaf elements includes: a standard dataset of thangka flower and leaf elements, and a hand-drawn sketch set of the standard dataset of thangka flower and leaf elements; wherein the hand-drawn sketch set of the standard dataset of thangka flower and leaf elements is a hand-drawn sketch of flower elements corresponding to the standard dataset of thangka flower and leaf elements; The processing module is used to input the hand-drawn sketch set of the thangka flower and leaf element standard data set into a preset thangka flower and leaf element generation network based on hand-drawn sketches, and output thangka flower and leaf element images that are adapted to the characteristics of thangka painting and have diversity.

Images