Color picture generation system and method based on texture and tone information attention mechanism

Abstract

The invention discloses a color picture generation system and method based on a texture and tone information attention mechanism, and belongs to the field of color picture intelligent generation, the system comprises a UVF-GAN model, and the UVF-GAN model comprises two paths of generative adversarial networks, each of the two paths of generative adversarial networks comprises a UNet-VI T generator based on color style modulation and an FFCS discriminator based on a feature statistical fusion consistency strategy, and the UNet-VI T generator and the FFCS discriminator translate unpaired images to images in a staggered manner. By adopting the color picture generation system and method based on the texture and tone information attention mechanism, the corresponding image of each drawing step can be generated on the premise of keeping the resolution and style by simulating the drawing teaching process in reality, so that the drawing teaching is assisted, and the teaching quality is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent generation of color paintings, and in particular to a color painting generation system and method based on a texture and hue information attention mechanism. Background Art

[0002] In today's global art education field, especially in painting training, the main challenges are due to the uneven distribution of educational resources and outdated teaching methods. Usually, painting teaching relies on on-site demonstration and guidance by teachers. This model is severely restricted by the number of teachers and time resources, and it is difficult to meet the personalized and diversified needs of students. In addition, the problems encountered by students in the learning process are not solved in time, which often weakens the students' learning motivation and the development of creativity.

[0003] At the same time, with the development of science and technology, especially the innovation of digital technology, a series of online platforms and applications to assist art education have emerged. Platforms such as "Painting Cloud" and "Art Express" have increased the diversity and accessibility of learning resources by providing video courses, interactive communities and real-time online tutoring. However, despite the convenience brought by these platforms, most of them still rely on traditional teaching content and methods. Coupled with technical limitations and poor user experience, it is difficult to meet students' expectations for instant feedback and personalized learning.

[0004] In the field of intelligent painting technology, academia and industry have proposed a series of innovative solutions to the problem of automatic color painting generation. For example, a method based on MDCM-GAN is used to reconstruct faces from sketches. This method expands the perception area through a multi-scale dilated convolution module and improves the quality of reconstruction through two mapping networks. Compared with traditional methods, it reduces the dependence on paired data sets and reduces data requirements, while performing well in detail restoration and texture expression. However, this method lacks artistic sense and is not suitable for the learning needs of art students. In addition, in terms of color painting generation, due to its higher complexity than sketching, the application effect of MDCM-GAN may be affected.

[0005] In addition, color painting occupies an important position in the entrance examination of art colleges. It not only accounts for a large proportion of the total score, but also tests the color perception ability, artistic imagination and expression skills of candidates. Therefore, teachers tend to build a systematic knowledge system to guide students to learn the basic theories and techniques of color painting in an orderly manner. Such a teaching strategy is conducive to students to establish connections between knowledge, simplify the learning path, improve learning efficiency, and thus significantly improve learning outcomes. However, in the traditional teaching method, there are several obvious problems in the teaching of teachers by manually drawing color paintings: first, the drawing efficiency is low. Teachers start from a blank canvas and gradually show the painting process. This process may take several hours. Long-term teaching can easily make students lose interest; second, the subjectivity of the work is strong. Different teachers have different interpretations and expressions of the same painting, which constitutes an obstacle for students to master objective color skills; third, the level of teachers is uneven. In order to save costs, some institutions will hire students who have not yet graduated as teachers. Even some senior teachers have their own expertise in different art fields, which may affect the integrity and scientificity of teaching.

[0006] In the context of Internet+ and educational innovation, color painting generation technology based on artificial intelligence is developing rapidly. Technologies such as generative adversarial networks (GANs) and deep convolutional neural networks (DCNNs) have been applied to artistic creation. For example, the DeepArt project uses deep convolutional neural networks to merge content images with style images to simulate the artist's unique style. However, many existing models still have limitations: they can usually only generate the final image once, which is of limited value for teaching.

[0007] In addition, most existing models are pre-trained based on datasets in non-art fields, and the generated results may be quite different from the actual needs of art education. At this time, style transfer technology based on convolutional neural networks, such as VGG-19, can achieve style transfer, but it is easy to ignore low-level details of the image, affecting the fineness of the generated image. Generative models that rely on paired datasets;

[0008] Although pix2pix can generate images in a specific style, this method has the following defects: (1) It only extracts high-level features of the image and discards low-level information, so the stylized image will lose details; although the attention mechanism can help the model focus on the main area of ​​the image, it ignores some subtle but important elements, such as fine textures and color gradients, resulting in the generated paintings lacking details or being overly simplified in some parts; (2) The model is sensitive to biases in the dataset and requires the dataset to be paired. If it is not a paired dataset, the generated works will also be affected and lack innovation and diversity; (3) The color painting generation method based on pix2pix is ​​unstable in adversarial training, making it difficult for the model to generate high-resolution images; (4) Although it can simulate the natural diffusion effect of paint, the amount of calculation in the algorithm iteration process is extremely large and the demand for computing resources is high, which is not practical in an environment with limited resources; (5) The large randomness may cause the final generated painting to deviate significantly from expectations. Summary of the invention

[0009] The purpose of the present invention is to provide a color painting generation system and method based on texture and hue information attention mechanism to solve the above technical problems.

[0010] To achieve the above objectives, the present invention provides a color painting generation system based on a texture and hue information attention mechanism, including a UVF-GAN model, the UVF-GAN model includes a two-way generative adversarial network, the two-way generative adversarial network both include a UNet-VIT generator based on color style modulation and a FFCS discriminator based on a feature statistics fusion consistency strategy, the UNet-VIT generator and the FFCS discriminator translate unpaired images to images in an interleaved manner.

[0011] Preferably, two UNet-VIT generators are set to be UNet-VIT generators G from domain A to domain B. A→B and UNet-VIT generator G from domain B to domain A B→A ;

[0012] To ensure that the images generated by the UNet-VIT generator are judged as real images by the FFCS discriminator, the generation loss function is used to calculate G B →A The generation loss and G A→B The generation loss is to calculate G B→A The generated image is passed to the FFCS discriminator D A The probability of judging it as a real image and calculating G A→B The generated image is passed to the FFCS discriminator D B The probability of judging it as a real image, where the loss function l is generated GANThe expression is as follows:

[0013] L Gen,A =E x～A [l GAN (D A (G B→ A(x)),1)] (1);

[0014] L Gen,B =E x～B [l GAN (D B (G A→B (x)),1)] (2);

[0015] Where, L Gen,A and L Gen,B Respectively represent G B→A and G A→B Generate loss; E x～A and E x～B They represent the expectation of the input sample x obeying domain A and domain B respectively; l GAN (.) represents the difference between the color painting generated by minimization and the real color painting data; G B→A (x), G A→B (x) represents G B→A Generator Function and G A→B Generator function; 1 represents the label of the real color painting in the dataset;

[0016] To pass G A→B and G B→A After two conversions, the loss of the original input sample x is restored and the cycle consistency loss function l is set cyc The expression is as follows:

[0017] L cyc,A =E x～A [l cyc (x,G B→ A(G A→B (x)))] (3);

[0018] L cyc,B =E x～B [l cyc (x,G A→B (G B→A (x)))] (4);

[0019] Where, L cyc,A and L cyc,B Represents G A→B and G B→A Cycle consistency loss;

[0020] In order to ensure that the generated color painting is consistent with the original image in structure, the identity loss function is used to perform the transformation within the same domain, where the identity loss function l id The expression is as follows:

[0021] L ID,A =E x～B [l id (x,G A→B (x))] (5);

[0022] L ID,B =E x～A [l id (x,G B→A (x))] (6);

[0023] Where, L ID,A and L ID,B Respectively represent G A→B and G B→A loss of identity;

[0024] The total loss function expression of the UNet-VIT generator obtained by combining formula (1)-formula (6) is as follows:

[0025] L Gen =(L Gen,A +L Gen,B )+γ c (L cyc,A +L cyc,B )+γ i (L ID,A +L ID,B )(7);

[0026] In the formula, γ c and γ i All represent combination coefficients;

[0027] And during the UNet-VIT generator training process, the pixel-by-pixel loss value L is used 1 Predict the original unmasked image:

[0028]

[0029] Where N is the total number of color paintings; y i represents the i-th true value; represents the i-th predicted value; |.| represents the absolute value operation.

[0030] Preferably, the UNet-VIT generator includes a preprocessing module, a U-net basic module, a VIT module combined with a bottleneck layer of the U-net basic module, and a multi-head attention mechanism and a color style modulation convolution module for modulating the decoding branch of the U-net basic module through style, wherein the U-net basic module is used to extract local detail features of the color painting, and the multi-head attention mechanism and the color style modulation convolution module are used to pay attention to the style transformation between the target color painting and the source color painting, so as to capture and transform the style changes between the color paintings at different stages;

[0031] The discriminator strengthens the feature extraction capability through a multi-layer convolutional neural network and introduces a feature statistics fusion consistency strategy.

[0032] Preferably, the U-net basic module includes an encoding path and a decoding path, and the encoding path and the decoding path are connected via a VIT module at the bottleneck layer;

[0033] The encoding path is a multi-level structure, and each layer of the encoding path contains a basic block and a downsampling block, where the basic block is mainly composed of two convolutional layers, and the downsampling block contains a convolutional layer with a stride of 2;

[0034] The decoding path is a multi-level structure, and each layer of the decoding path consists of an upsampling block and a style modulation convolution block;

[0035] The basic block input of the decoding path has a concatenated tensor and a style vector R a , the cascade tensor consists of the output of the upsampling layer and the skip connection tensor of the corresponding level in the encoding path, and the style vector R a Used to change the style of the output image by adjusting the weights of the convolutional layers of the basic blocks of the decoding path;

[0036] The output of the decoding path passes through a 1×1 convolutional post-processing layer and generates an image through a sigmoid activation function.

[0037] Preferably, the VIT module is composed of a multi-layer Transformer encoder for capturing the aesthetic information of the color painting at a fine-grained level; each layer of the Transformer encoder includes a layer normalization, a multi-head attention module, a layer normalization and a feedforward network arranged in sequence, and the feedforward network is configured to have a position encoding length F p =384, the new feature dimension F v =384, the hidden layer dimension F of the feedforward network h =4F v ;

[0038] When constructing the input of the VIT module, the global and local features of the image are captured and processed by flattening the image block into a feature sequence and adding position coding, while retaining the spatial relationship information; and the feature vector of the flattened image block is compared with the corresponding two-dimensional Fourier position coding F p Combine and map the combined feature vector to the new feature dimension F through a linear layer v ;

[0039] ReZero regularization and learnable scaling parameter α are introduced during the training process of the VIT module;

[0040] The output of the Transformer encoder stack is linearly projected back to the original dimension F. 0 , while maintaining spatial resolution.

[0041] Preferably, the layer normalization of the Transformer encoder is used to normalize different features of a single sample;

[0042] The number of attention heads of the multi-head attention module is set to i, that is, the multi-head attention module includes i self-attention modules, and the calculation formula of the self-attention module is as follows:

[0043] Attention(Q,K,V)=softmax(QK T )V (9);

[0044] MultiHead=Concat(Head 1 ,Head 2 ,…,Head i ) (10);

[0045] In the formula, Q, K, and V represent the query vector, key vector, and value vector respectively; Head 1 ,Head 2 ,…,Head i Represent the outputs of the 1st, 2nd,…,ith self-attention modules respectively;

[0046] The feed-forward network of the Transformer encoder consists of two linear layers for further feature extraction and processing of the output of the multi-head attention module.

[0047] Preferably, a style token R is added to the Transformer encoder, and at the output of the Transformer, the state of the style token R is regarded as a potential style to encode color style information;

[0048] In the decoding path of the U-net base module, the style token R is transformed into a style vector R through a trainable linear transformation. a, and through the provided style vector R a Calculate the weight w′ of the style modulated scaled convolution operator a,b,u,v :

[0049] w′ a,b,u,v =R a × a,b,u,v (11);

[0050] Where a, b represent the indexes of the input feature map and the output feature map respectively; u, v represent the indexes of the convolution kernel in the spatial dimension respectively.

[0051] Preferably, the FFCS discriminator uses adaptive instance normalization to fuse the content features and style features extracted by the UNet-VIT generator and performs linear interpolation to match the style features, thereby achieving style transfer;

[0052] The feature fusion calculation formula is as follows:

[0053] FFCS(x,y)=αx+(1-α)[AdaLIN(x,y)] (12);

[0054] In the formula, x and y represent two randomly selected color feature maps; α is the interpolation parameter;

[0055] AdaLIN converts the content feature map into a style map, and the channel mean and variance of the content feature map are the same as those of the style feature map, so as to generate a mixed feature from the two color feature maps x and y:

[0056]

[0057] Where D mix (x,y) represents the output result after n times of mixed features; Represents the mixed feature map of the nth layer of the FFCS discriminator Perform linear transformation to obtain the final output result;

[0058] The FFCS discriminator loss function expression is as follows:

[0059]

[0060] Where D(x) represents the original output; Represents the logarithmic distribution P data The expected value of x,y;

[0061] The FFCS discriminator is updated by back-propagating the loss between real and converted color paintings, and the back-propagated loss function l GAN The expression is as follows:

[0062] LD,A =E x～B [l GAN (D A (G B→A (x)),0]+E x～A [l GAN (D A (x),1]+0.01L FFCS (15);

[0063] L D,B =E x～A [l GAN (D B (G A→B (x)),0]+E x～B [l GAN (D B (x),1]+0.01L FFCS (16);

[0064] Where, L D,A and L D,B Denote the FFCS discriminator D A and FFCS discriminator D B loss.

[0065] The method of the color painting generation system based on the texture and hue information attention mechanism comprises the following steps:

[0066] S1. Constructing a dataset: Collect multiple art joint examination color paintings in different stages to form a dataset, and divide the dataset into a training set and a test set;

[0067] S2. Build UVF-GAN model;

[0068] S3. Generator pre-training of UVF-GAN model: two adjacent stages are divided into a training subset, and each training subset is input into the generator of the UVF-GAN model for pre-training;

[0069] S4, input the training set into the UVF-GAN model for training, and use the test set for verification;

[0070] S5. Use the verified UVF-GAN model to generate stage color paintings.

[0071] Preferably, in step S1, each of the collected art joint examination color paintings in stages includes a shape-forming stage, a three-face-finding stage, a color transition stage, and a detail-depicting stage;

[0072] In step S3, the training subset includes the stage from the shaping stage to the stage from finding the three major faces, from the stage from finding the three major faces to the color transition stage, and from the color transition stage to the detail depiction stage;

[0073] In step S5, when the color painting of the shaping stage is input into the UVF-GAN model, the color painting of the three major surfaces stage is output; when the color painting of the three major surfaces stage is input into the UVF-GAN model, the color painting of the color transition stage is output; when the color painting of the color transition stage is input into the UVF-GAN model, the color painting of the detail portrayal stage is output.

[0074] Therefore, the present invention adopts the above-mentioned color painting generation system and method based on the texture and hue information attention mechanism, which has the following beneficial effects:

[0075] 1. The intelligent step generation of color paintings is gradually realized through the improved recurrent generative adversarial network (GAN), that is, by combining the local feature extraction capability of U-net with the global feature capture capability of the visual transformer (Vision Transformer, VIT), the generated paintings can maintain high detail fidelity and global consistency; and the training weights of the model are fixed after the training is completed, ensuring that the generated effect is only related to the input painting, thereby ensuring the objectivity and consistency of the generated results in the teaching process;

[0076] 2. The combination of the visual transformer (VIT) and the color style modulation convolution block can fully capture the global information of the painting and the style changes at different stages. At the same time, the local features are extracted through U-net, so that the local and global information are organically combined, which effectively improves the model's ability to generate high-quality color paintings at different stages.

[0077] 3. Introducing a feature statistics fusion consistency strategy into the discriminator to make its style prediction more robust and improve the overall quality and consistency of generated images;

[0078] 4. The generator pre-training strategy was introduced before model training to further accelerate the convergence speed of the model;

[0079] 5. The constructed UVF-GAN model can be easily integrated into mini-programs or mobile applications, and supports real-time application of color painting generation (the system can generate an image within a dozen seconds), which greatly improves teaching efficiency. At this time, students can use it as a reference for copying and learning, and it also provides teachers with auxiliary teaching tools, alleviating the problem of insufficient teacher resources and subjectivity.

[0080] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 A generative adversarial network diagram of a color painting generation system based on a texture and hue information attention mechanism of the present invention;

[0082] Figure 2 This is a flow chart of the UNet-VIT generator of a color painting generation system based on the texture and hue information attention mechanism of the present invention;

[0083] Figure 3 This is a flow chart of a VI T visual converter of a color painting generation system based on a texture and hue information attention mechanism of the present invention;

[0084] Figure 4 A comparison diagram of the FFCS discriminator of the present invention and a traditional discriminator;

[0085] Figure 5 A flowchart of a method for a color painting generation system based on a texture and hue information attention mechanism of the present invention;

[0086] Figure 6 An example diagram of a color painting in stages of a method for a color painting generation system based on a texture and hue information attention mechanism of the present invention;

[0087] Figure 7 A schematic diagram of pre-training of a method for a color painting generation system based on a texture and hue information attention mechanism according to the present invention. DETAILED DESCRIPTION

[0088] In the description of the present invention, it should be noted that the terms "upper", "lower", "inside", "outside", etc. indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, or the positions or positional relationships in which the invented product is usually placed when in use. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In the description of the present invention, it should also be noted that, unless otherwise clearly specified and limited, the terms "setting", "installation", and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be a connection between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0089] The embodiments of the present invention are described in detail below in conjunction with the accompanying drawings.

[0090] like Figure 1-Figure 4As shown, a color painting generation system based on texture and hue information attention mechanism includes a UVF-GAN model. The UVF-GAN model includes two-way generative adversarial networks. Both of the two generative adversarial networks include a UNet-VIT generator based on color style modulation and a FFCS discriminator based on feature statistics fusion consistency strategy. The UNet-VIT generator and the FFCS discriminator translate unpaired images to images in an interleaved manner.

[0091] Set the two UNet-VIT generators to be UNet-VIT generators G from domain A to domain B A→B and UNet-VIT generator G from domain B to domain A B→A ;

[0092] To ensure that the images generated by the UNet-VIT generator are judged as real images by the FFCS discriminator, the generation loss function is used to calculate G B→A The generation loss and G A→B The generation loss is to calculate G B→A The generated image is passed to the FFCS discriminator D A The probability of judging it as a real image and calculating G A→B The generated image is passed to the FFCS discriminator D B The probability of judging it as a real image, where the loss function l is generated GAN The expression is as follows:

[0093] L Gen,A =E x～A [l GAN (D A (G B→ A(x)),1)] (1);

[0094] L Gen,B =E x～B [l GAN (D B (G A→B (x)),1)] (2);

[0095] Where, L Gen,A and L Gen,B Respectively represent G B→A and G A→B Generate loss; E x～A and E x～B They represent the expectation of the input sample x obeying domain A and domain B respectively; l GAN (.) represents the difference between the color painting generated by minimization and the real color painting data; G B→A (x), G A→B (x) represents G B→A Generator Function and GA→B Generator function; 1 represents the label of the real color painting in the dataset;

[0096] To pass G A→B and G B→A After two conversions, the loss of the original input sample x is restored and the cycle consistency loss function l is set cyc The expression is as follows:

[0097] L cyc,A =E x～A [l cyc (x,G B→A (G A→B (x)))] (3);

[0098] L cyc,B =E x～B [l cyc (x,G A→B (G B→A (x)))] (4);

[0099] Where, L cyc,A and L cyc,B Represents G A→B and G B→A Cycle consistency loss;

[0100] In order to ensure that the generated color painting is consistent with the original image in structure, the identity loss function is used to perform the transformation within the same domain, where the identity loss function l id The expression is as follows:

[0101] L ID,A =E x～B [l id (x,G A→B (x))] (5);

[0102] L ID,B =E x～A [l id (x,G B→A (x))] (6);

[0103] Where, L ID,A and L ID,B Respectively represent G A→B and G B→A loss of identity;

[0104] The total loss function expression of the UNet-VIT generator obtained by combining formula (1)-formula (6) is as follows:

[0105] L Gen =(L Gen,A +L Gen,B )+γc (L cyc,A +L cyc,B )+γ i (L ID,A +L ID,B )(7);

[0106] In the formula, γ c and γ i All represent combination coefficients;

[0107] And during the UNet-VIT generator training process, the pixel-by-pixel loss value L is used 1 Predict the original unmasked image:

[0108]

[0109] Where N is the total number of color paintings; y i represents the i-th true value; represents the i-th predicted value; |.| represents the absolute value operation.

[0110] Preferably, the UNet-VIT generator includes a preprocessing module, a U-net basic module, a VIT module combined with a bottleneck layer of the U-net basic module, and a multi-head attention mechanism and a color style modulation convolution module for modulating the decoding branch of the U-net basic module through style, wherein the U-net basic module is used to extract local detail features of the color painting, and the multi-head attention mechanism and the color style modulation convolution module are used to pay attention to the style transformation between the target color painting and the source color painting, so as to capture and transform the style changes between the color paintings at different stages;

[0111] The discriminator strengthens the feature extraction capability through a multi-layer convolutional neural network and introduces a feature statistics fusion consistency strategy.

[0112] In this embodiment, the preprocessing layer is used to convert the image into a tensor with a dimension of (W, H, F) = (256, 256, 48), where W and H are spatial positions and F is the number of feature channels;

[0113] The width and height of the preprocessed tensor are halved after each downsampling block in the encoding path, and the feature dimension is doubled after the last three downsampling blocks, so that the output dimension of the encoding path is (W0, H0, F0) = (W / 16, H / 16, 8F) = (16, 16, 384);

[0114] Preferably, the U-net basic module includes an encoding path and a decoding path, and the encoding path and the decoding path are connected via a VIT module at the bottleneck layer;

[0115] The encoding path is a multi-level structure, and each layer of the encoding path contains a basic block and a downsampling block, where the basic block is mainly composed of two convolutional layers, and the downsampling block contains a convolutional layer with a stride of 2;

[0116] The decoding path is a multi-level structure, and each layer of the decoding path consists of an upsampling block and a style modulation convolution block; the basic block input of the decoding path has a cascade tensor and a style vector R a , the cascade tensor consists of the output of the upsampling layer and the skip connection tensor of the corresponding level in the encoding path, and the style vector R a Used to change the style of the output image by adjusting the weights of the convolutional layer of the basic block of the decoding path; the decoding path output passes through a 1×1 convolution post-processing layer and generates an image through a sigmoid activation function.

[0117] The VIT module consists of a multi-layer Transformer encoder, which is used to capture the aesthetic information of the color painting at a fine-grained level; each layer of the Transformer encoder includes layer normalization, multi-head attention module, layer normalization and feedforward network, and the feedforward network is configured as the length F of the positional encoding. p =384, the new feature dimension F v =384, the hidden layer dimension F of the feedforward network h =4F v ;

[0118] When constructing the input of the VIT module, the global and local features of the image are captured and processed by flattening the image block into a feature sequence and adding position coding, while retaining the spatial relationship information; and the feature vector of the flattened image block is compared with the corresponding two-dimensional Fourier position coding F p Combine and map the combined feature vector to the new feature dimension F through a linear layer v ;

[0119] ReZero regularization and learnable scaling parameter α are introduced during the training process of the VIT module;

[0120] The output of the Transformer encoder stack is linearly projected back to the original dimension F. 0 , while maintaining the spatial resolution, that is, maintaining the width W of the input image 0 and height H 0 constant.

[0121] The layer normalization of the Transformer encoder is used to normalize the different features of a single sample;

[0122] The number of attention heads of the multi-head attention module is set to i, that is, the multi-head attention module includes i self-attention modules, and the calculation formula of the self-attention module is as follows:

[0123] Attention(Q,K,V)=softmax(QK T )V (9);

[0124] MultiHead=Concat(Head 1 ,Head 2 ,…,Head i ) (10);

[0125] In the formula, Q, K, and V represent the query vector, key vector, and value vector respectively; Head 1 ,Head 2 ,…,Head i Represent the outputs of the 1st, 2nd,…,ith self-attention modules respectively;

[0126] The feed-forward network of the Transformer encoder consists of two linear layers for further feature extraction and processing of the output of the multi-head attention module.

[0127] A style token R is added to the Transformer encoder, and at the output of the Transformer, the state of the style token R is regarded as a potential style to encode color style information;

[0128] In the decoding path of the U-net base module, the style token R is transformed into a style vector R through a trainable linear transformation. a , and through the provided style vector R a Calculate the weight w′ of the style modulated scaled convolution operator a,b,u,v :

[0129] w′ a,b,u,v =R a × a,b,u,v (11);

[0130] Where a, b represent the indexes of the input feature map and the output feature map respectively; u, v represent the indexes of the convolution kernel in the spatial dimension respectively.

[0131] The FFCS discriminator uses adaptive instance normalization to fuse the content features and style features extracted by the UNet-VIT generator, and performs linear interpolation to match the style features, thereby achieving style transfer; the feature fusion calculation formula is as follows:

[0132] FFCS(x,y)=αx+(1-α)[AdaLIN(x,y)] (12);

[0133] In the formula, x and y represent two randomly selected color feature maps; α is the interpolation parameter;

[0134] AdaLIN converts the content feature map into a style map, and the channel mean and variance of the content feature map are the same as those of the style feature map, so as to generate a mixed feature from the two color feature maps x and y:

[0135]

[0136] Where D mix (x,y) represents the output result after n times of mixed features; Represents the mixed feature map of the nth layer of the FFCS discriminator Perform linear transformation to obtain the final output result;

[0137] The FFCS discriminator loss function expression is as follows:

[0138]

[0139] Where D(x) represents the original output; Represents the logarithmic distribution P data The expected value of x,y;

[0140] The FFCS discriminator is updated by back-propagating the loss between real and converted color paintings, and the back-propagated loss function l GAN The expression is as follows:

[0141] L D,A =E x～B [l GAN (D A (G B→A (x)),0]+E x～A [l GAN (D A (x),1]+0.01L FFCS (15);

[0142] L D,B =E x～A [l GAN (D B (G A→B (x)),0]+E x～B [l GAN (D B (x),1]+0.01L FFCS (16);

[0143] Where, L D,A and L D,B Denote the FFCS discriminator DA and FFCS discriminator D B loss.

[0144] like Figure 5-Figure 7 As shown, the method of the color painting generation system based on the texture and hue information attention mechanism comprises the following steps:

[0145] S1. Constructing a dataset: Collect multiple art joint examination color paintings in different stages to form a dataset, and divide the dataset into a training set and a test set;

[0146] In step S1, each collected art joint examination color painting in stages includes a shaping stage, a three-face finding stage, a color transition stage, and a detail depiction stage; and in this embodiment, in order to ensure the performance of the model, the amount of image data collected in each step is kept balanced.

[0147] S2. Build UVF-GAN model;

[0148] S3. Generator pre-training of UVF-GAN model: two adjacent stages are divided into a training subset, and each training subset is input into the generator of the UVF-GAN model for pre-training;

[0149] In step S3, the training subset includes the stage from the shaping stage to the stage from the three major faces to the stage from the stage from the three major faces to the stage from the stage from the stage from the stage from the stage from the stage from the stage from the stage from the stage from the stage from the stage from the stage to the stage from the stage from the stage of ...

[0150] S4, input the training set into the UVF-GAN model for training, and use the test set for verification;

[0151] S5. Use the verified UVF-GAN model to generate stage color paintings.

[0152] In step S5, when the color painting of the shaping stage is input into the UVF-GAN model, the color painting of the three major surfaces stage is output; when the color painting of the three major surfaces stage is input into the UVF-GAN model, the color painting of the color transition stage is output; when the color painting of the color transition stage is input into the UVF-GAN model, the color painting of the detail portrayal stage is output.

[0153] In the UVF-GAN model, the U-net basic module is first used to extract the global features of the color painting, and then the VIT module is combined at its bottleneck to extract the texture, shape, edge and other features of the color painting. The color style modulation convolution block focuses on the style transformation between the target color painting and the source color painting; finally, in the FFCS discriminator part, a multi-layer convolutional network is used to extract features, and then the feature statistical fusion consistency strategy is applied to predict the style of the painting.

[0154] It should be noted that the trained and optimized UVF-GAN model can be integrated into a mini-program or application platform, where users only need to upload a color painting of a certain stage, and the system can generate the next stage of the painting. This deployment method is not only convenient and easy to use, but also supports students or teachers to learn painting and assist in teaching anytime and anywhere on different devices, greatly improving the application value and teaching efficiency of the system.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A color painting generation system based on texture and hue information attention mechanism, characterized by: Including the UVF-GAN model, the UVF-GAN model includes a two-way generative adversarial network, both of which include a UNet-VIT generator based on color style modulation and a FFCS discriminator based on feature statistics fusion consistency strategy. The UNet-VIT generator and the FFCS discriminator translate unpaired images to images in an interleaved manner.

2. The color painting generation system based on texture and hue information attention mechanism according to claim 1, characterized in that: Set the two UNet-VIT generators to be UNet-VIT generators G from domain A to domain B A→B and UNet-VIT generator G from domain B to domain A B→A ; To ensure that the images generated by the UNet-VIT generator are judged as real images by the FFCS discriminator, the generation loss function is used to calculate G B→A The generation loss and G A→B The generation loss is to calculate G B→A The generated image is passed to the FFCS discriminator D A The probability of judging it as a real image and calculating G A→B The generated image is passed to the FFCS discriminator D B The probability of being judged as a real image, where the loss function l is generated GAN The expression is as follows: L Gen,A =E x～A [l GAN (D A (G B→A (x)),1)] (1); L Gen,B =E x～B [l GAN (D B (G A→B (x)),1)] (2); Where, L Gen,A and L Gen,B Respectively represent G B→A and G A→B Generate loss; E x～A and E x～B They represent the expectation of the input sample x obeying domain A and domain B respectively; l GAN (.) represents the difference between the color painting generated by minimization and the real color painting data; G B→A (x), G A→B (x) represents G B→A Generator Function and G A→B Generator function; 1 represents the label of the real color painting in the dataset; To pass G A→B and G B→A After two conversions, the loss of the original input sample x is restored and the cycle consistency loss function l is set cyc The expression is as follows: L cyc,A =E x～A [l cyc (x,G B→A (G A→B (x)))] (3); L cyc,B =E x～B [l cyc (x,G A→B (G B→A (x)))] (4); Where, L cyc,A and L cyc,B Represents G A→B and G B→A Cycle consistency loss; In order to ensure that the generated color painting is consistent with the original image in structure, the identity loss function is used to perform the transformation within the same domain, where the identity loss function l id The expression is as follows: L ID,A =E x～B [l id (x,G A→B (x))] (5); L ID,B =E x～A [l id (x,G B→A (x))] (6); Where, L ID,A and L ID,B Respectively represent G A→B and G B→A loss of identity; The total loss function expression of the UNet-VIT generator obtained by combining formula (1)-formula (6) is as follows: L Gen =(L Gen,A +L Gen,B )+γ c (L cyc,A +L cyc,B )+γ i (L ID,A +L ID,B )(7); In the formula, γ c and γ i All represent combination coefficients; And during the UNet-VIT generator training process, the pixel-by-pixel loss value L1 is used to predict the original unmasked image: Where N is the total number of color paintings; y i represents the i-th true value; represents the i-th predicted value; |.| represents the absolute value operation.

3. The color painting generation system based on texture and hue information attention mechanism according to claim 2, characterized in that: The UNet-VIT generator includes a preprocessing module, a U-net basic module, a VIT module combined with the bottleneck layer of the U-net basic module, and a multi-head attention mechanism and a color style modulation convolution module for style modulation of the bottle decoding branch of the U-net basic module, wherein the U-net basic module is used to extract local detail features of the color painting, and the multi-head attention mechanism and the color style modulation convolution module are used to pay attention to the style transformation between the target color painting and the source color painting, so as to capture and transform the style changes between the color paintings at different stages; The discriminator strengthens the feature extraction capability through a multi-layer convolutional neural network and introduces a feature statistics fusion consistency strategy.

4. The color painting generation system based on texture and hue information attention mechanism according to claim 3, characterized in that: The U-net basic module includes an encoding path and a decoding path, and the encoding path and the decoding path are connected through the VIT module at the bottleneck layer; The encoding path is a multi-level structure, and each layer of the encoding path contains a basic block and a downsampling block, where the basic block is mainly composed of two convolutional layers, and the downsampling block contains a convolutional layer with a stride of 2; The decoding path is a multi-level structure, and each layer of the decoding path consists of an upsampling block and a style modulation convolution block; The basic block input of the decoding path has a concatenated tensor and a style vector R a , the cascade tensor consists of the output of the upsampling layer and the skip connection tensor of the corresponding level in the encoding path, and the style vector R a Used to change the style of the output image by adjusting the weights of the convolutional layers of the basic blocks of the decoding path; The output of the decoding path passes through a 1×1 convolutional post-processing layer and generates an image through a sigmoid activation function.

5. The color painting generation system based on texture and hue information attention mechanism according to claim 4, characterized in that: The VIT module consists of a multi-layer Transformer encoder, which is used to capture the aesthetic information of the color painting at a fine-grained level; each layer of the Transformer encoder includes layer normalization, multi-head attention module, layer normalization and feedforward network, and the feedforward network is configured as the length F of the positional encoding. p =384, the new feature dimension F v =384, the hidden layer dimension F of the feedforward network h =4F v ; When constructing the input of the VIT module, the global and local features of the image are captured and processed by flattening the image block into a feature sequence and adding position coding, while retaining the spatial relationship information; and the feature vector of the flattened image block is compared with the corresponding two-dimensional Fourier position coding F p Combine and map the combined feature vector to the new feature dimension F through a linear layer v ; ReZero regularization and learnable scaling parameter α are introduced during the training process of the VIT module; The output of the Transformer encoder stack is linearly projected back to the original dimension F0 while maintaining the spatial resolution.

6. The color painting generation system based on texture and hue information attention mechanism according to claim 5, characterized in that: The layer normalization of the Transformer encoder is used to normalize the different features of a single sample; The number of attention heads of the multi-head attention module is set to i, that is, the multi-head attention module includes i self-attention modules, and the calculation formula of the self-attention module is as follows: Attention(Q,K,V)=softmax(QK T )V (9); MultiHead=Concat(Head1,Head2,…,Head i ) (10); Where Q, K, and V represent query vectors, key vectors, and value vectors respectively; Head1, Head2, …, Head i Represent the outputs of the 1st, 2nd,…,ith self-attention modules respectively; The feed-forward network of the Transformer encoder consists of two linear layers for further feature extraction and processing of the output of the multi-head attention module.

7. The color painting generation system based on texture and hue information attention mechanism according to claim 6, characterized in that: A style token R is added to the Transformer encoder, and at the output of the Transformer, the state of the style token R is regarded as a potential style to encode color style information; In the decoding path of the U-net base module, the style token R is transformed into a style vector R through a trainable linear transformation. a , and through the provided style vector R a Calculate the weight w′ of the style modulated scaled convolution operator a,b,u,v : w′ a,b,u,v =R a ×w a,b,u,v (11); Where a, b represent the indexes of the input feature map and the output feature map respectively; u, v represent the indexes of the convolution kernel in the spatial dimension respectively.

8. The color painting generation system based on texture and hue information attention mechanism according to claim 7, characterized in that: The FFCS discriminator uses adaptive instance normalization to fuse the content features and style features extracted by the UNet-VIT generator and performs linear interpolation to match the style features, thereby achieving style transfer; The feature fusion calculation formula is as follows: FFCS(x,y)=αx+(1-α)[AdaLIN(x,y)] (12); In the formula, x and y represent two randomly selected color feature maps; α is the interpolation parameter; AdaLIN converts the content feature map into a style map, and the channel mean and variance of the content feature map are the same as those of the style feature map, so as to generate a mixed feature from the two color feature maps x and y: Where D mix (x,y) represents the output result after n times of mixed features; Represents the mixed feature map of the nth layer of the FFCS discriminator Perform linear transformation to obtain the final output result; The FFCS discriminator loss function expression is as follows: Where D(x) represents the original output; Represents the logarithmic distribution P data The expected value of x,y; The FFCS discriminator is updated by back-propagating the loss between real and converted color paintings, and the back-propagated loss function l GAN The expression is as follows: L D,A JE x～B [l GAN (D A (G B→A (x)),0]+E x～A [l GAN (D A (x),1]+0.01L FFCS (15) L D,B JE x～A [l GAN (D B (G A→B (x)),0]+E x～B [l GAN (D B (x),1]+0.01L FFCS (16) Where, L D,A and L D,B Denote the FFCS discriminator D A and FFCS discriminator D B loss.

9. The method of the color painting generation system based on the texture and hue information attention mechanism as claimed in claim 8, characterized in that: The following steps are involved: S1. Constructing a dataset: Collect multiple art joint examination color paintings in different stages to form a dataset, and divide the dataset into a training set and a test set; S2. Build UVF-GAN model; S3. Generator pre-training of UVF-GAN model: two adjacent stages are divided into a training subset, and each training subset is input into the generator of the UVF-GAN model for pre-training; S4, input the training set into the UVF-GAN model for training, and use the test set for verification; S5. Use the verified UVF-GAN model to generate stage color paintings.

10. The method of color painting generation system based on texture and hue information attention mechanism as claimed in claim 9, characterized in that: In step S1, each of the collected art joint examination color paintings in stages includes a shape-forming stage, a three-face-finding stage, a color transition stage, and a detail-depicting stage; In step S3, the training subset includes the stage from the shaping stage to the stage from finding the three major faces, from the stage from finding the three major faces to the color transition stage, and from the color transition stage to the detail depiction stage; In step S5, when the color painting of the shaping stage is input into the UVF-GAN model, the color painting of the three major surfaces stage is output; when the color painting of the three major surfaces stage is input into the UVF-GAN model, the color painting of the color transition stage is output; when the color painting of the color transition stage is input into the UVF-GAN model, the color painting of the detail portrayal stage is output.

Images