Color icon automatic generation method based on Transform

Abstract

The invention discloses a Transform-based icon style migration model, which is characterized in that a VGG encoder pre-trained by ImageNet is used for respectively extracting content characteristics of a content icon and style characteristics of a style icon, and a style encoder is designed for encoding the style characteristics to generate style values and style keys of style codes. The method comprises the steps that firstly, a content feature is obtained, then, a decoder based on Transform is constructed to stylize the content feature according to a style value and a style key, a content icon with a new style is decoded, and finally, the generated icon is further refined through a convolutional neural network decoder. According to the method, a new icon image can be generated only by inputting any two icons as contents and styles, the number of available icons is greatly increased, and the efficiency of software development UI interface design is improved.

Description

Technical Field

[0001] The present invention relates to the field of icon generation, and in particular to a transformer-based automatic icon generation method. Background Art

[0002] With the rapid development of computer technology and artificial intelligence, icons as a material for human-computer interface have become an indispensable part of modern applications and websites. Icons are a universal graphic symbol with intuitive and easy-to-understand characteristics that can quickly convey information and enhance user experience. In user interface design, a simple and beautiful icon can not only attract the user's attention, but also help users quickly identify functions and improve user efficiency. A software or website usually requires a set of icons with consistent style and harmonious color matching to build the software interface, but the design and creation of such a set of icons that meet the requirements requires professional artists to complete, which will incur high labor and time costs. Icons can be divided into 2D icons, 3D icons, and 2.5D icons in between according to their visual three-dimensional effects. Each type of icon can carry monochrome or multi-color attributes. The color change of 2D icons is relatively simple, usually in the form of monochrome, multi-color perspective, gradient multi-color, multi-color overlay, etc. 3D icons are more vivid and three-dimensional visually, with complex color distribution and structure. They can vividly simulate the three-dimensional sense and morphological changes of objects in the real world. Through reasonable lighting and shadow processing, 3D icons can present realistic and three-dimensional effects. 2.5D icons are between 2D and 3D, with a certain three-dimensional effect, but the sense of reality is not as strong as 3D icons. Because 3D icons have complex color distribution and structural characteristics, and some have facade effects such as shadows and gradients, manual rendering of 3D icons requires higher skills and requires more time and energy for creation. Although 2D icons and 2.5D icons are simpler in structure than 3D icons, it also takes a certain amount of manpower and time to generate icons with rich expression effects. If the deep learning method can be used to solve the problem of color icon generation, it can not only greatly reduce the workload of manual drawing, but also enrich the icon material library, providing more options for making harmonious and consistent UI interfaces. One method of icon generation is to color the monochrome icon sketch to generate the expected color icon image. Researchers have conducted preliminary explorations on this type of problem. For example, Sun et al. designed a dual GANs to colorize icon sketches based on reference images. The model can achieve good colorization effects for icons with simple structures, but the colorization results of some samples will have color overflow, and the colorization effect is very poor for icons with complex structures. Li et al. proposed a method for coloring icon sketches, separating style and structural features, and using a normalized flow network to generate diverse and realistic color icons. Another method of icon generation is to use a style transfer method, which transfers the style of another icon to the original icon while keeping the main body of the original content unchanged, achieving the effect of style transfer, thereby generating a new style of specific content icon.Gatys et al. found that different layers of convolutional neural networks can separate the content structure and style information of images, extract the style feature representation of images through the Gram matrix, and use optimization-based methods to continuously iterate and generate stylized images. In recent years, arbitrary style transfer has received more attention. Huang et al. added adaptive instance normalization (AdaIN) to the decoder-encoder framework, adjusted the mean and variance of content image features in the feature space to match the mean and variance of the style image, and realized real-time, arbitrary style image style transfer. AdaIN is widely used in image generation tasks to combine content and style features. Deng et al. introduced self-attention into the encoder and decoder framework based on the CNNs model to achieve the combination of content and style features. The method used in the present invention belongs to the second category, namely, the method of style transfer. The present invention extracts the content features of the content icon and the style features of the style icon respectively based on the VGG19 model, and encodes the style features with a style encoder to form style values ​​and keys. Finally, based on the Transformer decoder, the content encoding is stylized by the style encoding to generate an icon image after style transfer, thereby realizing the task of icon generation. Summary of the invention

[0003] The present invention designs a Transformer-based icon style transfer model, uses the VGG19 encoder pre-trained with the ImageNet dataset to extract the content features of the content icon and the style features of the style icon, and designs a style encoder to encode the style features to generate style values ​​and style keys of the style encoding. Then, a Transformer-based decoder is constructed to stylize the content features according to the style values ​​and style keys, decodes a content icon with a specific style, and finally uses a convolutional neural network decoder to further refine the generated icon. Specifically, it includes the following steps:

[0004] (1) Construct an icon dataset (101). Most of the dataset is collected from IconScout, Free3Dicon and Liutu websites, including 43,741 icons from 48 categories, including laboratories, charts, people and payments. Compared with the dataset composed of realistic images, the icons in the icon dataset are manually designed, covering different types, structures and color distributions. Some have three-dimensional effects, complex structures or gradient colors. The samples in the dataset are rich and diverse. In order to train and evaluate the model, the created icon dataset is divided into a training set and a test set in a ratio of 9:1, with 41,554 icons in the training set and 2,187 icons in the test set.

[0005] (2) Building a Transformer-based icon style transfer model (102). The icon generation method proposed in the present invention is implemented by using the Transformer-based icon style transfer model. The model structure is as follows: Figure 2 As shown, it includes a VGG19 encoder (201), a style encoder (202), a Transformer decoder (203), and a CNN decoder (204).

[0006] (3) Train the Transformer-based icon style transfer model (103). The training parameters of the model are set as follows: the learning rate is 0.0005, all parameters are optimized using the Adam optimizer, the optimizer parameters β1 and β2 are set to 0.5 and 0.999, the batch size is set to 8, and 160,000 iterations of training are performed. First, the training data in the dataset created in step 101 is randomly divided into two parts, 50% of which are used as content icons and the other 50% as style icons. After random pairing, they are input into the VGG19 encoder 201 pre-trained with ImageNet to extract the content features of the content icons and the style features of the style icons respectively; secondly, the style features are input into the style encoder 202, with the aim of discovering a set of specific style codes from the input style features, which include style keys K and style values ​​V. Structure of the style encoder 202 Figure 3 As shown, the style encoder 202 has two parallel branches, one for extracting the style value code V and the other for extracting the style key code K. In the branch encoding the style value, there are two convolutional layers with a step size of 2 (301) and three convolutional layers with a step size of 1 (302). In the branch encoding the style key, there are two convolutional layers with a step size of 2 (303); then, the style key K and style value V in the style code, together with the query matrix Q encoded by the convolutional layer of the content features, are input into the Transformer decoder 203 for decoding. Structure of the Transformer Decoder Figure 4As shown, the decoder includes three layers (205), each layer includes two multi-head attention modules (MSA) (401) and a feed-forward neural network (FFN) (402), the multi-head attention module 401 has N different attention modules, each attention head will learn different feature representations, the outputs of the N attention heads are connected, multiplied by the weight matrix for linear transformation, and the final output of the multi-head attention mechanism is obtained. The feed-forward neural network FFN includes two linear transformation layers, wherein the first linear transformation layer is followed by a RELU activation function. Q, K and V pass through the two multi-head attention modules MSA in the Transformer encoder, and then are processed by the feed-forward neural network FFN to form a new icon code, wherein both MSA and FFN use residual connection and normalization (403) processing. Finally, the new icon code is decoded and refined again by a three-layer CNN decoder 204, each layer of which includes a 3×3 convolution, ReLU and 2x upsampling operations, and the final output channel number is 3, and the size is consistent with the content icon after stylization.

[0007] (4) Test and apply the Transformer-based icon style transfer model to generate new icons (104). The Transformer-based icon style transfer model trained in step 103 is tested using the test set data constructed in step 101. If the test results do not show overfitting, the model can be used to generate new icons. The specific application method is as follows: If we have an icon A of a certain content and need to make it have the same style as icon B, we only need to input icon A as the content icon and icon B as the style icon into the model constructed by the present invention, and the model will output a new icon C with the same content as A and the same style as B.

[0008] The present invention applies Transformer to the task of icon style transfer, and adds a multi-head attention mechanism to the model's codec network to effectively capture the long-distance dependency of the image. The model only needs to be trained once, and users can input the icon and style icon of the content they want to retain, and the stylized icon can be output. It is convenient and efficient, and can generate icons that meet the requirements for users without design experience, and can provide design inspiration for designers with design experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A flowchart of a method for generating an icon based on Transformer according to an embodiment of the present invention;

[0010] Figure 2 is the overall network structure of the icon style transfer model according to an embodiment of the present invention;

[0011] Figure 3is a network structure of a style encoder according to an embodiment of the present invention;

[0012] Figure 4 Schematic diagram of a network structure based on a Transformer decoder according to an embodiment of the present invention. DETAILED DESCRIPTION

[0013] In order to more specifically describe the purpose, technical solutions and advantages of the present invention, the present invention is further described below in conjunction with the accompanying drawings and specific implementation methods. The process of the method of an embodiment is as follows: Figure 1 The present invention proposes an icon style transfer method based on Transformer, which includes the following steps:

[0014] (1) Collect a sufficient number of icon image samples of different types and color styles from the Internet.

[0015] (2) The obtained icon samples are divided into training set and test set in a ratio of 9:1. 30% of the samples in the training set and test set are used as style icons, and 70% of the samples are used as content icons. The sizes of style icons and content icons are normalized and uniformly scaled to 512×512 pixels. Then, a 256×256 pixel area is randomly cropped for model training and testing.

[0016] (3) Set the content icon I to 256×256 c and style icon I s Input into the ImageNet pre-trained VGG19 encoder to extract the content feature Z of the content icon c and style features of style icons Z s .

[0017] (4) The style feature Z s Input to the style encoder, output two parallel encoding branches, one extracts the style value V, and the other extracts the style key K. The network structure of the style encoder is as follows Figure 3 As shown, the branch of the style value encoding V consists of two convolutional layers with a stride of 2 and three convolutional layers with a stride of 1, and the branch of the style key encoding K consists of two convolutional layers with a stride of 2. Figure 3 Conv(C out ,K,S) in C out , K and S are the number of output channels, convolution kernel size and convolution step size respectively.

[0018] (5) The Transformer decoder maps the encoded content features into stylized feature encodings in a regression manner according to the encoded style features. The Transformer decoder consists of two multi-head attention modules (MSA) and a feed-forward neural network (FFN) that transform the content features Z c As the query vector Q, together with the style key K and style value V, it is input into the multi-head attention module. After passing through N different attention modules, each attention head will learn different representations of queries, keys, and values. The input sequence is weighted by calculating the attention weight, and then the weighted result is multiplied by the value matrix to obtain the output of each position. The outputs of the N attention heads are connected and multiplied by the weight matrix W. o Perform a linear transformation to obtain the final output of the multi-head attention mechanism. The output of the first multi-head attention module is normalized by residual connection and LayerNorm, and the output sequence X1 is input to the second multi-head attention module. It is also residually connected and normalized to output the sequence X2, and then input into the feedforward neural network FFN. FFN includes two linear transformation layers. The first linear transformation layer has a RELU activation function, and dropout is applied to each layer to prevent the neural network model from overfitting. Finally, a residual connection is applied to add the output of the feedforward neural network to the input, and the result is normalized using LayerNorm to obtain the final output stylized code Z cs The processing process is described by formulas (1)-(4):

[0019] MultiHead(Q,K,V)=Concat(Attention1(Q,K,V),...,Attention N (Q,K,V ))W o (1)

[0020] Z1=LayerNorm(MultiHead(Q,K,V)+Q) (2)

[0021] Z2=LayerNorm(MultiHead(Z1,K,V)+Z1) (3)

[0022] Z cs =LayerNorm(FFN(Z2)+Z2) (4)

[0023] Where N is the number of attention heads, FFN(·) is a fully connected feed-forward neural network, and LayerNorm(·) is layer normalization.

[0024] (6) The CNN decoder is used to decode the output sequence X of the Transformer decoder again, and after a 3×3 convolution, ReLU and 2x upsampling operation, the stylized image I is finally output. cs .

[0025] (7) The loss function of the training network consists of content loss, style loss, consistency loss, and gradient loss. In order to preserve the structural information of the content icon and the style information of the style icon, the content icon I is calculated. c and output icon I cs The difference between the content c and style icon I s and output icon I cs The style difference between s As shown in formulas (5)-(6):

[0026]

[0027] in, represents the layer used to calculate the style loss in VGG19, μ(·) and σ(·) represent the mean and variance of the extracted features, respectively, and N l It is the number of layers.

[0028] The consistency loss is used to help the generative model learn richer and more accurate content and style features. The content icon (or style icon) is input into the model as both content and style to extract content and style features and reconstruct the resulting icon I cc (or I ss ), the resulting icon should be the same as the original icon c (or I s ) is as consistent as possible, so two consistency loss terms are calculated to measure I c and I cc and I s and I ss The consistency loss is shown in formulas (7) and (8):

[0029] L identity1 =‖ I cc -I c ‖2+‖I ss -I s ‖2 (7)

[0030]

[0031] The gradient loss calculates the gradient difference between the content icon and the generated icon to maintain the color consistency of each region of the generated icon. It is obtained by applying the Sobel convolution kernel to calculate the gradient in the horizontal and vertical directions, as shown in formula (9):

[0032]

[0033] Among them, Conv(I,K) represents the convolution operation of image I and convolution kernel K, K x and K y are the Sobel convolution kernels in the horizontal and vertical directions respectively, and N is the number of pixels in the image.

[0034] By constantly minimizing the following L total Function to optimize the entire network:

[0035] L total =λ c L c + λ s L s + λ identity1 L identity1 + λ identity2 L identity2 +λ gd L gd (10)

[0036] Among them, λ c , s , identity1 and λ identity2 represent the loss weights and are set to 1, 1, 50 and 1 respectively.

[0037] The present invention can generate a new 3D icon by transferring the style of a 3D style icon to another 3D content icon, and can also generate a new 3D icon by transferring the style of a 3D style icon to another flat content icon. Two different specific implementations are given below:

[0038] Example 1

[0039] (1) Prepare training data

[0040] Prepare a sufficient number of 3D icons, resize all images to RGB images with an image size of 256×256 and 3 channels, and divide all data into training and test sets at a ratio of 9:1. Before each training, randomly divide the acquired 3D icon data training set, with 30% of the icons as style icons and 70% of the icons as content icons. Normalize the size of style icons and content icons, first adjust to 512×512 pixels, and then randomly crop out 256×256 pixel image areas for model training.

[0041] (2) Training a Transformer-based icon style transfer model

[0042] The preprocessed content icons and style icons are input into the ImageNet pre-trained VGG19 encoder to extract the content feature Z c and style features Z s . Style Features Z s The two branches of the style encoder generate the style value V and style key K, and encode the content Z c As the query matrix Q, it is input into the two multi-head attention modules shown in formulas (2)-(3) and outputs X2. After passing through the feedforward neural network shown in formula (4), the stylized sequence X is output. Finally, a CNN decoder composed of three convolutional layers decodes the stylized sequence X again to generate a stylized icon I with a channel number of 3 and a size consistent with the content icon. cs The loss of the training network includes: Using formulas (5) and (6) to calculate the content icon I c and output icon I cs The content loss between c And style image I s And the output image I cs The style loss L between s , use formulas (7) and (8) to calculate the consistency loss of the content icon and the style icon, and use formula (9) to calculate the gradient difference between the content icon and the output icon. Use the Adam optimization algorithm to continuously minimize the overall loss of formula (10) and train the entire network until the network converges.

[0043] (3) Save the network model trained in the previous step, prepare a 3D content icon and a 3D style icon, and use them as content and style inputs to the Transformer-based icon style transfer model to generate a new icon whose content is consistent with the content icon and whose style is consistent with the style icon.

[0044] Example 2

[0045] (1) Prepare training data.

[0046] Prepare a sufficient number of flat icons and 3D icons, resize all images to RGB images with an image size of 256×256 and 3 channels, and divide all data into training and test sets at a ratio of 9:1. Before each training, use the obtained 3D icons as style icons and flat icons as content icons. Normalize the size of style icons and content icons, first adjust them to 512×512 pixels, and then randomly crop out 256×256 pixel image areas for model training.

[0047] (2) Training Transformer-based encoders, decoders, and CNN-based decoders

[0048] The preprocessed content icons and style icons are input into the ImageNet pre-trained VGG19 encoder to extract the content feature Z c and style features Z s . Style Features Z s The two branches of the style encoder generate the style value V and style key K, and encode the content Z c As the query matrix Q, it is input into the two multi-head attention modules shown in formulas (2)-(3) and outputs X2. After passing through the feedforward neural network shown in formula (4), the stylized sequence X is output. Finally, a CNN decoder composed of three convolutional layers decodes the stylized sequence X again to generate a stylized icon I with a channel number of 3 and a size consistent with the content icon. cs The loss of the training network includes: Using formulas (5) and (6) to calculate the content icon I c and output icon I cs The content loss between c And style image I s And the output image I cs The style loss L between s , use formulas (7) and (8) to calculate the consistency loss of the content icon and the style icon, and use formula (9) to calculate the gradient difference between the content icon and the output icon. Use the Adam optimization algorithm to continuously minimize the overall loss of formula (10) and train the entire network until the network converges.

[0049] (3) Save the network model trained in the previous step, prepare a flat content icon and a 3D style icon, and use them as content and style inputs to the Transformer-based icon style transfer model to generate a new icon whose content is consistent with the content icon and whose style is consistent with the style icon.

[0050] The above description of the embodiments is to facilitate the understanding and application of the present invention by those skilled in the art. It is obvious that those skilled in the art can easily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative work. Therefore, the present invention is not limited to the above embodiments, and improvements and modifications made to the present invention by those skilled in the art based on the disclosure of the present invention should be within the scope of protection of the present invention.

Claims

1. A method for automatically generating color icons based on Transformer, comprising the following steps: (1) Obtain a sufficient amount of icon sample data, normalize the samples to the same size using scaling and cropping operations, and divide them into training and test sets in proportion; (2) Construct a Transformer-based icon style transfer model, which includes a VGG19 encoder, a style encoder, a Transformer decoder, a CNN decoder, and several convolutional layers and normalization layers; (3) Training the Transformer-based icon style transfer model. First, a portion of the training data in the dataset created in step (1) is used as content icons, and the other portion is used as style icons. These are randomly paired and input into the Transformer-based icon style transfer model for training until the model converges. (4) Generate a new icon using the Transformer-based icon style transfer model trained in step (3), input the content icon A and the style icon B into the model constructed by the present invention, and the model will output a new icon C whose content is consistent with A and whose style is consistent with B.

2. The Transformer-based 3D icon generation method according to claim 1, characterized in that: First, the VGG19 encoder pre-trained on ImageNet is used to extract content features and style features from content icons and style icons respectively.

3. The Transformer-based 3D icon generation method according to claim 2, characterized in that: The style features extracted by the VGG19 encoder from the style icon are further input into the style encoder for encoding. The style encoder has two parallel branches for extracting style values ​​and style keys respectively. In the branch encoding the style value, there are 2 convolutional layers with a stride of 2 and a convolution kernel size of 3×3, and 3 convolutional layers with a stride of 1 and a convolution kernel size of 3×3. In the branch encoding the style key, there are two convolutional layers with a stride of 2 and a convolution kernel size of 3×3.

4. The Transformer-based 3D icon generation method according to claim 3, characterized in that: The Transformer decoder consists of three layers, each of which includes two multi-head attention modules and a feedforward neural network. The multi-head attention module has multiple different attention heads, each of which learns different feature representations. The outputs of multiple attention heads are connected and multiplied by the weight matrix for linear transformation to obtain the final output of the multi-head attention mechanism; the feedforward neural network includes two linear transformation layers, the first of which is followed by a RELU activation function.

5. The Transformer-based 3D icon generation method according to claim 1, characterized in that: The Transformer decoder decodes the stylized icon code, which is further decoded into the icon image by a CNN decoder. The decoder is a three-layer CNN network. Each layer includes a 3×3 convolution, ReLU and a two-fold upsampling operation, and finally outputs a 3-channel stylized new icon with the same size as the content icon.

6. The Transformer-based 3D icon generation method according to claim 1, characterized in that: The loss function for training the entire Transformer-based icon style transfer model is composed of content loss L c , style loss L s , consistency loss L identity1 and L identity2 , gradient loss L gd Four parts composition: L total =λ c L c +λ s L s +λ identity1 L identity1 +λ identity2 L identity2 +λ gd L gd where λ c , s , identity1 and λ identity2 Represent the weight of each part of the loss; Loss L c By Computing Content Icon I c And the new icon generated I cs The difference between the content features is obtained, and the style loss L s By calculating the style icon I s And the new icon generated I cs The difference between the style features is: The consistency loss includes the consistency loss L of the content icon identity1 and the consistency loss L of the style icon identity2 : L identity1 =‖I cc -I c ‖2+‖I ss -I s ‖2 Among them I cc The content icon is used as both content and style input model to generate the icon, while I ss The style icon is used as both content and style input to the model to generate the icon; Gradient loss function L gd It is obtained by calculating the gradient difference between the content icon and the generated icon, that is, by applying the Sobel convolution kernel to calculate the horizontal and vertical gradients: Conv(I,K) represents the convolution operation between image I and convolution kernel K. x and K y are the Sobel convolution kernels in the horizontal and vertical directions respectively, and N is the number of pixels in the image.

Images