An end-to-end intelligent graphic design method based on deep learning

Abstract

The present invention discloses an end-to-end intelligent graphic design method based on deep learning. Using the collected poster data set, the original poster data set is first screened and divided, and the two subtasks of layout design and attribute confirmation are jointly learned. The training parameters are adjusted on the training set to make the model performance reach the optimal state; according to the input image and text information, by calling the trained model, the intelligent graphic design framework extracts the feature information of the image and text, searches for the appropriate design layout and confirms the composition text attributes, and automatically generates a harmonious graphic design. The present invention uses a unified joint training framework to prevent data distribution differences during the training process, reduce error propagation in the pipeline model, and give full play to the advantages of end-to-end training. It does not need to manually define the aesthetic rules of graphic design, but learns the aesthetic rules from the data, and does not rely on image saliency map detection, so that it can be better generalized to various graphic design tasks.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and specifically relates to an end-to-end intelligent plane design method based on deep learning. Background Art

[0002] Graphic design, taking poster design as an example, is widely used in our daily life as an important medium of visual communication. In order to produce harmonious and beautiful graphic designs, designers often need to invest a lot of time and energy, and the threshold is high for ordinary people without professional foundation, and they need to have high aesthetic requirements and professional design knowledge. With the rapid progress and development of computer vision technology, people are increasingly interested in computer intelligent design.

[0003] There are currently a variety of methods to help with graphic design, which can significantly reduce the time and manpower spent on basic design work. They can be mainly divided into traditional methods, rule-based methods, and data-driven methods.

[0004] Traditional methods of automatic graphic design usually adopt the method of design rules or structured data. When creating graphic designs, designers often follow design rules, such as aesthetic principles of layout and harmonious color models, but traditional methods are less intelligent and have limited scope of application. Rule-based methods use some design rules to assist specific design tasks, including layout synthesis, color generation, and image thumbnails. However, aesthetic rules are complex and difficult to be defined and constrained by specific rules. Ali et al. proposed a constraint-based recommendation system in the paper "Learning deconvolution network for semantic segmentation". The system can generate magazine cover designs based on user preferences, but this method is only applicable to specific magazine cover templates.

[0005] With the development of deep learning, data-driven methods are used to obtain specific design attributes from training images to solve graphic design problems. Yang et al. revealed the effectiveness of optimization methods using aesthetic design principles in the paper "Recommendation system for automatic design of magazine covers n". With the help of a large database, a learning-based method was proposed to generate magazine covers. However, there are many rule constraints and template problems.

[0006] Zheng et al. proposed the first content-aware deep generative model for graphic design layouts in the paper "Content-aware generative modeling of graphic design layouts". This model can synthesize graphic design layouts based on visual and text features, but it is mainly used for magazine typesetting and is not suitable for the generation of more extensive graphic posters.

[0007] Yang et al. designed a system in the paper "A simple framework for contrastive learning of visual representation" to automatically generate a digital magazine cover by summarizing a set of templates related to the topic and introducing a calculation formula, including a framework of key elements of layout design. However, the above method is a rule-based system with low intelligence.

[0008] Feng et al. proposed an interactive system DesignScape in the paper "A dataset and a baseline model for salient object detection", which can generate layout suggestions based on a set of user-input images and texts in order to enhance the interactivity of the automatic layout generation process. However, it is mainly based on saliency detection and has limitations for different graphic designs.

[0009] Although the above technologies have made certain contributions to generating appropriate text and image layouts, the layout rules derived from these technologies are not applicable to generating graphic design works, and ignore the key impact of design element attributes other than layout on graphic design. At the same time, mainstream intelligent solutions rely too much on saliency detection, which greatly limits the space for intelligent graphic design. Summary of the invention

[0010] Aiming at the four major problems in the current field of intelligent graphic design: first, over-reliance on the use of saliency image detection makes intelligent graphic design too limited; second, lack of consideration and solution for the style of design elements in graphic design; third, the use of pipeline model structures leads to accumulation of propagation errors; fourth, insufficient structured data of posters; the present invention provides an end-to-end intelligent graphic design method based on deep learning, which comprehensively considers given images and design elements to automatically generate harmonious and beautiful graphic designs.

[0011] An end-to-end intelligent plane design method based on deep learning comprises the following steps:

[0012] (1) After collecting semi-structured poster data and cleaning and screening it, the composition text attributes and background images of the posters are stored separately to provide the required training data for the intelligent graphic design model;

[0013] (2) Jointly train the two subtasks of layout design and attribute confirmation in the intelligent graphic design model, enabling the model to extract features from multimodal (visual and textual) views of poster data;

[0014] (3) The image and text features are fused using the image module and text module in the model, and the layout density inference graph is obtained after decoding;

[0015] (4) Determine the layout design using an approximate reasoning algorithm based on the layout density reasoning graph;

[0016] (5) The overall and local features of the image are integrated, and the attribute category of the composed text is determined based on the output of the classifier.

[0017] Furthermore, the specific implementation of step (1) is as follows: first, semi-structured poster data is collected from public information on web pages and screened, and the poster background image, title text sequence, relative coordinates of the text box, text font size information and other corresponding composition text attributes in the collected poster data set are recorded and stored as annotation information and self-supervision signals for training data, and the title text sequence and poster background image are used as inputs, and the rendering of the poster is used as the goal to provide the required training data for the neural network in the intelligent graphic design network framework. Using the self-supervision signals in the semi-structured posters for training can overcome the difficulty of obtaining annotations.

[0018] Furthermore, the specific implementation method of step (2) is as follows: first, the two main parts of the key to graphic poster design, namely, visual representation and text representation, are aggregated together, considering not only the features of the image but also the semantics of the input text; then, the aggregated multimodal features are used as the input of the decoder, and the decoder outputs a density map of the same size as the original image, in which each element value corresponds to the score of the original image pixel, representing the weight of the pixel selected to appear in the text area; and then, the objective function of the training process is designed, with the overall goal of minimizing the sum of the layout prediction loss and the attribute recognition loss, and the model parameters are learned through second-order gradient optimization.

[0019] Furthermore, the expression of the objective function is as follows:

[0020]

[0021]

[0022]

[0023] in: is the objective function, is the layout prediction loss function, is the attribute recognition loss function, M i,j is the element value of the pixel point (i, j) in the density map, G i,j is a binary indicator of whether the pixel (i, j) is located in the text area, For attribute α to belong to the appropriate category y a The probability of, σ() is the sigmoid function, Attributes represents the attribute set.

[0024] Furthermore, the intelligent graphic design model adopts an encoder-decoder architecture as a whole. In the image module, a convolutional neural network is used to extract image features of the poster. As the network goes deeper, it encodes graphic features in more channels and uses feature maps for encoding on a wider receptive field. In the text module, a pre-trained language model is used to extract context-aware text features, that is, a distributed vector representation is used to represent the semantic information of the input text, and then average pooling is used to obtain a fixed-dimensional distributed representation of the entire input sequence. A multi-layer perceptron is used to convert the text representation into a similar vector space of the graphic representation, and the final text representation is a vector. The image features and text features are fused at the channel level and decoded by a decoder to obtain a layout density reasoning graph.

[0025] Furthermore, the step (4) adopts a search-based layout design, that is, by searching on the layout density reasoning graph after feature fusion, the position and size of the composition text are predicted, and the score of each pixel is modeled to indicate the appropriate area of ​​the composition text. The layout design predicts the position and size of the composition text area by solving the optimization problem on the density reasoning graph; starting from the local maximum of the layout density reasoning graph, the area of ​​the rectangle is gradually expanded in a heuristic search manner to determine the area of ​​the text object, assuming that the correct position of the composition text input is approximately centered on the local maximum, and the score of the candidate area is almost convex in terms of the distance from the edge to the local maximum, and the locality of the density graph is utilized to determine the layout design using an approximate reasoning algorithm.

[0026] Furthermore, after determining the poster layout design, the step (5) discretizes the continuous attributes in the composition text attributes (including text font, color, font size, etc.) into several categories for separate design, and then uses the composition text attribute design model to collect features from two sources: on the one hand, collects image features hidden in the poster image text features for global aggregation of the overall input; on the other hand, collects local features from the weighted local view of the original image; since the color of text information is generally limited by the hue of the local text area, the composition text attribute design model uses a similar convolutional encoder to extract local view image features, fuses the overall image features and local features, and uses a multi-layer perceptron classifier for scoring and output, and jointly determines the category of the composition text attribute.

[0027] The present invention uses a unified joint training framework to prevent data distribution differences during the training process, reduce error propagation in the pipeline model, and give full play to the advantages of end-to-end training; at the same time, the present invention does not need to manually define the aesthetic rules of graphic design, but learns the aesthetic rules from data, and does not rely on image saliency map detection, so that it can be better generalized to various graphic design tasks.

[0028] Compared with the prior art, the present invention has the following characteristics and beneficial technical effects:

[0029] 1. This invention proposes an end-to-end framework for intelligent graphic design for the first time, avoiding the error propagation weaknesses of multi-step pipeline generation and the difficulty of maintaining aesthetic constraints.

[0030] 2. The present invention designs an end-to-end network to jointly learn layout design and attribute determination, and uses self-supervisory signals extracted from semi-structured posters to train the network, thus overcoming the difficulty of obtaining annotations.

[0031] 3. The experimental results on crawled data demonstrate the effectiveness of the framework of the present invention. Extensive experiments and analysis of the results demonstrate that the end-to-end intelligent graphic design method based on deep learning has outstanding superiority over previous saliency-based methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a schematic diagram of the overall process of the intelligent plane design method of the present invention.

[0033] Figure 2 Schematic diagram of self-supervisory signal for graphic design data.

[0034] Figure 3 Schematic diagram of the image and text feature fusion model.

[0035] Figure 4 Design a schematic for a search-based layout.

[0036] Figure 5 Schematic diagram of the attribute determination process based on fused features. DETAILED DESCRIPTION

[0037] In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0038] This example uses the collected poster dataset. First, the original poster dataset is filtered and divided, and the two subtasks of layout design and attribute confirmation are learned together. The training parameters are adjusted on the training set to optimize the model performance. Based on the input image and text information, the intelligent graphic design framework extracts the feature information of the image and text by calling the trained model, searches for the appropriate design layout and confirms the composition text attributes, and automatically generates a harmonious graphic design. Figure 1 As shown, the specific steps of this implementation are as follows:

[0039] (1) Collection and preprocessing of self-supervised graphic design data.

[0040] We collect and filter semi-structured poster data from public information on web pages, and record the poster background image, basic composition units, title text sequence, relative coordinates of the text box, text font size information, and other corresponding design element attributes in the collected poster data set as annotation information and self-supervision signals for training data. Taking text sequences and background images as input and rendering posters as the goal, we provide the required training signals for the neural network in the intelligent poster design network framework; using the self-supervision signals in semi-structured posters for training overcomes the difficulty of obtaining annotations. The data used in this example is poster data, and the basic data mainly includes: poster background image, basic composition units, text title content and text sequence, text font, font color, font size, font position, etc., as shown below. Figure 2 shown.

[0041] (2) Joint training of intelligent graphic design framework.

[0042] The intelligent graphic design framework uses a unified neural network for layout design and attribute confirmation, so it can enjoy the benefits of joint training of the two subtasks. The key to graphic poster design, the two main parts of visual representation and text representation, are aggregated together as shown below, considering not only the characteristics of the image but also the semantics of the input text:

[0043] F = Concat(H V ,REP(U),REP(L))

[0044] in: represents the hidden image feature map extracted by the encoder, C V, H and W represent the number of channels, height and width of the feature map respectively; the text representation is replicated in the height and width dimensions to align with the visual representation (repeated operation represented by REP), and an additional scalar feature is added as the input sequence Length.

[0045] M=Decoder(F)

[0046] Using the aggregated multimodal features F as the input of the decoder, the decoder outputs a density map of the same size as the original image Each element M i,j is the pixel I i,j The score represents the weight of selecting pixels that appear in the text area.

[0047] Goals of layout design Layout and composition text attribute confirmation The properties are as follows:

[0048]

[0049]

[0050]

[0051] Where: G i,j is a binary indicator of whether pixel (i,j) is in a text region, Is attribute a belonging to category y a The overall goal is to minimize the sum of layout design loss and attribute confirmation loss. The model parameters are learned through second-order gradient optimization. Joint training helps the smart graphic design network extract better features from multimodal (visual and textual) views of the data.

[0052] (3) Image-text feature fusion and layout density reasoning.

[0053] The intelligent graphic design model is trained on the collected poster data training set, and the overall structure adopts the encoder-decoder architecture, such as Figure 3 As shown in the figure, in the image module, a convolutional neural network is used to extract the image features of the poster; as the network goes deeper, it encodes the graphic features in more channels and uses feature maps for encoding over a wider receptive field:

[0054] H V =Encoder (I)

[0055] In the text module, a pre-trained language model is used to extract context-aware text features:

[0056] U = MLP(AVG({E1, E2, …, E n}))

[0057] where: The token embedding representation of each token T i is The distributed vector representation is expected to carry the semantic information of the input text. Then, average pooling (denoted by AVG) is used to obtain a fixed-dimensional distributed representation of the entire input sequence. The text representation is converted into a similar vector space of the graphical representation through a multi-layer perceptron (denoted by MLP). The final text representation is a vector The image features and text features are fused at the channel level and decoded by the decoder M to obtain the layout density inference graph.

[0058] (4) Search-based layout design.

[0059] The layout design module predicts the position and size of the compositional text by searching on the layout density map after feature fusion, and models the score of each pixel to indicate the appropriate area of the compositional text.

[0060] The layout design predicts the position and size of the compositional text area by solving the optimization problem on the density map; based on the output of the decoder M, assuming that σ(M i,j ) represents the probability that I i,j is located in the bounding box of the given text sequence, and σ represents the sigmoid function shown below:

[0061]

[0062] Given the density map M and the corresponding probability matrix σ(M), determine the corner coordinates of the text box: the lower left corner (x1, y1) and the upper right corner (x2, y2), and specify that x1 < x2, y1 < y2. Convert the corresponding prediction task into a constrained optimization problem. Starting from the local maximum of the layout density map, gradually expand the area of the rectangle in a heuristic search manner to determine the area of the text object. Assume that the correct position of the compositional text input is approximately centered on the local maximum, and the score of the candidate area is almost convex in terms of the distance from the edge to the local maximum. Utilize the locality of the density map and use an approximate inference algorithm to determine the layout design, as Figure 4 shown.

[0063] (5) Determination of compositional text attributes based on fused features.

[0064] Through the data-driven joint learning in step (2), the fused feature F contains not only the key information of the layout design, but also the key information of the composition text attributes. After determining the poster layout design, the continuous attributes of the composition text attributes (including text font, color, font size, etc.) are discretized into several categories for separate design. The composition text attribute design model mainly collects features from two sources: on the one hand, the image features hidden by the poster image text features are collected for global aggregation of the overall input; on the other hand, local features are collected from the weighted local views of the original image. In particular, since the color of the text information is generally limited by the hue of the local text area, the model can use a similar convolutional encoder to extract local view image features, such as Figure 5 As shown in the figure, the overall and local features of the image are integrated, and a multi-layer perceptron classifier is used for scoring and output to jointly determine the category of the composition text attribute.

[0065] The features F from the multimodal feature extraction network describe the global view of the input, while the local view of the image also has a key influence on the attribute determination. For example, the color of the text input is generally limited by the hue of the local text area. Weighted original image Apply the probability density map as the attention weight for each pixel of the original image, Represents the bitwise product, using a similar convolutional encoder to extract local view image features

[0066]

[0067] Connect global features and local features, and use MLP classifier to output logits. The logarithm is normalized to probability distribution through softmax function, p i Represents the probability that the attribute belongs to the i-th class:

[0068] logit i =MLP(F,F l )

[0069]

[0070] We study the performance of our end-to-end smart floor plan design framework on various mainstream models, using Test loss (lower is better), Jaccard similarity (higher is better), and ACC (higher is better) metrics to evaluate the overall quality, layout generation quality, and attribute design quality, respectively.

[0071] As the backbone structure becomes more powerful, the end-to-end intelligent plane design framework can obtain better design works in terms of overall quality, and the overall performance as well as the effectiveness of each component will become better. As shown in Table 1, the end-to-end intelligent plane design network of the present invention outperforms the saliency detection-based method on the layout prediction subtask, and the saliency-based method obtains lower Jaccard similarity during testing compared to all AuPoD series networks except AuPoD-FCN32. The results show that the end-to-end intelligent plane design framework can benefit from better saliency detection induced bias.

[0072] Table 1

[0073]

[0074] The above description of the embodiments is to facilitate the understanding and application of the present invention by those skilled in the art. It is obvious that those skilled in the art can easily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative work. Therefore, the present invention is not limited to the above embodiments. Improvements and modifications made by those skilled in the art to the present invention based on the disclosure of the present invention should be within the protection scope of the present invention.

Claims

1. An end-to-end intelligent plane design method based on deep learning, comprising the following steps: (1) After collecting semi-structured poster data and cleaning and screening it, the composition text attributes and background images of the posters are stored separately to provide the required training data for the intelligent graphic design model; The intelligent graphic design model adopts an encoder-decoder architecture as a whole. In the image module, a convolutional neural network is used to extract the image features of the poster. As the network goes deeper, it encodes the graphic features in more channels and uses feature maps for encoding on a wider receptive field. In the text module, a pre-trained language model is used to extract context-aware text features, that is, a distributed vector representation is used to represent the semantic information of the input text, and then average pooling is used to obtain a fixed-dimensional distributed representation of the entire input sequence. A multi-layer perceptron is used to convert the text representation into a similar vector space of the graphic representation, and the final text representation is a vector. The image features and text features are fused at the channel level and decoded by a decoder to obtain a layout density reasoning graph. (2) The two subtasks of layout design and attribute confirmation in the intelligent graphic design model are jointly trained to enable the model to extract features from the multimodal view of the poster data. The specific implementation method is as follows: first, the two main parts of the key to graphic poster design, namely visual representation and text representation, are aggregated together, considering not only the features of the image but also the semantics of the input text; then the aggregated multimodal features are used as the input of the decoder, and the decoder outputs a density map of the same size as the original image. The value of each element in the density map corresponds to the score of the original image pixel, representing the weight of the pixel selected to appear in the text area; then the objective function of the training process is designed, and the overall goal is to minimize the sum of the layout prediction loss and the attribute recognition loss. The model parameters are learned through second-order gradient optimization; the expression of the objective function is as follows: in: is the objective function, is the layout prediction loss function, is the attribute recognition loss function, M i,j is the element value of the pixel point (i, j) in the density map, G i,j is a binary indicator of whether the pixel (i, j) is located in the text area, For attribute a to belong to category y a The probability of, σ( ) is the sigmoid function, Attributes represents the attribute set; (3) The image and text features are fused using the image module and text module in the model, and the layout density inference graph is obtained after decoding; (4) According to the layout density reasoning graph, an approximate reasoning algorithm is used to determine the search-based layout design, that is, by searching on the layout density reasoning graph after feature fusion, the position and size of the composition text are predicted, and the score of each pixel is modeled to indicate the appropriate area of ​​the composition text. The layout design predicts the position and size of the composition text area by solving the optimization problem on the density reasoning graph; starting from the local maximum of the layout density reasoning graph, the area of ​​the rectangle is gradually expanded in a heuristic search manner to determine the area of ​​the text object. It is assumed that the correct position of the composition text input is approximately centered on the local maximum, and the score of the candidate area is almost convex in terms of the distance from the edge to the local maximum. The locality of the density graph is utilized to determine the layout design using an approximate reasoning algorithm; (5) The overall image features and local features are integrated, and the attribute category of the composition text is determined according to the output of the classifier. Specifically, the continuous attributes in the composition text attributes are discretized into several categories and designed separately. Then, the composition text attribute design model is used to collect features from two sources: on the one hand, the image features hidden in the poster image text features are collected for global aggregation of the overall input; on the other hand, local features are collected from the weighted local view of the original image. Since the color of text information is generally limited by the hue of the local text area, the composition text attribute design model uses a similar convolutional encoder to extract local view image features, integrates the overall image features and local features, and uses a multi-layer perceptron classifier for scoring and output, which jointly determines the category of the composition text attributes.

2. The end-to-end intelligent plane design method according to claim 1, characterized in that: The specific implementation method of step (1) is as follows: first, semi-structured poster data is collected from public information on web pages and screened, and the poster background image, title text sequence, relative coordinates of the text box, text font size information and other corresponding composition text attributes in the collected poster data set are recorded and stored as annotation information and self-supervision signals for training data, with the title text sequence and poster background image as input and the rendering of the poster as the goal, so as to provide the required training data for the neural network in the intelligent graphic design network framework.

3. The end-to-end intelligent plane design method according to claim 1, characterized in that: This method uses the collected poster dataset. First, it screens and divides the original poster dataset, conducts joint learning of the two subtasks of layout design and attribute confirmation, and performs training and parameter adjustment on the training set to optimize the model performance. According to the input image and text information, by calling the trained model, the intelligent graphic design framework extracts the feature information of the image and text, searches for the appropriate design layout and confirms the composition text attributes, and automatically generates a harmonious graphic design.

Images