Automatic poster generation method and system based on deep learning

Abstract

The present application relates to the field of image processing technology, and in particular to a method and system for automatically generating posters based on deep learning, the method comprising: selecting target template images of various template types from a template library according to a text description, the template types including title templates, background templates and slogan templates; inputting the template types and randomly initialized geometric parameters of each target template image into a trained layout model to obtain a layout scheme, and generating a poster according to the layout scheme; the layout model is Layout GAN. Through the technical solution of the present application, a poster that meets the text description can be automatically generated quickly and accurately.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and system for automatically generating posters based on deep learning. Background Art

[0002] Posters, as a visual communication tool, can not only attract the attention of the target audience, but also effectively convey information and emotions. They are one of the important ways to advertise a company or product. In the process of poster design, it is necessary to select multiple design elements such as pictures, texts, colors, etc., and to layout the selected multiple design elements, which is a very complicated task.

[0003] At present, a patent application document with application publication number CN111489422A discloses a poster generation and model training method, device, electronic device and storage medium, wherein the method includes: obtaining a first-scale sample poster and a second-scale sample poster; extracting semantic feature vectors and layout feature vectors of the first-scale sample poster and the second-scale sample poster; inputting the semantic feature vector and layout feature vector of the first-scale sample poster into a poster generation model, and outputting resampled semantic feature vectors and layout feature vectors; performing adversarial training on the poster generation model according to the resampled semantic feature vector and layout feature vector, the semantic feature vector and layout feature vector of the second-scale sample poster, the feature distribution of the resampling process and the preset distribution until the poster generation model converges, thereby obtaining a trained poster generation model.

[0004] The above method automatically generates a new-scale poster based on the original-scale version of the poster. Although it can automatically adjust the poster ratio, it still needs to pre-select various design elements of the poster. There are many design elements of the poster, which undoubtedly brings a huge workload to the designers, resulting in low efficiency of automatic poster generation. Summary of the invention

[0005] In order to solve the technical problem of low efficiency of automatic poster generation, the present application provides a method and system for automatic poster generation based on deep learning, which can quickly and accurately automatically generate posters that meet the text description.

[0006] In a first aspect, the present application provides a method for automatically generating posters based on deep learning, the generating method comprising: selecting a target template image of each template type from a template library according to a text description, the template types including a title template, a background template and a slogan template, comprising: extracting semantic features of the text description using a preset recurrent network, calculating the similarity between the template features and the semantic features of each template image in the template library; selecting the first K template images in each template type according to the similarity as candidate template images of each template type; combining the candidate template images of each template type to obtain a selection scheme, and calculating a selection score of each selection scheme, the selection score is positively correlated with the mean similarity in the selection scheme, and the correlation between each template feature; determining the target template image of each template type according to the selection scheme with the maximum selection score; inputting the template type and randomly initialized geometric parameters of each target template image into a trained layout model to obtain a layout scheme, and generating a poster according to the layout scheme, the geometric parameters including a scaling ratio, a rotation angle, and the upper left corner and lower right corner of a target template image bounding box; the layout model is Layout GAN.

[0007] Preferably, the selected solution Selection score satisfy: , is the number of template types, To select a solution Medium template type The similarity corresponding to the candidate template graph is and Select the solution Medium template type and template types The template features of the candidate template graph, express and The inner product of .

[0008] Preferably, any template image and the corresponding template type are input into a preset convolutional network, and the template features of the template image are output; the training method of the preset convolutional network includes: obtaining a text description of any sample poster and a template image of each template type, using a preset recurrent network to obtain the semantic features of the text description, and using a preset convolutional network to obtain the template features of the template image corresponding to each template type; calculating the sample poster and sample poster The contrast loss between the two networks is the sum of the consistency loss and the correlation loss; the preset convolution network and the preset recurrent network are iteratively trained according to the contrast loss until the contrast loss is less than the preset loss; the consistency loss for: , is the number of template types, For sample poster Medium template type The template features of For sample poster The semantic features of and For sample posters Semantic features and template types Template features; the correlation loss for: , For sample poster Correlation matrix including sample posters The inner product of template features of each template type; For sample poster Correlation matrix including sample posters The inner product of template features of each template type; For sample poster and sample poster Correlation matrix including sample posters and sample poster The inner product of template features of each template type; is a matrix with value 0, is a matrix with value 1.

[0009] Preferably, iteratively training the preset convolutional network and the preset recurrent network based on the contrast loss includes: freezing the preset recurrent network and updating the preset convolutional network using the gradient descent method; freezing the preset convolutional network and updating the preset recurrent network using the gradient descent method to complete one training.

[0010] Preferably, iteratively training the preset convolutional network and the preset recurrent network according to the contrast loss includes: using a gradient descent method to simultaneously update the preset recurrent network and the preset convolutional network to complete one training.

[0011] Preferably, the training method of the layout model includes: marking a rectangular box of each template type in a sample poster to obtain a layout label; inputting the layout label or random layout of the sample poster into the layout model, and after obtaining the output result of the layout model, calculating the layout loss, the layout loss is a weighted sum of the adversarial loss and the reconstruction loss, and the reconstruction loss is the Euclidean distance between the output result of the layout model and the layout label of the sample poster; training the layout model based on the layout loss; the random layout is a random value of the geometric parameters of the template graph of each template type in the sample poster.

[0012] Preferably, the layout model includes a generator and a discriminator, and the adversarial loss for: , represents the expected value of the distribution function, The layout tag representing the sample poster. represents a random layout of sample posters, For input layout label The output result of the discriminator is A random layout of the input sample poster The output result of the generator is for: , A random layout of the input sample poster The output of the generator is For random layout Layout tags corresponding to the sample poster.

[0013] Preferably, the layout loss also includes template loss, and the process of obtaining the template loss includes: obtaining the positional relationship between any template types in each sample poster, and calculating the mean and variance of the positional relationship between any template types; the template loss for: , is the number of template types, For template type and template types The variance of the position relationship between For template type and template types The mean of the position relationship between The template type in the output result of the layout model and template types The positional relationship between them.

[0014] Preferably, the positional relationship between any template types is the intersection-and-joint ratio of rectangular frames or the center point connection line.

[0015] In the second aspect of the present application, there is also provided a poster automatic generation system based on deep learning, comprising a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the poster automatic generation method based on deep learning according to the first aspect of the present application is implemented.

[0016] The technical solution of this application has the following beneficial technical effects:

[0017] The text description of the designer is obtained, and the target template images of each template type are selected from the template library according to the text description. The target template images of all template types can constitute a complete poster. In order to ensure that the target template image conforms to the content of the text description and the design styles of the target template images are highly adaptable, in the process of selecting the target template image, the selection score of each selection scheme is calculated according to the correlation between the features of each template and the similarity between the template features and the semantic features of the text description, and the target template image of each template type is determined according to the selection scheme with the maximum selection score. After obtaining each target template image, the template type and randomly initialized geometric parameters of each target template image are input into the trained layout model to obtain a layout scheme, realize the automatic typesetting layout of each target template image, and generate a poster according to the layout scheme, so as to quickly and accurately automatically generate a poster that meets the text description.

[0018] Furthermore, the template features of each template image in the template library and the semantic features of the text description directly affect the accuracy of the selection scheme; in the training process of the preset recurrent network and the preset convolutional network, the consistency loss is used to constrain the similarity between the semantic features in a sample poster and the template features of each template type to 1; the correlation loss is used to constrain the correlation of template features between each template type in a sample poster to 1, and at the same time constrain the correlation of template features of each template type between different sample posters to 0; ensure that the correlation between each template feature can reflect the adaptability between the template images, and the similarity between the template feature and the semantic features of the text description can reflect the degree of content conformity.

[0019] Furthermore, in the sample posters, the positional relationship between the various template types has a certain regularity; in order to make the layout scheme output by the layout model conform to the regularity of the positional relationship between the various template types, it is necessary to improve the layout loss of the layout model (i.e., Layout GAN) and introduce template loss to constrain the positional relationship between the various template types in the output results of the generator to conform to the distribution law. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become easy to understand. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0021] Figure 1 is a flowchart of a method for automatically generating posters based on deep learning according to an embodiment of the present application;

[0022] Figure 2 is a schematic diagram of a training method for a preset convolutional network according to an embodiment of the present application;

[0023] Figure 3 is a schematic diagram of a sample poster layout label according to an embodiment of the present application;

[0024] Figure 4 is a schematic diagram of a layout model training method according to an embodiment of the present application;

[0025] Figure 5 It is a structural block diagram of a poster automatic generation system based on deep learning according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0027] It should be understood that when the terms "first", "second", etc. are used in the claims, specification and drawings of the present application, they are only used to distinguish different objects, rather than to describe a specific order. The terms "include" and "comprise" used in the specification and claims of the present application indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their collections.

[0028] According to the first aspect of the present application, the present application provides a method for automatically generating posters based on deep learning. Figure 1 is a flowchart of a method for automatically generating posters based on deep learning according to an embodiment of the present application. Figure 1 As shown, the poster automatic generation method 100 based on deep learning includes steps S101 to S102, which are described in detail below.

[0029] S101, selecting a target template image of each template type from a template library according to the text description, wherein the template types include a title template, a background template and a slogan template.

[0030] In one embodiment, the text description is a paragraph of text describing the content of the poster, which is used to guide the automatic poster generation process, and the automatically generated poster conforms to the text description. For example, the text description may include prompt words such as title, main idea, design style, etc., and the designer can input text according to the prompt words to obtain the text description used to guide the automatic poster generation process.

[0031] In one embodiment, the template types include title templates, background templates and slogan templates; the template library includes multiple template images of each template type, and one template image corresponds to one template feature. Optionally, the template types may also include chart templates, subtitle templates, photo templates and icon templates.

[0032] It can be understood that in the same template type, the styles and colors of different template images are different, and all template types constitute a poster.

[0033] There are numerous template images in the template library with different design styles. In order to accurately and quickly screen out all target template images that meet the text description, it is necessary to select multiple template images in the template library based on the semantic features of the text description and the template features of the template images.

[0034] Specifically, selecting a target template graph for each template type from a template library according to a text description includes: extracting semantic features of the text description using a preset recurrent network, and calculating the similarity between template features and semantic features of each template graph in the template library; selecting the top K template graphs in each template type according to the similarity as candidate template graphs for each template type; combining the candidate template graphs for each template type to obtain a selection scheme, and calculating a selection score for each selection scheme, wherein the selection score is positively correlated with an average similarity value in the selection scheme and the correlation between each template feature; and determining a target template graph for each template type according to a selection scheme with the maximum selection score.

[0035] Among them, the preset recurrent network is an existing recurrent neural network such as LSTM, GRU or Transformer; through the similarity between the template features and semantic features of a template image, it can be judged whether the template image conforms to the content described in the text. The greater the similarity, the more the template image conforms to the content described in the text; the similarity adopts a similarity calculation method based on Euclidean distance.

[0036] The number of candidate template graphs in a selection scheme is equal to the number of template types, that is, a selection scheme includes any candidate template graph selected from each template type. The number of template types is denoted as , if the value of K is 5, then we can get A selection plan.

[0037] Among them, the correlation between the template features in a selection scheme can reflect the adaptability between different candidate template images. The design styles of different candidate template images are very different. When the correlation between the template features of two candidate template images is small, it means that the design styles of the two candidate template images are very different, that is, the adaptability is poor. Therefore, for a selection scheme, the greater the correlation between the template features and the greater the mean similarity of the template features, the better the adaptability of the selection scheme, and the more it conforms to the content of the text description, the greater the selection score.

[0038] Specifically, select the solution Selection score satisfy: , is the number of template types, To select a solution Medium template type The similarity corresponding to the candidate template graph is and Select the solution Medium template type and template types The template features of the candidate template graph, express and In the above formula, use and The inner product of Middle template features and template features of relevance.

[0039] Understandably, The selected solution Medium template type Template features of candidate template graph , and the similarity between semantic features.

[0040] In one embodiment, the template features of each template image in the template library and the semantic features of the text description directly affect the accuracy of the selection scheme. Therefore, it is crucial to accurately extract template features and semantic features. In the embodiment of the present application, a preset recurrent network is used to extract semantic features, and a preset convolutional network is used to extract template features of the template image.

[0041] The input of the preset convolutional network is the template image and the corresponding template type, and the template features of the template image are output. The preset convolutional network downsamples the template image to obtain intermediate features, and after splicing the intermediate features and the template type into multi-dimensional data, the dimension is transformed again to obtain the template features of the template image.

[0042] Understandably, the template type of the preset convolutional network input is one The 01 vector of row 1 column, is the number of template types, and the value of the template type to which the template image in the 01 vector belongs is 1, and the values ​​of other template types are 0.

[0043] In one embodiment, see Figure 2 , is a schematic diagram of a training method for a preset convolutional network according to an embodiment of the present application. The training method for the preset convolutional network includes: obtaining a text description of any sample poster and a template image of each template type, using a preset recurrent network to obtain semantic features of the text description, and using a preset convolutional network to obtain template features of the template image corresponding to each template type; calculating the sample poster and sample poster The contrast loss between the two networks is the sum of the consistency loss and the correlation loss; and the preset convolutional network and the preset recurrent network are iteratively trained according to the contrast loss until the contrast loss is less than the preset loss.

[0044] Among them, one sample poster corresponds to a semantic feature and a template feature of each template type; the template images of each template type in the sample poster all conform to the text description of the sample poster, and the template images of each template type have good adaptability. Therefore, under ideal conditions, for a sample poster, the similarity between its semantic feature and the template feature of each template type is 1, and the correlation between the template features of each template type is also 1.

[0045] The consistency loss is used to constrain the similarity between the semantic features in a sample poster and the template features of each template type to be 1. Specifically, the consistency loss for: , is the number of template types, For sample poster Medium template type The template features of For sample poster The semantic features of and For sample posters Semantic features and template types Template features.

[0046] in, For Constraints Sample Poster Semantic Features With template type Template features The Euclidean distance between them is equal to 0, which is the constraint semantic feature With template type The similarity between the template features is 1; For Constraints Sample Poster Semantic Features With template type The Euclidean distance between the template features is equal to 0, which is the constrained semantic feature With template type Template features The similarity between them is 1.

[0047] The correlation loss is used to constrain the correlation of template features between template types in a sample poster to be 1, and to constrain the correlation of template features between template types in different sample posters to be 0. Specifically, the correlation loss for: , For sample poster Correlation matrix including sample posters The inner product of template features of each template type; For sample poster Correlation matrix including sample posters The inner product of template features of each template type; For sample poster and sample poster Correlation matrix including sample posters and sample poster The inner product of template features of each template type; is a matrix with value 0, is a matrix with value 1.

[0048] in, For sample poster The correlation matrix of the sample posters The correlation of template features of each template type in is a square matrix with A rows and A columns. Line Columns represent sample posters Medium template type With template type The correlation between the template features is the template type With template type The inner product of the template features between ; similarly, Sample poster for reflection The correlation of template features of each template type is also a square matrix with A rows and A columns; Sample poster for reflection and sample poster The correlation between the template features of each template type, Line Columns represent sample posters Medium template type With sample poster Medium template type The correlation between template features.

[0049] Understandably, is a matrix with A rows and A columns whose values ​​are 1, and It is used to constrain the correlation of template features between template types in a sample poster to be 1, and is a matrix with A rows and A columns whose values ​​are 0, Used to constrain the differences between different sample posters (i.e. this poster and sample poster The correlation between template features of each template type is 0.

[0050] In one embodiment, the sum of the consistency loss and the correlation loss is used as the contrast loss to supervise the training of the preset convolutional network so that the template features of the template image meet both the consistency requirements and the correlation requirements. The iterative training of the preset convolutional network and the preset recurrent network based on the contrast loss includes: using the gradient descent method to simultaneously update the preset recurrent network and the preset convolutional network to complete one training. The preset convolutional network is continuously trained using multiple sample posters until the value of the contrast loss is less than the preset loss, indicating that the preset convolutional network can accurately extract the template features of the template image. The value of the preset loss is 0.01.

[0051] In another embodiment, the preset recurrent network and the preset convolutional network may be trained by an alternating training method. Specifically, iteratively training the preset convolutional network and the preset recurrent network according to the contrast loss includes: freezing the preset recurrent network and updating the preset convolutional network by using the gradient descent method; freezing the preset convolutional network and updating the preset recurrent network by using the gradient descent method to complete one training.

[0052] In this way, a preset recurrent network and a preset convolutional network are trained at the same time. The preset recurrent network is used to extract the semantic features of the text description input by the designer, and the preset convolutional network is used to extract the template features of each template image in the template library, and complete the screening of numerous template images based on the template features of each template image in the template library and the semantic features of the text description input by the designer, so as to quickly and accurately locate the target template images of each template type in the template library.

[0053] S102, input the template type and randomly initialized geometric parameters of each target template image into the trained layout model to obtain a layout plan, and generate a poster according to the layout plan. The geometric parameters include a scaling ratio, a rotation angle, and the upper left corner and lower right corner of the target template image boundary box.

[0054] In one embodiment, after obtaining the target template images of each template type, all target template images are laid out, and the geometric parameters of each target template image are determined, so that the position information of each template type in the poster can be obtained, and the poster can be automatically generated. The geometric parameters of the target template image include the upper left corner point and the lower right corner point of the bounding box, the scaling ratio, and the rotation angle.

[0055] Layout GAN is a generative adversarial network that uses data to learn layout and typesetting. Layout GAN is a well-known technology for those skilled in the art. Its detailed network structure is not described here. Layout GAN is used as a layout model to generate layout solutions for target template images of various template types. The layout model includes a generator and a discriminator. The input of the generator is the geometric parameters and template types of each target template image, and the output is a layout solution. The layout solution is a wireframe diagram, including the bounding box of each target template image; the input of the discriminator is the layout solution, and the output is the judgment result of the layout solution. The judgment result is true or false, which is used to judge whether the layout solution is the real layout of the poster.

[0056] In one embodiment, in order to make the layout model output a layout solution that conforms to the actual situation, the layout model is trained using sample posters. Specifically, the training method of the layout model includes: marking the rectangular boxes of each template type in the sample poster to obtain the layout label; inputting the layout label or random layout of the sample poster into the layout model, and after obtaining the output result, calculating the layout loss, the layout loss is the weighted sum of the adversarial loss and the reconstruction loss, and the reconstruction loss is the Euclidean distance between the output result and the sample poster layout label; training the layout model based on the layout loss; the random layout is the random value of the geometric parameters of the template map of each template type in the sample poster.

[0057] Here, a rectangular frame of each template type in the sample poster is drawn to obtain a layout label, which is also a wireframe diagram. For example, if the template types include icon templates, title templates, background templates, and chart templates, the schematic diagram of the sample poster layout label is as follows: Figure 3 As shown, the rectangular boxes of different template types in the layout tag have different colors, and the template types are distinguished by color.

[0058] See also Figure 4, is a schematic diagram of a layout model training method according to an embodiment of the present application. The adversarial loss is used to constrain the generator in the layout model to output a reasonable layout solution. for: , represents the expected value of the distribution function, The layout tag representing the sample poster, represents a random layout of sample posters, For input, layout label The output of the discriminator is A random layout of the input sample poster Furthermore, in order to make the output of the generator close to the layout label of the sample poster, the output of the generator is constrained by the reconstruction loss. for: , A random layout of the input sample poster The output of the generator is For random layout The layout label of the corresponding sample poster. The combination of adversarial loss and reconstruction loss is used as the layout loss to train the layout model, so that the generator in the layout model can output accurate and reasonable layout solutions.

[0059] Understandably, when the input of the layout model is the layout label of the sample poster When , the layout loss is , which is used to constrain the discriminator to accurately determine whether the layout scheme is the real layout of the poster; when the input of the layout model is the random layout of the sample poster When , the layout loss is , which is used to constrain the output of the generator to be the actual layout of the poster, and the output is equal to the layout label.

[0060] In another embodiment, in the sample poster, the positional relationship between the template types has a certain regularity, such as the title template is located above the subtitle template, etc. In order to make the layout scheme output by the layout model conform to the regularity of the positional relationship between the template types, the layout loss of the layout model (i.e., Layout GAN) needs to be improved.

[0061] Specifically, the layout loss also includes template loss; the template loss acquisition process includes: obtaining the positional relationship between any template types in each sample poster, and calculating the mean and variance of the positional relationship between any template types, then the template loss for: , is the number of template types, For template type and template types The variance of the position relationship between For template type and template types The mean of the position relationship between The template type in the output result and template types The positional relationship between them.

[0062] The positional relationship between any template types is the intersection-and-joint ratio of the rectangular frames or the center point connection line.

[0063] in, Used to characterize template types and template types The distribution law of the position relationship between Indicates the type of template and template types The smaller the value, the smaller the template type. and template types The stronger the regularity of the position relationship between middle Degree of attention The larger the size, the more types of templates in the output of the layout model. and template types The positional relationship between them conforms to the distribution law.

[0064] It is understandable that when the input of the layout model is the layout label of the sample poster When , the layout loss is , which is used to constrain the discriminator to accurately determine whether the layout scheme is the real layout of the poster; when the input of the layout model is the random layout of the sample poster When , the layout loss is , which is used to constrain the output result of the generator to be equal to the layout label, and the positional relationship between the template types in the output result conforms to the distribution law.

[0065] In this way, the training of the layout model is completed, and the trained layout model inputs the template types of multiple template images and the initialized geometric parameters, and outputs the layout solutions of the multiple template images.

[0066] In one embodiment, the template types and initialized geometric parameters of all target template images are input into the trained layout model to obtain a layout plan, and the target template images are placed at corresponding positions according to the layout plan to achieve automatic generation of posters.

[0067] The technical principles and implementation details of the poster automatic generation method based on deep learning of the present application are introduced above through specific embodiments. The text description of the designer is obtained, and the target template image of each template type is selected from the template library according to the text description. The target template images of all template types can constitute a complete poster; to ensure that the target template image conforms to the content of the text description, and the design style adaptability between the target template images is high, in the process of selecting the target template image, the selection score of each selection scheme is calculated according to the correlation between the features of each template and the similarity between the template features and the semantic features of the text description, and the target template image of each template type is determined according to the selection scheme with the maximum selection score; after obtaining each target template image, the template type and randomly initialized geometric parameters of each target template image are input into the trained layout model to obtain a layout scheme, realize the automatic typesetting layout of each target template image, and generate a poster according to the layout scheme, and quickly and accurately automatically generate a poster that meets the text description.

[0068] According to the second aspect of the present application, the present application also provides a poster automatic generation system based on deep learning. Figure 5 is a structural block diagram of a poster automatic generation system based on deep learning according to an embodiment of the present application. Figure 5 As shown, the system 50 includes a processor and a memory, the memory stores computer program instructions, and when the computer program instructions are executed by the processor, a method for automatically generating posters based on deep learning according to the first aspect of the present application is implemented. The system also includes other components well known to those skilled in the art, such as a communication bus and a communication interface, whose settings and functions are known in the art, so they are not described here.

[0069] In the present application, the aforementioned memory may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. For example, a computer-readable storage medium may be any suitable magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc., or any other medium that can be used to store the required information and can be accessed by an application, a module, or both. Any such computer storage medium may be part of a device or accessible or connectable to a device. Any application or module described in this application may be implemented using computer-readable / executable instructions that may be stored or otherwise maintained by such a computer-readable medium.

[0070] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0071] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.

Claims

1. A poster automatic generation method based on deep learning, characterized in that: The generation method comprises: Selecting a target template image of each template type from a template library according to the text description, wherein the template types include a title template, a background template, and a slogan template, including: extracting semantic features of the text description using a preset recurrent network, and calculating the similarity between the template features of each template image in the template library and the semantic features; selecting the first K template images in each template type according to the similarity as candidate template images of each template type; combining the candidate template images of each template type to obtain a selection scheme, and calculating a selection score of each selection scheme, wherein the selection score is positively correlated with the mean similarity value in the selection scheme and the correlation between each template feature; determining the target template image of each template type according to the selection scheme with the maximum selection score; Input the template type and randomly initialized geometric parameters of each target template image into the trained layout model to obtain a layout plan, and generate a poster according to the layout plan, wherein the geometric parameters include a scaling ratio, a rotation angle, and the upper left corner and the lower right corner of the target template image boundary box; The layout model is Layout GAN; The training method of the layout model includes: marking a rectangular frame of each template type in a sample poster to obtain a layout label; inputting the layout label or random layout of the sample poster into the layout model, and after obtaining the output result of the layout model, calculating the layout loss, wherein the layout loss is a weighted sum of the adversarial loss and the reconstruction loss, and the reconstruction loss is the Euclidean distance between the output result of the layout model and the layout label of the sample poster; training the layout model according to the layout loss; the random layout is a random value of the geometric parameters of the template graph of each template type in the sample poster; The layout loss also includes template loss, and the process of obtaining the template loss includes: Obtain the positional relationship between any template types in each sample poster, and calculate the mean and variance of the positional relationship between any template types; The template loss for: , is the number of template types, For template type and template types The variance of the position relationship between For template type and template types The mean of the position relationship between The template type in the output result of the layout model and template types The positional relationship between them; the positional relationship between any template types is the intersection ratio of the rectangular boxes or the center point connection line.

2. The method for automatically generating posters based on deep learning according to claim 1, characterized in that: Select a plan Selection score satisfy: , is the number of template types, To select a solution Medium template type The similarity corresponding to the candidate template graph is and Select the solution Medium template type and template types The template features of the candidate template graph, express and The inner product of .

3. The method for automatically generating posters based on deep learning according to claim 1, characterized in that: Input any template image and the corresponding template type into a preset convolutional network, and output the template features of the template image; The training method of the preset convolutional network includes: Obtain the text description of any sample poster and the template graph of each template type, use the preset recurrent network to obtain the semantic features of the text description, and use the preset convolutional network to obtain the template features of the template graph corresponding to each template type; Calculation sample poster and sample poster The contrast loss between the two is the sum of the consistency loss and the correlation loss; Iteratively train the preset convolutional network and the preset recurrent network according to the contrast loss until the contrast loss is less than the preset loss; The consistency loss for: , is the number of template types, For sample poster Medium template type The template features of For sample poster The semantic features of and For sample posters Semantic features and template types Template features; the correlation loss for: , For sample poster Correlation matrix including sample posters The inner product of template features of each template type; For sample poster Correlation matrix including sample posters The inner product of template features of each template type; For sample poster and sample poster Correlation matrix including sample posters and sample poster The inner product of template features of each template type; is a matrix with value 0, is a matrix with value 1.

4. The method for automatically generating posters based on deep learning according to claim 3, characterized in that: Iteratively training a preset convolutional network and a preset recurrent network based on contrastive loss includes: Freeze the preset recurrent network and use the gradient descent method to update the preset convolutional network; freeze the preset convolutional network and use the gradient descent method to update the preset recurrent network to complete one training.

5. The method for automatically generating posters based on deep learning according to claim 3, characterized in that: Iteratively training a preset convolutional network and a preset recurrent network based on contrastive loss includes: The gradient descent method is used to simultaneously update the preset recurrent network and the preset convolutional network to complete a training session.

6. The method for automatically generating posters based on deep learning according to claim 1, characterized in that: The layout model includes a generator and a discriminator, and the adversarial loss for: , represents the expected value of the distribution function, The layout tag representing the sample poster, represents a random layout of sample posters, For input layout label The output of the discriminator is A random layout of the input sample poster The output result of the generator when The reconstruction loss for: , A random layout of the input sample poster The output of the generator is For random layout Layout tags corresponding to the sample poster.

7. A poster automatic generation system based on deep learning, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, a method for automatically generating posters based on deep learning according to any one of claims 1 to 6 is implemented.

Images