Travel magazine ad layout autogenerator
By combining fully convolutional neural networks and generative adversarial networks, the problem of automated layout generation for travel magazines has been solved, achieving efficient and accurate automatic layout generation, reducing manual intervention and improving design efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA JILIANG UNIV
- Filing Date
- 2023-03-10
- Publication Date
- 2026-06-26
AI Technical Summary
Existing technologies struggle to automatically generate tourism magazine layouts that meet design goals, especially in the layout of elements in specified locations and sizes, which requires significant manual intervention and time.
By combining fully convolutional neural networks and generative adversarial networks (GANs), we automatically generate tourism magazine advertisement layouts that meet the constraints by crawling tourism website data, tag segmentation, semantic processing, and training the GAN.
It enables the automated generation of high-quality travel magazine layouts that conform to design style and reading order, reducing manual intervention and improving the efficiency and accuracy of layout generation.
Smart Images
Figure CN117151783B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the fields of text data mining, data analysis, visualization and visualization analysis technology, and specifically relates to an intelligent automatic generator for tourism magazine advertising layout. Background Technology
[0002] Good design is visually pleasing and clearly conveys information, but high-quality design relies on the expertise of experienced designers. In a magazine, designers often need to create pages where various elements intersect, with similar design styles but different layouts. Designers typically start with the magazine's materials (such as the number of images and text descriptions), then design the layout based on the quantity of materials and image sizes, and finally design a style that matches the magazine's materials. This process is extremely time-consuming and labor-intensive. Organizing a given set of elements (images, text, titles, etc.) into a layout offers many possibilities. A good layout must meet design goals (reading order, harmony, etc.), and changing one element may require rearranging many others. Therefore, generating custom travel magazine layouts is of great significance, greatly liberating labor. Users can design suitable layouts based on the quantity of materials they have for designing travel magazine advertisements, or they can design layouts according to their desired style. Layout modeling is a crucial step in graphic design, and methods for generating graphic layouts have made progress, especially Generative Adversarial Networks (GANs). However, designing elements with specified positions and sizes often involves constraints on element attributes such as area, aspect ratio, and reading order. Automating attribute-based conditional graphic layout remains a complex and unresolved problem. Summary of the Invention
[0003] To address the problems described above, this invention provides an automatic generator for travel magazine advertising layouts.
[0004] To achieve the above-mentioned technical objectives, the technical solution adopted by the present invention is as follows:
[0005] This invention discloses an automatic generator for travel magazine advertisement layouts, comprising the following steps:
[0006] Step 1: Traverse each travel website to be crawled and obtain travel magazine ads from the travel websites; set a size threshold for travel magazine ads, filter out travel magazine ads in the database whose size does not meet the threshold requirements, and then filter out blurry travel magazine ads.
[0007] Step 2: Use a fully convolutional neural network to divide the travel magazine ad layout into six categories of tags: travel magazine ad text, travel magazine ad title, image, title on image, text on image, and background. Refine each tag and then annotate the semantic information of each travel magazine ad layout as the constraint text for each travel magazine ad layout.
[0008] Step 3: Convert the constraint text of each travel magazine advertisement layout obtained in Step 2 into a vector using a text encoder. Then, convert each travel magazine advertisement layout obtained in Step 2 into a feature vector using a layout encoder. Next, construct and train a generative adversarial network (GAN). A mixer is introduced into the GAN to form three inputs to the discriminator: the vector Y converted from the constraint text. i The vector X corresponding to the actual travel magazine ad layout and matching constraint text. i The vector pair (Y) i ,X i ), the vector Y of the text transformation under constraints i The vector X corresponding to the actual travel magazine ad layout and the mismatch constraint text j The vector pair (Y) i ,X j ), the vector Y of the text transformation under constraints i and the layout p generated by the generator i The vector X output by the layout encoder i The vector pair (Y) formed by ′ i ,X i The discriminator determines whether the input vector pair contains a vector corresponding to the generated layout or a vector corresponding to the actual travel magazine ad layout. After the generative adversarial network is trained, the generator automatically generates travel magazine ad layouts that match the constraint text.
[0009] Preferably, in step 1, the process of traversing each travel website to be crawled and obtaining travel magazine advertisements from those websites is as follows: Using the Python-based Scrapy crawling framework, a travel magazine advertisement search project is created. The "engine" requests the travel websites to be crawled from the crawler file and submits them to the scheduler to add them to the travel website queue. After processing the request, the scheduler returns the travel website to the engine. The engine then submits the travel website to the downloader to download the response object. After obtaining the response object, the downloader submits the response result to the engine. Upon receiving the response, the engine submits the response result to the crawler file via the spider middleware. The crawler file processes and analyzes the response result and extracts the required travel magazine advertisements, which are then submitted to the pipeline file for storage in the database.
[0010] Preferably, in step 1, blurry travel magazine advertisements are filtered out using the Sobel operator edge detection algorithm.
[0011] Preferably, the specific process of step 2 is as follows:
[0012] Step 2.1: Set six categories of tags for travel magazine advertisements: text, title, image, title on image, text on image, and background, represented by yellow, green, red, purple, blue, and gray areas respectively. Then, manually divide a portion of the travel magazine advertisement layouts obtained in Step 1 into these six categories as a training set to train a fully convolutional neural network. Next, use the trained fully convolutional neural network to semantically segment the remaining travel magazine advertisement layouts obtained in Step 1 into the six categories.
[0013] Step 2.2: Refine each label after semantic segmentation. The specific process is as follows: First, identify the internal noise points in each label after semantic segmentation using color recognition technology, and then fill the internal noise points with the color of the corresponding label. Next, remove boundary noise points from each label after removing internal noise points. Finally, correct the boundaries of each label after removing boundary noise points.
[0014] Step 2.3: Calculate the proportion of each tag for the travel magazine ad layouts manually divided into six categories in Step 2.1 and the travel magazine ad layouts obtained after processing in Step 2.2, and calculate which categories of tags each travel magazine ad layout consists of and the number of each category of tags. The semantic information of the travel magazine ad layouts is described in nine sentences in Table 1, serving as the constraint text for the travel magazine ad layouts.
[0015] Table 1
[0016]
[0017] More preferably, during the training of a fully convolutional neural network, data augmentation techniques are used to augment the training set.
[0018] More preferably, the process of correcting the boundaries of each label after removing boundary noise points is as follows: Use the cv2.convexHull function in the OpenCV package of Python to obtain the point set of the four boundaries of the label, then calculate the mean point of each boundary, take the mean points of the upper and lower boundaries as two vertical axis coordinates, take the mean points of the left and right boundaries as two horizontal axis coordinates, and combine each horizontal axis coordinate with the two vertical axis coordinates to obtain a total of four point coordinates, which are used as four vertices. The boundary of the rectangular area enclosed by the four vertices is the boundary of the label after correction.
[0019] More preferably, the layout of travel magazine advertisements is divided into seven types of compositions: nested composition, grid composition, symmetrical composition, three-column composition, combined composition, segmented composition, and two-column composition; in the segmented composition, one label occupies most of the page.
[0020] Preferably, step 3 is performed as follows:
[0021] Step 3.1: Convert the constraint text of each travel magazine advertisement layout obtained in Step 2 into a vector using a text encoder. The specific process is as follows: Use the jieba library in Python to segment the constraint text of the travel magazine advertisement layout into words, and create a vocabulary table using one-hot encoding. Use a loop to read all the constraint text of the travel magazine advertisement layout, and determine whether each phrase in the constraint text of the travel magazine advertisement layout is in the vocabulary table. If the phrase is not in the vocabulary table, update the vocabulary table, add the phrase to the vocabulary table, and assign a number to the phrase and replace the phrase with the number. If the phrase is in the vocabulary table, directly replace the phrase with the number in the vocabulary table, thereby converting the constraint text of each travel magazine advertisement layout into a vector x. i ={x1,x2,…,x m ,…,x M}, M represents the constraint on the layout of travel magazine advertisements, the number of phrases in the text, and x m Let Y be the index corresponding to the m-th word group in the constraint text; then pass this vector through a word embedding layer, and then through a recurrent neural network layer, finally outputting vector Y. i .
[0022] Step 3.2: The layout encoder transforms the various travel magazine ad layouts obtained in Step 2 into feature vectors. Specifically, it first uses a spatial attention mechanism to make the travel magazine layouts salient, and then transforms them into feature vectors through a CNN network layer. In the layout encoder, the input is a vector P composed of various travel magazine ad layouts, and the output is the feature vector X of each travel magazine ad layout, where P = [p1, p2, ..., p...]. i ,…,p l In P, each element represents the layout of travel magazine advertisements, and l represents the total number of travel magazine advertisement layouts; the output vector of the spatial attention mechanism. s i For p i In the intermediate hidden state of the spatial attention mechanism; the resulting vector The input is fed into a CNN network layer for learning, and the final output is X = [X1, X2, ..., X...]. i ,…,X l], CNN() represents the convolution operation of a CNN network layer.
[0023] Step 3.3: Construct and train a Generative Adversarial Network (GAN). The GAN consists of a generator and a discriminator. After training, the generator is used to automatically generate a travel magazine advertisement layout that matches the constraint text.
[0024] More preferably, in step 3.1:
[0025] Y i =RNN(s)
[0026] The output s of the word embedding layer i ={s m |m=1,2,…,M},s m Is with x M The corresponding word embedding layer output vector; RNN() is the operation of the recurrent neural network layer; Y i ={y m |m=1,2,…,M},y m Is with x m The corresponding output vector of the recurrent neural network layer.
[0027] More preferably, in step 3.3, the generative adversarial network first samples from noise that follows a Gaussian distribution N(0,1), and then transforms the sampled noise and the constraint text into a vector Y. i The input is fed into the generator to generate a layout p of text that conforms to the given constraints. i ′.
[0028] The beneficial effects of this invention are as follows:
[0029] 1. This invention describes the layout of each real travel magazine advertisement obtained through filtering as a constraint text consisting of nine short sentences, and then converts it into a vector Y using a text encoder. i Real travel magazine advertising layout p i The feature vector X is then generated by the layout encoder. i Then, the vector Y and a segment of Gaussian noise z are input into the generator to generate the layout p′. i The generated layout, the matching real travel magazine ad layout, and the non-matching real travel magazine ad layout are paired with the constraint text and input into the discriminator to determine whether each pairing matches, thereby improving the discriminator's discrimination ability and ultimately ensuring that the generator's generated layout matches the constraint text better.
[0030] 2. In the layout encoder, the spatial attention mechanism is first used to make the layout of the travel magazine saliency to generate a high-quality layout image. Then, it is transformed into a feature vector through a CNN network layer. The recurrent neural network learns the features of the layout image to improve the performance of the generator, which also makes the generated layout more consistent with the text conditions.
[0031] 3. This invention introduces a mixer into a generative adversarial network consisting of a generator and a discriminator. Three types of inputs are added to the discriminator, including vector pairs with mismatched text and the real layout, which improves the performance of the discriminator and can also learn to optimize the constraint relationship between the layout and the text. Attached Figure Description
[0032] Figure 1 This is a schematic diagram of the transmission structure between the text encoder, layout encoder, and generative adversarial network in this invention.
[0033] Figure 2 This is a diagram illustrating the layout of several real travel magazine advertisements.
[0034] Figure 3 This is a schematic diagram of the structure of the fully convolutional neural network in this invention.
[0035] Figure 4 This is a schematic diagram of the structure of the text encoder in this invention.
[0036] Figure 5 This is a schematic diagram of the layout encoder in this invention. Detailed Implementation
[0037] The embodiments will now be described in more detail with reference to the accompanying drawings.
[0038] like Figure 1 As shown, the automatic generator for generating travel magazine ad layouts uses nine short sentences to describe each filtered real travel magazine ad layout as a constraint text, which is then converted into a vector Y by a text encoder. i Real travel magazine advertising layout p i The feature vector X is then generated by the layout encoder. i Then vector Y i Along with a segment of Gaussian noise z, the data is input into the generator to generate the layout p′. i The generated layout, the matching real travel magazine ad layout, and the non-matching real travel magazine ad layout are paired with the constraint text and input into the discriminator D to determine whether each pairing matches, thereby improving the discriminator's discrimination ability and ultimately ensuring that the generator's generated layout better matches the constraint text. The specific steps include the following:
[0039] Step 1: Traverse each travel website to be crawled and obtain travel magazine ads from the travel websites; set a size threshold for travel magazine ads, filter out travel magazine ads in the database whose size does not meet the threshold requirements, and then use the Sobel operator edge detection algorithm to filter out blurry travel magazine ads.
[0040] Step 2: Use a fully convolutional neural network to divide the travel magazine ad layout into six categories of tags: travel magazine ad text, travel magazine ad title, image, title on image, text on image, and background. Refine each tag and then annotate the semantic information of each travel magazine ad layout as the constraint text for each travel magazine ad layout.
[0041] Step 3: The constraint text of each travel magazine advertisement layout obtained in Step 2 is converted into a vector by a text encoder. Then, the layout encoder is used to convert each travel magazine advertisement layout obtained in Step 2 into a feature vector. A generative adversarial network (GAN) is then constructed and trained. A mixer is introduced into the GAN to form three inputs to the discriminator D. After the GAN is trained, the generator is used to automatically generate travel magazine advertisement layouts that match the constraint text.
[0042] In a preferred embodiment, step 1 involves traversing each travel website to be crawled and obtaining travel magazine advertisements from those websites. The specific process is as follows:
[0043] Using the Python-based Scrapy web crawling framework, a travel magazine ad search project is created. The "engine" requests travel websites (URLs) to be crawled from the crawler file and passes them to the scheduler to add them to the travel website queue. After processing the request, the scheduler returns the travel website to the engine. The engine then passes the travel website to the downloader to download the response object. After receiving the response object, the downloader returns the response result to the engine. Upon receiving the response, the engine passes the response result to the crawler file via the spider middleware. The crawler file processes and analyzes the response result and extracts the required travel magazine ads, which are then stored in the database via a pipeline file.
[0044] As a preferred embodiment, the specific process of step 2 is as follows:
[0045] Step 2.1: Set six categories of tags for the travel magazine advertisement: text, title, image, title on image, text on image, and background. Use yellow, green, red, purple, blue, and gray areas respectively to represent these categories. The layout should look like this: Figure 2As shown; this invention distinguishes title elements from other text elements because title elements play an important role in graphic design layout. Then, a portion of the travel magazine ad layouts obtained in step 1 is manually divided into six categories of tags as a training set to train a fully convolutional neural network (FCN). Next, the trained FCN is used to semantically segment the remaining travel magazine ad layouts obtained in step 1 into six categories of tags, such as... Figure 3 As shown. Preferably, since the amount of training data is relatively small, data augmentation techniques (including random resize, random horizontal flip, and random pruning) are used to augment the training set during the training of the fully convolutional neural network (FCN).
[0046] Step 2.2: Refine each label after semantic segmentation. The specific process is as follows: First, identify the internal noise points in each label after semantic segmentation using color recognition technology (such as the OpenCV package in Python), and then fill the internal noise points with the color of the corresponding label. Next, remove boundary noise points from each label after removing internal noise points. Finally, correct the boundaries of each label after removing boundary noise points, specifically as follows: Use the cv2.convexHull function in the OpenCV package in Python to obtain the point set of the four boundaries of the label, then calculate the mean point of each boundary. Use the mean points of the upper and lower boundaries as two vertical axis coordinates, and the mean points of the left and right boundaries as two horizontal axis coordinates. Combine each horizontal axis coordinate with the two vertical axis coordinates to obtain four point coordinates, which are used as four vertices. The boundary of the rectangular area enclosed by the four vertices is the corrected boundary of the label.
[0047] Step 2.3: Calculate the proportion of each tag for the travel magazine ad layouts manually divided into six categories in Step 2.1 and the travel magazine ad layouts obtained after processing in Step 2.2, and calculate which categories of tags each travel magazine ad layout consists of and the number of each category of tags. The semantic information of the travel magazine ad layouts is described in nine sentences in Table 1, serving as the constraint text for the travel magazine ad layouts.
[0048] Table 1
[0049]
[0050]
[0051] The survey on layout classification revealed that layouts can be divided into seven types: nested layouts, grid layouts (including four-grid, six-grid, and nine-grid layouts), symmetrical layouts, three-column layouts, combined layouts (combinations of multiple layout forms), segmented layouts (where one label occupies most of the page), and two-column layouts.
[0052] As a preferred embodiment, step 3 is performed as follows:
[0053] Step 3.1: Extract the constraint text T for each travel magazine advertisement layout obtained in Step 2. i ={t1,t2,t3,…,t9} is converted into a vector by a text encoder. Each element in T represents a phrase in the constraint text. The goal is to learn text features from the constraint text content of the travel magazine advertisement layout and use the learned text features to guide the generation of the travel magazine advertisement layout. The specific process is as follows: The constraint text of the travel magazine advertisement layout is segmented using the jieba library in Python. A vocabulary table (containing phrases a1, a2, a3,…,x) is created using one-hot encoding. K Let K be the total number of phrases in the vocabulary table. A loop is used to read the constraint text of all travel magazine ad layouts. Each phrase in the constraint text is checked against the vocabulary table. If the phrase is not in the vocabulary table, it is updated, added, and numbered, and this number is used to replace the phrase. If the phrase is in the vocabulary table, its number is used to replace the phrase. This process transforms the constraint text of each travel magazine ad layout into a vector x. i ={x1,x2,…,x m ,…,x M}, x m Let be the index of the m-th word group in the constraint text; then pass this vector through a word embedding layer to output s. i Then, it passes through a recurrent neural network (RNN) layer, and finally outputs a vector Y. i ,like Figure 4 As shown.
[0054] Y i =RNN(s)
[0055] Where s i ={s m |m=1,2,…,M}, where M is the constraint condition for the layout of travel magazine advertisements, and s is the number of phrases in the text. m Is with x M The corresponding word embedding layer output vector; RNN() is the operation of the recurrent neural network layer; Y i ={y m |m=1,2,…,M},y m Is with x m The corresponding output vector of the recurrent neural network layer.
[0056] Step 3.2: The layout encoder transforms the various travel magazine advertisement layouts obtained in Step 2 into feature vectors. Specifically, it first uses a spatial attention mechanism to salientize the travel magazine layouts, and then transforms them into feature vectors through a CNN network layer. If only a layout encoder constructed with a single CNN is used, it is difficult to learn the layout features. The generated layouts will have overlapping regions and severely distorted borders. Therefore, a spatial attention mechanism is added to the CNN-constructed layout encoder. First, the spatial attention mechanism salientizes the travel magazine layouts, and then the CNN network layer can better learn the layout features. CNN network models based on spatial attention mechanisms have become popular in various computer vision and machine learning tasks, including neural machine translation, image classification, image segmentation, image and video captioning, and visual question answering. Attention improves the performance of all these tasks by encouraging the model to focus on the most relevant parts of the input. In the layout encoder, the input is a vector P composed of various travel magazine advertisement layouts, and the output is the feature vector X of each travel magazine advertisement layout, such as... Figure 5 As shown. Where P = [p1, p2, ..., p i ,…,p l In P, each element represents the layout of travel magazine advertisements, and l represents the total number of travel magazine advertisement layouts; the output vector of the spatial attention mechanism. s i For p i In the intermediate hidden state of the Spatial Attention Mechanism (SAM); the resulting vector The inputs are fed into CNN network layers to learn the global and local behaviors that capture labels, and the final output is X = [X1, X2, ..., X...]. i ,…,X l ], CNN() represents the convolution operation of a CNN network layer.
[0057] Step 3.3: Construct and train a Generative Adversarial Network (GAN). The GAN consists of a generator and a discriminator (D). After training, the generator automatically generates travel magazine advertisement layouts that match the constraint text. The purpose of using a dual-system GAN is to allow the generator to confuse the discriminator as much as possible, while allowing the discriminator (D) to judge the input source as much as possible. The two are in an adversarial relationship, striving to improve by trying to defeat each other. The generator can obtain feedback from the discriminator (D) on whether the generated result matches the dataset, while the discriminator (D) can obtain more training samples from the generator.
[0058] Generative Adversarial Networks (GANs) first sample noise z following a Gaussian distribution N(0,1), and then transform the sampled noise and the constraint text into a vector Y. i The input is fed into the generator to generate a layout p of text that conforms to the given constraints. i ′.
[0059] This invention introduces a mixer into existing generative adversarial networks to form three inputs to the discriminator D: a vector Y representing the text transformation under constraints. i The vector X corresponding to the actual travel magazine ad layout and matching constraint text. i The vector pair (Y) i ,X i ), the vector Y of the text transformation under constraints i The vector X corresponding to the actual travel magazine ad layout and the mismatch constraint text j The vector pair (Y) i ,X j ), the vector Y of the text transformation under constraints i and the layout p generated by the generator i The vector X output by the layout encoder i The vector pair (Y) formed by ′ i ,X i The discriminator determines whether the input vector pair contains a vector G corresponding to the generated layout or a vector R corresponding to the actual travel magazine advertisement layout. This ensures the discriminator doesn't ignore vectors transformed from constraint text and avoids misinterpreting feature vectors as X. i The layout of the travel magazine advertisement is not directly determined as a generated layout and thus output by the generator, but rather it is combined with the constraint text Y. i Correspondingly, determine whether the constraint condition text Y is satisfied. i Only when the conditions are met will the generator output the ad. Therefore, this invention introduces a mixer to improve the performance of the discriminator, thereby making the layout of the travel magazine ad generated by the generator more closely match the constraint text.
[0060] The discriminator first performs multi-layer convolutions using spatial batch normalization, then reduces the dimension of the constraint text to a fully connected layer using ReLU, and finally performs correction. When the spatial dimension of the discriminator is 4×4, the constraint text embedding is copied spatially and deep connections are performed, followed by 1×1 convolutions, then correction and 4×4 convolutions to calculate the discriminator's final score. Batch normalization is performed on all convolutional layers.
[0061] As a preferred embodiment, the user enters the desired layout style in the 'Design Style' panel of the user interface. The user can select the number of each tag based on the amount of materials they have. After selecting, clicking the 'OK' button generates a layout that meets the requirements based on the user's input design style and number of tags, and recommends it to the user. The user can then design a travel magazine advertisement based on the layout.
Claims
1. An automatic generator for travel magazine ad layouts, characterized by: Includes the following steps: Step 1: Traverse each travel website to be crawled and obtain travel magazine ads from the travel websites; set a size threshold for travel magazine ads, filter out travel magazine ads in the database whose size does not meet the threshold requirements, and then filter out blurry travel magazine ads. Step 2: Use a fully convolutional neural network to divide the travel magazine ad layout into six categories of tags: travel magazine ad text, travel magazine ad title, image, title on image, text on image, and background. Refine each tag and then annotate the semantic information of each travel magazine ad layout as the constraint text for each travel magazine ad layout. Step 3: Convert the constraint text of each travel magazine advertisement layout obtained in Step 2 into a vector using a text encoder. Then, convert each travel magazine advertisement layout obtained in Step 2 into a feature vector using a layout encoder. Next, construct and train a generative adversarial network (GAN). A mixer is introduced into the GAN to form three inputs to the discriminator: the vector Y converted from the constraint text. i The vector X corresponding to the actual travel magazine ad layout and matching constraint text. i The vector pair (Y) i ,X i ), the vector Y of the text transformation under constraints i The vector X corresponding to the actual travel magazine ad layout and the mismatch constraint text j The vector pair (Y) i ,X j ), the vector Y of the text transformation under constraints i and the layout p generated by the generator i The vector X output by the layout encoder i The vector pair (Y) formed by ′ i ,X i The discriminator determines whether the input vector pair contains a vector corresponding to the generated layout or a vector corresponding to the actual travel magazine advertisement layout. After the generative adversarial network is trained, the generator automatically generates a travel magazine advertisement layout that matches the constraint text.
2. The automatic generator for generating travel magazine advertisement layouts according to claim 1, characterized in that: In step 1, the various travel websites to be crawled are traversed to obtain travel magazine advertisements from them. The specific process is as follows: Using the Python-based Scrapy crawling framework, a travel magazine advertisement search project is created. The "engine" requests the travel websites to be crawled from the crawler file and hands them over to the scheduler to add them to the travel website queue. After the scheduler processes the request, it returns the travel website to the engine. The engine then hands the travel website over to the downloader to download the response object. After the downloader obtains the response object, it hands the response result to the engine. Upon receiving the response result, the engine passes it to the crawler file via the spider middleware. The crawler processes and analyzes the response results, extracts the necessary travel magazine advertisements, and passes them to the pipeline file for storage in the database.
3. The automatic generator for generating travel magazine advertisement layouts according to claim 1, characterized in that: In step 1, blurry travel magazine advertisements are filtered out using the Sobel operator edge detection algorithm.
4. The automatic generator for generating travel magazine advertisement layouts according to claim 1, characterized in that: The specific process of step 2 is as follows: Step 2.1: Set six categories of tags for travel magazine advertisements: text, title, image, title on image, text on image, and background, represented by yellow, green, red, purple, blue, and gray areas respectively. Then, manually divide a portion of the travel magazine advertisement layouts obtained in Step 1 into these six categories as a training set to train a fully convolutional neural network. Next, use the trained fully convolutional neural network to semantically segment the remaining travel magazine advertisement layouts obtained in Step 1 into the six categories. Step 2.2: Refine each label after semantic segmentation. The specific process is as follows: First, identify the internal noise points in each label after semantic segmentation using color recognition technology, and then fill the internal noise points with the color of the corresponding label; next, remove the boundary noise points from each label after removing the internal noise points; finally, correct the boundaries of each label after removing the boundary noise points. Step 2.3: Calculate the proportion of each tag for the travel magazine ad layout manually divided into six categories in Step 2.1 and the travel magazine ad layout obtained after processing in Step 2.2, and calculate which categories of tags each travel magazine ad layout consists of and the number of each category of tags; use nine sentences to describe the semantic information of the travel magazine ad layout as the constraint text for the travel magazine ad layout; the nine sentences are: composition type, proportion of travel magazine ad text area, proportion of image area, proportion of travel magazine ad title area, proportion of background area, proportion of title area on image, proportion of text area on image, tag categories that make up the layout, and the number of each category of tags except the background.
5. The automatic generator for generating travel magazine advertisement layouts according to claim 4, characterized in that: During the training of a fully convolutional neural network, data augmentation techniques are used to enhance the training set.
6. The automatic generator for generating travel magazine advertisement layouts according to claim 4, characterized in that: The specific process for correcting the boundaries of each label after removing boundary noise points is as follows: The cv2.convexHull function in the OpenCV package of Python is used to obtain the point set of the four boundaries of the label. Then, the mean point of each boundary is calculated. The mean points of the upper and lower boundaries are used as two vertical axis coordinates, and the mean points of the left and right boundaries are used as two horizontal axis coordinates. Each horizontal axis coordinate is combined with the two vertical axis coordinates to obtain four point coordinates, which are used as four vertices. The boundary of the rectangular area enclosed by the four vertices is the boundary of the label after correction.
7. The automatic generator for generating travel magazine advertisement layouts according to claim 4, characterized in that: The layout of travel magazine advertisements can be divided into seven types: nested layout, grid layout, symmetrical layout, three-column layout, combined layout, segmented layout, and two-column layout. In a segmented layout, one label occupies most of the page.
8. The automatic generator for generating travel magazine advertisement layouts according to claim 1, characterized in that: Step 3 is as follows: Step 3.1: Convert the constraint text of each travel magazine advertisement layout obtained in Step 2 into a vector using a text encoder. The specific process is as follows: Use the jieba library in Python to segment the constraint text of the travel magazine advertisement layout into words, and create a vocabulary table using one-hot encoding. Use a loop to read all the constraint text of the travel magazine advertisement layout, and determine whether each phrase in the constraint text of the travel magazine advertisement layout is in the vocabulary table. If the phrase is not in the vocabulary table, update the vocabulary table, add the phrase to the vocabulary table, and assign a number to the phrase and replace the phrase with the number. If the phrase is in the vocabulary table, directly replace the phrase with the number in the vocabulary table, thereby converting the constraint text of each travel magazine advertisement layout into a vector x. i ={x1,x2,…,x m ,…,x M }, M represents the constraint on the layout of travel magazine advertisements, the number of phrases in the text, and x m Let Y be the index corresponding to the m-th word group in the constraint text; then pass this vector through a word embedding layer, and then through a recurrent neural network layer, finally outputting vector Y. i ; Step 3.2: The layout encoder transforms the various travel magazine ad layouts obtained in Step 2 into feature vectors. Specifically, it first uses a spatial attention mechanism to make the travel magazine layouts salient, and then transforms them into feature vectors through a CNN network layer. In the layout encoder, the input is a vector P composed of various travel magazine ad layouts, and the output is the feature vector X of each travel magazine ad layout, where P = [p1, p2, ..., p...]. i ,…,p l In P, each element represents the layout of travel magazine advertisements, and l represents the total number of travel magazine advertisement layouts; the output vector of the spatial attention mechanism. s i For p i In the intermediate hidden state of the spatial attention mechanism; the resulting vector The input is fed into a CNN network layer for learning, and the final output is X = [X1, X2, ..., X...]. i ,…,X l ], CNN() represents the convolution operation of a CNN network layer; Step 3.3: Construct and train a generative adversarial network (GAN). The GAN consists of a generator and a discriminator. After the GAN is trained, the generator is used to automatically generate a travel magazine advertisement layout that matches the constraint text.
9. The automatic generator for generating travel magazine advertisement layouts according to claim 8, characterized in that: In step 3.1: Y i =RNN(s) The output s of the word embedding layer i ={s m |m=1,2,…,M},s m Is with x M The corresponding word embedding layer output vector; RNN() is the operation of the recurrent neural network layer; Y i ={y m |m=1,2,…,M},y m Is with x m The corresponding output vector of the recurrent neural network layer.
10. The automatic generator for generating travel magazine advertisement layouts according to claim 8, characterized in that: In step 3.3, the generative adversarial network first samples from noise that follows a Gaussian distribution N(0,1), and then transforms the sampled noise and the constraint text into a vector Y. i The input is fed into the generator to generate a layout p of text that conforms to the given constraints. i ′.