Transform-based page layout generative adversarial network method and related system

Abstract

The invention belongs to the field of page layout and generative adversarial networks, and discloses a transformer-based page layout generative adversarial network method and a related system, which can automatically generate page layout through training of a TGAN model and reduce the demand of manual intervention. In the traditional design, a designer needs to manually adjust the layout of each element, the TGAN network based on the Transform can automatically generate the layout conforming to the design specification according to the training data, the design efficiency is improved, the Transform model is good at capturing the long-term dependency relationship, the complex structure and the long-distance relationship can be processed, and the design efficiency is improved. This enables it to better understand the relationship between different elements, and to more accurately generate designs that meet logic and aesthetic requirements during automatic layout.

Description

Technical Field

[0001] The present invention belongs to the field of page layout and generative adversarial networks, and in particular relates to a transformer-based page layout generative adversarial network method and a related system. Background Art

[0002] With the continuous development of electronic devices such as computers and mobile phones, various pages have entered everyone's life, and page layout design has become one of the important tasks in many fields. Layout design is to effectively combine text, graphics, tables, etc. in a limited space, so that the layout looks colorful, the layout style is solemn and steady or dynamic and lively, which can attract readers' attention, increase reading interest, and enable readers to directly feel the main idea conveyed by the layout visually. Layout design is applied to many fields of design planes and images such as web pages, advertisements, posters, magazines, documents, newspapers, presentations, etc. Designs are often difficult to create because they must clearly convey information while meeting the requirements of aesthetic goals. In fact, CAD is the result of earlier computer-aided page design. In recent years, thanks to the rapid development of image processing technology based on deep learning, some scholars have begun to use deep neural networks to promote the task of automatic page layout. The automatic page layout technology has evolved from early statistical learning based on limited samples to machine learning and deep learning based on deep neural networks, reducing the manual dependence in the design process and improving the accuracy of data generation. Since 2014, generative adversarial networks have attracted widespread attention for their powerful image generation capabilities. In the initial stage of research, the limitation of generative technology was that it could only generate images with strong structure, such as faces or pattern textures. By continuously improving the network structure of generative adversarial networks and proposing new loss functions, the generation ability of the model has been further enhanced, and applications in multiple fields such as natural image generation, product design, and 3D printing have been achieved. At present, related research is moving towards the development of large models such as mutual conversion between multiple modalities of text, images and videos, high-resolution image generation, and style transfer. However, the technical challenges have not been fully discussed. How to balance the coordination of different elements in such a diverse design, ensure the readability of the text, the beauty of the image, and the harmony of the entire layout are technical obstacles that need to be overcome when applying deep learning to page design. Summary of the invention

[0003] The purpose of the present invention is to overcome the deficiency of coordination balance of different elements in diversity design, and to provide a transformer-based page layout generation adversarial network method and related system.

[0004] In order to achieve the above object, the present invention adopts the following technical solution: In a first aspect, the present invention provides a transformer-based page layout generation adversarial network method, comprising the following steps: Obtain instance graphs of different page layouts, preprocess the instance graphs of the page layouts, and obtain a unified abstract page layout graph; Divide the abstract page layout graphs into training and testing sets; Build a transformer-based TGAN network model for automatic page layout generation, convert the training set of abstract page layout graphs into tensor data, and train the transformer-based TGAN network model for automatic page layout generation; The test set is used to test the trained transformer-based automatic page layout generation TGAN network model to obtain a simulated page layout that meets the requirements; The simulated page layouts are scored, and the effectiveness and robustness of the trained adversarial network are judged based on the scoring results.

[0005] A further improvement of the present invention is that the specific method of obtaining instance graphs of different page layouts and preprocessing the instance graphs of the page layouts to obtain a unified abstract page layout graph is as follows: Get instance images of different page layouts; Unify instance images of different page layouts to the target pixel size; Unify the page layout elements in the example diagrams of different page layouts into boxes of different colors; The boxes with uniform target pixel size and different colors are used as abstract page layout diagrams.

[0006] A further improvement of the present invention is that the working method of generating the TGAN network model based on the transformer-based automatic page layout is as follows: Get the test set and extract the dataset label pair of the test set; The transformer module is used to generate pseudo samples and pseudo label pairs that match the label pairs of the test set dataset; Establish the variable relationship between the dataset label pair and the pseudo sample and pseudo label pair to form a combined conditional variable; Input the dataset label pair into the combined condition variable to get the output result.

[0007] A further improvement of the present invention is that the working method of the transformer module is as follows: The dataset label pairs of the test set are vectorized through embedding processing, and then the corresponding data in the dataset label pairs of the test set are positionally encoded; The dataset is mapped to triples Q, K, and V through the attention mechanism for mapping representation and outputting pseudo samples and pseudo label pairs.

[0008] A further improvement of the present invention is that the scoring criteria for scoring the simulated page layout are as follows:

[0009] Among them, the design layout To simulate the position, metadata, width and height of each object in the page layout, are model parameters, is the energy term, Energy term The weight of is a nonlinear parameter.

[0010] A further improvement of the present invention is that the energy term It includes alignment energy term, balance energy term, white space energy term, scale energy term, overlap energy term and boundary energy term.

[0011] In a second aspect, the present invention provides a transformer-based page layout generation adversarial network system, comprising: A preprocessing module is used to obtain instance graphs of different page layouts, preprocess the instance graphs of the page layouts, and obtain a unified abstract page layout graph; A data differentiation module, used to divide the abstract page layout graph into a training set and a test set; The model training module is used to build a transformer-based TGAN network model for automatic page layout generation, convert the training set of the abstract page layout graph into tensor data, and train the transformer-based TGAN network model for automatic page layout generation; The model testing module is used to test the trained transformer-based automatic page layout generation TGAN network model using a test set to obtain a simulated page layout that meets the requirements; The model scoring module is used to score the simulated page layout and judge the effectiveness and robustness of the trained adversarial network based on the scoring results.

[0012] A further improvement of the present invention is that the function of the preprocessing module is realized by the following method: Get instance images of different page layouts; Unify instance images of different page layouts to the target pixel size; Unify the page layout elements in the example diagrams of different page layouts into boxes of different colors; The boxes with uniform target pixel size and different colors are used as abstract page layout diagrams.

[0013] A further improvement of the present invention is that the function of generating the TGAN network model based on the transformer-based automatic page layout in the model training module is realized by the following method: Get the test set and extract the dataset label pair of the test set; The transformer module is used to generate pseudo samples and pseudo label pairs that match the label pairs of the test set dataset; Establish the variable relationship between the dataset label pair and the pseudo sample and pseudo label pair to form a combined conditional variable; Input the dataset label pair into the combined condition variable to get the output result.

[0014] A further improvement of the present invention is that the function of the model scoring module is realized by the following method:

[0015] Among them, the design layout To simulate the position, metadata, width and height of each object in the page layout, are model parameters, is the energy term, Energy term The weight of is a nonlinear parameter.

[0016] In a third aspect, the present invention provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a transformer-based page layout generation adversarial network method when executing the computer program.

[0017] In a fourth aspect, the present invention provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a transformer-based page layout generation adversarial network method.

[0018] Compared with the prior art, the present invention has the following beneficial effects: The present invention can automatically generate page layouts through the training of the TGAN model, reducing the need for manual intervention. In traditional design, designers need to manually adjust the layout of each element, while the Transformer-based TGAN network can automatically generate a layout that meets the design specifications based on training data, thereby improving the efficiency of the design. The Transformer model is good at capturing long-term dependencies and can handle complex structures and long-distance relationships, which enables it to better understand the relationship between different elements and more accurately generate designs that meet logical and aesthetic requirements during automatic layout. Through training, TGAN can dynamically adjust the position and size of different elements, automatically optimize the spacing and alignment between elements, and ensure the visual balance of the page. In particular, when facing layout requirements for different screen sizes or devices, the flexibility of Transformer enables it to generate an adaptive layout based on different input information to ensure the coordination of the design. Generative adversarial networks (GANs) themselves have strong robustness and can generate a variety of layout designs and adapt to different types of inputs. Even in the face of complex or unknown layout styles, TGAN can still maintain a high generation quality, which makes it have strong fault tolerance and adaptability when facing different inputs or changing scenarios. In conclusion, the Transformer-based page layout generative adversarial network (TGAN) has significant advantages in dealing with the coordination and diversity of design elements. Through its powerful modeling capabilities, it can automatically generate page layouts that conform to design principles, enhance the coordination between different elements, adapt to diverse and personalized needs, and improve the consistency, creativity, and robustness of the design. In addition, the introduction of feedback mechanisms and scoring systems helps to further optimize the design process and ensure that the generated page layouts meet actual needs. Therefore, TGAN provides an innovative and efficient solution to the coordination and balance problems in traditional layout design. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of the present invention; Figure 2 is a system diagram of the present invention; Figure 3 It is the principle diagram of the present invention; Figure 4 is a schematic diagram of a generator in the present invention; Figure 5 is a schematic diagram of a discriminator in the present invention; Figure 6 Generate the training process of the TGAN network model for transformer-based page automatic layout; Figure 7 An example diagram of the sampling process of a network dataset for transformer-based page layout adversarial generation; Figure 8 This is a system diagram of Example 6. DETAILED DESCRIPTION

[0020] In order to further understand the content of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the embodiments are only for explaining the present invention and are not intended to limit it.

[0021] See also Figure 1 , a transformer-based page layout generation adversarial network method includes the following steps: S1, obtain instance graphs of different page layouts, preprocess the instance graphs of the page layouts, and obtain a unified abstract page layout graph.

[0022] S2, divides the abstract page layout graph into training set and test set.

[0023] S3, builds a transformer-based TGAN network model for automatic page layout generation, converts the training set of the abstract page layout graph into tensor data, and trains the transformer-based TGAN network model for automatic page layout generation.

[0024] S4, use the test set to test the trained transformer-based page automatic layout generation TGAN network model to obtain a simulated page layout that meets the requirements.

[0025] S5, scores the simulated page layout, and judges the effectiveness and robustness of the trained adversarial network based on the scoring results.

[0026] See also Figure 2 , a transformer-based page layout generation adversarial network system, including: A preprocessing module is used to obtain instance graphs of different page layouts, preprocess the instance graphs of the page layouts, and obtain a unified abstract page layout graph; A data differentiation module, used to divide the abstract page layout graph into a training set and a test set; The model training module is used to build a transformer-based TGAN network model for automatic page layout generation, convert the training set of the abstract page layout graph into tensor data, and train the transformer-based TGAN network model for automatic page layout generation; The model testing module is used to test the trained transformer-based automatic page layout generation TGAN network model using a test set to obtain a simulated page layout that meets the requirements; The model scoring module is used to score the simulated page layout and judge the effectiveness and robustness of the trained adversarial network based on the scoring results.

[0027] Embodiment 1: At present, with the continuous development of image algorithms, a variety of GAN variant models have been derived, making the training process more stable and significantly improving the quality of generated images. The present invention introduces the transformer mechanism based on the above method. The specific TGAN network diagram is as follows: Figure 3 As shown in . Among them, the model diagram of the generator is as follows Figure 4 As shown in . The generator has two inputs, namely, a 100-dimensional vector randomly sampled from the latent space (left) and a class label y (right). The model uses a neural network structure with two layers of deconvolution and one layer of convolution. First, a random vector of length 100 is input to the generator, and the vector dimension is converted to 320 dimensions through the fully connected layer. The function of the fully connected layer is to fuse the image features and further obtain the high-level meaning of the feature. This process is achieved through multiple pooling layers and convolution layers. The vector input enters the reshape layer to change its shape through matrix transformation, and the output is sent to the transformer_block layer for feature extraction. The data processed by the Transformer module will be flattened through the flatten layer to make the vector one-dimensional. The network layer that processes the Transformer module consists of the above dense layer, reshape layer, transformer_block layer and flatten layer. Its main function is to extract image features more carefully, thereby improving the accuracy of the image generated by the generator and ensuring faster convergence of the model.

[0028] When the generator converges, freeze the parameters of the generator and iterate the discriminator instead. The model diagram of the discriminator is as follows: Figure 5As shown in . The discriminator takes an image of 224×224 pixels as input and outputs a binary prediction of whether the image is real (class=1) or fake (class=0). An output of 1 means that the discriminator believes that the generated image is close to the real image, and an output of 0 means that the discriminator believes that the generated image is not good. The discriminator model is designed as a moderate convolutional neural network, using a LeakyReLU activation function with a slope of 0.2 and a 2×2 step for downsampling. As shown in Figure 5, the model has two inputs. The left input is the class label (y) and an embedding layer with a dimension of 50. The input uses an integer as the class label of the image, so as to achieve the effect of conditioning the input image on it. The right input is an image of size 224×224 defined by the CGAN discriminator model. The class label passes through an embedding layer with a dimension of 50, where each class will be mapped to a different 50-dimensional vector representation, which will be learned by the discriminator. The embedded output is then passed to a fully connected layer with linear activation. After the fully connected layer, the output is reshaped into an activation map of size 224×224 through the reshape layer and connected to the input image. When input to the next convolutional layer, the above operation can achieve the effect of a dual-channel input image. The define_discriminator() function is used to update the discriminator. The parameterized shape of the input image is used to define the activation of the fully connected layer after the embedding layer to reshape its output. Finally, by defining and compiling the discriminator and returning it, the number of classes of the model is also parameterized in the function and collection.

[0029] Repeat the above iterations until the model converges. Finally, use the trained model to generate a simulated page layout diagram, and then use the proposed subjective and objective scores to score the model.

[0030] Embodiment 2: There are many types of content contained in the page layout, mainly including titles, subtitles, tables, graphics, text, etc. In addition, the actual sizes of different page layouts are not consistent, so it is difficult to use different page layout samples as object information as input data. In view of the problem of inconsistent actual sizes of all page layouts, all real data are unified to 224×224 pixels. The elements in the page layout, such as titles, subtitles, tables, graphics, and text, are abstracted into boxes of different colors to ignore the factors such as text, images, and formats inside the elements that affect the model's generation of the overall page layout.

[0031] Embodiment 3: In the task of page layout, convolutional neural networks can process local features of images very well, while transformers can focus more on long-distance feature information. Combining the advantages of the two will help to better process image tasks.

[0032] Therefore, the present invention proposes a TGAN network. The starting point of the generator learning of TGAN is no longer just the unknown noise distribution, but the label or more detailed features of the image, and then synthesizes fake samples. During the training of the TGAN model, the generator learns to generate realistic samples that match the labels of the dataset, and the discriminator learns to distinguish between the fake samples and label pairs from the generator and the real samples and label pairs from the training set. By introducing conditional variables into the generator and the discriminator, and All form combined conditional variables, and the objective function of TGAN becomes:

[0033] Among them, D is the discriminator, whose purpose is to distinguish between real data and generated data. G is the generator, whose purpose is to generate data that is difficult to distinguish from real data. is the objective function, From the real data distribution The sample x sampled from Output a probability value for the discriminator D, which indicates the probability that the input x belongs to the real data under the given condition y. The purpose of taking the logarithm of the discriminator’s output is to maximize the accuracy of the discriminator on real data. From the prior noise distribution The noise z sampled in The generator G generates samples according to the input noise z and condition y, The discriminator D judges the samples generated by the generator The probability of truth given condition y. is the logarithm of the output probability of the discriminator, which is used to punish the discriminator for misclassifying the generated sample as a real sample. x is the real data sample, is the probability distribution of the real data, is the conditional variable, is the noise vector, is the prior distribution of noise, is the output of the discriminator, is the output of the generator.

[0034] Embodiment 4: The Transformer module consists of an encoder and a decoder. The encoder maps the input sequence x = (x1, ..., xn) to a continuous sequence z = (z1, ..., zn). The decoder generates a sequence = (y1, ..., ym) based on a given value z. The input data is represented by a vector through Embedding, and then the corresponding data in the sequence is positionally encoded to prevent data confusion. The data is mapped to the triple Q (Query), K (key), and V (Value) through self-Attention (attention mechanism) for mapping and representation and output vectors. The formula of the self-attention mechanism is as follows:

[0035] In the above formula is the query matrix, is the key matrix, and the two are matched by inner product, and the result is the attention matrix, which effectively reflects and The semantic relevance between . In addition, Represents the channel dimension, and the weight of the value calculated by the Softmax function is further used to obtain the value of self-attention. The multi-head attention mechanism (Multi-HeadAttention) is composed of multiple attention mechanisms. It learns the relationship between each data through parallel training. Its calculation formula is as follows:

[0036]

[0037] In addition to the multi-head self-attention mechanism, each layer of the Transformer encoder and decoder also contains a fully connected feedforward network. The fully connected feedforward network is composed of a ReLU activation function through two linear transformations, and its calculation formula is as follows:

[0038] Although the linear transformations at different positions in the fully connected feedforward network layers are the same, the network parameters used are different.

[0039] The role of position encoding is to obtain accurate information about the position of the input sequence, thereby making up for the lack of the recursive network and convolutional neural network of the Transformer model. By adding "position encoding" to the bottom input of the encoder and decoder, the bottom input and the position encoding can have the same dimension for addition, so that the position encoding can accurately describe the relative and absolute position of the input sequence data. There are many ways to perform position encoding, and the Transformer model uses cosine and sine functions of different frequencies.

[0040]

[0041]

[0042] in represents the dimension, Represents the position, which means that each dimension of the position encoding has a corresponding sine function.

[0043] Embodiment 5: See also Figure 7 , ordinary users scored the slides in the page layout samples, with a score range of 0 to 10, and higher scores indicate better layout quality. In practice, users mainly judge the layout of presentations visually and subjectively. Combining aesthetic principles and layout-related criteria, without considering the overall coordination of background and color, users subjectively evaluated the layout quality of the samples based on the following two aspects: Reasonableness of object position: whether the object position looks beautiful, whether the objects are aligned, and whether the objects are in a safe position on the page. Aligning objects can show the connection between objects and make it easier for readers to understand visually; on the other hand, the entire layout will appear beautiful and orderly.

[0044] Text readability: Is there any overlap between text or images that affects reading?

[0045] The objective evaluation of layout quality is mainly based on aesthetic principles and layout-related criteria. , using the parameter To evaluate the layout of your presentation The overall quality of the layout is measured by the weighted sum of the energy terms:

[0046] Among them, the design layout Defined as the position, metadata, width and height of each object, are model parameters. Model parameters are divided into two groups: Each energy term The weight of are all positive, and most of the terms have nonlinear parameters The energy function enforces many design principles, including alignment, balance, white space, scale, overlap, and margins. Each energy term is defined below.

[0047] Alignment Energy Term: The alignment energy term is crucial for design. In a limited space, in order to efficiently transmit information and organize the structure to look beautiful, the first thing to consider is the alignment of various elements. The present invention defines six possible alignment types: Left, X-center, Right, Top, Y-center and Bottom. Before defining the alignment energy term, an analysis phase is performed, which marks all aligned objects and alignment groups. A simple heuristic algorithm is used to calculate the alignment of the object bounding box. First, the difference between the edge or center position of the bounding box must be less than the threshold:

[0048] Where 𝑎 is the alignment type. represents the distance between two objects 𝑖 and 𝑗, depending on the type of alignment using the bounding boxes of the objects. For example, if 𝑎 = 𝐿𝑒𝑓𝑡, then 𝐿𝑒𝑓𝑡 measures the difference between the left edges of the bounding boxes of two objects. The threshold is set to = 0.065, slightly misaligned objects are still marked as aligned.

[0049]

[0050] in, yes and Then, define the alignment indicators. and The alignment indicator variable between is represented as , is a combination of the above terms:

[0051] In each axis, two objects can generally only be aligned in a single way. Minimum alignment distance is set to 1 and the other two to 0. However, if all types are perfectly aligned (all types of = 0), then all three indices are set to 1. Next, an alignment group is defined as a set of connected alignment objects. If an object and Alignment, and objects and Aligned by the same type, 𝑖 and 𝑗 are set to be aligned, that is: .

[0052] Finally, an alignment energy term is defined that measures the proportion of objects that are aligned with a particular alignment type. Larger alignment groups are encouraged because they produce simpler designs with more uniformity between objects. The fraction of pairs of objects whose types are aligned:

[0053] in, is the number of objects, Representation Object and By type Alignment. Different alignment types define separate energy terms. express Is it by type 𝑎 internally aligned text objects. Each feature is represented by Transformation, it is a smooth step function, the formula is shown in (13). The parameter 𝛼 controls the smoothness of the step. When the value of 𝛼 is large, the energy is more sensitive to smaller changes.

[0054]

[0055] Balance energy term: Balance refers to the even distribution of weight on the page, i.e. the symmetry (or asymmetry) of the page in terms of color, size, shape and texture. The present invention mainly considers the symmetry of the size of objects on the page. The global balance is measured using binary mapping (flipping along the axis) of text or graphic objects. Axisymmetric, its equilibrium energy term is:

[0056]

[0057] in, Represents pixels Is it A binary variable, Represents pixels In the image of The symmetric counterpart on the axis. Axisymmetric, a similar equilibrium energy term is defined .in, Denotes the 𝑦-axis symmetric term.

[0058] White space energy term: White space in graphic design is fundamental to readability and aesthetics. The distance between objects is closely related to relevance. The closer the objects are, the more likely they are to be related to each other. White space also affects the overall design style, and many modern designs use significant white space. However, white space between objects can also distract readers. White space should occupy about half of the entire page area. The energy term of white space is defined as:

[0059] in, is a binary variable representing the pixel Whether it is covered by the object, and is the width and height of the entire page.

[0060] Scale energy term: The size of an object determines its practicality and style. Objects must be large enough to be easy to see, but not too large, which can easily make the design cluttered and unsightly. The types of objects in a layout can be roughly divided into two categories: text and graphics. The size of the object is defined according to the different types. The size of a text object is defined as:

[0061] in, Indicates the height of the text object. Indicates the number of lines of text object. is the scaling parameter and is determined by the page height In the present invention, = 1. Use a smaller value, such as =0.4, will increase the size of the text object. The size of the graphic object is defined as:

[0062] in, and Represents the width and height of the graphics object's bounding box, and are the width and height of the page. The scale energy term of the text object is defined as:

[0063] in, is the number of text objects. The scale energy term of the graphic objects Definition and similar.

[0064] Overlap energy term: Overlapping objects are common in many designs, but some overlaps can affect information acquisition. Three types of overlap are defined, including overlap between text objects, overlap between graphic and text objects, and overlap between graphic objects. Separate energy terms are defined for each overlap type and are added to the overlapped pixels. The overlapping energy term between text objects is defined as:

[0065]

[0066] Among them, 𝐴𝑝𝑡 represents the pixels where text objects overlap with each other. and Similarly, it is defined as the overlap between text and graphic objects, and the overlap between graphics and graphic objects.

[0067] Boundary energy term: The extent to which an object can extend beyond its boundaries is controlled by calculating the portion of the object that exceeds its boundaries. Graphic boundary energy terms and text boundary energy terms are defined. The definition of the graphic boundary energy term is:

[0068]

[0069] in, represents the sum of all pixels in object 𝑖, Representation Object The sum of pixels within the page boundary. Text boundary energy term The definition of is similar to that of the graph boundary energy term.

[0070] By defining energy items for alignment, balance, white space, scale, overlap, and border, we can objectively evaluate the details of the presentation layout. The value range of each energy item is [0,1]. Each energy item is weighted =1 / 6 to measure the overall quality of the layout. The closer the weighted sum of the energy terms is to 0, the better the layout quality.

[0071] Embodiment 6: See also Figure 8 As shown, the present invention also provides an electronic device 100 for a transformer-based page layout generation adversarial network method; the electronic device 100 includes a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on the at least one processor 102, and at least one communication bus 104.

[0072] The memory 101 can be used to store the computer program 103. The processor 102 implements the steps of the transformer-based page layout generation adversarial network method described in Example 1 by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101. The memory 101 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data (such as audio data) created according to the use of the electronic device 100, etc. In addition, the memory 101 can include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices.

[0073] The at least one processor 102 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 102 may be a microprocessor or any conventional processor, etc. The processor 102 is the control center of the electronic device 100, and uses various interfaces and lines to connect various parts of the entire electronic device 100.

[0074] The memory 101 in the electronic device 100 stores a plurality of instructions to implement a transformer-based page layout generation adversarial network method, and the processor 102 may execute the plurality of instructions to implement: Obtain instance graphs of different page layouts, preprocess the instance graphs of the page layouts, and obtain a unified abstract page layout graph; Divide the abstract page layout graphs into training and testing sets; Build a transformer-based TGAN network model for automatic page layout generation, convert the training set of abstract page layout graphs into tensor data, and train the transformer-based TGAN network model for automatic page layout generation; The test set is used to test the trained transformer-based automatic page layout generation TGAN network model to obtain a simulated page layout that meets the requirements; The simulated page layouts are scored, and the effectiveness and robustness of the trained adversarial network are judged based on the scoring results.

[0075] Embodiment 7: If the module / unit integrated in the electronic device 100 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory and read-only memory (ROM, Read-Only Memory).

[0076] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0077] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0078] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0079] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. Transformer-based page layout generation adversarial network method, characterized by: The following steps are involved: Obtain instance graphs of different page layouts, preprocess the instance graphs of the page layouts, and obtain a unified abstract page layout graph; Divide the abstract page layout graphs into training and testing sets; Build a transformer-based TGAN network model for automatic page layout generation, convert the training set of abstract page layout graphs into tensor data, and train the transformer-based TGAN network model for automatic page layout generation; The test set is used to test the trained transformer-based automatic page layout generation TGAN network model to obtain a simulated page layout that meets the requirements; The simulated page layouts are scored, and the effectiveness and robustness of the trained adversarial network are judged based on the scoring results.

2. The transformer-based page layout generation adversarial network method according to claim 1, characterized in that: The specific method of obtaining instance graphs of different page layouts, preprocessing the instance graphs of page layouts, and obtaining a unified abstract page layout graph is as follows: Get instance images of different page layouts; Unify instance images of different page layouts to the target pixel size; Unify the page layout elements in the example diagrams of different page layouts into boxes of different colors; The boxes with uniform target pixel size and different colors are used as abstract page layout diagrams.

3. The transformer-based page layout generation adversarial network method according to claim 1, characterized in that: The working method of the transformer-based page automatic layout generation TGAN network model is as follows: Get the test set and extract the dataset label pair of the test set; The transformer module is used to generate pseudo samples and pseudo label pairs that match the label pairs of the test set dataset; Establish the variable relationship between the dataset label pair and the pseudo sample and pseudo label pair to form a combined conditional variable; Input the dataset label pair into the combined condition variable to get the output result.

4. The transformer-based page layout generation adversarial network method according to claim 3, characterized in that: The transformer module works as follows: The dataset label pairs of the test set are vectorized through embedding processing, and then the corresponding data in the dataset label pairs of the test set are positionally encoded; The dataset is mapped into triples Q, K, and V through the attention mechanism for mapping representation and outputting pseudo samples and pseudo label pairs.

5. The transformer-based page layout generation adversarial network method according to claim 1, characterized in that: The scoring criteria for scoring the simulated page layouts are as follows: Among them, the design layout To simulate the position, metadata, width and height of each object in the page layout, are model parameters, is the energy term, Energy term The weight of is a nonlinear parameter.

6. The transformer-based page layout generation adversarial network method according to claim 1, characterized in that: Energy term It includes alignment energy term, balance energy term, white space energy term, scale energy term, overlap energy term and boundary energy term.

7. Transformer-based page layout generation adversarial network system, characterized by: include: A preprocessing module is used to obtain instance graphs of different page layouts, preprocess the instance graphs of the page layouts, and obtain a unified abstract page layout graph; A data differentiation module, used to divide the abstract page layout graph into a training set and a test set; The model training module is used to build a transformer-based TGAN network model for automatic page layout generation, convert the training set of the abstract page layout graph into tensor data, and train the transformer-based TGAN network model for automatic page layout generation; The model testing module is used to test the trained transformer-based automatic page layout generation TGAN network model using a test set to obtain a simulated page layout that meets the requirements; The model scoring module is used to score the simulated page layout and judge the effectiveness and robustness of the trained adversarial network based on the scoring results.

8. The transformer-based page layout generation adversarial network system according to claim 7, characterized in that: The functions of the preprocessing module are implemented in the following ways: Get instance images of different page layouts; Unify instance images of different page layouts to the target pixel size; Unify the page layout elements in the example diagrams of different page layouts into boxes of different colors; The boxes with uniform target pixel size and different colors are used as abstract page layout diagrams.

9. The transformer-based page layout generation adversarial network system according to claim 7, characterized in that: The function of generating the TGAN network model based on the transformer-based automatic page layout in the model training module is implemented by the following method: Get the test set and extract the dataset label pair of the test set; The transformer module is used to generate pseudo samples and pseudo label pairs that match the label pairs of the test set dataset; Establish the variable relationship between the dataset label pair and the pseudo sample and pseudo label pair to form a combined conditional variable; Input the dataset label pair into the combined condition variable to get the output result.

10. The transformer-based page layout generative adversarial network system according to claim 7, characterized in that: The functions of the model scoring module are implemented through the following methods: Among them, the design layout To simulate the position, metadata, width and height of each object in the page layout, are model parameters, is the energy term, Energy term The weight of is a nonlinear parameter.

11. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the transformer-based page layout generation adversarial network method described in any one of claims 1 to 6 are implemented.

12. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the transformer-based page layout generation adversarial network method described in any one of claims 1 to 6 are implemented.

Images