Method, device, equipment and medium for unified multi-task and multi-field layout generation

Abstract

The application discloses a method, device and equipment for unified multi-task and multi-field page generation and a medium, wherein the method comprises the following steps: acquiring page generation data of multiple fields; flattening information of all elements of each page into a sequence, inputting a random mask of the sequence as an input, and inputting a complete sequence as a label; inputting the sequence and the label into a large language model for training, and mixing page data of different fields for use in the training; and inputting part of page information generated according to different task requirements and different field requirements into the trained model, so that the model generates a complete page sequence. The application uses deep learning and sequence generation technology based on a large language model to unify multi-page generation tasks and page generation of multiple fields, and realizes a general and good page generation engine. The application can be widely applied to the fields of deep learning and pattern recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of deep learning and pattern recognition, and in particular to a method, device, equipment and medium for unified multi-task and multi-domain layout generation. BACKGROUND

[0002] Layout generation refers to automatically generating a structured document layout with reasonable layout according to user requirements. Specifically, a complete layout is composed of multiple layout elements, and each layout element contains five pieces of information: element category (text, table, picture, etc.), x and y coordinates of the top left corner of the box, and width and height of the box. The user can give part of the information of the layout element, such as only giving the element category or the width and height of the box, and the program generates a complete document layout according to these preset and incomplete information. This technology is widely used in various practical scenarios, such as webpage and UI design (quickly generating webpage layout or user interface design according to natural language description), advertising poster generation (automatically generating marketing posters, advertising copy layout, etc.), education and publishing (automatically generating test papers, teaching materials, book catalogues, etc.), and the like.

[0003] Most of the existing layout generation methods only focus on single-task layout generation, such as inputting the category of all elements and letting the model automatically infer the x, y coordinates and width and height of the element. In recent years, layout generation models that take into account multiple tasks have received more and more attention, showing better versatility and flexibility. However, these multi-task models only focus on single-domain layout generation, such as generating only paper layouts or magazine layouts. SUMMARY

[0004] To at least partially solve one of the technical problems existing in the prior art, the purpose of the present application is to provide a method, device, equipment and medium for unified multi-task and multi-domain layout generation based on a large language model.

[0005] The first technical solution adopted by the present application is:

[0006] A method for unified multi-task and multi-domain layout generation, comprising the following steps:

[0007] Obtaining layout generation data in multiple domains;

[0008] Flattening the information of all elements of each layout into a sequence, and inputting the sequence with random masks as input and the complete sequence as label;

[0009] Inputting the sequence and the label into a large language model for training, and mixing layout data in different domains for use in training;

[0010] Input the partial layout information generated according to different task requirements and different field requirements into the trained model to make the model generate a complete layout sequence.

[0011] Further, the layout generation data includes paper layout, mobile phone App UI layout, magazine layout and slide layout.

[0012] Each complete layout is composed of N layout elements, and the format of each layout element is (c, x, y, w, h), wherein c is the category of the current element, such as text, table, image, etc., (x, y) is the top-left corner coordinate of the current element in the entire layout canvas, w is the width of the current element, and h is the height of the current element; the entire layout is represented as: {x1, y1, w1, h1, …, xN, yN, wN, hN}. N N N N

[0013] Further, the information of all elements of each layout is flattened into a sequence, and the sequence is randomly masked as input, and the complete sequence is taken as a label, including:

[0014] An arbitrary layout prompt template is constructed, and after being masked or noise-added, the template is taken as the input of the model;

[0015] A unified layout answer template is constructed, and the sequence of complete and noise-free layout element information is taken as the unified layout answer.

[0016] Interval quantization position coding is used to avoid using placeholders.

[0017] Further, the arbitrary layout prompt template is composed of two parts: a prefix part and a main part.

[0018] The prefix part is composed of a "denoising flag", a "layout type", an "element number" and a "column number", and the main part is composed of a plurality of layout elements and a description of the relationship between different elements, and the format of each element is (c, x, y, w, h).

[0019] The information of each element is randomly masked or noise-added to simulate any layout generation condition; the masked information is directly discarded, and the noise-added means that random noise is added to x, y, w and h, and the model is required to denoise the noise after noise addition; the unmasked layout element information is directly spliced together as the input of the model, so that the model generates complete layout element information through autoregressive manner.

[0020] Further, the interval quantization position coding is used to avoid using placeholders, including:

[0021] ​​​​Set an interval value l large enough to be greater than the length and width of all page canvases; then encode the x, y, w and h of each page element according to the following rules:

[0022] x = x + 0 x l

[0023] y = y + 1 x l

[0024] w = w + 2 x l

[0025] h = h + 3 x l

[0026] The encoding rules make x, y, w, h each in a different numerical interval, that is, x∈[0, l), y∈[l, 2l), w∈[2l, 3l), h∈[3l, 4l).

[0027] Further, the sequence and the label are input to the large language model for training, and page data in different fields are mixed for training, including:

[0028] The GPT2-XL model is used as the large language model, and the model is trained:

[0029] Any page prompt template is used as the input of the GPT2-XL model, a unified page answer template is used as the label, and the model is required to generate a complete specified field page for any task and any field generation requirement.

[0030] Further, the following steps are included:

[0031] The generated page is compared with the real page to calculate the index, or the generated page sequence is rendered into a two-dimensional page picture.

[0032] The second technical solution adopted by the application is:

[0033] An apparatus for unified multi-task and multi-field page generation, comprising:

[0034] A data acquisition module for acquiring page generation data in multiple fields;

[0035] An input-output construction module for flattening the information of all elements of each page into a sequence, randomly masking the sequence as input, and taking the complete sequence as a label;

[0036] A model training module for inputting the sequence and the label to a large language model for training, and mixing page data in different fields for training;

[0037] A model inference module for inputting part of the page information generated according to different task requirements and different field requirements into the trained model to make the model generate a complete page sequence.

[0038] The third technical solution adopted by the present application is:

[0039] An electronic device, comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the method for unified multi-task and multi-field layout generation as described above.

[0040] The fourth technical solution adopted by the present application is:

[0041] A computer-readable storage medium, the storage medium storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by a processor to implement the method for unified multi-task and multi-field layout generation as described above.

[0042] The fifth technical solution adopted by the present application is:

[0043] A computer program product or computer program, comprising computer instructions stored in a computer-readable storage medium. The processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to make the computer device execute the method for unified multi-task and multi-field layout generation as described above.

[0044] The beneficial effects of the present application are: the present application uses deep learning and sequence generation technology based on large language model, unifies the tasks of multi-page generation and the page generation of multiple fields, and realizes a general and good performance page generation engine. In addition, the performance of the model is enhanced by compressing the information of the prompt, which can even surpass other single-scene page generation methods in more difficult unified scenes. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following introduces the drawings of the related technical solutions in the embodiments of the present application or the prior art. It should be understood that the drawings in the following introduction are only for the convenience of clearly describing part of the embodiments of the technical solutions of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0046] Figure 1 is a step flow chart of a method for unified multi-task and multi-field layout generation in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The embodiments of the present application are described below in detail, examples of which are shown in the drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary only, for the purpose of explanation, and are not to be understood as a limitation of the present application. For the step numbers in the following embodiments, they are only set for the convenience of explanation, and no limitation is made on the order between the steps, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0048] In the description of the present application, it should be understood that the orientation description, such as the orientation or position relationship indicated by up, down, front, back, left, right, etc. is based on the orientation or position relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as a limitation of the present application, which does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application.

[0049] In the description of the present application, the meaning of several is one or more, and the meaning of multiple is more than two, greater than, less than, more than, etc. are understood as not including the number, and above, below, etc. are understood as including the number. If it is described as first, second, it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated or the order of technical features indicated.

[0050] In the description of the present application, unless otherwise explicitly limited, the words such as setting, installing, connecting, etc. should be understood in a broad sense, and those skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical solution.

[0051] In order to realize more general and comprehensive page generation, the present application proposes a method of unifying multi-task and multi-domain page generation. Due to the greater difficulty brought by the unification of multi-domain and multi-task, it is difficult to obtain good generation effect with existing small models, therefore the present application proposes to realize this method based on a large language model. This method accommodates the conditions of any page generation task and the requirements of multi-domain page generation by proposing an arbitrary page prompt template, and uses interval quantization position coding to compress the length of the input to improve generation efficiency. The method of the present application uses a large language model as a generation engine, which can achieve performance beyond existing methods in the complex multi-task and multi-domain unified scene.

[0052] Example 1

[0053] As Figure 1As shown, the embodiment provides a method for unified multi-task and multi-field layout generation based on a large language model, including the following steps:

[0054] S1, obtain layout generation data of multiple fields.

[0055] As an implementation manner, the embodiment considers four fields of layout data, including paper layout, mobile phone App UI layout, magazine layout, and slide layout. It should be noted that the present application is not limited to these four types of layout data, and can include any type of layout data. Each complete layout is composed of N layout elements, and the format of each layout element is (c, x, y, w, h), where c is the category of the current element, such as text, table, image, etc., (x, y) is the top-left corner coordinate of the current element in the entire layout canvas, w is the width of the current element, and h is the height of the current element. The entire layout can be represented as {x1, y1, w1, h1, …, xN, yN, wN, hN}. N N N N

[0056] S2, flatten the information of all elements of each layout into a sequence, and input the random mask of the sequence as the label.

[0057] In some embodiments, step S2 specifically includes steps S21-S23:

[0058] S21, construct an arbitrary layout prompt template.

[0059] The embodiment proposes an arbitrary layout prompt template as the input of the model, which consists of two parts: a prefix part and a main part. The prefix part consists of a "denoising flag", a "layout type", an "element number", and a "column number", where the "layout type" is "paper", "App UI", "magazine", and "slide". The "element number" is N in step S1. The main part consists of multiple layout elements and descriptions of the relationships between different elements, and the format of each element is (c, x, y, w, h) as described above.

[0060] The embodiment randomly masks or adds noise to the information of each element to simulate arbitrary layout generation conditions. The masked information is discarded directly, and the noise added to x, y, w, and h requires the model to denoise the added noise. The unmasked layout element information is directly spliced together as input. For example, assuming that a complete layout sequence is "denoising; paper; N; 2; text x1, y1, w1, h1; …; table x2, y2, w2, h2; …; image xN, yN, wN, hN", the input of the model is "denoising; paper; N; 2; text x1, y1, w1, h1; …; table x2, y2, w2, h2; …; image xN, yN, wN, hN". N N N ​​​​​​h1", after masking and adding noise, the possible input is "denoising; thesis; N; 2; text w1; h2; …; table y N By changing the "layout type", the model can learn to generate a specific type of layout; by inputting any layout generation condition, the model can learn to generate a layout under any condition, and the combination of the two can train the model to have the ability of unified multi-domain and multi-task layout generation.

[0061] S22, construct a unified layout answer template.

[0062] As described in step S21, we construct a sequence of complete, noise-free layout element information as a unified layout answer, that is, regardless of the given input, the model is required to output a complete layout, such as "text x1, y1, w1, h1; …; table x N ,y N ,w N ,h N ". Combining any layout prompt template with the unified layout answer template can enable the model to output a complete layout under any task and any specified domain, achieving the ability of unified layout generation.

[0063] S23, interval quantization position coding.

[0064] In the description of step S21, the embodiment of the present application will directly discard the masked information, splice the remaining information that is not masked together, and then let the model generate complete layout element information through autoregressive. However, this will face a problem, that is, the model may not be able to infer which one of x, y, w, and h the unmasked value is. The conventional solution is to replace the position of a masked information with a placeholder, which is equivalent to telling the model that there is a vacant position that needs to be predicted. However, this approach will introduce a large number of placeholders, reducing the density of layout information in the input instruction and affecting the performance of generation. Therefore, the embodiment of the present application proposes a method that can avoid confusion of unmasked values by the model and avoid the use of placeholders, that is, interval quantization position coding. Specifically, the embodiment sets a large enough interval value l, which is greater than the length and width of all layout canvases. Then, the x, y, w, and h of each layout element are encoded according to the following rules:

[0065] x=x+0×l

[0066] y=y+1×l

[0067] w=w+2×l

[0068] h=h+3×l

[0069] The encoding rule makes x, y, w, h each in a different numerical interval, that is, x∈[0, l), y∈[l, 2l), w∈[2l, 3l), h∈[3l, 4l). The model can accurately determine what the unmasked elements are and what the information to be predicted is through the size of the numerical value, so that the placeholder does not need to be used to replace the masked information. The interval quantization position encoding can only keep the effective layout information in the input, greatly improve the density of the layout information, and improve the performance of the model; at the same time, it can shorten the length of the input sequence, improve the efficiency of the input and output, and further improve the training and inference efficiency of the model.

[0070] S3, input the sequence and label into the large language model for training, and the layout data in different fields are mixed in the training.

[0071] The existing large language model is used as a sequence generation model. In recent years, large language models are mainly developed based on a pure decoder architecture, such as GPT2, LLaMA, etc., which has shown quite amazing text generation capabilities. The embodiment of the present application adopts the GPT2-XL model as the generation engine, and the GPT2-XL is a pure decoder Transformer with 1.5 billion parameters, including 48 Transformer decoder layers, a hidden layer dimension of 1600, and 25 attention heads. The autoregressive paradigm is used for text generation. In this embodiment, any layout prompt template is used as the input of GPT2-XL, and a unified layout answer template is used as the label, and the model is required to generate a complete specified field layout for any task and any field generation requirement.

[0072] Since it is more difficult to simultaneously unify multi-task and multi-field layout generation than to only unify one of the scenes or single-task, single-field layout generation, we hope to use the strong reasoning ability of the large language model to solve this problem, and experiments have proved that using a large language model can obtain better performance than using a small language model.

[0073] As an implementation manner, the training parameters are as follows:

[0074] 1) Iteration steps: 23000

[0075] 2) Optimizer: AdamW

[0076] 3) Learning rate: 0.0001, the learning rate is reduced to 0 at the last iteration step by using the cosine annealing strategy.

[0077] S4, input part of the layout information generated according to different task requirements and different field requirements into the trained model to make the model generate a complete layout sequence.

[0078] Model inference: input the partial layout information generated according to different task requirements and different field requirements into the model, let the model generate a complete layout sequence, and compare the generated layout with the real layout to calculate the index, or render the layout sequence into a two-dimensional layout picture.

[0079] Specifically, according to a given layout generation task and a given layout data type, the corresponding input can be generated according to any layout prompt template. For example, the generation task is to give only 10 types of layout elements c, generate the x, y, w, h of these elements, and the generated layout data type is a paper, then the input is “non-noise; paper; 10; 2; text; table; …; image”. After inputting the model, a sequence is generated, which should be “text x1, y1, w1, h1; table x2, y2, w2, h2; …; image x N ,y N ,w n ,h N ”. The generated sequence can be compared with the real sequence to calculate the index, or the generated sequence can be rendered on a blank canvas according to the coordinates and width and height to visualize each layout element.

[0080] In summary, the present application discloses a layout generation method, which includes an arbitrary layout prompt template, a unified layout answer template and an interval quantization position coding. The arbitrary layout prompt template includes any combination of layout generation requirements, can support any layout generation task, and can specify the type of layout data to be generated (such as magazines, papers, etc.) through specific type indicators, support layout generation in multiple fields, and therefore can support layout generation in multiple tasks and multiple fields. The interval quantization position coding encodes the position information (x coordinate, y coordinate, width and height) of the layout elements into independent numerical intervals, maintains the discriminability between different position information without using placeholders, shortens the length of the input information and improves the information density, and enhances the performance and input / output efficiency of the model. The present application inputs the layout information according to the arbitrary layout prompt template into the large language model, generates a complete layout as output, and realizes unified layout generation in multiple fields and multiple tasks. The present application first proposes a method for unified layout generation in multiple tasks and multiple fields, and enhances the performance of the model by compressing the prompt information, so that it can surpass other layout generation methods in single scene even in more difficult unified scene.

[0081] Compared with the prior art, the method of the present application has at least the following advantages and beneficial effects:

[0082] (1) The method proposed in the present application is the first method to unify multi-task and multi-domain layout generation, while previous methods only focus on the unification of multi-task or multi-domain, and do not unify the two scenarios. At the same time, even in the case of higher difficulty brought by the two unifications, the present method can still achieve better performance than existing methods that only unify multi-task or multi-domain.

[0083] (2) The present application proposes an arbitrary layout prompt template and a unified layout answer template. The arbitrary layout prompt template consists of a prefix part and a main part. The prefix part can specify the type of layout data to be generated, so it can accommodate the requirements of multi-domain layout generation; the main part can simulate any layout generation requirement by randomly masking the information (i.e. c, x, y, w, h) in the layout elements, and each requirement can be regarded as a task, so this can cover any layout generation task. The two parts work together as the input of the model, and by requiring the model to output a complete layout (i.e. the unified layout answer template), the multi-task and multi-domain unified layout generation capability can be trained.

[0084] (3) The present application proposes an interval quantization encoding method, which encodes the values of x, y, w, h into non-overlapping intervals, so that the model can accurately determine what the unmasked elements in the input sequence are and what information needs to be predicted by the size of the values, avoiding the use of placeholders to replace the masked information. In this way, on the one hand, only valid layout information is retained without the need for placeholders, increasing the density of layout information in the input and enhancing the generation performance of the model, on the other hand, the length of the input sequence can be shortened, improving the training and inference efficiency of the model.

[0085] (4) The present application uses a large language model as the generation engine, which is one of the few explorations of using a large language model for layout generation. Since simultaneously unifying multi-task and multi-domain layout generation is more difficult than only unifying one of the two scenarios or single-task and single-domain layout generation, the present application makes good use of the strong reasoning ability of the large language model itself to solve this difficult problem, and achieves better performance than other methods.

[0086] Embodiment 2

[0087] The present embodiment provides a device for unifying multi-task and multi-domain layout generation, comprising:

[0088] a data acquisition module for acquiring multi-domain layout generation data;

[0089] an input-output construction module for flattening the information of all elements of each layout into a sequence, randomly masking the sequence as input, and taking the complete sequence as a label;

[0090] The model training module is configured to input the sequence and the label into the large language model for training, and the layout data in different fields are mixed for use in the training.

[0091] The model inference module is configured to input part of the layout information generated according to different task requirements and different field requirements into the trained model, so that the model generates a complete layout sequence.

[0092] Since the device is a device for unified multi-task and multi-field layout generation according to the embodiment of the application, and the principle of solving the problem of the device is similar to that of the method, the implementation of the device can be referred to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be described again.

[0093] Embodiment 3

[0094] The embodiment of the application further provides an electronic device, which comprises a processor and a memory, and the memory stores at least one instruction, at least one program, a code set or an instruction set. Figure 1 The processor loads and executes the at least one instruction, the at least one program, the code set or the instruction set to implement a method for unified multi-task and multi-field layout generation as shown in

[0095] It can be understood that the memory can include a random access memory (RAM) and can also include a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, codes, code sets or instruction sets. The memory can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for at least one function, instructions for implementing various method embodiments, etc.; and the data storage area can store data created according to the use of the server, etc.

[0096] The processor can include one or more processing cores. The processor connects various parts within the entire server by various interfaces and lines, executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Alternatively, the processor can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor can be integrated with one or several combinations of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes operating systems and application programs; the modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor, but be implemented by a single chip.

[0097] Since the electronic device is an electronic device corresponding to the method for generating a unified multitask and multi-field page according to the embodiments of the present application, and the principle of solving problems of the electronic device is similar to that of the method, the implementation of the electronic device can be referred to the implementation process of the above-mentioned method embodiments, and the repeated parts will not be described again.

[0098] Embodiment 4

[0099] The embodiments of the present application also provide a computer readable storage medium, wherein at least one instruction, at least one program, a code set or an instruction set are stored in the storage medium, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by a processor to implement a method for generating a unified multitask and multi-field page as shown in Figure 1

[0100] ​Those skilled in the art can understand that all or part of the steps of various methods in the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer readable storage medium, including Read-Only Memory (ROM), Random Access Memory (RAM), Programmable Read-only Memory (PROM), Erasable Programmable Read Only Memory (EPROM), One-time Programmable Read-Only Memory (OTPROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, magnetic disk storage, magnetic tape storage, or any other medium that can be used to carry or store data which can be read by a computer.

[0101] Since the storage medium is a storage medium corresponding to the method for generating a unified multi-task and multi-field layout according to an embodiment of the application, and the principle of solving the problem is similar to the method, the implementation of the storage medium can refer to the implementation process of the above method embodiment, and the repeated parts will not be described again.

[0102] Embodiment 5

[0103] In some possible implementation manners, various aspects of the method according to an embodiment of the application can also be implemented in the form of a program product, which includes program codes for causing a computer device to execute the steps of the method for generating a unified multi-task and multi-field layout according to various exemplary embodiments of the present application described in the specification when the program product is run on the computer device. Wherein, the executable computer program codes or "codes" for executing various embodiments can be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (for example, Transact-SQL), Perl, or in various other programming languages.

[0104] It should be understood that various aspects of the application can be implemented in hardware, software, firmware or a combination of them. In the above embodiments, various steps or methods can be implemented in software or firmware which is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, or their combination, can be used: discrete logic circuitry having logic gates for implementing logic functions upon an application of data signals, application-specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field-programmable gate arrays (FPGA), and so on.

[0105] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. Furthermore, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples, without contradiction.

[0106] The above embodiments are only for the purpose of illustrating the technical concept and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and to implement it, and cannot limit the protection scope of the present application. Any equivalent changes or modifications made according to the essence of the present application should be covered within the protection scope of the present application.

Claims

1. A method for unified multi-task and multi-domain layout generation, characterized in that, The method comprises the following steps: Obtaining layout generation data in multiple fields; Flattening information of all elements of each layout into a sequence, performing random masking on the sequence as input, and taking the complete sequence as a label; Inputting the sequence and the label into a large language model for training, and mixing layout data in different fields during training; Inputting part of the layout information generated according to different task requirements and different field requirements into the trained model to enable the model to generate a complete layout sequence; The sequence is flattened into a sequence, and the sequence is randomly masked as input, Taking the complete sequence as a label, comprising: Constructing an arbitrary layout prompt template and performing masking or noise adding processing, and taking the template as input of the model; Constructing a unified layout answer template, and taking the sequence composed of complete and noise-free layout element information as a unified layout answer; Avoiding the use of placeholders by interval quantization position coding; The interval quantization position coding avoids the use of placeholders, comprising: Setting an interval value l, which is greater than the length and width of all layout canvases; then encoding x, y, w and h of each layout element according to the following rules: x = x + 0 x l y = y + 1 x l w = w + 2 x l h = h + 3 x l The encoding rules make x, y, w and h each in a different numerical interval, i.e. x ∈ [0, l), y ∈ [l, 2l), w ∈ [2l, 3l), h ∈ [3l, 4l).

2. The method for unified multitasking and multi-domain layout generation of claim 1, wherein, The layout generation data comprises paper layout, mobile phone App UI layout, magazine layout and slide layout; Each complete page is composed of N page elements, and the format of each page element is (c, x, y, w, h), wherein c is the category of the current element, (x, y) is the upper left corner coordinate of the current element in the entire page canvas, w is the width of the current element, and h is the height of the current element; the entire page is represented as: {x1, y1, w1, h1, …, xN, yN, wN, hN}. N N N N}​​​ 3. The method of claim 1, wherein the method further comprises: The arbitrary layout prompt template comprises two parts: a prefix part and a main part; The prefix part comprises a "denoising flag", a "layout type", a "number of elements" and a "number of columns"; the main part comprises a plurality of layout elements and a description of the relationship between different elements, and the format of each element is (c, x, y, w, h); Randomly masking or adding noise to the information of each element to simulate any layout generation condition; the masked information is directly discarded, and the noise added to x, y, w and h requires the model to denoise the added noise; the unmasked layout element information is directly spliced together as input of the model, so that the model generates complete layout element information in an autoregressive manner.

4. The method of claim 1, wherein the method further comprises: The sequence and the label are input into a large language model for training, and layout data in different fields are mixed during training, comprising: Using a GPT2-XL model as the large language model to train the model: Taking an arbitrary layout prompt template as input of the GPT2-XL model, and taking a unified layout answer template as a label, the model is required to generate a complete specified field layout for any task and any field generation requirement.

5. The method for unified multitasking and multi-domain layout generation of claim 1, wherein, Further comprising the following steps: Comparing the generated layout with the real layout to calculate an index, or rendering the generated layout sequence into a two-dimensional layout picture.

6. A device for unified multi-tasking and multi-domain layout generation, characterized in that, Comprising: A data acquisition module for acquiring layout generation data in multiple fields; An input-output construction module is configured to flatten information of all elements of each page into a sequence, perform random masking on the sequence as input, and take the complete sequence as a label; A model training module is configured to input the sequence and the label into a large language model for training, and mix page data of different fields in the training; A model inference module is configured to input part of page information generated according to different task requirements and different field requirements into the trained model, so that the model generates a complete page sequence; The flattening of information of all elements of each page into a sequence, the performing of random masking on the sequence as input, and the taking of the complete sequence as a label, The taking of the complete sequence as a label includes: Constructing an arbitrary page prompt template and performing masking or noise processing to serve as input of the model; Constructing a unified page answer template, and taking a sequence of complete and noise-free page element information as a unified page answer; Interval quantization position coding is used to avoid using placeholders; The interval quantization position coding to avoid using placeholders includes: An interval value l is set, which is greater than the length and width of all page canvases; then x, y, w and h of each page element are encoded according to the following rules: x=x+0×l y=y+1×l w=w+2×l h=h+3×l The encoding rules make x, y, w and h each in a different numerical interval, i.e., x∈[0,l), y∈[l,2l), w∈[2l,3l), h∈[3l,4l).

7. An electronic device, comprising: The electronic device includes a processor and a memory, and the memory stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the method of any one of claims 1 to 5.

Images