A training method for a poster typesetting model, a training model and a poster typesetting method

Abstract

The present invention provides a method for training a poster typesetting model, including: obtaining a finished poster data set and a poster foreground text typesetting data set; obtaining a text-image patching model and a pre-trained optical character recognition model of a pre-trained latent variable diffusion model; inputting the poster foreground text typesetting data set based on the text-image patching model of the pre-trained latent variable diffusion model to obtain a poster background image; inputting the poster foreground typesetting data set into the pre-trained optical character recognition model to obtain a processed poster foreground text typesetting data set; establishing a poster automatic typesetting model, and training the poster automatic typesetting model based on the poster background image and the processed poster foreground typesetting data set to obtain a trained poster automatic typesetting model L. The present invention also includes a corresponding training model and a poster automatic typesetting method. The scheme provided by the present invention is flexible and adjustable, and can annotate a large amount of poster data.

Description

Technical Field

[0001] The invention belongs to the field of artificial intelligence, and in particular relates to a training method for an automatic poster typesetting model, a training model and an automatic poster typesetting method. Background Art

[0002] Posters are currently very important information transmission media in the fields of marketing, advertising, and self-media, which involve the design of the background, the drawing of foreground elements, and the layout of foreground text. On the one hand, one or more professional design tools are needed in the production of posters and elements in posters. Relying on the automatic typesetting model of artificial intelligence algorithms, it is first necessary to collect a large number of posters and complete the annotation of elements in each poster. On the other hand, since the visual presentation effect of the poster depends largely on the design experience of professional designers, in addition to the design of relatively independent elements such as pictures and texts, the staggered layout of elements also plays an important role.

[0003] The main defects of the existing technology are: (1) Poster annotation includes the separation of background layer and foreground layer, the position annotation of foreground elements, the font annotation of foreground text, the content annotation of foreground text, etc. For a poster that may have several or even dozens of foreground elements, the cost of annotating a poster is much higher than annotating a general picture; (2) There are currently few public algorithm models for automatic poster layout. Among the few works that are available, most of them are also aimed at the layout of foreground elements. The layout of foreground elements of posters must not only consider the relationship between elements, but also the relationship between elements and the overall composition of the background picture. Another problem with the existing algorithm work is that the automatic layout of posters is achieved under the premise of ignoring a large amount of foreground element information. In particular, for the layout of text, it is necessary to consider information such as the font and content of the text in order to more reasonably achieve local layout.

[0004] Based on the above, this application provides a technical solution to solve the above technical problems. Summary of the invention

[0005] In view of the difficulty in labeling poster data and the lack of information for automatic poster layout in the prior art, the present invention provides a method for training a poster layout model, comprising the following steps:

[0006] Step S1, obtaining a finished poster dataset and a poster foreground text layout dataset;

[0007] Step S2, obtaining a text-image inpainting model and a pre-trained optical character recognition model of a pre-trained latent variable diffusion model;

[0008] Step S3, inputting the poster foreground text typesetting dataset into the text-image inpainting model based on the pre-trained latent variable diffusion model to obtain the poster background image;

[0009] Step S4, inputting the poster foreground text typeset data set into the pre-trained optical character recognition model to obtain a processed poster foreground text typeset data set;

[0010] Step S5, establishing a poster automatic typesetting model L1, and training the poster automatic typesetting model L1 based on the poster background image and the processed poster foreground typesetting data set to obtain a trained poster automatic typesetting model L.

[0011] In a preferred embodiment of the present invention, step S3 further includes:

[0012] Step S3.1, converting the poster foreground text layout into a foreground mask;

[0013] Step S3.2: multiply the finished poster by the foreground mask to obtain the finished poster filtered by the mask

[0014] Step S3.3, inputting the finished poster after mask filtering into the text-image inpainting model of the pre-trained latent variable diffusion model;

[0015] Step S3.4: input predetermined text and generate a poster background image.

[0016] In a preferred embodiment of the present invention, step S4 further includes:

[0017] Step S4.1, select the foreground text of the poster;

[0018] Step S4.2, appropriately scaling the selected text portion image so that it can be correctly recognized by the pre-trained optical character recognition model;

[0019] Step S4.3, inputting the scaled poster foreground text into a pre-trained optical character recognition model;

[0020] Step S4.4, the pre-trained optical character recognition model outputs a poster foreground text layout dataset with font labels and text content.

[0021] In a preferred embodiment of the present invention, step S4.2 further includes a pixel restoration step and a confidence threshold screening step:

[0022] The pixel restoration step includes: if the resolution of the characters in the poster is greater than a predetermined maximum threshold, the characters are reduced to an average resolution of the pre-trained optical character recognition model during pre-training; if the resolution of the characters in the poster is less than a predetermined minimum threshold, the characters are enlarged to a minimum resolution of the pre-trained optical character recognition model during pre-training through super-resolution;

[0023] The confidence threshold screening step includes: if the text confidence is greater than the confidence threshold, taking the label with the largest confidence as the font label of the current text; if the maximum confidence of the text is still less than the confidence threshold, completing the font label marking by manual annotation.

[0024] In a preferred embodiment of the present invention, after step S4, the method further includes:

[0025] Step A: Obtain a pre-trained text encoder, encode the text content of the poster text typesetting dataset, and obtain a text content feature vector.

[0026] In a preferred embodiment of the present invention, step S5 further includes:

[0027] The poster automatic typesetting model L is based on the Transformer structure.

[0028] Step S5.1, input the poster background image and the text content feature vector into the poster automatic typesetting model L, and output the typesetting boxes and font labels corresponding to all foreground texts;

[0029] Step S5.2, introduce the cross-attention and self-attention mechanisms, calculate the correlation between the poster background image and the input foreground text content feature vector and the poster background image itself, and typeset and render the poster foreground text according to the output of the poster automatic typesetting model L.

[0030] In a preferred embodiment of the present invention, in step S5.1, the output content includes coordinates (x, y), width and height (w, h), and font category label;

[0031] Step S5.2 also includes: applying Hungarian matching loss l to the poster automatic typesetting model L Hungarian , constraint box loss l bbox , classification loss l class , according to the coordinates, width, height and font category label c, the poster foreground text is typeset and rendered on the poster background image.

[0032] In a preferred embodiment of the present invention, it also includes:

[0033] Hungarian matching loss l Hungarian The calculation formula is as follows:

[0034]

[0035] Constraint box loss l bbox The calculation formula is as follows:

[0036]

[0037] Classification loss class The calculation formula is as follows:

[0038]

[0039] in,

[0040] N is the number of predicted poster foreground text box instances; M is the number of poster text fonts; b i The coordinates, width and height of the real constraint box of the i-th text box; is the index sequence between the real text element and the predicted text box; For the The poster text box coordinate width and height of the index; p ic is the true category of the i-th text box of the symbol function. If the true category is c, it takes 1, otherwise it takes 0; The predicted index is The predicted probability that the font category of the text box belongs to c; Represents a real text box instance; l Hungarian represents the Hungarian matching loss; y i represents the i-th text box in the model's current predicted text box instance; c i Represents the category label of the i-th text box in the model's current predicted text box instance.

[0041] The present invention also provides a training model for a poster typesetting model, comprising the following modules:

[0042] A dataset acquisition module is used to acquire a finished poster dataset and a poster foreground text layout dataset module;

[0043] Get model modules for pre-training text-image inpainting models of latent variable diffusion models and pre-training optical character recognition models;

[0044] A poster foreground acquisition module, based on the text-image inpainting model of the pre-trained latent variable diffusion model, inputs the poster foreground text typesetting dataset to obtain a poster background image;

[0045] A poster marking module, inputting the poster foreground typesetting data set into the pre-trained optical character recognition model to obtain a processed poster foreground text typesetting data set;

[0046] The poster automatic typesetting model training module establishes the poster automatic typesetting model L1, and trains the poster automatic typesetting model L1 based on the poster background image and the processed poster foreground typesetting data set to obtain the trained poster automatic typesetting model L.

[0047] The present invention also provides a poster typesetting method, which adopts a poster automatic typesetting model L, inputs the data to be typeset into the poster automatic typesetting model L, and outputs a poster with a completed typeset, wherein the poster automatic typesetting model L is trained by the aforementioned poster automatic typesetting model training method.

[0048] The present invention can bring at least one of the following beneficial effects:

[0049] (1) The present invention achieves low-cost poster reverse structuring by pre-training a text-image inpainting model of a latent variable diffusion model and a pre-training optical character recognition model, thereby obtaining a large amount of deconstructed poster data;

[0050] (2) The poster automatic typesetting model contains a structure based on cross-attention and self-attention mechanisms, which can add information of any modality during the training process and is flexible and adjustable. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The preferred implementation scheme will be described below in a clear and understandable manner with reference to the accompanying drawings to further illustrate the above-mentioned characteristics, technical features, advantages and their implementation methods.

[0052] Figure 1 A schematic diagram of the steps of a method for training a poster automatic typesetting model according to a first embodiment of the present invention;

[0053] Figure 2 Schematic diagram of step S3 of the first embodiment of the present invention;

[0054] Figure 3 Schematic diagram of step S4 of embodiment 1 of the present invention;

[0055] Figure 4 This is a structural diagram of a training model for an automatic poster typesetting model according to Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0056] Various aspects of the present invention are described in further detail below.

[0057] Unless otherwise defined or indicated, all professional and scientific terms used herein have the same meanings as those familiar to those skilled in the art. In addition, any methods and materials similar or equivalent to those described herein can be applied to the present invention.

[0058] The following is an explanation of the terms.

[0059] Unless otherwise clearly specified and limited, the "or" mentioned in the present invention includes the relationship of "and". The "and" is equivalent to the Boolean logic operator "AND", the "or" is equivalent to the Boolean logic operator "OR", and "AND" is a subset of "OR".

[0060] It is to be understood that although the terms "first", "second", etc. may be used herein to describe different elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. Therefore, the first element may be referred to as the second element without departing from the teachings of the present invention.

[0061] In the present invention, the terms "comprising", "including" or "comprising" indicate that various components can be applied together in the mixture or composition of the present invention. Therefore, the term "consisting mainly of..." is included in the terms "comprising", "including" or "comprising".

[0062] Unless otherwise clearly specified and limited, the terms "connected", "connected" and "connection" in the present invention should be understood in a broad sense, for example, it can be a fixed connection, or it can be connected through an intermediary medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0063] For example, if an element (or component) is referred to as being on, coupled to, or connected to another element, the element may be directly formed on, coupled to, or connected to the other element, or there may be one or more intervening elements between them. In contrast, if the expressions "directly on," "directly coupled to," and "directly connected to" are used herein, then no intervening elements are indicated. Other words used to describe the relationship between elements should be similarly interpreted, such as "between" and "directly between," "attached" and "directly attached," "adjacent" and "directly adjacent," and the like.

[0064] It should also be noted that the words "front", "rear", "left", "right", "upper" and "lower" used in the following description refer to directions in the drawings. The words "inner" and "outer" used refer to directions toward or away from the geometric center of a particular component, respectively. It can be understood that these terms are used here to describe the relationship of one element, layer or region relative to another element, layer or region as shown in the drawings. In addition to the orientations described in the drawings, these terms should also include other orientations of the device.

[0065] Other aspects of the present invention will be apparent to those skilled in the art in view of the disclosure herein.

[0066] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the specific implementation methods of the present invention will be described below with reference to the accompanying drawings. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings and other implementation methods can be obtained based on these drawings without creative work.

[0067] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. The drawings only show the components related to the present application and are not drawn according to the number, shape and size of the components in actual implementation. The type, quantity and proportion of each component in actual implementation may be changed arbitrarily, and the component layout may also be more complicated. For example, the thickness of the components in the drawings may be exaggerated for the sake of clarity.

[0068] The following is an explanation of the terms involved in this disclosure:

[0069] Poster layout: A collection of target frames used to represent the position, size, and type of foreground elements in a poster. Specific types may include title text, decorative elements, background, descriptive text, etc.

[0070] Foreground mask: Another way to represent the layout of a poster. Specifically, a white area is rendered on a black canvas of the same size as the original poster according to the position and size of the foreground element target box, which is used to intuitively indicate the specific position of the foreground element in the poster.

[0071] Optical Character Recognition Model: A neural network used to predict the text content and font type in an image, and can also provide a scalar of the model's confidence in the prediction results.

[0072] Automatic layout network: Given a poster background image and some foreground element content information, a neural network is used to predict the appropriate layout of foreground elements.

[0073] Embodiment 1

[0074] In view of the difficulty of labeling poster data and the lack of information for poster layout in the existing technology, as shown in the attached Figure 1 As shown, the present invention provides a method for training a poster typesetting model, comprising the following steps:

[0075] Step S1, obtaining a finished poster dataset and a poster foreground text layout dataset.

[0076] Specifically, the finished poster dataset and the poster foreground text layout dataset can be collected from network resources or created by the author himself, and the present invention does not limit this.

[0077] Step S2: Obtain a pre-trained latent variable diffusion model's text-image inpainting model and a pre-trained optical character recognition model.

[0078] Among them, the training method steps of the text-image inpainting model of the pre-trained latent variable diffusion model are as follows:

[0079] Step S21: Obtain a pairing condition-image dataset and a large-scale image-text pairing dataset.

[0080] Step S22, obtaining a pre-trained latent variable diffusion generation model G1, wherein the pre-trained latent variable diffusion generation model G1 includes a one-stage autoencoder network VAE, a denoising network U-Net, and a text encoder Text Encoder.

[0081] Step S23: Based on the large-scale image-text pairing dataset, design a variety of random mask generation strategies, and fine-tune the training to obtain the latent variable diffusion patching model G based on the pre-trained latent variable diffusion generation model G1. inpaint .

[0082] In addition, the pre-trained optical character recognition model may include the following recognition steps: image preprocessing step, character segmentation step, character recognition step, and performance evaluation step. The image preprocessing step may include: denoising, grayscale, binarization, correction, etc.; the character recognition step may be based on gray level co-occurrence matrix, Hu invariant moment or vector machine, etc., which is not limited in the present invention.

[0083] By pre-training the text-image inpainting model of the latent variable diffusion model and the pre-trained optical character recognition model, low-cost poster reverse structuring can be achieved and a large amount of deconstructed poster data can be obtained.

[0084] It should be understood that the above-mentioned examples of pre-trained optical character recognition models are only examples, including but not limited to the steps, and any form of addition or deletion should be included in the subject matter of the present invention as long as it does not limit the purpose of the invention.

[0085] Step S3: Based on the text-image inpainting model of the pre-trained latent variable diffusion model, the poster foreground text typesetting dataset is input to obtain a poster background image.

[0086] In a preferred embodiment of the present invention, as shown in the attached Figure 2 As shown, step S3 also includes:

[0087] Step S3.1, converting the poster foreground text layout into a foreground mask.

[0088] Step S3.2: multiply the finished poster by the foreground mask to obtain a finished poster filtered by the mask.

[0089] Step S3.3: input the finished poster after mask filtering into the text-image inpainting model of the pre-trained latent variable diffusion model.

[0090] Step S3.4: input predetermined text and generate a poster background image.

[0091] Specifically, the predetermined text may be empty or may be a description word describing the background of the poster, and the present invention does not limit this.

[0092] Step S4: input the poster foreground typesetting dataset into the pre-trained optical character recognition model to obtain a poster foreground text typesetting dataset.

[0093] In a preferred embodiment of the present invention, as shown in the attached Figure 3 As shown, step S4 also includes:

[0094] Step S4.1, select the foreground text of the poster.

[0095] Step S4.2: appropriately scale the selected text portion image so that it can be correctly recognized by the pre-trained optical character recognition model.

[0096] In a preferred embodiment of the present invention, step S4.2 further includes a pixel restoration step and a confidence threshold screening step:

[0097] The pixel restoration step includes: if the resolution of the text in the poster is greater than a predetermined maximum threshold, the text is reduced to the average resolution of the pre-trained optical character recognition model during pre-training; if the resolution of the text in the poster is less than a predetermined minimum threshold, the text is enlarged to the minimum resolution of the pre-trained optical character recognition model during pre-training through super resolution.

[0098] The confidence threshold screening step includes: if the text confidence is greater than the confidence threshold, taking the label with the largest confidence as the font label of the current text; if the maximum confidence of the text is still less than the confidence threshold, completing the font label marking by manual annotation.

[0099] In this embodiment, the pixel grayscale step and the confidence threshold screening step can ensure that the pre-trained optical character recognition model correctly recognizes the selected text, thereby improving the accuracy and implementation efficiency of the poster automatic typesetting model.

[0100] Step S4.3: input the scaled poster foreground text into the pre-trained optical character recognition model.

[0101] Step S4.4, the pre-trained optical character recognition model outputs a poster foreground text layout dataset with font labels and text content.

[0102] In a preferred embodiment of the present invention, after step S4, the method further includes:

[0103] Step A: Obtain a pre-trained text encoder, encode the text content of the poster text typesetting dataset, and obtain a text content feature vector.

[0104] The present invention adopts a pre-trained text encoder to obtain vectorized encoding of the poster foreground text, which can improve the text feature representation capability and further reduce the text data processing time and computing power consumption during the training process.

[0105] Step S5, establishing a poster automatic typesetting model L1, and training the poster automatic typesetting model L1 based on the poster background image and the processed poster foreground typesetting data set to obtain a trained poster automatic typesetting model L.

[0106] In a preferred embodiment of the present invention, step S5 further includes:

[0107] The poster automatic typesetting model L is based on the Transformer structure.

[0108] Step S5.1: Input the poster background image and the text content feature vector into the poster automatic typesetting model L, and output the typesetting boxes and font labels corresponding to all foreground texts.

[0109] Step S5.2, introduce the cross-attention and self-attention mechanisms, calculate the correlation between the poster background image and the input foreground text content feature vector and the poster background image itself, and typeset and render the poster foreground text according to the output of the poster automatic typesetting model L.

[0110] In a preferred embodiment of the present invention, in step S5.1, the output content includes coordinates (x, y), width and height (w, h), and a font category label.

[0111] Step S5.2 also includes: applying Hungarian matching loss l to the poster automatic typesetting model L Hungarian , constraint box loss l bbox , classification loss l class , according to the coordinates, width, height and font category label c, the poster foreground text is typeset and rendered on the poster background image.

[0112] In a preferred embodiment of the present invention, it also includes:

[0113] Hungarian matching loss l Hungarian The calculation formula is as follows:

[0114]

[0115] Constraint box loss l bbox The calculation formula is as follows:

[0116]

[0117] Classification loss class The calculation formula is as follows:

[0118]

[0119] in,

[0120] N is the number of predicted poster foreground text box instances; M is the number of poster text fonts; b i The coordinates, width and height of the real constraint box of the i-th text box; is the index sequence between the real text element and the predicted text box; For the The poster text box coordinate width and height of the index; p ic is the true category of the i-th text box of the symbolic function. If the true category is c, it takes 1, otherwise it takes 0; The predicted index is The predicted probability that the font category of the text box belongs to c; Represents a real text box instance; l Hungarian represents the Hungarian matching loss; y i represents the i-th text box in the model's current predicted text box instance; c i Represents the category label of the i-th text box in the model's current predicted text box instance.

[0121] In this embodiment, the poster automatic typesetting model L calculates the correlation between the poster background image and the input foreground text content feature vector and the poster background image itself based on the cross-attention and self-attention mechanisms. It can add information of any modality during the training process, making the model flexible and adjustable.

[0122] In the poster typesetting model training method proposed in Example 1, low-cost poster reverse structuring is first achieved through a pre-trained latent variable diffusion model text-image inpainting model and a pre-trained optical character recognition model to obtain a large amount of deconstructed poster data; the poster automatic typesetting model includes a structure based on an attention mechanism, which can add information of any modality during the training process and is flexible and adjustable; the present invention uses a pre-trained text encoder to obtain a vectorized encoding of the poster foreground text, which can improve the text feature representation capability and further reduce the text data processing time and computing power consumption during the training process.

[0123] Embodiment 2

[0124] The present invention also provides a training model for a poster layout model, as shown in the attached Figure 4 As shown, it includes the following modules:

[0125] The dataset acquisition module is used to obtain the finished poster dataset and the poster foreground text layout dataset module.

[0126] Get the model modules for pre-training the text-image inpainting model of the latent variable diffusion model and the pre-training optical character recognition model.

[0127] The poster foreground acquisition module, based on the text-image inpainting model of the pre-trained latent variable diffusion model, inputs the poster foreground text typesetting data set to obtain the poster background image.

[0128] The poster marking module inputs the poster foreground typesetting data set into the pre-trained optical character recognition model to obtain a processed poster foreground text typesetting data set.

[0129] The poster automatic typesetting model training module establishes the poster automatic typesetting model L1, and trains the poster automatic typesetting model L1 based on the poster background image and the processed poster foreground typesetting data set to obtain the trained poster automatic typesetting model L.

[0130] Embodiment 3

[0131] The present invention also provides a method for automatic poster typesetting, which adopts an automatic poster typesetting model L, inputs the data to be typeset into the automatic poster typesetting model L, and outputs a poster with a completed typeset, wherein the automatic poster typesetting model L is trained by the aforementioned training method for the automatic poster typesetting model.

[0132] It should be noted that a single or multiple technical features of each specific embodiment of the present invention may be combined with a single or multiple technical features of other embodiments, and these combinations all fall within the protection scope of the present invention.

[0133] In summary, the present invention has achieved the following effects:

[0134] (1) The present invention achieves low-cost poster reverse structuring by pre-training a text-image inpainting model of a latent variable diffusion model and a pre-training optical character recognition model, thereby obtaining a large amount of deconstructed poster data.

[0135] (2) The poster automatic typesetting model contains a structure based on cross-attention and self-attention mechanisms, which can add information of any modality during the training process and is flexible and adjustable.

[0136] (3) The present invention uses a pre-trained text encoder to obtain vectorized encoding of the poster foreground text, which can improve the text feature representation capability and further reduce the text data processing time and computing power consumption during the training process.

[0137] Based on this application, it should be understood by those skilled in the art that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number and aspect described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0138] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0139] It should be noted that the above embodiments can be freely combined as needed. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention.

[0140] All documents mentioned in the present invention are cited as references in this application, just as each document is cited as reference separately. In addition, it should be understood that after reading the above content of the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the claims attached to this application.

Claims

1. A method for training a poster layout model, characterized in that: The following steps are involved: Step S1, obtaining a finished poster dataset and a poster foreground text layout dataset; Step S2, obtaining a pre-trained latent variable diffusion model text-image inpainting model and a pre-trained optical character recognition model; wherein the training method steps of the pre-trained latent variable diffusion model text-image inpainting model are as follows: Step S21, obtaining a pairing condition-image dataset and a large-scale image-text pairing dataset; Step S22, obtaining a pre-trained latent variable diffusion generation model G1, where the pre-trained latent variable diffusion generation model G1 includes a one-stage autoencoder network VAE, a denoising network U-Net, and a text encoder Text Encoder; Step S23, designing a variety of random mask generation strategies based on a large-scale image-text pairing dataset, and fine-tuning the training to obtain a latent variable diffusion patching model Ginpaint based on the pre-trained latent variable diffusion generation model G1; Step S3, inputting the poster foreground text typesetting dataset into the text-image inpainting model based on the pre-trained latent variable diffusion model to obtain the poster background image; Step S4, inputting the poster foreground text typeset data set into the pre-trained optical character recognition model to obtain a processed poster foreground text typeset data set; Step S5, establishing a poster automatic typesetting model L1, and training the poster automatic typesetting model L1 based on the poster background image and the processed poster foreground typesetting data set to obtain a trained poster automatic typesetting model L, wherein the poster automatic typesetting model L is based on a Transformer structure and includes: Step S5.1, input the poster background image and the text content feature vector into the poster automatic typesetting model L, and output the typesetting boxes and font labels corresponding to all foreground texts; Step S5.2, introduce the cross-attention and self-attention mechanisms, calculate the correlation between the poster background image and the input foreground text content feature vector and the poster background image itself, and typeset and render the poster foreground text according to the output of the poster automatic typesetting model L.

2. The poster layout model training method according to claim 1, characterized in that: Step S3 also includes: Step S3.1, converting the poster foreground text layout into a foreground mask; Step S3.2, multiplying the finished poster by the foreground mask to obtain a finished poster filtered by the mask; Step S3.3, inputting the finished poster after mask filtering into the text-image inpainting model of the pre-trained latent variable diffusion model; Step S3.4: input predetermined text and generate a poster background image.

3. The method for training a poster layout model according to claim 1, characterized in that: Step S4 also includes: Step S4.1, select the foreground text of the poster; Step S4.2, appropriately scaling the selected text portion image so that it can be correctly recognized by the pre-trained optical character recognition model; Step S4.3, inputting the scaled poster foreground text into a pre-trained optical character recognition model; Step S4.4, the pre-trained optical character recognition model outputs a poster foreground text layout dataset with font labels and text content.

4. The poster layout model training method according to claim 3, characterized in that: Step S4.2 also includes a pixel restoration step and a confidence threshold screening step: The pixel restoration step includes: if the resolution of the characters in the poster is greater than a predetermined maximum threshold, the characters are reduced to an average resolution of the pre-trained optical character recognition model during pre-training; if the resolution of the characters in the poster is less than a predetermined minimum threshold, the characters are enlarged to a minimum resolution of the pre-trained optical character recognition model during pre-training through super-resolution; The confidence threshold screening step includes: if the text confidence is greater than the confidence threshold, taking the label with the largest confidence as the font label of the current text; if the maximum confidence of the text is still less than the confidence threshold, completing the font label marking by manual annotation.

5. The method for training a poster layout model according to claim 1, characterized in that: After step S4, the method further includes: Step A: Obtain a pre-trained text encoder, encode the text content of the poster text typesetting dataset, and obtain a text content feature vector.

6. The method for training a poster layout model according to claim 5, characterized in that: In step S5.1, the output content includes coordinates (x, y), width and height (w, h), and font category label; Step S5.2 also includes: applying Hungarian matching loss to the poster automatic typesetting model L , constraint box loss , classification loss , according to the coordinates, width, height and font category label c, the poster foreground text is typeset and rendered on the poster background image.

7. The method for training a poster layout model according to claim 6, characterized in that: Also includes: Hungarian matching loss The calculation formula is as follows: ； Constraint Box Loss The calculation formula is as follows: ； Classification Loss The calculation formula is as follows: ； in, ; N is the number of predicted poster foreground text box instances; M is the number of poster text fonts; The coordinates, width and height of the real constraint box of the i-th text box; is the index sequence between the real text element and the predicted text box; For the The coordinate width and height of the poster text box of the index; is the true category of the i-th text box of the symbolic function. If the true category is c, it takes 1, otherwise it takes 0; The predicted index is The predicted probability that the font category of the text box belongs to c; Represents a real text box instance; represents the Hungarian matching loss; Represents the i-th text box in the model's current predicted text box instance; Represents the category label of the i-th text box in the model's current predicted text box instance.

8. A poster layout model training device, characterized in that: Includes the following modules: The dataset acquisition module is used to acquire the finished poster dataset and the poster foreground text layout dataset; Get model modules for pre-training text-image inpainting models of latent variable diffusion models and pre-training optical character recognition models; The poster foreground acquisition module, based on the text-image inpainting model of the pre-trained latent variable diffusion model, inputs the poster foreground text typesetting data set to obtain the poster background image; wherein the training method steps of the text-image inpainting model of the pre-trained latent variable diffusion model are as follows: Obtain pairing condition-image dataset and large-scale image-text pairing dataset; Obtain a pre-trained latent variable diffusion generation model G1, which includes a one-stage autoencoder network VAE, a denoising network U-Net, and a text encoder Text Encoder; Based on a large-scale image-text pairing dataset, we designed a variety of random mask generation strategies, and based on the pre-trained latent variable diffusion generation model G1, we fine-tuned the training to obtain the latent variable diffusion inpainting model Ginpaint. A poster marking module, inputting the poster foreground typesetting data set into the pre-trained optical character recognition model to obtain a processed poster foreground text typesetting data set; The poster automatic typesetting model training module establishes the poster automatic typesetting model L1, and trains the poster automatic typesetting model L1 based on the poster background image and the processed poster foreground typesetting data set to obtain the trained poster automatic typesetting model L; the poster automatic typesetting model L is based on the Transformer structure and includes: Input the poster background image and the text content feature vector into the poster automatic typesetting model L, and output the typesetting boxes and font labels corresponding to all foreground texts; The cross-attention and self-attention mechanisms are introduced to calculate the correlation between the poster background image and the input foreground text content feature vector and the poster background image itself, and the poster foreground text is typeset and rendered according to the output of the poster automatic typesetting model L.

9. A poster typesetting method, characterized in that: A poster automatic typesetting model L is used, the data to be typeset is input into the poster automatic typesetting model L, and the model outputs a poster with completed typeset, wherein the poster automatic typesetting model L is trained by the training method of the poster automatic typesetting model described in any one of claims 1-7.

Images