A method for quickly generating a magazine cover or inside page layout based on a large language model

Abstract

The application discloses a method for generating magazine cover or inner page layout based on a large language model, and has the characteristics that the method comprises the following steps: a) generating texts describing elements and position coordinates on a magazine page in batches by using a large language model; b) generating elements one by one by using Hyper-SD according to the content of the texts; c) performing weakly supervised labeling on a generated image of a single object by using PP-LiteSeg; d) combining single elements into a complete magazine page according to the position description of the generated elements, and combining the weakly supervised labels of the single elements; e) verifying the generated labels by using a segformor corrected by a self-labeled data set; and f) evaluating the generated data labeled based on crowdsourcing. Compared with the prior art, the application has the advantages of fast generation speed, accurate labeling of generated images, satisfaction of human aesthetic feeling of the generated images and the like, and has a good application prospect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic document layout generation, in particular to a method for quickly generating magazine cover or inner page layout using large language models. BACKGROUND

[0002] In the field of publishing and content creation, automatic document layout generation technology can significantly improve the efficiency and personalization of content production, especially in the generation of magazine covers and inner pages with diverse and precise design requirements. Document generation systems aim to automatically create page layout descriptions that meet specific theme, style, and element requirements through computers, allowing content creators to obtain consistent and visually appealing page designs in a short time.

[0003] Traditional page design and layout generation usually relies on manually designed templates and static layout rules. Template-driven layout generation methods require users to fill in text and image elements in predefined frames, which has poor flexibility and is difficult to adapt to dynamic content layout and precise arrangement requirements between elements. In addition, the content generated by templates often lacks precision in details, especially in element position, size, and properties, making it difficult to meet the requirements of complex document generation. With the rapid development of large language models, automatic content generation methods based on natural language generation have gradually become a new trend. Large language models can automatically generate semantically coherent and content-rich document descriptions by learning the relationship between language structure and visual semantics from a large amount of text and visual data, especially suitable for page generation tasks that contain rich information and multi-level layout. For example, by setting conditions such as theme, element type (such as background, text, chart), and specific requirements (such as position and size), large language models can generate document layouts containing specific page descriptions, covering the precise position and arrangement of elements such as background, graphics, and text.

[0004] However, existing large language models still have limitations in generating fine-grained page layout descriptions. In particular, it is extremely challenging to directly generate content with labels (i.e., data containing weak labels), as complex layouts and relationships between multiple elements require highly consistent and accurate descriptions. Large language models often lack consistency when generating such content, making it difficult to achieve the required precision in position, size, and element details. This limitation makes it difficult to generate high-quality magazine inner page content with weak labels. Therefore, how to use large language models to generate high-quality content that meets page design requirements and contains accurate descriptions and labels without affecting generation speed is the focus of current research in automatic document generation technology. SUMMARY

[0005] The purpose of the present application is to provide a method for generating magazine cover or inside page layout based on large language model to overcome the shortcomings of the prior art, which comprises: having controllable layout, diverse elements and weak annotation methods, using large language model to generate elements and position coordinates on magazine pages in batches, generating elements one by one using Hyper-SD according to the description content, weakly supervising and annotating the generated image of a single object using PP-LiteSeg, combining single elements into a complete magazine page according to the generated element position description, combining their weakly supervised labels, verifying the generated annotation using segformor corrected by self-annotation dataset, and evaluating the generated data with labels based on crowdsourcing. The method can generate document page descriptions containing various elements according to preset requirements, and can generate flexible and controllable layout and diverse elements to make up for the lack of current labeled data, provide more abundant data support for model training and content generation, accurately generate page layout and element description on the premise of meeting user's customized requirements, thereby providing basic weak annotation information for data generation process, providing high flexibility in layout and element design, significantly reducing the workload of subsequent annotation, and improving the applicability and quality of generated content. It has the advantages of fast generation speed, accurate annotation of generated images, and generated images meeting human aesthetic standards, and good application prospect.

[0006] The specific technical solution to achieve the purpose of the present application is: a method for generating magazine cover or inside page layout based on large language model, characterized in that the method comprises the following steps:

[0007] Step 1: using large language model to generate text description of magazine pages in batches, including elements and their positions

[0008] Relationships, specifically including:

[0009] 1.1: define description template and constraints

[0010] Set the template for the large language model to generate content, which needs to limit the number of elements to be consistent with reality; textually constrain the element categories of the generated image, so that the generated magazine page or inside page layout has diversity, rather than just a single category of elements; ensure that the page contains at least two of figure, text, table, and chart; ensure that the page must contain a background element (which can be a blank background).

[0011] 1.2: guide model output format

[0012] Provide the model with standardized description output requirements, including element categories, positions (x, y coordinates), sizes (pixel values), hierarchical relationships, and other fields, for subsequent processing.

[0013] Step 2: Efficiently generate individual elements in the description using Hyper-SD, including:

[0014] 2.1: Generate elements one by one based on the description using the Hyper-SD model

[0015] According to the description information in Step 1, use Hyper-SD to generate each element in the page individually, including background, characters, charts, text, and other content.

[0016] Step 3: For the generated image of individual objects, use PP-LiteSeg for weakly supervised labeling, including:

[0017] 3.1: Object detection and segmentation preprocessing

[0018] Preprocess each generated single object image, including adjusting contrast and resolution, to ensure that the segmentation effect of PP-LiteSeg is more accurate, especially in identifying complex boundaries and tiny details.

[0019] 3.2: Weakly supervised segmentation labeling

[0020] Input each preprocessed object image into the PP-LiteSeg model. PP-LiteSeg uses a weakly supervised method to label images, allowing each object to obtain accurate segmentation contours and labels, including boundary information and shape attributes.

[0021] 3.3: Generate multiple formats of annotation files

[0022] Save the annotation information, including segmented boundary coordinates, object categories, and other attributes, and generate multiple annotation file formats such as JSON or XML for subsequent data storage and analysis.

[0023] Step 4: Layout multiple elements in a picture according to the generated element position description to form a complete magazine page or inner page layout, including:

[0024] 4.1: Element position matching and layout construction

[0025] According to the description file in Step 1, position each generated element image according to the accurate x, y coordinates to ensure that each element is placed in the predetermined position, avoiding position deviation or overlap.

[0026] 4.2: Layer adjustment and background synthesis

[0027] According to the hierarchical relationship of elements (such as background at the bottom layer and text in the foreground), sort and superimpose the layers to ensure that the page layout is clear and meets the expected visual hierarchy.

[0028] 4.3: Edge processing and color coordination

[0029] Anti-aliasing processing is performed on the edges of each element to ensure that the edges are clear and smooth, avoiding a rough splicing feeling. In addition, the color of each element is adjusted to be consistent with the overall style of the page, avoiding excessive color difference between elements affecting the visual effect.

[0030] Step 5: Manually annotate a dataset and improve Segformer and subsequent user verification of label quality, including:

[0031] 5.1: Data annotation of magazine page or inner page layout

[0032] Introduce traditional auxiliary methods to perform fine-grained data annotation on magazine page or inner page layout. The annotated semantic types include: figure, text, table, chart and background, ensuring that the semantic information of each type is clear to support the segmentation performance of the model on different categories. During the annotation process, combine binary cutting and foreground-background segmentation to pre-process complex areas and refine edges through progressive segmentation to provide auxiliary support for annotation.

[0033] 5.2: Feature fusion

[0034] An AFPAN (Adaptive Feature Pyramid Attention Network) module is added to the SegFormer semantic segmentation model to adaptively fuse features at multiple scales.

[0035] 5.3: Extraction of high-frequency feature components

[0036] Fourier transform is introduced in SegFormer, by introducing Fourier transform in the MLP layer of SegFormer, the features are mapped to the frequency domain and high-frequency components are extracted. These high-frequency components focus on the details and edge information in the image, which helps to capture smaller and complex structures. After these high-frequency features are fused into the MLP, the segmentation performance of the model in detail and edge areas is optimized, while the overall model complexity does not increase significantly, thus improving the segmentation accuracy on the premise of ensuring efficiency.

[0037] 5.4: Model fine-tuning and quality verification

[0038] Fine-tune SegFormer using annotated data to ensure its adaptability to specific tasks and verify the accuracy of the segmentation results, so that the segmentation information meets the expectations.

[0039] Step 6: Evaluate the quality of the generated dataset to verify the accuracy of its data labeling and whether the generated images meet human aesthetics, including:

[0040] 6.1: Data labeling accuracy verification

[0041] Perform labeling accuracy check on each sample in the generated dataset using Segformer to ensure that its weak label is consistent with the label given by Segformer.

[0042] 6.2: Human visual aesthetic evaluation

[0043] Conduct visual aesthetic scoring through a questionnaire, inviting evaluators to score the generated magazine page or inside page layout image from layout, color, and overall coordination, to verify whether the generated magazine image meets human visual aesthetic standards.

[0044] Compared with the prior art, the present application generates a document page description containing multiple elements according to preset requirements, and through flexible and controllable layout and diversified element generation, it makes up for the current lack of labeled data, provides more abundant data support for model training and content generation, can accurately generate page layout and element description on the premise of meeting user's customized needs, thereby providing basic weak labeling information for data generation process, not only providing high flexibility in layout and element design, but also significantly reducing the workload of subsequent labeling, improving the applicability and quality of generated content. Compared with the original SegFormer MLP decoder, AFPAN provides a more flexible feature fusion mechanism that can extract and integrate important information from features of different scales. This fusion method helps the model to be more efficient in processing details and global information, especially in scenes with multiple scale objects and rich details, and has the advantages of fast generation speed, accurate labeling of generated images, and meeting human aesthetic standards, as well as good application prospects. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 The flowchart of the present application;

[0046] Figure 2 The flowchart of the present application;

[0047] DETAILED DESCRIPTION

[0048] Referring to Figure 1 , the present application generates a magazine cover or inside page layout quickly according to the following steps:

[0049] Step 1: Use a large language model to generate text descriptions of magazine pages in batches, including elements and their positions

[0050] relationship, specifically including:

[0051] 1.1: Define description templates and constraints

[0052] Set templates for large language models to generate content, limiting the number of elements to match reality; impose textual constraints on the element categories generated for images, allowing for diverse magazine page or interior page layouts rather than single-element categories; ensure that pages contain at least two elements from the set {figure, text, table, chart}; and ensure that the page must contain a background element (which can be a blank background).

[0053] 1.2: Guide the output format of the model

[0054] Provide the model with standardized description output requirements, including element categories, positions (x, y coordinates), sizes (pixel values), hierarchical relationships, and other fields, for subsequent processing.

[0055] Step 2: Use Hyper-SD to efficiently generate individual elements in the description, including:

[0056] 2.1: Generate elements one by one using the Hyper-SD model based on the description

[0057] According to the description information in Step 1, use Hyper-SD to generate each element in the page individually, including background, characters, charts, and text.

[0058] Step 3: For the generated image of a single object, use PP-LiteSeg for weakly supervised labeling, including:

[0059] 3.1: Object detection and segmentation preprocessing

[0060] Preprocess each generated single object image, including adjusting contrast and resolution, to ensure that the segmentation effect of PP-LiteSeg is more accurate, especially in identifying complex boundaries and small details.

[0061] 3.2: Weakly supervised segmentation labeling

[0062] Input each preprocessed object image into the PP-LiteSeg model. PP-LiteSeg uses a weakly supervised method to label images, allowing each object to obtain accurate segmentation contours and labels, including boundary information and shape attributes.

[0063] 3.3: Generate multiple formats of annotation files

[0064] Save the annotation information, including the segmented boundary coordinates, object categories and other attributes, and generate various annotation file formats such as JSON or XML for subsequent data storage and analysis.

[0065] Step 4: Layout multiple elements in a picture according to the generated element position description, forming a complete magazine page, including:

[0066] 4.1: Element position matching and layout construction

[0067] According to the description file in step 1, position each generated element image according to the accurate x, y coordinates, ensuring that each element is placed in the predetermined position, avoiding position offset or overlap.

[0068] 4.2: Layer adjustment and background synthesis

[0069] According to the hierarchical relationship of elements (such as background at the bottom layer and text in the foreground), sort and superimpose the layers to ensure clear page layout and meet the expected visual hierarchy.

[0070] 4.3: Edge processing and color coordination

[0071] Anti-aliasing processing is performed on the edges of each element to ensure clear and smooth edges, avoiding rough splicing. In addition, adjust the color of each element to be consistent with the overall style of the page, avoiding too large color difference between elements affecting the visual effect.

[0072] Step 5: Manually annotate a dataset using the improved Segformer, and the user verifies the label quality later, including:

[0073] 5.1: Data annotation of magazine page or inner page layout

[0074] Introduce traditional auxiliary methods to perform fine-grained data annotation on magazine page or inner page layout. The semantic types of annotation include: figure, text, table, chart and background, ensuring that the semantic information of each type is clear to support the segmentation performance of the model on different categories. During the annotation process, combine binary cutting and foreground-background segmentation to pre-process complex areas and refine edges through progressive segmentation to provide auxiliary support for annotation.

[0075] 5.2: Feature fusion

[0076] Add AFPAN (Adaptive Feature Pyramid Attention Network) module to the SegFormer semantic segmentation model to adaptively fuse features at multiple scales.

[0077] 5.3: Extraction of high-frequency feature components

[0078] The Fourier transform is introduced in SegFormer. By introducing the Fourier transform in the MLP layer of SegFormer, the features are mapped to the frequency domain and high-frequency components are extracted. These high-frequency components focus on the details and edge information in the image, which helps to capture smaller and complex structures. After fusing these high-frequency features into the MLP, the model's segmentation performance in detail and edge areas is optimized, while the overall model complexity does not increase significantly, thus improving the segmentation accuracy while ensuring efficiency.

[0079] 5.4: Model fine-tuning and quality verification

[0080] Fine-tune SegFormer with labeled data to ensure its adaptability to specific tasks and verify the accuracy of the segmentation results, so that the segmentation information meets the expectations.

[0081] Step 6: Evaluate the quality of the generated dataset to verify the accuracy of the data labeling and whether the generated images meet human aesthetic standards, including:

[0082] 6.1: Verification of data labeling accuracy

[0083] Perform accuracy checks on each sample in the generated dataset using Segformer to ensure that its weak labels are consistent with the labels given by Segformer.

[0084] 6.2: Human visual aesthetic evaluation

[0085] Conduct visual aesthetic scoring through a questionnaire, inviting evaluators to score the generated magazine page or inside page layout images from layout, color, and overall harmony, to verify whether the generated magazine images meet human visual aesthetic standards.

[0086] The present application will be further described in combination with the following specific embodiments and drawings. The process, conditions, experimental methods, etc. for implementing the present application are generally known and commonly understood, except for the following specifically mentioned content, and the present application does not have special restrictions. Embodiments

[0087] Reference Figure 2 This embodiment demonstrates the following specific process for quickly generating magazine pages using large language models:

[0088] 1) The system preprocesses the user's set page requirements, including page theme, element type (such as charts, text, graphics), and layout requirements of each element. After data preparation is complete, these requirements are input into a large language model to generate a text description of the page. This description includes specific information about each element, such as location (accurate to x, y coordinates), size (pixels), category (such as figure, text, chart, etc.), and independence constraints of the element (such as "ensure that this description contains only a single object without other objects"), to ensure the clarity and accuracy of each element's layout.

[0089] 2) After generating the page description, the system uses the Hyper-SD model to generate page elements one by one according to the element information in the description. To ensure image quality, the system checks each generated element image for content to ensure that it meets the description requirements and has no other interfering objects. For elements that do not meet the requirements, they are regenerated to ensure the integrity of single elements. Each generated image and its description file is saved as basic data for the next step of page assembly.

[0090] 3) Use PP-LiteSeg to weakly supervise the labeling of the single generated element image. First, preprocess the image to adjust contrast, resolution, and other parameters to ensure that the labeling results of PP-LiteSeg are more accurate, especially in boundary detection and micro-detail recognition. After preprocessing, input the image into PP-LiteSeg for weakly supervised segmentation to generate accurate segmentation contours and labels for each element. After labeling is complete, the final labeling results are saved in JSON and XML formats for subsequent processing and analysis.

[0091] 4) After all element labeling is complete, the system combines each element image into a complete magazine page or inner page layout according to the position coordinates in the generated description. According to the hierarchical relationship in the description (such as background in the bottom layer, text and graphics in the foreground), the layers are sorted and superimposed, while adjusting the transparency, color tone, etc. of the elements to ensure the coordination of visual effects. The system performs anti-aliasing and color harmonization processing on the entire page to ensure that the page layout is clear and the style is consistent. The image after page synthesis is previewed, and the position, size, etc. are adjusted to ensure that the page meets the user's visual requirements.

[0092] 4) To verify the accuracy of the supervision label in the subsequent verification, a segmentation model is introduced, and the SegFormer model is also improved in this embodiment to improve the performance of the segmentation model. By introducing the AFPAN (Adaptive Feature Pyramid Attention Network) module, adaptive fusion of multi-scale features is realized; in addition, the Fourier transformation (Fourier Feature Transformation Module, FFTP) is added in the LP, which further enhances the model's ability to capture high-frequency details, so as to improve the segmentation accuracy of complex structures and edges. The improved SegFormer is fine-tuned using labeled data to ensure its adaptability and the accuracy of the segmentation results.

[0093] 5) The quality of the generated data set is evaluated to verify the accuracy of the data labeling and whether the generated image meets the human visual aesthetic. Specifically, it includes two aspects: (1) data labeling accuracy verification: check the labeling of each sample to ensure that all labeled semantic types (such as figure, text, table, chart, background, etc.) meet the expectations, and the edges and contours are accurately aligned; (2) human visual aesthetic evaluation: visual aesthetic scoring is conducted through a questionnaire, inviting evaluators to score the generated magazine page images from layout, color, and overall coordination, to ensure that the page design meets the human visual aesthetic standards.

[0094] The above is only a further description of the present application and is not intended to limit the patent. Any equivalent implementation of the present application shall be included within the scope of the claims of the present patent.

Claims

1. A method for generating a layout of a magazine cover or inside page based on a large language model, characterized in that, The method specifically comprises the following steps: Step 1: Batch generate text descriptions of magazine pages using a large language model, including elements and their positional relationships; Step 2: Generate individual elements in the text description using Hyper-SD; Step 3: For the generated image of a single object, use PP-LiteSeg for weakly supervised labeling one by one, and finally generate a dataset; Step 4: According to the generated element position description, layout multiple elements in a picture to form a complete magazine page or inner page layout; Step 5: Fine-tune SegFormer with AFPAN module using the manually annotated dataset; Step 6: Quality assessment of the dataset generated in step 3 to verify the accuracy of the data labeling and whether the generated image meets human aesthetic standards; The step 1 specifically comprises: Step 1.1: Define the description template and constraints Set the template for the large language model to generate content, which needs to limit the number of elements to be consistent with reality; Textually constrain the element categories of the generated image so that the generated magazine page or inner page layout contains at least two of the elements background, figure, text, table, and chart; Step 1.2: Guide the model output format Provide the large language model with a standardized description output requirement that includes the category, x, y coordinates, pixel value, and hierarchical relationship of the elements; The step 5 specifically comprises: Step 5.1: Data labeling of magazine pages or inner page layouts Perform fine-grained data labeling on magazine pages or inner page layouts, with semantic types including figure, text, table, chart, and background. During the labeling process, use binary segmentation and foreground-background segmentation to pre-process complex areas and refine edges through progressive segmentation; Step 5.2: Feature fusion Add the AFPAN module to the SegFormer semantic segmentation model to adaptively fuse features at multiple scales; Step 5.3: Extraction of high-frequency feature components Introduce Fourier transform in the SegFormer semantic segmentation model by introducing Fourier transform in the MLP layer of SegFormer to map features to the frequency domain and extract high-frequency feature components; Step 5.4: Model fine-tuning and quality verification Fine-tune SegFormer using labeled data and verify the accuracy of the segmentation results to ensure that the segmentation information meets expectations; The step 6 specifically comprises: Step 6.1: Data labeling accuracy verification Check the labeling accuracy of each sample in the generated dataset using Segformer to ensure that the weak labels are consistent with the labels provided by Segformer; Step 6.2: Human visual aesthetic evaluation Conduct visual aesthetic scoring through a questionnaire, inviting evaluators to score the generated magazine page or inner page layout image in terms of layout, color, and overall harmony to verify whether the generated magazine image meets human visual aesthetic standards. 2.The method of claim 1, wherein, The step 2 specifically comprises: Step 2.1: Generating elements one by one based on the description using the Hyper-SD model Each element in the page is generated individually using Hyper-SD, including background, characters, charts, and text content. 3.The method of claim 1, wherein, The step 3 specifically includes: Step 3.1: Object detection and segmentation For each generated single object image, perform object detection and segmentation preprocessing including adjusting contrast and resolution; Step 3.2: Weakly supervised segmentation labeling Input each preprocessed object image into the PP-LiteSeg model and use a weakly supervised method to label the image, so that each object obtains a segmentation contour and label including boundary information and shape attributes; Step 3.3: Generate multiple formats of annotation files Save the annotation information including the boundary coordinates of the segmentation, object categories and other attributes, and generate multiple annotation file formats in JSON or XML. 4.The method of claim 1, wherein, The step 4 specifically includes: Step 4.1: Element position matching and layout construction Position each generated element image according to the x, y coordinates to place each element in the predetermined position; Step 4.2: Layer adjustment and background synthesis According to the hierarchical relationship that the background of the element is at the bottom layer and the text is in the foreground, sort and superimpose the layers to make the page layout clear and consistent with the expected visual hierarchy; Step 4.3: Edge processing and color coordination Anti-aliasing processing is performed on the edge of each element to make the edge clear and smooth, avoid rough splicing feeling, and adjust the color of each element to make it consistent with the overall style of the page.

Images