Method for quickly generating magazine cover or inside page layout based on large language model

Abstract

The invention discloses a method for generating a magazine cover or inside page layout based on a large language model. The method is characterized by comprising the following steps of: a) generating text descriptions of elements and position coordinates on a magazine page in batches by utilizing the large language model; b) generating elements one by one by using Hyper-SD according to the description content; c) carrying out weak supervision labeling on the generated graph of the single object by using PP-LiteSeg; d) combining the single elements into a complete magazine page according to the generated element position description, and combining the weak supervision labels of the single elements; e) verifying the generated annotation by using the segform corrected by the self-annotation data set; and f) evaluating the labeled generation data based on crowdsourcing. Compared with the prior art, the method has the advantages that the generation speed is high, the generated image has accurate annotation, the generated image meets the aesthetic appreciation of human beings, and the like, and has a good application prospect.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic document layout generation, in particular to a method for rapidly generating magazine cover or inner page layout using a large language model. Background Art

[0002] In the field of publishing and content creation, automated document layout generation technology can significantly improve the efficiency and personalization of content production, especially in the generation of magazine covers and inner pages with diverse and precise design requirements. The document generation system aims to create page layout descriptions that meet specific themes, styles, and elements through computer automation, so that content creators can obtain consistent and visually beautiful page designs in a short period of time.

[0003] Traditional page design and layout generation usually rely on manually designed templates and static layout rules. Template-driven layout generation methods require users to fill in text and image elements in predefined frames, which are less flexible and difficult to adapt to dynamic content layout and precise arrangement requirements between elements. In addition, the content generated by templates often lacks precision in details, especially in terms of element position, size and attributes, which makes it difficult to meet the requirements of complex document generation. With the rapid development of large language models, automatic content generation methods based on natural language generation have gradually become a new trend. Large language models can automatically generate semantically coherent and rich document descriptions by learning the relationship between language structure and visual semantics from a large amount of text and visual data. They are particularly suitable for page generation tasks that contain rich information and multi-level layouts. For example, by setting conditions such as themes, element types (such as background, text, and graphics), and specific requirements (such as position and size), large language models can generate document layouts containing specific page descriptions, covering the precise position and arrangement of elements such as background, graphics, and text.

[0004] However, existing large language models still have limitations when generating fine-grained page layout descriptions. In particular, directly generating labeled content (i.e., data containing weak labels) is extremely challenging because complex layouts and the relationship processing of multiple elements require highly consistent and precise descriptions, while large language models often lack consistency when generating these contents and have difficulty achieving the required accuracy in position, size, and element details. This limitation makes it difficult to generate high-quality magazine page content with weak labels. Therefore, how to use large language models to generate high-quality content that meets page design requirements and contains precise descriptions and labels without affecting the generation speed is the current research focus of automated document generation technology. Summary of the invention

[0005] The purpose of the present invention is to provide a method for generating magazine cover or inner page layout based on a large language model in view of the deficiencies in the prior art. The method adopts methods including: having controllable layout, diverse elements and weak annotation, and using a large language model to batch generate text descriptions of elements and position coordinates on magazine pages. Hyper-SD is used to generate elements one by one according to the description content. For the generated graph of a single object, PP-LiteSeg is used for weak supervision annotation. According to the generated element position description, the single elements are combined into a complete magazine page, and their weak supervision labels are combined at the same time. The generated annotations are verified using segformor corrected by a self-annotated dataset, and the generated data with labeling is evaluated based on crowdsourcing. This method can generate a document page description containing multiple elements according to preset requirements. Through flexible and controllable layout and diversified element generation, it makes up for the current shortage of labeled data, provides richer data support for model training and content generation, and can accurately generate page layout and element descriptions on the premise of meeting user customization requirements, thereby providing basic weak annotation information for the data generation process. It not only provides a high degree of flexibility in layout and element design, but also significantly reduces the workload of subsequent annotation, and improves the applicability and quality of generated content. It has the advantages of fast generation speed, accurate annotation of generated images, and generated images that meet human aesthetics, as well as good application prospects.

[0006] The specific technical solution for achieving the purpose of the present invention is: a method for generating a magazine cover or inner page layout based on a large language model, which is characterized in that the method specifically includes the following steps: Step 1: Use a large language model to batch generate text descriptions of magazine pages, including elements and their locations Relationships, including: 1.1: Defining description templates and constraints When setting up a template for large language model-generated content, the number of elements must be limited to be realistic; text constraints must be placed on the element categories of the generated images so that the generated magazine pages or inner page layouts have diversity, rather than just a single category of elements; ensure that the page contains at least two of the following elements: figure, text, table, or chart; and ensure that the page must contain a background element (which can be a blank background).

[0007] 1.2: Guidance model output format Provide the model with standardized description output requirements, including element category, location (x, y coordinates), size (pixel value), hierarchical relationship and other fields for subsequent processing.

[0008] Step 2: Use Hyper-SD to efficiently generate individual elements in the description based on the description content, including: 2.1: Generate elements one by one using the Hyper-SD model based on the description According to the description in step 1, use Hyper-SD to generate each element in the page separately, including background, characters, charts, text and other contents.

[0009] Step 3: For the generated graph of a single object, use PP-LiteSeg for weakly supervised annotation, including: 3.1: Object detection and segmentation pre-processing Each generated single object image is preprocessed, including adjusting contrast and resolution, to ensure that PP-LiteSeg’s segmentation effect is more accurate, especially in identifying complex boundaries and tiny details.

[0010] 3.2: Weakly supervised segmentation annotation Each preprocessed object image is fed into the PP-LiteSeg model. PP-LiteSeg uses a weakly supervised approach to annotate images, giving each object an accurate segmentation outline and label, including boundary information and shape attributes.

[0011] 3.3: Generate annotation files in multiple formats Save annotation information, including segmented boundary coordinates, object categories, and other attributes, and generate multiple annotation file formats such as JSON or XML for subsequent data storage and analysis.

[0012] Step 4: Lay out multiple elements in one picture according to the generated element position description to form a complete magazine page or inner page layout, including: 4.1: Element position matching and layout construction According to the description file in step 1, position each generated element image according to the precise x, y coordinates to ensure that each element is placed at the predetermined position to avoid position offset or overlap.

[0013] 4.2: Layer adjustment and background synthesis Sort and overlay layers according to the hierarchical relationship of elements (such as background at the bottom and text in the foreground) to ensure that the page layout is clear and meets the expected visual hierarchy.

[0014] 4.3: Edge processing and color coordination Anti-aliasing is performed on the edges of each element to ensure that the edges are clear and smooth, and to avoid a rough sense of splicing. In addition, the color of each element is adjusted to be consistent with the overall style of the page to avoid excessive color differences between elements that affect the visual effect.

[0015] Step 5: Manually annotate a dataset and improve the quality of labels for Segformer and subsequent user verification, including: 5.1: Data annotation of magazine page or inner page layout Traditional auxiliary methods are introduced to perform fine-grained data annotation on magazine pages or inner page layouts. The annotated semantic types include: figure, text, table, chart and background, ensuring that the semantic information of each type is clear to support the segmentation performance of the model on different categories. In the annotation process, binary cutting and foreground-background segmentation are combined to pre-process complex areas, and the edges are refined through progressive segmentation to provide auxiliary support for annotation.

[0016] 5.2: Feature Fusion The AFPAN (Adaptive Feature Pyramid Attention Network) module is added to the SegFormer semantic segmentation model to adaptively fuse features at multiple scales.

[0017] 5.3: Extraction of high-frequency feature components Fourier transform is introduced in SegFormer. By introducing Fourier transform in the MLP layer of SegFormer, the features are mapped to the frequency domain and high-frequency components are extracted. These high-frequency components focus on the details and edge information in the image, which helps to capture small and complex structures. After integrating these high-frequency features into MLP, the segmentation performance of the model in detail and edge areas is optimized, and the overall model complexity will not increase significantly, thereby improving the segmentation accuracy while ensuring efficiency.

[0018] 5.4: Model fine-tuning and quality verification Use the labeled data to fine-tune SegFormer to ensure its adaptability to specific tasks and verify the accuracy of the segmentation results so that the segmentation information meets expectations.

[0019] Step 6: Evaluate the quality of the generated dataset to verify the accuracy of its data annotation and whether the generated images meet human aesthetics, including: 6.1: Verification of data annotation accuracy Each sample in the generated dataset is labeled with Segformer to check its accuracy and ensure that its weak label is consistent with the label given by Segformer.

[0020] 6.2: Human Visual Aesthetics Assessment Visual aesthetics are scored through a questionnaire, and evaluators are invited to score the generated magazine page or inner page layout images in terms of layout, color, and overall coordination to verify whether the generated magazine images meet human visual aesthetic standards.

[0021] Compared with the prior art, the present invention generates a document page description containing multiple elements according to preset requirements. Through flexible and controllable layout and diversified element generation, it makes up for the current shortage of labeled data, provides richer data support for model training and content generation, and can accurately generate page layout and element description on the premise of meeting user customization requirements, thereby providing basic weak annotation information for the data generation process. It not only provides a high degree of flexibility in layout and element design, but also significantly reduces the workload of subsequent annotation, and improves the applicability and quality of generated content. Compared with the MLP decoder of the original SegFormer, AFPAN provides a more flexible feature fusion mechanism, which can extract and integrate important information from features of different scales. This fusion method helps the model to be more efficient in processing details and global information, especially in multi-scale objects and scenes with rich details. It has the advantages of fast generation speed, accurate annotation of generated images, and generated images that meet human aesthetics, as well as good application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flow chart of the present invention; Figure 2 Schematic diagram of the process of the embodiment.

[0023] Specific examples See also Figure 1 The present invention quickly generates a magazine cover or inner page layout according to the following steps: Step 1: Use a large language model to batch generate text descriptions of magazine pages, including elements and their locations Relationships, including: 1.1: Defining description templates and constraints When setting up a template for large language model-generated content, the number of elements must be limited to be realistic; text constraints must be placed on the element categories of the generated images so that the generated magazine pages or inner page layouts have diversity, rather than just a single category of elements; ensure that the page contains at least two of the following elements: figure, text, table, or chart; and ensure that the page must contain a background element (which can be a blank background).

[0024] 1.2: Guidance model output format Provide the model with standardized description output requirements, including element category, location (x, y coordinates), size (pixel value), hierarchical relationship and other fields for subsequent processing.

[0025] Step 2: Use Hyper-SD to efficiently generate individual elements in the description based on the description content, including: 2.1: Generate elements one by one using the Hyper-SD model based on the description According to the description in step 1, use Hyper-SD to generate each element in the page separately, including background, characters, charts, text and other contents.

[0026] Step 3: For the generated graph of a single object, use PP-LiteSeg for weakly supervised annotation, including: 3.1: Object detection and segmentation pre-processing Each generated single object image is preprocessed, including adjusting contrast and resolution, to ensure that PP-LiteSeg’s segmentation effect is more accurate, especially in identifying complex boundaries and tiny details.

[0027] 3.2: Weakly supervised segmentation annotation Each preprocessed object image is fed into the PP-LiteSeg model. PP-LiteSeg uses a weakly supervised approach to annotate images, giving each object an accurate segmentation outline and label, including boundary information and shape attributes.

[0028] 3.3: Generate annotation files in multiple formats Save annotation information, including segmented boundary coordinates, object categories, and other attributes, and generate multiple annotation file formats such as JSON or XML for subsequent data storage and analysis.

[0029] Step 4: Lay out multiple elements in a picture according to the generated element position description to form a complete magazine page, including: 4.1: Element position matching and layout construction According to the description file in step 1, position each generated element image according to the precise x, y coordinates to ensure that each element is placed at the predetermined position to avoid position offset or overlap.

[0030] 4.2: Layer adjustment and background synthesis Sort and overlay layers according to the hierarchical relationship of elements (such as background at the bottom and text in the foreground) to ensure that the page layout is clear and meets the expected visual hierarchy.

[0031] 4.3: Edge processing and color coordination Anti-aliasing is performed on the edges of each element to ensure that the edges are clear and smooth, and to avoid a rough sense of splicing. In addition, the color of each element is adjusted to be consistent with the overall style of the page to avoid excessive color differences between elements that affect the visual effect.

[0032] Step 5: Manually annotate a dataset using the improved Segformer, and users subsequently verify the label quality, including: 5.1: Data annotation of magazine page or inner page layout Traditional auxiliary methods are introduced to perform fine-grained data annotation on magazine pages or inner page layouts. The annotated semantic types include: figure, text, table, chart and background, ensuring that the semantic information of each type is clear to support the segmentation performance of the model on different categories. In the annotation process, binary cutting and foreground-background segmentation are combined to pre-process complex areas, and the edges are refined through progressive segmentation to provide auxiliary support for annotation.

[0033] 5.2: Feature Fusion The AFPAN (Adaptive Feature Pyramid Attention Network) module is added to the SegFormer semantic segmentation model to adaptively fuse features at multiple scales.

[0034] 5.3: Extraction of high-frequency feature components Fourier transform is introduced in SegFormer. By introducing Fourier transform in the MLP layer of SegFormer, the features are mapped to the frequency domain and high-frequency components are extracted. These high-frequency components focus on the details and edge information in the image, which helps to capture small and complex structures. After integrating these high-frequency features into MLP, the segmentation performance of the model in detail and edge areas is optimized, and the overall model complexity will not increase significantly, thereby improving the segmentation accuracy while ensuring efficiency.

[0035] 5.4: Model fine-tuning and quality verification Use the labeled data to fine-tune SegFormer to ensure its adaptability to specific tasks and verify the accuracy of the segmentation results so that the segmentation information meets expectations.

[0036] Step 6: Evaluate the quality of the generated dataset to verify the accuracy of its data annotation and whether the generated images meet human aesthetics, including: 6.1: Verification of data annotation accuracy Each sample in the generated dataset is labeled with Segformer to check its accuracy and ensure that its weak label is consistent with the label given by Segformer.

[0037] 6.2: Human Visual Aesthetics Assessment Visual aesthetics are scored through a questionnaire, and evaluators are invited to score the generated magazine page or inner page layout images in terms of layout, color, and overall coordination to verify whether the generated magazine images meet human visual aesthetic standards.

[0038] The present invention is further described in detail with reference to the following specific examples and drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the contents specifically mentioned below, are all common knowledge and common common sense in the art and are not particularly limited by the present invention. Example

[0039] See also Figure 2 This example shows the following specific process of using a large language model to quickly generate magazine pages: 1) The system pre-processes the page requirements set by the user, including the page theme, element type (such as charts, text, graphics), and layout requirements of each element. After the data preparation is completed, these requirements are input into the large language model to generate a text description of the page. The description includes specific information of each element, such as position (accurate to x, y coordinates), size (pixels), category (such as figure, text, chart, etc.) and independence constraints of the element (such as "make sure this description contains only a single object and no other objects") to ensure the clear identifiability and layout accuracy of each element.

[0040] 2) After the page description is generated, the system uses the Hyper-SD model to generate page elements one by one according to the element information in the description. To ensure the image quality, the system verifies the content of each element image generated to ensure that it meets the description requirements and there are no other interfering objects. For non-compliant elements, they are regenerated to ensure the integrity of the single element. Each generated image and its description file are saved as the basic data for the next page combination.

[0041] 3) Use PP-LiteSeg to perform weakly supervised annotation on the individually generated element images. First, preprocess the image and adjust the contrast, resolution and other parameters to ensure that the annotation results of PP-LiteSeg are more accurate, especially in boundary detection and identification of small details. After preprocessing, the image is input into PP-LiteSeg for weakly supervised segmentation to generate accurate segmentation contours and labels for each element. After the annotation is completed, the final annotation results are saved in JSON and XML formats for subsequent processing and analysis.

[0042] 4) After all elements are marked, the system combines the images of each element into a complete magazine page or inner page layout according to the position coordinates in the generated description. According to the hierarchical relationship in the description (such as the background in the bottom layer, text and graphics in the foreground), the layers are sorted and superimposed, and the transparency and color tone of the elements are adjusted to ensure the coordination of the visual effect. The system performs anti-aliasing and color blending on the entire page to ensure that the page layout is clear and consistent in style. The synthesized image of the page is previewed, and the position, size and other details are finally adjusted to ensure that the page meets the user's visual needs.

[0043] 4) To verify the accuracy of the supervised labels in the future, a segmentation model is introduced. In order to improve the performance of the segmentation model, this embodiment also improves the SegFormer model. By introducing the AFPAN (Adaptive Feature Pyramid Attention Network) module, the adaptive fusion of multi-scale features is realized; in addition, the Fourier Feature Transformation Module (FFTM) is added to the LP to further enhance the model's ability to capture high-frequency details, so as to improve the segmentation accuracy of complex structures and edges. The improved SegFormer is fine-tuned using the labeled data to ensure its adaptability and the accuracy of the segmentation results.

[0044] 5) Evaluate the quality of the generated dataset to verify the accuracy of the data annotation and whether the generated images meet human visual aesthetics. This includes two aspects: (1) Verify the accuracy of data annotation: Check the annotation of each sample to ensure that the semantic types of all annotations (such as figure, text, table, chart, background, etc.) meet expectations and that the edges and contours are accurately aligned; (2) Human visual aesthetics evaluation: Visual aesthetics scoring is conducted through a questionnaire, and evaluators are invited to score the generated magazine page images in terms of layout, color, and overall coordination to ensure that the page design meets human visual aesthetic standards.

[0045] The above is only a further explanation of the present invention and is not intended to limit this patent. Any equivalent implementation of the present invention should be included in the scope of the claims of this patent.

Claims

1. A method for generating magazine cover or inner page layout based on a large language model, characterized in that: The method specifically comprises the following steps: Step 1: Use a large language model to batch generate text descriptions of magazine pages, which include elements and their positional relationships; Step 2: Generate individual elements in the text description using Hyper-SD; Step 3: For the generated graphs of a single object, use PP-LiteSeg to perform weak supervision annotation one by one, and finally generate a dataset; Step 4: According to the generated element position description, multiple elements are laid out in one picture to form a complete magazine page or inner page layout; Step 5: Fine-tune SegFormer with the AFPAN module using the manually annotated dataset; Step 6: Perform a quality assessment on the dataset generated in step 3 to verify the accuracy of its data annotation and whether the generated images meet human aesthetic standards.

2. The method for generating magazine cover or inner page layout based on a large language model according to claim 1, characterized in that: The step 1 specifically includes: 1.1: Define description templates and constraints When setting a template for generating content using a large language model, the number of elements must be limited to be realistic. Text constraints must be imposed on the element categories of the generated images, so that the generated magazine page or inner page layout contains background elements and at least two elements from figure, text, table, and chart. 1.2: Guidance model output format Provide the large language model with a standardized description output requirement including element categories, x, y coordinates, pixel values, and hierarchical relationships.

3. The method for generating magazine cover or inner page layout based on a large language model according to claim 1, characterized in that: The step 2 specifically includes: 2.1 Generate elements one by one using the Hyper-SD model based on the description Hyper-SD is used to generate each element in the page individually, including background, characters, graphics and text content.

4. The method for generating magazine cover or inner page layout based on a large language model according to claim 1, characterized in that: The step 3 specifically includes: 3.1: Object Detection and Segmentation Each generated single object image is subjected to object detection and segmentation preprocessing including contrast and resolution adjustment; 3.2: Weakly supervised segmentation annotation Each preprocessed object image is input into the PP-LiteSeg model, and the image is annotated using a weakly supervised method so that each object obtains a segmentation outline and label including boundary information and shape attributes; 3.3: Generate annotation files in multiple formats Save annotation information including segmented boundary coordinates, object categories and other attributes, and generate multiple annotation file formats such as JSON or XML.

5. The method for generating magazine cover or inner page layout based on a large language model according to claim 1, characterized in that: The step 4 specifically includes: 4.1: Element position matching and layout construction Position each generated element image according to the x, y coordinates so that each element is placed at the predetermined position; 4.2: Layer adjustment and background synthesis According to the hierarchical relationship of the elements, the background is at the bottom layer and the text is in the foreground, so that the page layout is clear and meets the expected visual hierarchy; 4.3: Edge processing and color coordination Perform anti-aliasing on the edges of each element to make them clear and smooth, avoid rough stitching, and adjust the color of each element to make it consistent with the overall style of the page.

6. The method for generating magazine cover or inner page layout based on a large language model according to claim 1, characterized in that: The step 5 specifically includes: 5.1: Data annotation of magazine page or inner page layout Fine-grained data annotation of magazine pages or inner page layouts. The annotated semantic types include: figure, text, table, chart and background. The annotation process combines binary cutting and foreground-background segmentation, pre-processes complex areas, and refines edges through progressive segmentation. 5.2: Feature Fusion Add the AFPAN module to the SegFormer semantic segmentation model to adaptively fuse features at multiple scales; 5.3: Extraction of high-frequency feature components Fourier transform is introduced in SegFormer semantic segmentation. By introducing Fourier transform in the MLP layer of SegFormer, the features are mapped to the frequency domain and high-frequency feature components are extracted; 5.4: Model fine-tuning and quality verification Use the labeled data to fine-tune SegFormer and verify the accuracy of the segmentation results to ensure that the segmentation information meets expectations.

7. The method for generating magazine cover or inner page layout based on a large language model according to claim 1, characterized in that: The step 6 specifically includes: 6.1: Verification of data annotation accuracy Use Segformer to check the accuracy of annotation for each sample in the generated dataset to ensure that its weak label is consistent with the label given by Segformer; 6.2: Human Visual Aesthetics Assessment Visual aesthetics are scored through a questionnaire, and evaluators are invited to score the generated magazine page or inner page layout images based on layout, color, and overall coordination to verify whether the generated magazine images meet human visual aesthetic standards.

Images