Model training method for layout generation and electronic equipment

Abstract

The invention provides a model training method for layout generation and electronic equipment, and relates to the technical field of computers and Internet. The method comprises the steps of obtaining a training image; coding layout elements in the training image into color blocks with transparency to obtain an original layout element image; performing main body identification and mask extraction on the training image to obtain a main body mask image; extracting a structure boundary and a texture contour of a main body in the training image, and generating a main body edge image; adding noise based on the original layout element image to obtain an image with noise; in at least one of the original layout element image, the main body mask image and the main body edge image, denoising the image with noise by using an attention mechanism of a diffusion model so as to reconstruct a layout element image; and training a diffusion model based on the original layout element image and the reconstructed layout element image, wherein the trained diffusion model is used for layout generation. According to the embodiment of the invention, the model capable of automatically generating the layout in a high-quality, diversified and efficient manner can be obtained.

Description

Technical Field

[0001] This disclosure relates to the fields of computer and internet technology, and in particular to a model training method and electronic device for layout generation. Background Technology

[0002] This section is intended to provide background or context for the embodiments of this disclosure as set forth in the claims. The description herein is not intended to be a prior art simply because it is included in this section.

[0003] Existing layout generation methods all have obvious limitations: rule-based or template-based layout generation methods are inflexible and difficult to adapt to diverse design needs; data-driven layout generation models are constrained by the scarcity and forgery of "clean canvas" data, resulting in insufficient generalization ability and defects such as overlap and occlusion; while layout generation methods based on large language models have strong semantic understanding capabilities, they perform poorly in terms of computational overhead and fine-grained spatial awareness.

[0004] Therefore, there is an urgent need for a new content-aware layout generation solution that can balance stability, detail perception, and adaptability to complex scenarios. Summary of the Invention

[0005] The purpose of this disclosure is to provide a model training method, apparatus, electronic device, computer-readable storage medium, and computer program product for layout generation, which can achieve high-quality, diverse, and efficient automated layout generation.

[0006] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part by practice of this disclosure.

[0007] This disclosure provides a method for training a model for layout generation, comprising: acquiring a training image, the training image including one or more layout elements; encoding the layout elements in the training image as color blocks with transparency to obtain an original layout element image; performing subject recognition and mask extraction on the training image to obtain a subject mask image; extracting the structural boundary and texture contour of the subject in the training image to generate a subject edge image; the structural boundary and texture contour are extracted using an edge algorithm; adding noise to the original layout element image to obtain a noisy image; denoising the noisy image based on at least one of the original layout element image, the subject mask image, and the subject edge image using the attention mechanism of a diffusion model to reconstruct the layout element image; wherein the diffusion model is used for a layout generation task; and training the diffusion model based on the original layout element image and the reconstructed layout element image.

[0008] In some embodiments, based on at least one of the original layout element image, the subject mask image, and the subject edge image, the noisy image is denoised using the attention mechanism of a diffusion model to reconstruct the layout element image. This includes: vectorizing the original layout element image and the subject mask image using the diffusion model to obtain layout element features and subject mask features; concatenating the subject mask features and the layout element features along the channel dimension using the diffusion model to obtain enhanced layout element features; processing the enhanced layout element features using the attention mechanism of the diffusion model to extract a first attention feature; and reconstructing the layout element image based on the first attention feature.

[0009] In some embodiments, based on at least one of the original layout element image, the main body mask image, and the main body edge image, the noisy image is denoised using the attention mechanism of a diffusion model to reconstruct the layout element image. This includes: vectorizing the original layout element image using the diffusion model to obtain layout element features; performing feature processing on the layout element features using the cross-attention mechanism in the diffusion model to obtain an initial query vector, a key vector, and a value vector; vectorizing the main body edge image using the diffusion model to obtain edge features; enhancing the initial query vector using the edge features to obtain an enhanced query vector; and reconstructing the layout element image based on the enhanced query vector, the key vector, and the value vector.

[0010] In some embodiments, based on at least one of the original layout element image, the main body mask image, and the main body edge image, the noisy image is denoised using the attention mechanism of a diffusion model to reconstruct the layout element image. This includes: obtaining layout element features by vectorizing the original layout element image using the diffusion model; performing feature processing on the layout element features using the cross-attention mechanism in the diffusion model to obtain an initial query vector for the cross-attention mechanism; performing feature mapping on the layout element features using a first mapping structure and a second mapping structure in the diffusion model to obtain a key vector and a value vector in the cross-attention mechanism; and reconstructing the layout element image based on the initial query vector, the key vector, and the value vector.

[0011] In some embodiments, based on at least one of the original layout element image, the main body mask image, and the main body edge image, the noisy image is denoised using the attention mechanism of a diffusion model to reconstruct the layout element image. This includes: vectorizing the original layout element image and the main body mask image using the diffusion model to obtain layout element features and main body mask features; concatenating the main body mask features and the layout element features along the channel dimension using the diffusion model to obtain enhanced layout element features; vectorizing the main body edge image using the diffusion model to obtain edge features; processing the enhanced layout element features using the cross-attention mechanism in the diffusion model to obtain an initial query vector for the cross-attention mechanism; enhancing the initial query vector using the edge features to obtain an enhanced query vector; performing feature mapping on the layout element features using a first mapping structure and a second mapping structure in the diffusion model to obtain a key vector and a value vector in the cross-attention mechanism; and reconstructing the layout element image based on the enhanced query vector, the key vector, and the value vector.

[0012] In some embodiments, the method further includes: acquiring target noise information; and performing denoising processing on the target noise information based on the diffusion model to obtain a target layout image.

[0013] In some embodiments, the method further includes: obtaining at least one of a preset key element layout map, a preset main body mask map, and a preset main body edge map; obtaining target noise information; and, guided by at least one of the preset key element layout map, the preset main body mask map, and the preset main body edge map, performing denoising processing on the target noise information using a diffusion model to obtain a target layout image.

[0014] In some embodiments, encoding layout elements in the training image as semi-transparent color blocks to obtain an original layout element image includes: encoding different categories of layout elements in the training image as semi-transparent color blocks of different colors to obtain a layout element image; or, encoding different layout elements in the training image as semi-transparent color blocks of different colors to obtain a layout element image.

[0015] In some embodiments, the method further includes: segmenting the reconstructed layout element image according to color channels to obtain at least one color channel image; performing binarization processing on each color channel image to generate at least one layout element mask image; and determining a structured description of the reconstructed layout element image based on the at least one layout element mask image.

[0016] This disclosure provides a model training apparatus for layout generation, including: a training image acquisition module, an original layout element image determination module, a main body mask image determination module, a main body edge image determination module, a noisy image determination module, a denoising module, and a training module.

[0017] The training image acquisition module is used to acquire training images, which include one or more layout elements; the original layout element image determination module can be used to encode the layout elements in the training image into semi-transparent color blocks to obtain an original layout element image; the subject mask image determination module can be used to perform subject recognition and mask extraction on the training image to obtain a subject mask image; the subject edge image determination module can be used to extract the structural boundaries and texture contours of the subject in the training image to generate a subject edge image; the noisy image determination module can be used to add noise to the original layout element image to obtain a noisy image; the denoising module can be used to denoise the noisy image based on at least one of the original layout element image, the subject mask image, and the subject edge image, using the attention mechanism of a diffusion model to reconstruct the layout element image; the training module can be used to train the diffusion model based on the original layout element image and the reconstructed layout element image, wherein the trained diffusion model is used for layout generation.

[0018] This disclosure provides an electronic device comprising: a memory and a processor; the memory for storing computer program instructions; and the processor for calling the computer program instructions stored in the memory to implement the model training method for layout generation described above.

[0019] This disclosure provides a computer-readable storage medium storing computer program instructions to implement the model training method for layout generation as described in any of the preceding embodiments.

[0020] This disclosure provides a computer program product or computer program that includes computer program instructions stored in a computer-readable storage medium. The computer program instructions are read from the computer-readable storage medium, and a processor executes the computer program instructions to implement the aforementioned model training method for layout generation.

[0021] The model training method, apparatus, electronic device, computer-readable storage medium, and computer program product for layout generation provided in this disclosure construct a visual representation by encoding layout elements as semi-transparent color blocks, enabling the diffusion model to explicitly learn and flexibly control the overlapping and occlusion relationships between layout elements. At the same time, under the precise guidance of multiple perception conditions such as subject mask and subject edge image, the model can intelligently avoid improper occlusion of the core area and generate a diverse layout that is spatially reasonable, visually harmonious, and supports irregular shapes.

[0022] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this disclosure. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0024] Figure 1 A schematic diagram of a scenario is shown that can be applied to a model training method or apparatus for layout generation in embodiments of this disclosure.

[0025] Figure 2 This is a flowchart illustrating a model training method for layout generation according to an exemplary embodiment.

[0026] Figure 3 This is an image of an original layout element shown according to an exemplary embodiment.

[0027] Figure 4 This is a schematic diagram of a subject mask according to an exemplary embodiment.

[0028] Figure 5 This is a schematic diagram illustrating a subject edge image according to an exemplary embodiment.

[0029] Figure 6 This is a flowchart illustrating a noise reduction method according to an exemplary embodiment.

[0030] Figure 7 This is a flowchart illustrating a noise reduction method according to an exemplary embodiment.

[0031] Figure 8 This is a flowchart illustrating a noise reduction method according to an exemplary embodiment.

[0032] Figure 9This is a flowchart illustrating a noise reduction method according to an exemplary embodiment.

[0033] Figure 10 This is a flowchart illustrating a model training method for layout generation according to an exemplary embodiment.

[0034] Figure 11 This is a flowchart illustrating a model training method for layout generation according to an exemplary embodiment.

[0035] Figure 12 This is a block diagram illustrating a model training apparatus for layout generation according to an exemplary embodiment.

[0036] Figure 13 A schematic diagram of the structure of an electronic device suitable for implementing embodiments of the present disclosure is shown. Detailed Implementation

[0037] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0038] Those skilled in the art will recognize that embodiments of this disclosure can be a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0039] The features, structures, or characteristics described in this disclosure can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more specific details omitted, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0040] In this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0041] The accompanying drawings are merely illustrative of this disclosure, and the same reference numerals in the drawings denote the same or similar parts, thus omitting repeated descriptions of them. Some block diagrams shown in the drawings do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0042] The flowchart shown in the accompanying drawings is merely illustrative and does not necessarily include all content and steps, nor does it require execution in the described order. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0043] In the description of this disclosure, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences; the terms "contains," "includes," and "has" are used to indicate an open-ended meaning of inclusion and refer to the existence of additional elements / components / etc. besides those listed.

[0044] To better understand the above-mentioned objectives, features and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0045] Existing layout generation methods can be broadly classified into the following categories, each of which has certain limitations in practical applications.

[0046] Rule-based or template-based methods.

[0047] These methods primarily rely on manually defined energy functions, typography rules, and template libraries to generate layouts. Specifically, researchers typically summarize a series of constraints on the arrangement of layout elements based on experience, such as the relative positions of text and images, white space ratios, and margin sizes. They then use heuristic search or optimization algorithms to obtain a layout that satisfies these constraints. The advantages of this type of method are its simplicity and high computational efficiency. However, due to its heavy reliance on prior human knowledge, it lacks flexibility and struggles to handle the complex and diverse content and styles in real-world design scenarios. For example, in e-commerce advertising and social media posters, the types of elements are numerous and highly uncertain; template-based solutions often fail to generate innovative and diverse layout effects.

[0048] Data-driven generative models.

[0049] In recent years, with the development of deep learning, a class of generative model-based methods has emerged, such as Generative Adversarial Networks (GANs) or Diffusion Models. These methods typically use bounding box coordinates, saliency maps, and canvas background images as input conditions to directly learn and generate layout results that conform to the semantics of the canvas content. Compared to rule-based methods, these methods have significantly improved in adaptability and automation. However, their effectiveness is limited by the training data: most models rely on so-called "clean canvas" datasets, i.e., clean background canvas images that do not contain text, logos, or other elements. However, in practical applications, due to copyright issues, obtaining large-scale, high-quality clean canvases is extremely difficult. Researchers often synthesize fake canvases through methods such as inpainting, which inevitably introduces artifacts and biases, causing inconsistencies between the model's training data and real-world data. This leads to insufficient generalization ability of the model in complex real-world scenarios, and the generated results are prone to problems such as layout overlap, subject occlusion, and lack of diversity.

[0050] A method based on a large language model.

[0051] With the rise of Large Language Models (LLMs), some studies have attempted to leverage the powerful knowledge reasoning capabilities of LLMs for layout generation. These methods typically transform the layout generation problem into sequence modeling or tree structure modeling tasks, utilizing the contextual understanding capabilities of LLMs to output layout schemes. While these methods demonstrate certain advantages in semantic reasoning and cross-modal tasks, they still face two main challenges in specific image layout generation: first, they incur high computational costs, with the reasoning process consuming significant resources (more than 24GB of GPU memory), making them difficult to deploy in low-computing-power scenarios; second, they lack sensitivity to fine-grained image structures, such as the precise location of subject edges and the spatial relationships between different elements, often failing to capture these details, resulting in a lack of spatial organization and robustness in the generated results.

[0052] In summary, existing methods are insufficient in terms of flexibility, generalization, and fine-grained visual feature perception. There is an urgent need for a new content-aware layout generation scheme that can balance stability, detail perception, and adaptability to complex scenes.

[0053] To address the aforementioned problems, this application proposes the following technical solutions.

[0054] Below, this application will provide a detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings.

[0055] Figure 1 A schematic diagram of a scenario is shown that can be applied to a model training method or apparatus for layout generation in embodiments of this disclosure.

[0056] Please refer to Figure 1 The diagram illustrates an implementation environment provided by an exemplary embodiment of this disclosure.

[0057] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0058] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, desktop computers, wearable devices, virtual reality devices, smart home devices, etc.

[0059] Server 105 can be a server that provides various services, such as a backend management server that supports the devices operated by users using terminal devices 101, 102, and 103. The backend management server can analyze and process received requests and other data, and feed the processing results back to the terminal devices.

[0060] A server can be a standalone physical server, a server cluster or a distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. This disclosure does not impose any restrictions on this.

[0061] Server 105 may, for example, acquire a training image, which includes one or more layout elements; server 105 may, for example, encode the layout elements in the training image into semi-transparent color blocks to obtain an original layout element image; server 105 may, for example, perform subject recognition and mask extraction on the training image to obtain a subject mask image; server 105 may, for example, extract the structural boundaries and texture contours of the subject in the training image to generate a subject edge image; server 105 may, for example, add noise to the original layout element image to obtain a noisy image; server 105 may, for example, use a diffusion model to denoise the noisy image under the guidance of at least one of the original layout element image, the subject mask image, and the subject edge image to reconstruct the layout element image; server 105 may, for example, train the diffusion model based on the original layout element image and the reconstructed layout element image, wherein the trained diffusion model is used for layout generation.

[0062] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Server 105 can be a single physical server or a combination of multiple servers. Depending on actual needs, it can have any number of terminal devices, networks, and servers.

[0063] Figure 2 This is a flowchart illustrating a model training method for layout generation according to an exemplary embodiment. The method provided in this disclosure can be executed by any electronic device with computing power, for example, the method can be performed by the above-described... Figure 1 The execution can be performed by a server or terminal device in the embodiments, or it can be performed by both a server and a terminal device. In the following embodiments, the server is used as the execution subject for illustration, but this disclosure is not limited to this.

[0064] Reference Figure 2The model training method for layout generation provided in this disclosure may include the following steps.

[0065] Step S202: Obtain a training image, which includes one or more layout elements.

[0066] The training images mentioned above can be source images containing the complete layout.

[0067] In some embodiments, the training image may include a background (also referred to as a canvas) and one or more layout elements.

[0068] In some embodiments, the above-mentioned layout elements may refer to the basic visual components or entities that constitute a complete visual layout (such as a poster, advertising banner, UI (User Interface) interface, etc.).

[0069] In some embodiments, layout elements may include, but are not limited to, the following categories.

[0070] Text elements, such as headings, subheadings, body paragraphs, quotations, and tags.

[0071] Image / graphic elements, such as product images, background images, icons, illustrations, shapes, etc.

[0072] Brand identification elements, such as logos, trademarks, watermarks, etc.

[0073] Interactive elements (in UI / UX design), such as buttons, input boxes, menus, sliders, etc.

[0074] Step S204: Encode the layout elements in the training image into color blocks with transparency to obtain the original layout element image.

[0075] In some embodiments, the purpose of step S204 is to convert a complex training image containing real images and text into a simple line drawing that only cares about the position, shape, category, and overlap of elements.

[0076] In some implementations, the layout elements in the training image are encoded as semi-transparent color blocks to obtain the original layout element image. This may include: encoding different categories of layout elements in the training image as semi-transparent color blocks of different colors to obtain the layout element image; or encoding different layout elements in the training image as semi-transparent color blocks of different colors to obtain the layout element image.

[0077] In some embodiments, the layout elements in the training image can be encoded as semi-transparent color blocks through the following steps to obtain the original layout element image.

[0078] Step 1: Identify and locate layout elements.

[0079] For product images: Object detection (such as YOLO) or image segmentation (such as Segment AnythingModel) models can be used to accurately obtain the outer contour polygons of the product.

[0080] For text: OCR (Optical Character Recognition) technology can be used to obtain the rectangular bounding box containing the text.

[0081] For logos, object detection or image matching techniques can also be used for location.

[0082] Step 2: Define the color coding scheme.

[0083] You can predefine which color represents which type (or which element), creating a "color-category (or object)" mapping dictionary. For example: product -> yellow; title text -> cyan; logo -> pink.

[0084] Step 3: Create a blank canvas and draw color blocks.

[0085] Create a completely black blank image with the same dimensions as the training image.

[0086] For each layout element identified in step 1: select the corresponding color from the dictionary based on its category. Fill the area on the blank canvas with this color, depending on its position and shape (whether it's a rectangle or an irregular polygon). Key point: When filling, do not use a solid color, but rather a semi-transparent one (e.g., set the alpha value to approximately 75%, or approximately 190 / 255).

[0087] Step 4: Generate the "Original Layout Element Image".

[0088] Once all elements have been drawn, the resulting color image is the "original layout element image".

[0089] Effect: Where the product, text, and logo overlap, you can see color mixing (for example, yellow and cyan overlapping will produce green), which clearly tells the model "there are two overlapping elements here".

[0090] This image contains no real textures or details, only pure, color-coded layout information.

[0091] In some embodiments, after identifying one or more layout elements from the training image, a semi-transparent color block can be directly encoded at the location of the layout element to cover the original position, and then all content outside the layout element can be replaced with a uniform color (such as white or black).

[0092] Figure 3 This is an image of an original layout element shown according to an exemplary embodiment.

[0093] like Figure 3 As shown, Figure 3 Different shades of color blocks can correspond to different colors (in this application). Figure 3 (The colors themselves are not shown.) Different colors can represent different categories or objects.

[0094] Specifically, different types of layout elements (such as text, logos, etc.) can be encoded as semi-transparent color blocks of different colors and mapped to the feature space of the diffusion model through an image encoder. This method can not only intuitively express the type and quantity of elements, but also support irregular and free-form layout conditions, thereby breaking through the limitations of bounding box representation and significantly improving the diversity and expressiveness of layouts generated by the diffusion model.

[0095] In some embodiments, semi-transparent filled shapes can be used instead of colors to represent different categories or different objects.

[0096] Step S206: Perform subject recognition and mask extraction on the training image to obtain the subject mask image.

[0097] The "subject" can refer to the most central, prominent, and visually focal object or area in a given training image (such as a poster or an advertisement). It is the "visual center" and "semantic core" of the entire layout, and other elements in the layout (such as text or logos) should be organized around it, avoiding any critical occlusion.

[0098] To give a specific example.

[0099] E-commerce product posters: The main element is the product itself that is being sold, such as a bag, a mobile phone, or a bottle of cosmetics.

[0100] Movie posters: The main subject may be the protagonist of the film or a symbolic prop (such as the hero's shield or the villain's weapon).

[0101] Tourism promotional images: The main subject may be a landmark building or a beautiful natural landscape (such as a specific beach).

[0102] Portrait poster: The main subject is the person's face and upper body.

[0103] The main subject mask extracts the main objects (such as people or products) in the poster through detection, segmentation, or manual annotation, and serves as an explicit spatial constraint input model during the diffusion generation process. This condition effectively prevents elements such as text and logos from obscuring the core subject, thus ensuring that the visual focus is clearly visible, and improving the rationality of the layout and the user's perception without affecting the overall flexibility.

[0104] In some embodiments, the mask image can be a binary image (typically containing only black and white pixels), where the white areas can precisely outline the subject, while the black areas can represent the background or other non-subject parts.

[0105] In some embodiments, the overall outline and main structural boundaries of the subject in the image can be identified and delineated first to define its basic geometric shape and spatial occupancy. Secondly, the detailed features and internal texture changes of the subject's surface need to be captured to reflect the visual lines formed by its material, pattern, or local features. Finally, the extracted overall outline and internal texture information are effectively fused and optimized to form a complete, coherent, and clear edge representation image, which is used as a key condition for describing the subject's visual structure and input into the subsequent generative model.

[0106] Figure 4 This is a schematic diagram of a subject mask according to an exemplary embodiment.

[0107] refer to Figure 4 The white area can be used to represent the position of the main subject, and the black area can be used to represent the position of non-main subjects.

[0108] Step S208: Extract the structural boundaries and texture contours of the main body in the training image to generate the main body edge image; the structural boundaries and texture contours are extracted using an edge algorithm.

[0109] In some embodiments, the structural boundaries and texture contours of the subject in the training image can be extracted using the Canny (Canny Edge Detector) and HED (Holistically-Nested Edge Detection) algorithms to generate a subject edge image. This subject edge image significantly improves the model's sensitivity to background structure, reduces element-boundary conflicts or unreasonable overlaps, and makes the layout more balanced, coherent, and visually appealing.

[0110] Figure 5 This is a schematic diagram illustrating a subject edge image according to an exemplary embodiment.

[0111] Step S210: Add noise to the original layout element image to obtain a noisy image.

[0112] In some embodiments, random noise (such as Gaussian noise) can be gradually added to the original layout element image according to a preset number of times N to obtain at least one (such as N) noisy images, where N is an integer greater than 1.

[0113] In some embodiments, at least one noisy image can be denoised using a diffusion model to reconstruct at least one layout element image. Then, the diffusion model can be trained based on the layout element image before reconstruction and the at least one layout element image after reconstruction, so that the diffusion model has the ability to generate layouts.

[0114] Below, this embodiment will use one of the at least one layout element images as an example to explain how to perform noise reduction processing.

[0115] Step S212: Based on at least one of the original layout element image, the main body mask image, and the main body edge image, the noisy image is denoised using the attention mechanism of the diffusion model to reconstruct the layout element image; wherein the diffusion model is used for the layout generation task.

[0116] In some embodiments, at least one of the original layout element image, the main body mask image, and the main body edge image can be input into the diffusion model in a certain way, so as to guide the denoising process of the diffusion model through at least one of the original layout element image, the main body mask image, and the main body edge image.

[0117] Step S214: Train the diffusion model based on the original layout element image and the reconstructed layout element image.

[0118] The technical solution proposed in this embodiment achieves high-quality and high-efficiency layout generation by encoding layout elements as semi-transparent color blocks to construct a visual representation and combining multiple conditions such as subject mask and edge image to guide the training of the diffusion model. This method not only effectively avoids element overlap and subject occlusion, but also supports the generation of diverse layouts with irregular shapes. While maintaining excellent visual aesthetics, it has good computational efficiency and cross-scene adaptability, providing a reliable automated solution for visual design scenarios such as e-commerce advertising and social media posters.

[0119] Figure 6 This is a flowchart illustrating a noise reduction method according to an exemplary embodiment.

[0120] refer to Figure 6 Guided by at least one of the original layout element image, the main body mask image, and the main body edge image, a diffusion model is used to denoise the noisy image in order to reconstruct the layout element image, including the following steps.

[0121] Step S602: The layout element image and the main body mask image are vectorized and represented respectively in the diffusion model to obtain the layout element features and the main body mask features.

[0122] In some embodiments, the diffusion model may include a vectorization module and an attention mechanism (which may be a self-attention mechanism or a cross-attention mechanism).

[0123] The vectorization module can be used to vectorize the original layout element image and the main body mask image to obtain layout element features and main body mask features.

[0124] Step S604: The main body mask features and layout element features are concatenated in the channel dimension using a diffusion model to obtain the enhanced layout element features.

[0125] Step S606: Use the attention mechanism of the diffusion model to process the layout element features after the subject enhancement and extract the first attention feature.

[0126] The attention mechanism of the diffusion model can process the layout element features after the subject is enhanced and extract the first attention feature.

[0127] Step S608: Reconstruct the layout element image based on the first attention feature.

[0128] In some embodiments, the diffusion model can construct a layout element image based on features processed by the attention mechanism.

[0129] This method combines the main subject mask features with the layout element features in the channel dimension and uses an attention mechanism to extract deep semantic relationships from the fused features. This allows the diffusion model to fully perceive and respect the spatial constraints of the main subject region in the image during the denoising and reconstruction process, thereby effectively generating a high-quality layout where the main subject is not occluded, the elements are arranged reasonably, and the principle of visual focus priority is followed.

[0130] Figure 7 This is a flowchart illustrating a noise reduction method according to an exemplary embodiment.

[0131] refer to Figure 7 Guided by at least one of the original layout element image, the main body mask image, and the main body edge image, the noisy image is denoised using a diffusion model to reconstruct the layout element image, which may include the following steps.

[0132] Step S702: Obtain layout element features by vectorizing the original layout element image in the diffusion model.

[0133] Step S704: The layout element features are processed through the cross-attention mechanism in the diffusion model to obtain the initial query vector, key vector and value vector.

[0134] Step S706: Vectorize the subject edge image using a diffusion model to obtain edge features.

[0135] Step S708: Enhance the initial query vector using edge features to obtain the enhanced query vector.

[0136] Step S710: Reconstruct the layout element image based on the enhanced query vector, key vector, and value vector.

[0137] This denoising method enhances the spatial rationality of layout generation by precisely injecting the subject's edge structure information into the cross-attention mechanism of the diffusion model. Specifically, the original layout element image is first vectorized to obtain its basic feature representation, and then the initial query, key, and value vectors required by the cross-attention mechanism are generated. Simultaneously, the subject's edge image is vectorized to extract its edge features representing the image's geometric structure and texture contours. Subsequently, these edge features are used to modulate and enhance the initial query vector, making the query process more focused on the image's true boundaries and structural information. Finally, attention calculation is performed based on the enhanced query vector and the original key and value vectors, driving the diffusion model to reconstruct a layout element image that is highly coordinated with the underlying structure of the image.

[0138] This method significantly enhances the model's ability to perceive the geometric structure and spatial boundaries of the image by deeply integrating the subject's edge features into the attention mechanism of the diffusion model. This effectively avoids structural conflicts between elements and the background subject during the layout generation process, ensuring a high degree of visual harmony and structural consistency between the generated layout and the canvas content.

[0139] Figure 8 This is a flowchart illustrating a noise reduction method according to an exemplary embodiment.

[0140] refer to Figure 8 Guided by at least one of the original layout element image, the main body mask image, and the main body edge image, the noisy image is denoised using a diffusion model to reconstruct the layout element image, which may include the following steps.

[0141] Step S802: Obtain layout element features by vectorizing the original layout element image in the diffusion model.

[0142] Step S804: The layout element features are processed through the cross-attention mechanism in the diffusion model to obtain the initial query vector of the cross-attention mechanism.

[0143] Step S806: Use the first and second mapping structures in the diffusion model to perform feature mapping on the layout element features to obtain the key vector and value vector in the cross-attention mechanism.

[0144] The first mapping structure mentioned above can be an image encoder, and the second mapping structure can be a multilayer perceptron (MLP), but this application is not limited to these.

[0145] Step S808: Reconstruct the layout element image based on the initial query vector, key vector, and value vector.

[0146] This method constructs a self-guided feature interaction mechanism by mapping the original layout element image to query, key, and value vectors in the cross-attention mechanism. This enables the diffusion model to autonomously explore and strengthen the inherent semantic and spatial relationships between layout elements during the denoising and reconstruction process, thereby effectively improving the structural coordination and visual coherence of the generated layout and achieving a more reasonable and natural element arrangement effect.

[0147] Figure 9 This is a flowchart illustrating a noise reduction method according to an exemplary embodiment.

[0148] refer to Figure 9 Guided by at least one of the original layout element image, the main body mask image, and the main body edge image, the noisy image is denoised using a diffusion model to reconstruct the layout element image, which may include the following steps.

[0149] Step S902: The layout element image and the main body mask image are vectorized and represented in the diffusion model to obtain the layout element features and the main body mask features.

[0150] Step S904: The main body mask features and layout element features are concatenated in the channel dimension using a diffusion model to obtain the enhanced layout element features.

[0151] Step S909: Vectorize the subject edge image using a diffusion model to obtain edge features.

[0152] Step S908: Process the features of the layout elements after the subject enhancement through the cross-attention mechanism in the diffusion model to obtain the initial query vector of the cross-attention mechanism.

[0153] Step S910: Enhance the initial query vector using edge features to obtain the enhanced query vector.

[0154] Step S912: Use the first and second mapping structures in the diffusion model to perform feature mapping on the layout element features to obtain the key vector and value vector in the cross-attention mechanism.

[0155] Step S914: Reconstruct the layout element image based on the enhanced query vector, key vector, and value vector.

[0156] This method achieves accurate layout generation through a multi-stage feature fusion and enhancement mechanism: First, the main body mask features and layout element features are concatenated in the channel dimension to form an enhanced feature that includes the spatial constraints of the main body; then, the edge structure features are deeply fused into the query vector through a cross-attention mechanism, while the dual-path mapping structure is used to maintain the feature representation capability of key-value pairs. Finally, at the attention level, a unified modeling of the relationship between the main body region, boundary structure and elements is completed, enabling the diffusion model to generate a visually harmonious layout that avoids occlusion of the main body and conforms to the geometric structure of the image.

[0157] Figure 10 This is a flowchart illustrating a model training method for layout generation according to an exemplary embodiment.

[0158] This embodiment proposes a content-aware automated layout generation method. Its core lies in transforming the layout generation problem into an image generation problem and combining multiple content-aware conditions to improve the generation effect. (Reference) Figure 10 The above layout generation method may include the following steps.

[0159] Step 1: Layout image representation construction (direct input to the diffusion model).

[0160] For layout representation of training data (such as training images), existing technologies typically use rectangular bounding boxes, describing element positions only with coordinates and category information, which fails to fully reflect the spatial relationships and semantic features between elements. This embodiment proposes a graphical layout representation method that encodes different types of elements (such as the main body, text, logo, etc.) as semi-transparent color blocks of different colors, which are then uniformly drawn in the layout image. To better express the overlapping relationships between elements, all color blocks are set to approximately 75% transparency, thus visually and intuitively showing the relative positions and coverage of elements. Compared with bounding boxes, graphical representation not only fully preserves the spatial structure and visual semantics of the layout but also integrates more naturally into the image generation process of the diffusion model, improving the model's ability to learn complex layouts and the diversity and realism of the generated results.

[0161] Step 2: Extraction of multiple content-aware conditions.

[0162] To enable the diffusion model to possess fine-grained perception of the canvas subject, structural boundaries, and element categories / shapes during the denoising and layout image generation process, this step introduces three complementary content-aware conditions: subject mask, edge map, and element image. These are structurally fused at key points in the diffusion generation process to jointly constrain spatial reasoning. This design significantly enhances sensitivity to fine-grained visual cues while maintaining semantic consistency and supports the generation of irregular / non-rectangular layouts.

[0163] The main subject mask extracts the main objects (such as people or products) in the poster through detection, segmentation, or manual annotation, and serves as an explicit spatial constraint input model during the diffusion generation process. This condition effectively prevents elements such as text and logos from obscuring the core subject, thus ensuring that the visual focus is clearly visible, and improving the rationality of the layout and the user's perception without affecting the overall flexibility.

[0164] Edge maps are used to extract the structural boundaries and texture contours of the canvas using the Canny and HED algorithms. The query features of the model are enhanced through a cross-attention mechanism, making the generated results more consistent with the real spatial structure of the canvas. This condition significantly improves the model's sensitivity to background structure, reduces element conflicts or unreasonable overlaps with boundaries, and makes the layout more balanced, coherent, and visually appealing.

[0165] Image-based conditional elements encode different categories of layout elements (such as text and logos) into semi-transparent color blocks of different colors. These blocks are then mapped to the feature space of a diffusion model using an image encoder, replacing traditional text-based / bounding box-based conditional elements. This approach not only more intuitively expresses element categories and quantities but also supports irregular and free-form layout conditions, thus overcoming the limitations of bounding box representations and significantly improving the diversity and expressiveness of the generated results.

[0166] In some embodiments, the element image here may be consistent with the layout image representation in step one.

[0167] Step 3: Layout generation based on the diffusion model.

[0168] To achieve stable generation of layout images while accommodating diversity, this embodiment employs a diffusion model as the core generator within the overall framework, using pre-trained text-to-image generation model weights (Stable Diffusion 1.5) as initialization parameters. Furthermore, multiple content-aware conditions are effectively integrated into the diffusion model's denoising process, thereby achieving precise guidance for layout generation.

[0169] Specifically: First, the subject mask condition, after being processed by a variational autoencoder (VAE), is concatenated along the channel dimension with the latent space features obtained after processing the layout element image by the diffusion model. This concatenated feature serves as the explicit spatial prior input in the progressive denoising process. This concatenated feature can then be used as input to the cross-attention mechanism to obtain its query vector. This approach provides strong constraints with extremely low computational overhead, effectively preventing the subject area from being occluded during layout generation while maintaining flexibility in arranging other elements. Second, the edge map condition extracts the binary edge structure map of the canvas using the Canny and HED methods. After downsampling and transformation, this map is added to the query vector obtained in the cross-attention mechanism, thereby enhancing the model's sensitivity to real boundaries and texture demarcations. This reduces conflicts or unreasonable overlaps between layout elements and background structures, ensuring good contextual consistency and visual balance in complex canvases. Furthermore, the element image conditions are mapped to a cross-attention mechanism compatible feature space via an image encoder and a multilayer perceptron (MLP), replacing the original text condition features and serving as the Key and Value in the cross-attention mechanism for computation. This design not only provides constraints comparable to text conditions but also overcomes the limitations of text in expressing irregular shapes, enabling the model to generate conditions with arbitrary layouts, thus exhibiting greater flexibility and scalability.

[0170] Furthermore, to address the limitations of current content-aware layout datasets in terms of size and insufficient model generalization, this application proposes a subjectless data augmentation strategy (e.g., a subject mask image that does not contain a subject). By synthesizing layout images that do not contain subject regions but have a reasonable layout arrangement, and using them together with real layout images for training, the robustness and generalization ability of the model in small sample sizes and complex scenarios can be effectively improved.

[0171] The method proposed in this embodiment has significant differences and advantages in the following aspects.

[0172] (1) Get rid of the dependence on fake canvas and transform the layout regression task into an image generation task.

[0173] Unlike traditional data-driven models, this embodiment no longer relies on a fake "clean canvas" generated through redrawing. Instead, it employs a diffusion model as the core generation framework, transforming the original prediction and regression task for rectangular bounding boxes into an image generation task. Specifically, this embodiment constructs a color-coded layout image representation, mapping elements such as text, logos, and main subjects onto the layout image as semi-transparent color blocks. This allows the layout generation problem to be directly modeled as an image synthesis problem. This approach not only fully preserves the spatial relationships, overlaps, and category distinctions between elements but also avoids the structural artifacts and biases introduced by the fake canvas, fundamentally improving the model's training stability and cross-scene generalization ability. Furthermore, leveraging the progressive denoising generation characteristics of the diffusion model, it can more naturally learn the global distribution and local details of layout elements in high-dimensional space, thereby generating more reasonable, realistic, and diverse results.

[0174] (2) Diverse layout generation driven by multiple content awareness conditions.

[0175] This embodiment introduces three complementary conditions—subject mask, edge map, and element image—into the diffusion model. The subject mask guides the model to avoid occluding the core region, the edge map enhances the model's sensitivity to background structure and texture boundaries, and the element image replaces the text condition, guiding the model to generate layouts that conform to the category conditions, while also supporting the generation of irregular and non-rectangular layouts. Combined with a color-coded image representation, this scheme breaks through the limitations of traditional methods that rely solely on rectangular bounding boxes, enabling the model to learn and generate more complex and diverse layout structures. This multi-condition fusion not only improves the rationality of spatial organization and semantic consistency but also significantly enhances the aesthetic expressiveness of the generated results and its applicability in various scenarios such as e-commerce advertising and social media posters.

[0176] (3) Balancing computational efficiency and robustness.

[0177] Compared to methods relying on large language models, this embodiment uses a diffusion model as its core, combined with lightweight conditional inputs. This significantly reduces computational overhead while maintaining generation quality, offering advantages in terms of memory consumption (less than 10GB) and inference speed. Furthermore, based on color-coded image representation, this embodiment proposes a subjectless data augmentation strategy. This involves synthesizing images without subject regions and jointly training them with real data, enabling the model to maintain robustness and good generalization performance even in small sample sizes and complex environments.

[0178] In some embodiments, after obtaining the trained diffusion model, the layout can be generated by the following methods: obtaining target noise information; and denoising the target noise information based on the diffusion model to obtain the target layout image.

[0179] Figure 11 This is a flowchart illustrating a model training method for layout generation according to an exemplary embodiment.

[0180] refer to Figure 11 The above-mentioned model training method for layout generation may include the following steps.

[0181] Step S1102: Obtain at least one of the following: a preset key element layout map, a preset main body mask map, and a preset main body edge map.

[0182] A preset key element layout diagram is a predefined visual template that encodes the category, location, and shape information of key elements through visual symbols such as color or shape, providing clear design intent input for layout generation.

[0183] Preset subject mask image: A predefined binary image in which the pixel-level outline and spatial position of the core subject in the image are accurately marked by white areas, serving as an explicit spatial constraint to avoid occlusion during layout generation.

[0184] Preset Subject Edge Map: A predefined structured image in which the outline and internal texture of the core subject are extracted and presented in the form of lines, serving as a geometric reference to guide the alignment of layout elements with the image structure.

[0185] Step S1104: Obtain target noise information.

[0186] Step S1106: Guided by at least one of the preset key element layout map, preset main body mask map, and preset main body edge map, the target noise information is denoised using a diffusion model to obtain the target layout image.

[0187] In some embodiments, reference may be made to Figures 2-10 The embodiments determine how to guide the denoising process of the diffusion model through at least one of the preset key element layout map, preset main body mask map, and preset main body edge map, which will not be described in detail in this embodiment.

[0188] This method introduces a preset key element layout map, a subject mask map, and a subject edge map as generation conditions, enabling the diffusion model to controllably denoise the target noise based on clear spatial constraints and structural guidance. This achieves a precise transformation from abstract noise to a structured layout image, effectively ensuring the comprehensive performance of the generated results in terms of visual rationality, subject protection, and layout coordination.

[0189] In some embodiments, the reconstructed layout element image can also be structurally described by the following method: segmenting the reconstructed layout element image according to color channels to obtain at least one color channel image; performing binarization processing on each color channel image to generate at least one layout element mask image; and determining the structural description of the reconstructed layout element image based on at least one layout element mask image.

[0190] Specifically, after the diffusion model generates the layout image, this embodiment employs differentiated post-processing methods based on different layout types to obtain structured output results. For regular rectangular layout elements, connected component detection and boundary fitting methods can be used to directly extract their circumscribed rectangles, thereby obtaining the element's bounding box coordinate information. For irregular or free-form layout elements, color-coded channels are used for segmentation to output the corresponding layout mask, accurately representing the element's true shape and coverage area. Through the above methods, the generated layout image can ultimately be transformed into a standardized structured description, providing reliable support for subsequent rendering, editing, and multi-scenario applications.

[0191] It should be particularly noted that the steps in the various embodiments of the model training method for layout generation described above can be interchanged, substituted, added, or deleted from each other. Therefore, these reasonable permutations and combinations of the model training method for layout generation should also fall within the protection scope of this disclosure, and the protection scope of this disclosure should not be limited to the described embodiments.

[0192] Based on the same inventive concept, this disclosure also provides a layout generation apparatus, as described in the following embodiments. Since the principle by which this apparatus solves the problem is similar to that of the method embodiments described above, the implementation of this apparatus embodiment can refer to the implementation of the method embodiments described above, and repeated details will not be elaborated further.

[0193] Figure 12 This is a block diagram illustrating a layout generation apparatus according to an exemplary embodiment. (Refer to...) Figure 12 The layout generation apparatus 1200 provided in this embodiment may include: a training image acquisition module 1201, an original layout element image determination module 1202, a main body mask image determination module 1203, a main body edge image determination module 1204, a noisy image determination module 1205, a denoising module 1206, and a training module 12012.

[0194] The training image acquisition module 1201 can be used to acquire a training image, which includes one or more layout elements; the original layout element image determination module 1202 can be used to encode the layout elements in the training image into semi-transparent color blocks to obtain an original layout element image; the subject mask image determination module 1203 can be used to perform subject recognition and mask extraction on the training image to obtain a subject mask image; the subject edge image determination module 1204 can be used to extract the structural boundary and texture contour of the subject in the training image to generate a subject edge image; the noisy image determination module 1205 can be used to add noise to the original layout element image to obtain a noisy image; the denoising module 1206 can be used to denoise the noisy image based on at least one of the original layout element image, the subject mask image, and the subject edge image, using the attention mechanism of the diffusion model to reconstruct the layout element image; the training module 12012 can be used to train the diffusion model based on the original layout element image and the reconstructed layout element image, wherein the trained diffusion model is used for layout generation.

[0195] It should be noted that the training image acquisition module 1201, the original layout element image determination module 1202, the main body mask image determination module 1203, the main body edge image determination module 1204, the noisy image determination module 1205, the denoising module 1206, and the training module 1207 mentioned above correspond to S202 to S214 in the method embodiment. The examples and application scenarios implemented by the above modules and corresponding steps are the same, but they are not limited to the content disclosed in the above method embodiment. It should be noted that the above modules, as part of the device, can be executed in a computer system such as a set of computer-executable instructions.

[0196] In some embodiments, based on at least one of the original layout element image, the subject mask image, and the subject edge image, the noisy image is denoised using the attention mechanism of a diffusion model to reconstruct the layout element image. This includes: vectorizing the original layout element image and the subject mask image using the diffusion model to obtain layout element features and subject mask features; concatenating the subject mask features and the layout element features along the channel dimension using the diffusion model to obtain enhanced layout element features; processing the enhanced layout element features using the attention mechanism of the diffusion model to extract a first attention feature; and reconstructing the layout element image based on the first attention feature.

[0197] In some embodiments, based on at least one of the original layout element image, the main body mask image, and the main body edge image, the noisy image is denoised using the attention mechanism of a diffusion model to reconstruct the layout element image. This includes: vectorizing the original layout element image using the diffusion model to obtain layout element features; performing feature processing on the layout element features using the cross-attention mechanism in the diffusion model to obtain an initial query vector, a key vector, and a value vector; vectorizing the main body edge image using the diffusion model to obtain edge features; enhancing the initial query vector using the edge features to obtain an enhanced query vector; and reconstructing the layout element image based on the enhanced query vector, the key vector, and the value vector.

[0198] In some embodiments, based on at least one of the original layout element image, the main body mask image, and the main body edge image, the noisy image is denoised using the attention mechanism of a diffusion model to reconstruct the layout element image. This includes: obtaining layout element features by vectorizing the original layout element image using the diffusion model; performing feature processing on the layout element features using the cross-attention mechanism in the diffusion model to obtain an initial query vector for the cross-attention mechanism; performing feature mapping on the layout element features using a first mapping structure and a second mapping structure in the diffusion model to obtain a key vector and a value vector in the cross-attention mechanism; and reconstructing the layout element image based on the initial query vector, the key vector, and the value vector.

[0199] In some embodiments, based on at least one of the original layout element image, the main body mask image, and the main body edge image, the noisy image is denoised using the attention mechanism of a diffusion model to reconstruct the layout element image. This includes: vectorizing the original layout element image and the main body mask image using the diffusion model to obtain layout element features and main body mask features; concatenating the main body mask features and the layout element features along the channel dimension using the diffusion model to obtain enhanced layout element features; vectorizing the main body edge image using the diffusion model to obtain edge features; processing the enhanced layout element features using the cross-attention mechanism in the diffusion model to obtain an initial query vector for the cross-attention mechanism; enhancing the initial query vector using the edge features to obtain an enhanced query vector; performing feature mapping on the layout element features using a first mapping structure and a second mapping structure in the diffusion model to obtain a key vector and a value vector in the cross-attention mechanism; and reconstructing the layout element image based on the enhanced query vector, the key vector, and the value vector.

[0200] In some embodiments, the apparatus further includes a noise acquisition module and a noise processing module.

[0201] The noise acquisition module can be used to acquire target noise information; the noise processing module can be used to denoise the target noise information based on the diffusion model to obtain the target layout image.

[0202] In some embodiments, the apparatus further includes: a preset image acquisition module, a noise acquisition module, and a noise reduction module.

[0203] The preset image acquisition module can be used to acquire at least one of a preset key element layout image, a preset main body mask image, and a preset main body edge image; the noise acquisition module can be used to acquire target noise information; and the denoising module can be used to denoise the target noise information using a diffusion model under the guidance of at least one of the preset key element layout image, the preset main body mask image, and the preset main body edge image to obtain a target layout image.

[0204] In some embodiments, encoding layout elements in the training image as semi-transparent color blocks to obtain an original layout element image includes: encoding different categories of layout elements in the training image as semi-transparent color blocks of different colors to obtain a layout element image; or, encoding different layout elements in the training image as semi-transparent color blocks of different colors to obtain a layout element image.

[0205] In some embodiments, the apparatus further includes a segmentation module, a binarization processing module, and a reconstruction module.

[0206] The segmentation module can be used to segment the reconstructed layout element image according to color channels to obtain at least one color channel image; the binarization processing module can be used to perform binarization processing on each color channel image to generate at least one layout element mask image; and the reconstruction module can be used to determine the structured description of the reconstructed layout element image based on the at least one layout element mask image.

[0207] Since the functions of the device 1200 have been described in detail in their respective method embodiments, they will not be repeated here.

[0208] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a portion of a module or program segment containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer program instructions.

[0209] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0210] Figure 13 A schematic diagram of an electronic device suitable for implementing embodiments of the present disclosure is shown. It should be noted that... Figure 13 The illustrated electronic device 1300 is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0211] like Figure 13 As shown, the electronic device 1300 includes a central processing unit (CPU) 1301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1302 or a program loaded from a storage section 1308 into a random access memory (RAM) 1303. The RAM 1303 also stores various programs and data required for the operation of the electronic device 1300. The CPU 1301, ROM 1302, and RAM 1303 are interconnected via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0212] The following components are connected to I / O interface 1305: an input section 1306 including a keyboard, mouse, etc.; an output section 1307 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1308 including a hard disk, etc.; and a communication section 1309 including a network interface card such as a LAN card, modem, etc. The communication section 1309 performs communication processing via a network such as the Internet. Drive 1310 is also connected to I / O interface 1305 as needed. Removable media 1311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1310 as needed so that computer programs read from them can be installed into storage section 1308 as needed.

[0213] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing computer program instructions for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1309, and / or installed from removable medium 1311. When the computer program is executed by central processing unit (CPU) 1301, it performs the functions defined above in the system of this disclosure.

[0214] It should be noted that the computer-readable storage medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable computer program instructions. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable storage medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Computer program instructions contained on a computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0215] In another aspect, this disclosure also provides a computer-readable storage medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable storage medium carries one or more programs that, when executed by the device, enable the device to perform the following functions: acquiring a training image, the training image including one or more layout elements; encoding the layout elements in the training image as semi-transparent color blocks to obtain an original layout element image; performing subject recognition and mask extraction on the training image to obtain a subject mask image; extracting the structural boundaries and texture contours of the subject in the training image to generate a subject edge image; adding noise to the original layout element image to obtain a noisy image; denoising the noisy image using a diffusion model under the guidance of at least one of the original layout element image, the subject mask image, and the subject edge image to reconstruct the layout element image; and training the diffusion model based on the original layout element image and the reconstructed layout element image, wherein the trained diffusion model is used for layout generation.

[0216] According to one aspect of this disclosure, a computer program product or computer program is provided, comprising computer program instructions stored in a computer-readable storage medium. The computer program instructions are read from the computer-readable storage medium, and a processor executes the computer program instructions to implement the methods provided in various optional implementations of the above embodiments.

[0217] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions of the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or portable hard drive) and includes several computer program instructions to cause an electronic device (such as a server or terminal device) to execute the method according to the embodiments of this disclosure.

[0218] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0219] It should be understood that this disclosure is not limited to the detailed structures, drawing arrangements or implementations shown herein; rather, this disclosure is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. A model training method for layout generation, characterized in that, include: Acquire a training image, wherein the training image includes one or more layout elements; The layout elements in the training image are encoded as color blocks with transparency to obtain the original layout element image; Perform subject recognition and mask extraction on the training images to obtain subject mask images; Extract the structural boundaries and texture contours of the main subject in the training image to generate a subject edge image; the structural boundaries and texture contours are extracted using an edge algorithm. Noise is added to the original layout element image to obtain a noisy image; Based on at least one of the original layout element image, the main body mask image, and the main body edge image, the noisy image is denoised using the attention mechanism of the diffusion model to reconstruct the layout element image; The diffusion model mentioned above is used for layout generation tasks; The diffusion model is trained based on the original layout element image and the reconstructed layout element image.

2. The method according to claim 1, characterized in that, Based on at least one of the original layout element image, the main body mask image, and the main body edge image, the noisy image is denoised using the attention mechanism of a diffusion model to reconstruct the layout element image, including: By vectorizing the original layout element image and the main body mask image in the diffusion model, layout element features and main body mask features are obtained. The main body mask features and the layout element features are concatenated in the channel dimension using the diffusion model to obtain the enhanced layout element features. The attention mechanism of the diffusion model is used to process the features of the layout elements after the main body is enhanced, and the first attention feature is extracted. Reconstruct the layout element image based on the first attention feature.

3. The method according to claim 1, characterized in that, Based on at least one of the original layout element image, the main body mask image, and the main body edge image, the noisy image is denoised using the attention mechanism of a diffusion model to reconstruct the layout element image, including: The layout element features are obtained by vectorizing the original layout element image in the diffusion model. The layout element features are processed using the cross-attention mechanism in the diffusion model to obtain the initial query vector, key vector, and value vector. The subject edge image is vectorized using the diffusion model to obtain edge features; The initial query vector is enhanced using the edge features to obtain an enhanced query vector; The layout element image is reconstructed based on the enhanced query vector, the key vector, and the value vector.

4. The method according to claim 1, characterized in that, Based on at least one of the original layout element image, the main body mask image, and the main body edge image, the noisy image is denoised using the attention mechanism of a diffusion model to reconstruct the layout element image, including: The layout element features are obtained by vectorizing the original layout element image in the diffusion model. The layout element features are processed using the cross-attention mechanism in the diffusion model to obtain the initial query vector of the cross-attention mechanism; The first and second mapping structures in the diffusion model are used to perform feature mapping on the layout element features to obtain the key vector and value vector in the cross-attention mechanism. Based on the initial query vector, the key vector, and the value vector, the layout element image is reconstructed.

5. The method according to claim 1, characterized in that, Based on at least one of the original layout element image, the main body mask image, and the main body edge image, the noisy image is denoised using the attention mechanism of a diffusion model to reconstruct the layout element image, including: By vectorizing the original layout element image and the main body mask image in the diffusion model, layout element features and main body mask features are obtained. The main body mask features and the layout element features are concatenated in the channel dimension using the diffusion model to obtain the enhanced layout element features. The subject edge image is vectorized using the diffusion model to obtain edge features; The features of the enhanced layout elements of the subject are processed by the cross-attention mechanism in the diffusion model to obtain the initial query vector of the cross-attention mechanism; The initial query vector is enhanced using the edge features to obtain an enhanced query vector; The first and second mapping structures in the diffusion model are used to perform feature mapping on the layout element features to obtain the key vector and value vector in the cross-attention mechanism. Based on the enhanced query vector, the key vector, and the value vector, the layout element image is reconstructed.

6. The method according to claim 1, characterized in that, The method further includes: Obtain target noise information; The target noise information is denoised based on the diffusion model to obtain the target layout image.

7. The method according to claim 1, characterized in that, The method further includes: Obtain at least one of the following: a preset key element layout image, a preset main body mask image, and a preset main body edge image; Obtain target noise information; Guided by at least one of the preset key element layout diagram, the preset main body mask diagram, and the preset main body edge diagram, the target noise information is denoised using a diffusion model to obtain a target layout image.

8. The method according to claim 1, characterized in that, Encoding the layout elements in the training image into color blocks with transparency to obtain the original layout element image includes: The layout elements of different categories in the training image are encoded as semi-transparent color blocks of different colors to obtain a layout element image; or, Different layout elements in the training image are encoded as semi-transparent color blocks of different colors to obtain layout element images.

9. The method according to claim 1, characterized in that, The method further includes: The reconstructed layout element image is segmented according to color channels to obtain at least one color channel image; Binarize each color channel image to generate at least one layout element mask image; Based on the at least one layout element mask image, a structured description of the reconstructed layout element image is determined.

10. An electronic device, characterized in that, include: Memory and processor; The memory is used to store computer program instructions; the processor calls the computer program instructions stored in the memory to implement the model training method for layout generation as described in any one of claims 1-9.

Images