Poster generation method and device, equipment and medium
Through the combination of feature encoding, feature fusion, graph neural network and layout transformer, the problems of low efficiency and difficult quality quantification of traditional poster design are solved, and high-quality automatic poster generation is achieved.
Patent Information
- Application Number
- CN202510830629.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-19
- Publication Date
- 2025-10-17
AI Technical Summary
Traditional poster design relies on manual layout adjustment, which is inefficient, costly, and difficult to quantify in quality. Neural network-based poster generation models face challenges in generating high-quality posters.
The poster element features are extracted through the feature encoder, and the feature fusion is performed using the feature fuser. The region grouping and layout transformer optimization are performed in combination with the graph neural network module. Finally, the target poster image is generated through the gated adjustment module, realizing automated design from macro layout to micro alignment.
The quality of poster generation is improved, the coordination of the overall layout and the accuracy of detail processing are ensured, and high-quality automated poster design is achieved.
Smart Images
Figure CN120807706A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of poster design and the field of financial technology, and in particular to a poster generation method, device, equipment and medium. BACKGROUND
[0002] As an important form of visual communication, poster design, especially in the fields of financial technology, medical health and old-age care, for example, in the financial technology scenario, posters often need to carry complex and critical information, such as the characteristics of financial products, the yield situation, risk prompts, etc. These information needs to be ensured through reasonable visual levels and professional layout design so that users can quickly and accurately obtain key content. In the medical health and old-age care scenario, poster design not only needs reasonable visual levels and professional layout design, but also needs to take into account the readability of old users. Therefore, the quality of poster design directly affects the effect of information transmission and user experience.
[0003] Traditional poster design usually needs to rely on manual layout adjustment by designers, and such a way usually has problems such as low efficiency, high cost, and difficulty in quantitative evaluation of quality. In recent years, with the application of deep learning in automatic poster layout, the generation model based on neural network can learn the layout rules from a large number of design samples, and then realize the automatic generation of posters. However, in actual application, due to the complexity of poster design elements and the limitation of model performance, there are still many challenges in realizing high-quality poster generation, which makes the final output effect difficult to meet the actual demand. Therefore, how to improve the quality of poster generation has become a problem to be solved. SUMMARY
[0004] The embodiments of the present application provide a poster generation method, device, computer equipment and storage medium to solve the problem of how to improve the quality of poster generation.
[0005] A poster generation method comprises: Obtaining elements of a poster image to be generated, inputting all elements into a feature encoder of a target poster generation model, outputting element features of each element through the feature encoder, inputting all element features into a feature fusioner of the target poster generation model, and outputting fusion features of all elements through the feature fusioner; inputting the fusion features into a first layout transformer of the target poster generation model, outputting a region division result of a layout space required by all elements through the first layout transformer, inputting the fusion features and the region division result into a graph neural network module of the target poster generation model, and outputting a region grouping result of all elements through the graph neural network module; The region grouping result is input into a second layout transformer of the target poster generation model, a region layout result of all elements is output through the second layout transformer, the region division result, the region grouping result and the region layout result are input into a gate adjustment module of the target poster generation model, a target layout result of all elements is output through the gate adjustment module, and a target poster image is generated according to the target layout result.
[0006] A poster generation apparatus comprises: A feature encoding module is configured to acquire elements of a poster image to be generated, input all elements into a feature encoder of a target poster generation model, output element features of each element through the feature encoder, and input all element features into a feature fusioner of the target poster generation model, and output fusion features of all elements through the feature fusioner. A layout generation module is configured to input the fusion features into a first layout transformer of the target poster generation model, output a region division result of layout spaces required by all elements through the first layout transformer, and input the fusion features and the region division result into a graph neural network module of the target poster generation model, and output a region grouping result of all elements through the graph neural network module. A poster generation module is configured to input the region grouping result into a second layout transformer of the target poster generation model, output a region layout result of all elements through the second layout transformer, input the region division result, the region grouping result and the region layout result into a gate adjustment module of the target poster generation model, output a target layout result of all elements through the gate adjustment module, and generate a target poster image according to the target layout result.
[0007] A computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the above poster generation method when executing the computer program.
[0008] A computer readable storage medium stores a computer program, and the computer program is executable on a processor to implement the above poster generation method.
[0009] The poster generation method and device, computer device and storage medium, by inputting the elements of the poster image to be generated into a feature encoder of a target poster generation model, obtaining element features of each element, inputting all element features into a feature fusioner, obtaining fusion features of all elements, inputting the fusion features into a first layout transformer, obtaining a region division result, inputting the fusion features and the region division result into a graph neural network module, obtaining a region grouping result, inputting the region grouping result into a second layout transformer, obtaining a region layout result, inputting the region division result, the region grouping result and the region layout result into a gate adjustment module, obtaining a target layout result, and generating a target poster image according to the target layout result.
[0010] The first layout transformer performs region division on the layout space at a macro level, the graph neural network module dynamically performs region grouping on the elements, the second layout transformer optimizes the layout of the elements in the region grouping at a micro level, and finally the gate adjustment module performs interactive adjustment to output the target layout result, so as to generate the target poster image, realize automatic poster design from macro layout to micro alignment, and realize hierarchical generation, that is, complete region division in the global planning stage and optimize element alignment in the local refinement stage, so as to ensure the coordination of the overall layout and the accuracy of the detail processing, thereby improving the quality of the generated poster. BRIEF DESCRIPTION OF DRAWINGS
[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained according to these drawings without creative labor for those skilled in the art.
[0012] Figure 1 is a schematic diagram of an application environment of a poster generation method in an embodiment of the present application; Figure 2 is a flowchart of a poster generation method in an embodiment of the present application; Figure 3 is an architecture diagram of a target poster generation model in an embodiment of the present application; Figure 4 is another flowchart of a poster generation method in an embodiment of the present application; Figure 5 is another flowchart of a poster generation method in an embodiment of the present application; Figure 6 is another flowchart of a poster generation method in an embodiment of the present application; Figure 7 is another flowchart of a poster generation method in an embodiment of the present application; Figure 8 is another flow chart of a poster generation method according to one embodiment of the present invention; Figure 9 is a schematic diagram of a poster generating device according to an embodiment of the present invention; Figure 10 FIG. 1 is a schematic diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0013] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort shall fall within the scope of protection of the present invention.
[0014] The poster generation method provided by the embodiment of the present invention can be applied as follows: Figure 1 Specifically, the poster generation method is applied in a poster generation system, which includes the following: Figure 1 The client and server shown communicate over a network to solve the problem of improving the quality of poster generation. The client, also known as the user end, is the program that corresponds to the server and provides local services to clients. The client can be installed on, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented as a standalone server or a server cluster consisting of multiple servers.
[0015] For example, in the poster generation task in the field of financial technology, it is necessary to generate financial product promotion posters based on elements such as financial product yield data, risk warning clauses, bank institution logos, and yield trend curves. At this time, the poster generation method of the present invention can be used to automatically generate financial product promotion posters that ensure the coordination of the overall layout and the accuracy of detail processing.
[0016] In one embodiment, if Figure 2 As shown, a poster generation method is provided, which is applied in Figure 1 The server in the example is used as an example, and the steps are as follows: Step S201: Obtain the elements of the poster image to be generated, input all elements into the feature encoder of the target poster generation model, output the element features of each element through the feature encoder, input all element features into the feature fuser of the target poster generation model, and output the fusion features of all elements through the feature fuser.
[0017] In this embodiment, the elements to be generated into the poster image can refer to independent visual units in the poster image, for example, the elements can include financial text, medical popular science text, financial data image, medical image, technology element image and health element image in the poster image, the target poster generation model can refer to a neural network model that has been trained for poster image generation, the feature encoder can refer to an encoder for feature extraction of the elements, the element feature can refer to a vector or abstract description representing the visual, semantic or structural features of the elements, the feature fusioner can refer to a module for feature fusion of the element features of all elements, and the fused feature can refer to a feature after fusion of the element features of all elements.
[0018] Specifically, the elements of data types such as vision, text and voice to be generated into the poster image are obtained, all the elements are input into the feature encoder of the target poster generation model corresponding to the data type, the feature of each element is obtained by feature extraction of the elements through the feature encoder, and the fused feature of all the elements is obtained by inputting the element features of all the elements into the feature fusioner and performing feature fusion of all the element features through the feature fusioner.
[0019] Step S202: input the fused feature into the first layout transformer of the target poster generation model, output the region division result of the layout space required for all elements through the first layout transformer, input the fused feature and the region division result into the graph neural network module of the target poster generation model, and output the region grouping result of all elements through the graph neural network module.
[0020] In this embodiment, the layout space can refer to a two-dimensional region in the poster canvas for placing elements, the first layout transformer can refer to a module for predicting the global layout planning of the poster according to the fused feature, the region division result can refer to the result of region division of the layout space, indicating that the layout space is divided into different two-dimensional regions, for example, the layout space is divided into a main image area, a title area and a secondary text area, etc., the graph neural network module can refer to a module for region grouping of all elements according to the fused feature and the region division result, and the region grouping result can refer to the result of region grouping of all elements, indicating the region to which the elements are allocated, for example, the financial data image and the technology element image are allocated to the title area, and the financial text is allocated to the secondary text area.
[0021] Specifically, the fused feature is input into the first layout transformer of the target poster generation model to obtain the region division result of the layout space required for all elements, and the fused feature and the region division result are input into the graph neural network module to obtain the grouping result of all elements in different regions.
[0022] Step S203: input the region grouping result into a second layout transformer of the target poster generation model, output the region layout result of all elements by the second layout transformer, input the region division result, the region grouping result and the region layout result into a gating adjustment module of the target poster generation model, output the target layout result of all elements by the gating adjustment module, and generate the target poster image according to the target layout result.
[0023] In this embodiment, the second layout transformer can refer to a module for optimizing the spatial layout of elements in the corresponding region according to the region grouping result, the region layout result can refer to the result of optimizing the layout of elements in the corresponding region, indicating the spatial position of elements in the corresponding region, the gating adjustment module can refer to a module for globally optimizing the region division result, the region grouping result and the region layout result, the target layout result can refer to the overall layout result of all elements optimized by the gating adjustment module, and the target poster image can refer to a poster image generated by the target poster generation model based on all elements.
[0024] Specifically, the region grouping result is input into the second layout transformer of the target poster generation model to obtain the region layout result of all elements in the corresponding region, the region division result, the region grouping result and the region layout result are input into the gating adjustment module to obtain the target layout result of all elements, and the target poster image is rendered according to the target layout result.
[0025] For example, as shown in Figure 3 The architecture diagram of the target poster generation model provided in this embodiment can be as follows: 1) input all elements into a feature encoder to extract features to obtain element features, input all element features into a feature fusioner to fuse features to obtain fused features; 2) input the fused features into a first layout transformer to divide the layout space to obtain a region division result, input the fused features and the region division result into a graph neural network module to group elements to obtain a region grouping result; 3) input the region grouping result into a second layout transformer to adjust the region layout to obtain a region layout result, input the region division result, the region grouping result and the region layout result into a gating adjustment module to adjust to obtain a target layout result, and generate a target poster image according to the target layout result.
[0026] In the embodiment, the layout space is regionally divided at a macro level by the first layout transformer, the elements are dynamically regionally grouped by the graph neural network module, the layout of the elements in the regionally grouped elements is optimized at a micro level by the second layout transformer, and finally the target layout result is output after interactive adjustment by the gating adjustment module, so as to generate the target poster image, realize the automatic poster design from the macro layout to the micro alignment, and through the hierarchical generation mode, the region division is completed in the global planning stage and the element alignment is optimized in the local refinement stage, so that the coordination of the overall layout is ensured and the accuracy of the detail processing is ensured, thereby improving the quality of the generated poster.
[0027] In an embodiment, as shown in Figure 4 A poster generation method is provided, the elements include visual elements and text elements, the step S201 is that all the elements are input into a feature encoder of a target poster generation model, the element features of each element are output by the feature encoder, all the element features are input into a feature fusioner of the target poster generation model, and the fusion features of all the elements are output by the feature fusioner, including the following steps. Step S401: all the visual elements are input into a visual feature encoder of a target poster generation model, and the visual features of each visual element are output by the visual feature encoder.
[0028] Step S402: all the text elements are input into a text feature encoder of a target poster generation model, and the text features of each text element are output by the text feature encoder.
[0029] Step S403: all the elements are input into a geometric feature encoder of a target poster generation model, and the geometric features of each element are output by the geometric feature encoder.
[0030] Step S404: all the visual features, text features and geometric features are input into a feature fusioner of a target poster generation model, and the fusion features are output by the feature fusioner.
[0031] In this embodiment, the visual element can refer to an element in the poster that transmits information through visual media such as images, images, etc. For example, fund yield curve, insurance coverage radar chart, and disease interpretation diagram, etc. The text element can refer to an element in the poster that transmits information through language and text, such as insurance promotion slogans, investment risk warnings, and disease popular science scripts, etc. The visual feature encoder can refer to an encoder that extracts features of visual elements. The text feature encoder can refer to an encoder that extracts features of text elements. The geometric feature encoder can refer to an encoder that extracts geometric properties (such as center point, size, etc.) of elements. The visual feature can refer to an abstract description of the characteristics of the visual element. The text feature can refer to an abstract description of the characteristics of the text element. The position feature can refer to an abstract description of the geometric properties of the element.
[0032] Specifically, all visual elements are input into the visual feature encoder of the target poster generation model to obtain the visual features of the corresponding visual elements. All text elements are input into the text feature encoder to obtain the text features of the corresponding text elements. All elements are input into the geometric feature encoder to obtain the geometric features of the corresponding elements. All visual features, text features, and geometric features are input into the feature fusioner, and the fusion features are output by the feature fusioner.
[0033] In this embodiment, the visual feature encoder, the text feature encoder, and the geometric feature encoder are used to extract features of multi-modal visual elements and text elements, and multi-modal feature fusion is performed based on the feature fusioner to obtain fusion features, which realizes fine classification and extraction of multi-modal features, improves the accuracy of feature extraction, provides an accurate data basis for subsequent generation of high-quality poster images, and thus improves the quality of the generated poster.
[0034] In an embodiment, as shown in Figure 5 a poster generation method is provided. In step S202, the fusion features are input into the first layout transformer of the target poster generation model, and the region division results of the layout space required by all elements are output by the first layout transformer, including the following steps: Step S501: input the fusion features into the encoder in the first layout transformer, and output the aggregated features representing the attention relationship between all elements by the encoder.
[0035] Step S502: input the aggregated features into the first multi-layer perception module in the first layout transformer, and output the spatial regions divided by the layout space and the region parameters of the corresponding spatial regions by the first multi-layer perception module. All spatial regions and region parameters of the corresponding spatial regions form the region division results.
[0036] In this embodiment, the aggregated feature can refer to an abstract representation of global attention relationships between all elements, the spatial region can refer to a two-dimensional region obtained by dividing the layout space, the region parameter can refer to a parameter representing the geometric properties (such as center, width and height) of the spatial region, and the first multi-layer perception module can refer to a module that maps the aggregated feature to the spatial region and the corresponding region parameter.
[0037] Specifically, the fused feature In the encoder in the first layout transformer, the aggregated feature representing the attention relationships between all elements is output by the encoder The calculation formula can be wherein Q is a query matrix, K is a key matrix, and V is a value matrix, Q, K and V are linear projections of the fused feature , is a scaling factor, the aggregated feature is input into the first multi-layer perception module, and the aggregated feature is non-linearly transformed by the first multi-layer perception module to be mapped to the spatial region obtained by dividing the layout space and the region parameter of the corresponding spatial region, i.e. wherein is a spatial region, m is the number of spatial regions, R is a layout space, and the region parameter corresponding to each spatial region can include the center coordinates, width and height of the spatial region, and a score representing the importance of the spatial region.
[0038] In this embodiment, the fused feature is converted into the aggregated feature by the encoder of the first layout transformer based on the global attention mechanism, capturing the semantic and spatial dependency relationships between elements, so that when the fused feature is converted into the region division result by the first multi-layer perception module, the region division is adapted to the global relationships between elements on the basis of realizing end-to-end mapping from semantics to physical space, improving the accuracy of region division and thus improving the quality of the generated poster.
[0039] In an embodiment, as shown in Figure 6 A poster generation method is provided, and in step S202, the fused feature and the region division result are input into the graph neural network module of the target poster generation model, and the region grouping result of all elements is output by the graph neural network module, including the following steps. Step S601: The fused feature and the region division result are input into the graph construction submodule in the graph neural network module, and an element relationship graph representing the relationships between all elements is output by the graph construction submodule.
[0040] Step S602: For an element represented by any node in the element relationship graph, the attention weight between the element and any element other than the element is calculated according to the attribute feature of the element and the attribute feature of each element other than the element.
[0041] Step S603: Determine the number of all spatial regions in the region division result, and select a certain number of elements from all elements as initial cluster centers of the corresponding spatial regions according to the number.
[0042] Step S604: Cluster all elements according to the attention weights between all elements, all initial cluster centers and the preset threshold to obtain cluster groups corresponding to the spatial regions, and form the cluster groups of all spatial regions into regional grouping results.
[0043] In this embodiment, the graph construction submodule may refer to a module used to model the relationships between all elements, the element relationship graph may refer to a graph structure that represents the relationships between all elements, each node in the element relationship graph represents an element and the attribute characteristics of the corresponding element, the attribute characteristics may refer to the element characteristics of the element, each edge represents the spatial relationship between elements, the clustering group may refer to a set composed of any group of elements obtained by clustering all elements, and the preset threshold may refer to a preset threshold that needs to be met by the attention weight between elements that are clustered into the same clustering group.
[0044] Specifically, the fusion features and region division results are input into the graph construction submodule in the graph neural network module to obtain an element relationship graph that represents the relationship between all elements. For any element, the attention mechanism is used to select the element based on its attribute characteristics. and the attribute characteristics of any element except this element , calculate the attention weight between two elements The calculation formula can be ,in, is the projection matrix, is the attention vector, T is the vector transpose, || represents splicing, As the activation function, determine the number of all spatial regions, select the same elements as the number of spatial regions from all elements as the initial cluster centers, according to the formula Cluster all elements to obtain cluster groups corresponding to the spatial region, where is the cluster group, is an element in any cluster group, are elements outside the cluster group, is the preset threshold, that is, the element Assigned to cluster groups The conditions are: element With cluster group Each element in Attention weights between Exceeds the preset threshold , thus, all elements are divided into a plurality of cluster groups, each cluster group corresponding to a spatial region, and forming a region grouping result by the cluster groups of all spatial regions.
[0045] In this embodiment, the graph construction submodule of the graph neural network module constructs an element relationship graph based on the fused features and the region grouping result, captures the spatial and semantic correlation between elements, calculates the attention weight between elements based on the attribute features, quantifies the correlation degree between elements, and thus when clustering according to the attention weight to obtain the cluster group corresponding to the spatial region, the region grouping is adapted to the correlation degree between elements on the basis of ensuring the consistency of the grouping and the macro layout, the rationality of the region grouping is improved, and thus the quality of the generated poster is improved.
[0046] In an embodiment, as shown in Figure 7 a poster generation method is provided, and the region grouping result is input into a second layout transformer of a target poster generation model in step S203, and a region layout result of all elements is output by the second layout transformer, including the following steps. Step S701: inputting the region grouping result into the second layout transformer, and determining, for any element in any cluster group, the neighboring elements of the element from all elements in the cluster group.
[0047] Step S702: updating the attribute features of the element according to the attribute features of the element, the attribute features of each neighboring element, and the attention weight between the element and each neighboring element, to obtain updated attribute features.
[0048] Step S703: inputting the updated attribute features into a second multi-layer perceptron module in the second layout transformer, and outputting, by the second multi-layer perceptron module, a geometric adjustment amount of the element in the corresponding spatial region, and forming a region layout result by the geometric adjustment amounts of all elements in the corresponding spatial region.
[0049] In this embodiment, for any element, the neighboring elements can refer to the elements closest to the element in the cluster group where the element is located, the updated attribute features can refer to the updated attribute features of the element, the geometric adjustment amount can refer to a parameter representing the geometric properties such as coordinates and dimensions of the element in the spatial region, and the second multi-layer perceptron module can refer to a module that maps the updated attribute features to the geometric adjustment amount of the element in the spatial region.
[0050] Specifically, the region grouping result is input into the second layout transformer, and for any element in any cluster, the k neighboring elements closest to the element are determined from all elements in the cluster group, k is an integer greater than zero, and the attribute features of the element , the attribute features of each neighboring element , and the attention weight between the element and each neighboring element The attribute feature of the element is updated The attribute feature is updated to obtain an updated attribute feature That is, Wherein, The updated attribute feature is input into a second multi-layer perception module, and the updated attribute feature is nonlinearly transformed by the second multi-layer perception module to map the updated attribute feature to a geometric adjustment amount of the element in the corresponding spatial region, that is, Wherein, The geometric adjustment amount is used to represent the relative change of the center coordinates and the size of the element in the spatial region, and the geometric adjustment amounts of all elements in the corresponding spatial region form a region layout result.
[0051] In this embodiment, the attribute feature is updated based on the attention weight between the adjacent elements in the clustering group, the feature interaction of the adjacent elements in the clustering group is realized, and when the updated attribute feature is mapped to the region layout result by the second perception module, the local correlation between the elements in the clustering group can be further captured, a more natural visual arrangement is realized, the structural of the overall layout of the poster is maintained, and the local layout details are optimized, thereby improving the quality of the generated poster.
[0052] In an embodiment, as shown in Figure 8 A poster generation method is provided, and the region division result, the region grouping result and the region layout result in step S203 are input into a gating adjustment module of a target poster generation model, and the target layout result of all elements is output by the gating adjustment module, including the following steps. Step S801: The region division result and the region grouping result are input into a first adjustment submodule in the gating adjustment module, and a first correction coefficient of the region division result to the region grouping result is output by the first adjustment submodule.
[0053] Step S802: The region division result and the region grouping result are input into a second adjustment submodule in the gating adjustment module, and a second correction coefficient of the region grouping result to the region division result is output by the second adjustment submodule.
[0054] Step S803: The target layout result is obtained according to the first correction coefficient, the second correction coefficient and the region layout result.
[0055] In this embodiment, the first adjustment submodule can refer to a module for adjusting the region grouping result according to the region division result, the second adjustment submodule can refer to a module for adjusting the region division result according to the region grouping result, the first correction coefficient can refer to a parameter value representing the adjustment degree of the region grouping result according to the region division result, and the second correction coefficient can refer to a parameter value representing the adjustment degree of the region division result according to the region grouping result.
[0056] Specifically, the region division result and the region grouping result are input into the first adjustment submodule of the gating adjustment module, and the first correction coefficient for adjusting the region grouping result according to the region division result is output through the first adjustment submodule, that is, wherein, is the first correction coefficient, is a learnable parameter, R is the region division result, and C is the region grouping result. The region division result and the region grouping result are input into the second adjustment submodule, and the second correction coefficient for adjusting the region division result according to the region grouping result is output through the second adjustment submodule, that is, wherein, is the second correction coefficient, is a learnable parameter, and the target layout result is obtained according to the first correction coefficient, the second correction coefficient, and the region layout result.
[0057] Optionally, in the process of obtaining the target layout result according to the first correction coefficient, the second correction coefficient, and the region layout result, the region division result and the region grouping result can be adjusted according to the first correction coefficient and the second correction coefficient to obtain an updated region division result and an updated region grouping result. The updated region division result, the updated region grouping result, and the region layout result are input into the fusion submodule in the gating adjustment module, and the target layout result is output through the fusion submodule.
[0058] wherein, the updated region grouping result can refer to a result obtained by adjusting the region grouping result according to the first correction coefficient, the updated region division result can refer to a result obtained by adjusting the region division result according to the second correction coefficient, for example, if the elements in the spatial region are too dense, the spatial region can be expanded according to the second correction coefficient, and the fusion submodule can refer to a module for feature fusion of the updated region division result, the updated region grouping result, and the region layout result.
[0059] In the embodiment, through the bidirectional correction mechanism of the first adjustment submodule and the second adjustment submodule, dynamic collaborative optimization of the region division result and the region grouping result is realized, the rationality of the overall layout and the local details is ensured, and thus when the updated region division result, the updated region grouping result and the region layout result are fused by the fusion submodule to obtain the target layout result, the quality of the target layout is improved, and thus the quality of the generated poster is improved.
[0060] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the application.
[0061] In an embodiment, a poster generation device is provided, which corresponds to the poster generation method in the above embodiment. As shown in the figure, the poster generation device includes a feature encoding module 91, a layout generation module 92 and a poster generation module 93. The functions of each module are described in detail as follows: Figure 9 The feature encoding module 91 is configured to obtain elements of an image to be generated, input all the elements into a feature encoder of a target poster generation model, output element features of each element through the feature encoder, input all the element features into a feature fusioner of the target poster generation model, and output fusion features of all the elements through the feature fusioner. The layout generation module 92 is configured to input the fusion features into a first layout transformer of the target poster generation model, output a region division result of a layout space required for all the elements through the first layout transformer, input the fusion features and the region division result into a graph neural network module of the target poster generation model, and output a region grouping result of all the elements through the graph neural network module. The poster generation module 93 is configured to input the region grouping result into a second layout transformer of the target poster generation model, output a region layout result of all the elements through the second layout transformer, input the region division result, the region grouping result and the region layout result into a gate adjustment module of the target poster generation model, output a target layout result of all the elements through the gate adjustment module, and generate a target poster image according to the target layout result. Optionally, the feature encoding module 91 includes:
[0062] A visual encoding unit is configured to input all the visual elements into a visual feature encoder of the target poster generation model, and output visual features of each visual element through the visual feature encoder. a text encoding unit, configured to input all text elements into a text feature encoder of the target poster generation model, and output text features of each text element through the text feature encoder; a geometry encoding unit, configured to input all elements into a geometry feature encoder of the target poster generation model, and output geometry features of each element through the geometry feature encoder; a first feature fusion unit, configured to input all visual features, text features, and geometry features into a feature fusioner of the target poster generation model, and output the fusion features through the feature fusioner.
[0063] Optionally, the layout generation module 92 includes: a first calculation unit, configured to input the fusion features into an encoder in the first layout transformer, and output aggregated features representing attention relationships among all elements through the encoder; a first mapping unit, configured to input the aggregated features into a first multi-layer perception module in the first layout transformer, and output spatial regions divided from the layout space and region parameters of corresponding spatial regions through the first multi-layer perception module, so as to form the region division result.
[0064] Optionally, the layout generation module 92 includes: a graph construction unit, configured to input the fusion features and the region division result into a graph construction submodule in the graph neural network module, and output an element relationship graph representing relationships among all elements through the graph construction submodule, each node in the element relationship graph representing an element and an attribute feature of a corresponding element, and each edge representing a spatial relationship between elements; a second calculation unit, configured to calculate, for an element represented by any node in the element relationship graph, an attention weight between the element and any element other than the element according to the attribute feature of the element and the attribute features of all elements other than the element; a selection unit, configured to determine a number of all spatial regions in the region division result, and select, according to the number, a number of elements from all elements as initial clustering centers of corresponding spatial regions; a clustering unit, configured to cluster all elements according to the attention weights among all elements, all initial clustering centers, and a preset threshold, to obtain clustering groups of corresponding spatial regions, and form the region grouping result by using clustering groups of all spatial regions.
[0065] Optionally, the poster generation module 93 includes: determining, for any element in any cluster group, neighboring elements of the element from all elements in the cluster group; updating, according to the attribute feature of the element, the attribute feature of each neighboring element, and the attention weight between the element and each neighboring element, the attribute feature of the element to obtain an updated attribute feature; inputting the updated attribute feature into a second multi-layer perception module in the second layout transformer, and outputting, by the second multi-layer perception module, a geometric adjustment amount of the element in the corresponding spatial region, and forming the region layout result by using the geometric adjustment amounts of all elements in the corresponding spatial region.
[0066] Optionally, the poster generation module 93 includes: inputting the region division result and the region grouping result into a first adjustment submodule in the gating adjustment module, and outputting, by the first adjustment submodule, a first correction coefficient of the region division result to the region grouping result; inputting the region division result and the region grouping result into a second adjustment submodule in the gating adjustment module, and outputting, by the second adjustment submodule, a second correction coefficient of the region grouping result to the region division result; obtaining the target layout result according to the first correction coefficient, the second correction coefficient, and the region layout result.
[0067] Optionally, the third adjustment unit includes: adjusting, according to the first correction coefficient and the second correction coefficient, the region division result and the region grouping result to obtain an updated region division result and an updated region grouping result; inputting the updated region division result, the updated region grouping result, and the region layout result into a fusion submodule in the gating adjustment module, and outputting, by the fusion submodule, the target layout result.
[0068] The specific limitations of the poster generation device can refer to the limitations of the poster generation method described above, which will not be repeated here. Each module in the poster generation device described above can be realized by software, hardware, and combinations thereof, in whole or in part. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.
[0069] In one embodiment, a computer device is provided, which can be a server, and an internal structure diagram of the computer device can be as shown in Figure 10 The computer device includes a processor, a memory, a network interface and a database connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store elements of a poster image to be generated. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is configured to be executed by the processor to implement a poster generation method.
[0070] In one embodiment, a computer device is provided, which includes a memory, a processor and a computer program stored in the memory and executable on the processor. The processor is configured to implement the poster generation method in the above embodiments when executing the computer program, for example, Figure 2 S201-S203 shown in Figures 3 to 8 For brevity, details are not repeated here. Alternatively, the processor is configured to implement the functions of the modules / units in the poster generation apparatus embodiment when executing the computer program, for example, Figure 9 the functions of the feature encoding module 91, the layout generation module 92 and the poster generation module 93 shown in
[0071] In one embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is configured to be executed by a processor to implement the poster generation method in the above embodiments, for example, Figure 2 S201-S203 shown in Figures 3 to 8 For brevity, details are not repeated here. Alternatively, the computer program is configured to implement the functions of the modules / units in the poster generation apparatus embodiment when executed by the processor, for example, Figure 9 the functions of the feature encoding module 91, the layout generation module 92 and the poster generation module 93 shown in
[0072] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments of each method. Any reference to memory, storage, database or other medium used in each embodiment provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0073] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.
[0074] The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, but not to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.
Claims
1. A poster generation method, characterized in that: include: Obtain elements of a poster image to be generated, input all elements into a feature encoder of a target poster generation model, output element features of each element through the feature encoder, input all element features into a feature fusion device of the target poster generation model, and output fusion features of all elements through the feature fusion device; Inputting the fused features into the first layout transformer of the target poster generation model, outputting the regional division result of the layout space required for all elements through the first layout transformer, inputting the fused features and the regional division result into the graph neural network module of the target poster generation model, and outputting the regional grouping result of all elements through the graph neural network module; The region grouping result is input into the second layout converter of the target poster generation model, and the region layout results of all elements are output through the second layout converter. The region division result, the region grouping result and the region layout result are input into the gating adjustment module of the target poster generation model, and the target layout results of all elements are output through the gating adjustment module. According to the target layout result, a target poster image is generated.
2. The poster generation method according to claim 1, wherein: The elements include visual elements and text elements, inputting all elements into a feature encoder of a target poster generation model, outputting element features of each element through the feature encoder, inputting all element features into a feature fusion device of the target poster generation model, and outputting fusion features of all elements through the feature fusion device, including: Input all visual elements into the visual feature encoder of the target poster generation model, and output the visual feature of each visual element through the visual feature encoder; Input all text elements into the text feature encoder of the target poster generation model, and output the text feature of each text element through the text feature encoder; Input all elements into the geometric feature encoder of the target poster generation model, and output the geometric features of each element through the geometric feature encoder; All visual features, text features and geometric features are input into the feature fusion device of the target poster generation model, and the fused features are output through the feature fusion device.
3. The poster generation method according to claim 1, wherein: Inputting the fused features into a first layout transformer of the target poster generation model, and outputting a result of dividing the layout space required for all elements through the first layout transformer, includes: Inputting the fused features into an encoder in the first layout transformer, and outputting aggregated features representing the attention relationships between all elements through the encoder; The aggregated features are input into a first multi-layer perceptron module in the first layout transformer, and the first multi-layer perceptron module outputs the spatial regions into which the layout space is divided and the region parameters of the corresponding spatial regions, and all the spatial regions and the region parameters of the corresponding spatial regions form the region division result.
4. The poster generation method according to claim 3, wherein: The step of inputting the fusion features and the region division results into the graph neural network module of the target poster generation model, and outputting the region grouping results of all elements through the graph neural network module, includes: Inputting the fusion features and the region division results into a graph construction submodule in the graph neural network module, and outputting an element relationship graph representing the relationship between all elements through the graph construction submodule, wherein each node in the element relationship graph represents an element and the attribute features of the corresponding element, and each edge represents the spatial relationship between the elements; For an element represented by any node in the element relationship graph, calculating the attention weight between the element and any element other than the element according to the attribute characteristics of the element and the attribute characteristics of each element other than the element; Determine the number of all spatial regions in the region division result, and select the number of elements from all elements as initial cluster centers of the corresponding spatial regions according to the number; All elements are clustered according to the attention weights between all elements, all initial cluster centers and a preset threshold to obtain cluster groups corresponding to the spatial regions, and the cluster groups of all spatial regions are formed into the regional grouping result.
5. The poster generation method according to claim 4, wherein: Inputting the region grouping result into the second layout converter of the target poster generation model, and outputting the region layout results of all elements through the second layout converter, includes: Inputting the region grouping result into the second layout converter, and determining, for any element in any cluster group, neighboring elements of the element from all elements in the cluster group; updating the attribute feature of the element according to the attribute feature of the element, the attribute feature of each neighboring element, and the attention weight between the element and each neighboring element to obtain an updated attribute feature; The updated attribute features are input into the second multi-layer perceptron module in the second layout transformer, and the geometric adjustment amounts of the elements in the corresponding spatial area are output through the second multi-layer perceptron module, so that the geometric adjustment amounts of all elements in the corresponding spatial area form the regional layout result.
6. The poster generation method according to claim 1, wherein: The step of inputting the region division result, the region grouping result, and the region layout result into a gate adjustment module of the target poster generation model, and outputting target layout results of all elements through the gate adjustment module, comprises: Inputting the region division result and the region grouping result into a first adjustment submodule in the gate adjustment module, and outputting a first correction coefficient of the region division result to the region grouping result through the first adjustment submodule; Inputting the region division result and the region grouping result into a second adjustment submodule in the gate adjustment module, and outputting a second correction coefficient of the region grouping result to the region division result through the second adjustment submodule; The target layout result is obtained according to the first correction coefficient, the second correction coefficient and the regional layout result.
7. The poster generation method according to claim 6, wherein: Obtaining the target layout result according to the first correction coefficient, the second correction coefficient, and the regional layout result includes: Adjusting the area division result and the area grouping result according to the first correction coefficient and the second correction coefficient to obtain an updated area division result and an updated area grouping result; The update region division result, the update region grouping result and the region layout result are input into a fusion submodule in the gate adjustment module, and the target layout result is output through the fusion submodule.
8. A poster generating device, characterized in that: include: A feature encoding module is used to obtain elements of the poster image to be generated, input all elements into a feature encoder of a target poster generation model, output element features of each element through the feature encoder, input all element features into a feature fusion device of the target poster generation model, and output fusion features of all elements through the feature fusion device; a layout generation module, configured to input the fused features into a first layout transformer of the target poster generation model, output a regional division result of the layout space required for all elements through the first layout transformer, input the fused features and the regional division result into a graph neural network module of the target poster generation model, and output a regional grouping result of all elements through the graph neural network module; A poster generation module is used to input the area grouping result into the second layout converter of the target poster generation model, output the area layout results of all elements through the second layout converter, input the area division result, the area grouping result and the area layout result into the gate adjustment module of the target poster generation model, output the target layout results of all elements through the gate adjustment module, and generate a target poster image based on the target layout result.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the poster generating method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the poster generating method according to any one of claims 1 to 7 is implemented.