Poster generation method and device, equipment and medium

Abstract

The invention discloses a poster generation method and device, equipment and a medium, and the method comprises the steps: inputting elements of a poster image to be generated into a feature encoder of a target poster generation model, outputting the element features of each element, inputting all the element features into a feature fusion device of the target poster generation model, outputting the fusion features of all the elements, and outputting the fusion features of all the elements; inputting the fusion features into a graph neural network module of a target poster generation model, outputting a target relation graph representing relations among all elements, respectively inputting the target relation graph into a first layout generator and a second layout generator of the target poster generation model, generating a first type of layout and a second type of layout, and fusing the first type of layout and the second type of layout to obtain a target poster image, the method and the device can be applied to a poster propaganda generation system in the business fields of financial science and technology, medical health and the like, and the quality of generated posters is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of poster design and the field of financial technology, and in particular to a poster generation method, device, equipment and medium. BACKGROUND

[0002] As an important form of visual communication, poster design, especially in the business fields of financial technology, medical health and old-age care, for example, in the financial technology scene, posters often need to carry complex and critical information, such as the characteristics of financial products, the yield situation, risk prompts, etc. These information needs to be ensured that users can quickly and accurately obtain key content through reasonable visual levels and professional layout design. In the medical health and old-age care scene, poster design not only needs reasonable visual levels and professional layout design, but also needs to take into account the readability of old users. Therefore, the quality of poster design directly affects the effect of information transmission and user experience.

[0003] Traditional poster design usually needs to rely on manual layout adjustment by designers, and such a way usually has problems such as low efficiency, high cost, and difficulty in quantitative evaluation of quality. In recent years, with the application of deep learning in automatic poster layout, the generation model based on neural network can learn the layout rules from a large number of design samples, and then realize the automatic generation of posters. However, in actual application, due to the complexity of poster design elements and the limitation of model performance, there are still many challenges in realizing high-quality poster generation, which makes the final output effect difficult to meet the actual demand. Therefore, how to improve the quality of poster generation has become a problem to be solved. SUMMARY

[0004] The embodiments of the present application provide a poster generation method, device, computer equipment and storage medium to solve the problem of how to improve the quality of poster generation.

[0005] A poster generation method comprises: Obtaining elements of a poster image to be generated, inputting all elements into a feature encoder of a target poster generation model, outputting element features of each element through the feature encoder, inputting all element features into a feature fusioner of the target poster generation model, and outputting fusion features of all elements through the feature fusioner; Inputting the fusion features into a graph neural network module of the target poster generation model, and outputting a target relationship graph representing the relationship between all elements through the graph neural network module; Inputting the target relationship graph into a first layout generator and a second layout generator of the target poster generation model respectively, generating a first type of layout through the first layout generator, generating a second type of layout through the second layout generator, and performing layout fusion on the first type of layout and the second type of layout to obtain a target poster image.

[0006] A poster generation apparatus, comprising: a feature encoding module configured to obtain elements of a poster image to be generated, input all the elements into a feature encoder of a target poster generation model, output element features of each element through the feature encoder, input all the element features into a feature fusioner of the target poster generation model, and output fusion features of all the elements through the feature fusioner; a graph construction module configured to input the fusion features into a graph neural network module of the target poster generation model, and output a target relation graph representing relations among all the elements through the graph neural network module; a poster generation module configured to input the target relation graph into a first layout generator and a second layout generator of the target poster generation model respectively, generate a first type of layout through the first layout generator, generate a second type of layout through the second layout generator, and perform layout fusion on the first type of layout and the second type of layout to obtain a target poster image.

[0007] A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the poster generation method when executing the computer program.

[0008] A computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executable by a processor to implement the poster generation method.

[0009] The poster generation method, apparatus, computer device, and storage medium, by inputting elements of a poster image to be generated into a feature encoder of a target poster generation model, obtaining element features of each element, inputting all the element features into a feature fusioner of the target poster generation model, obtaining fusion features of all the elements, inputting the fusion features into a graph neural network module of the target poster generation model, obtaining a target relation graph, inputting the target relation graph into a first layout generator and a second layout generator of the target poster generation model respectively, obtaining a first type of layout and a second type of layout, and performing fusion on the first type of layout and the second type of layout to obtain a target poster image.

[0010] The graph neural network module constructs a relation graph representing relations among elements according to the extracted and fused element features, realizes dynamic interactive modeling of the relations among the elements, adaptively captures and optimizes the relations among the elements, and thus, when the double layout generator generates layouts based on the relation graph, the quality of each layout can be improved according to the relations among the elements on the basis of generating diversified layouts, and when a plurality of layouts are fused to generate a target poster image, the overall quality of the generated poster is improved. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments of the present application will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0012] Figure 1 is a schematic diagram of an application environment of a poster generation method in an embodiment of the present application; Figure 2 is a flowchart of a poster generation method in an embodiment of the present application; Figure 3 is an architectural diagram of a target poster generation model in an embodiment of the present application; Figure 4 is another flowchart of a poster generation method in an embodiment of the present application; Figure 5 is another flowchart of a poster generation method in an embodiment of the present application; Figure 6 is another flowchart of a poster generation method in an embodiment of the present application; Figure 7 is another flowchart of a poster generation method in an embodiment of the present application; Figure 8 is a schematic diagram of a poster generation device in an embodiment of the present application; Figure 9 is a schematic diagram of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0013] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0014] The poster generation method provided by the embodiments of the present application can be applied in an application environment as shown in Figure 1 . Specifically, the poster generation method is applied in a poster generation system, and the poster generation system includes a poster generation device as shown in Figure 1The illustrated client and server communicate through a network to solve the problem of how to improve the quality of poster generation. Among them, the client, also known as the user end, is a program that provides local services to the client corresponding to the server. The client can be installed on, but not limited to, various personal computers, notebook computers, smartphones, tablet computers and portable wearable devices. The server can be implemented by a stand-alone server or a server cluster composed of multiple servers.

[0015] For example, in the poster generation task in the financial technology field, it is necessary to generate a poster based on visual elements: dynamic yield curve graph, blockchain network icon, security shield icon, and text elements: "intelligent financial management, stable income", "funds are stored by banks". At this time, the poster generation method of the application can be used to automatically generate a financial technology poster image that not only meets the aesthetic characteristics but also accurately expresses the semantics. For example, the yield curve graph is placed as the main visual in the upper left corner, the "intelligent financial management" title is right-aligned with the yield graph, the blockchain icon is embedded in the curve graph background as a symbol of technology, the security shield icon is combined with the "funds are stored" text and placed at the bottom, and the risk prompt is fixed at the bottom of the poster.

[0016] In an embodiment, as Figure 2 shown, a poster generation method is provided. Taking the server in Figure 1 as an example, the method includes the following steps: Step S201: Obtain the elements of the poster image to be generated, input all the elements into the feature encoder of the target poster generation model, output the element features of each element through the feature encoder, input all the element features into the feature fusioner of the target poster generation model, and output the fusion features of all the elements through the feature fusioner.

[0017] In this embodiment, the elements of the poster image to be generated can refer to independent visual units in the poster image. For example, the elements can include financial text, medical popular science text, financial data images, medical images, technology element graphics, and health element graphics, etc. The target poster generation model can refer to a neural network model that has been trained for poster image generation. The feature encoder can refer to an encoder for extracting features of the elements. The element features can refer to vectors or abstract descriptions representing the visual, semantic or structural features of the elements. The feature fusioner can refer to a module for fusing the element features of all the elements. The fusion features can refer to the features after fusing the element features of all the elements.

[0018] Specifically, elements of visual, textual, speech and other data types of the poster image to be generated are obtained, all elements are input into the feature encoder of the corresponding data type of the target poster generation model, the elements are extracted through the feature encoder to obtain the element features of each element, the element features of all elements are input into the feature fuser, the feature fuser is used to fuse all element features to obtain the fusion features of all elements.

[0019] Step S202: Input the fused features into the graph neural network module of the target poster generation model, and output the target relationship graph representing the relationship between all elements through the graph neural network module.

[0020] Step S203: Input the target relationship graph into the first layout generator and the second layout generator of the target poster generation model respectively, generate a first type of layout through the first layout generator, generate a second type of layout through the second layout generator, and perform layout fusion on the first type of layout and the second type of layout to obtain the target poster image.

[0021] In this embodiment, the graph neural network module may refer to a neural network used to model and optimize the spatial, semantic or style relationships among all elements, the target relationship graph may refer to a topological structure graph that characterizes the relationships (such as semantics, space) among all elements, the first layout generator may refer to a module used to generate a poster layout that conforms to aesthetic rules, the second layout generator may refer to a module used to generate a poster layout that conforms to semantic coherence, the first type of layout may refer to a layout that conforms to aesthetic rules, the second type of layout may refer to a layout that conforms to semantic coherence, and the target poster image may refer to a poster image generated based on all elements by the target poster generation model.

[0022] Specifically, the fusion features are input into the graph neural network module of the target poster generation model, and a target relationship graph representing the relationship between all elements is output. The target relationship graph is input into the first layout generator to obtain a first type of layout. The target relationship graph is input into the second layout generator to obtain a second type of layout. The first type of layout and the second type of layout are weightedly fused, and the fused layout is rendered to obtain the target poster image.

[0023] For example, Figure 3 As shown in FIG. 1 , a schematic diagram of a target poster generation model of this embodiment is shown. The overall process of generating the poster can be: The visual elements are input into a visual tokenizer for feature extraction to obtain visual features, the text elements are input into a text tokenizer for feature extraction to obtain text features, and the visual elements and the text elements are input into a position encoder for feature extraction to obtain position features; 2) the visual features, the text features, and the position features are input into a feature fusioner for feature fusion to obtain fused features; 3) the fused features are input into a graph constructor to construct an initial relationship graph, and the initial relationship graph is input into a graph attention subnetwork to update a target relationship graph; 4) the target relationship graph is input into a first layout generator and a second layout generator respectively, a first type of layout conforming to an aesthetic rule is obtained through the first layout generator, a second type of layout conforming to semantic coherence is obtained through the second layout generator, and the first type of layout and the second type of layout are input into a layout fusioner for fusion to obtain a target poster image.

[0024] In the embodiment, the relationship graph representing the relationship between the elements is constructed by the graph neural network module according to the extracted and fused element features, the dynamic interaction modeling between the elements is realized, the relationship between the elements is adaptively captured and optimized, and therefore when the layout is generated based on the relationship graph by the double layout generator, the quality of each layout can be improved according to the relationship between the elements on the basis of the diversified layout, and when the target poster image is generated by fusing the multiple layouts, the overall quality of the generated poster is improved.

[0025] In an embodiment, as shown in FIG. 4, a poster generation method is provided, and the elements include visual elements and text elements. Figure 4 In the step S201, all the elements are input into a feature encoder of a target poster generation model, the element features of each element are output by the feature encoder, all the element features are input into a feature fusioner of the target poster generation model, and the fused features of all the elements are output by the feature fusioner. Step S401: All the visual elements are input into a visual feature encoder of a target poster generation model, and the visual features of each visual element are output by the visual feature encoder.

[0026] Step S402: All the text elements are input into a text feature encoder of a target poster generation model, and the text features of each text element are output by the text feature encoder.

[0027] Step S403: All the elements are input into a position feature encoder of a target poster generation model, and the position features of each element are output by the position feature encoder.

[0028] Step S404: All the visual features, the text features, and the position features are input into a feature fusioner of a target poster generation model, and the fused features are output by the feature fusioner.

[0029] In this embodiment, the visual element can refer to an element in the poster that transmits information through visual media such as images, images, etc. For example, fund yield curve, insurance coverage radar chart, and disease interpretation diagram, etc. The text element can refer to an element in the poster that transmits information through language and text, such as insurance promotion slogans, investment risk tips, and disease popular science scripts, etc. The visual feature encoder can refer to an encoder for extracting features of visual elements. The text feature encoder can refer to an encoder for extracting features of text elements. The position encoder can refer to an encoder for extracting spatial features (such as position, size) of elements. The visual feature can refer to an abstract description of the characteristics of the visual element. The text feature can refer to an abstract description of the characteristics of the text element. The position feature can refer to an abstract description of the spatial characteristics of the element.

[0030] Specifically, for any visual element (such as picture, shape, etc.), first segment the visual element to obtain an image block of a specific size, and then extract global and local features of each visual element through the visual feature encoder to obtain the visual feature of each visual element , wherein, represents the visual feature of the i-th visual element, is the dimension of the visual feature, and n is the number of visual elements.

[0031] For any text element, first perform word segmentation processing on the text element to obtain a word of a specific size, and then extract semantic features of each text element through the text feature encoder to obtain the text feature of each text element , wherein, represents the text feature of the j-th text element, is the dimension of the text feature, and m is the number of text elements.

[0032] For any element, extract the spatial features (such as position, size) of each element through the position feature encoder, wherein the position feature encoder adopts a combination of sine function and pre-function to extract absolute and relative position information of the element to obtain the position feature of each element , wherein, represents the spatial feature of the k-th element, is the dimension of the spatial feature.

[0033] Input all visual features, text features, and position features into the feature fusioner of the target poster generation model, and output the fusion features through the feature fusioner.

[0034] In this embodiment, feature extraction is performed on multimodal visual elements and text elements through a visual feature encoder, a text feature encoder, and a position feature encoder, and multimodal feature fusion is performed based on a feature fuser to obtain fused features, thereby achieving refined classification extraction of multimodal features, improving the accuracy of feature extraction, and providing an accurate data basis for the subsequent generation of high-quality poster images, thereby also improving the quality of generated posters.

[0035] In one embodiment, if Figure 5 As shown, a poster generation method is provided. In the above step S404, all visual features, text features and position features are input into the feature fusion device of the target poster generation model, and the fusion features are output through the feature fusion device, including the following steps: Step S501: Input all visual features, text features and position features into the feature fusion device of the target poster generation model, map all visual features, text features and position features to the same feature dimension, and splice all visual features, text features and position features of the same feature dimension to obtain joint features.

[0036] Step S502: Determine the number of attention heads based on the feature dimension, and The features are linearly transformed to obtain the query matrix, key matrix and value matrix. According to the number of attention heads, the query matrix, key matrix and value matrix are split respectively to obtain the query sub-matrix, key sub-matrix and value sub-matrix.

[0037] Step S503: Form an attention head with any query sub-matrix, key sub-matrix and value sub-matrix. For any attention head, calculate the attention weights between all elements, and concatenate the attention weights between all elements corresponding to all attention heads to obtain a fusion feature.

[0038] In this embodiment, the joint feature may refer to a feature obtained by concatenating visual features, text features, and position features of the same feature dimension.

[0039] Specifically, all visual features , , text features , , and location features , In the input feature fusion device, all visual features, text features and position features are mapped to the same feature dimension and spliced ​​to obtain the joint feature , perform linear transformation on the joint features to obtain the query matrix , key matrix Sum Matrix ,in, 、 and is a learnable parameter matrix. , , The feature dimension of is evenly divided into h parts (h is an integer greater than or equal to 1), to obtain query sub-matrix , key sub-matrix and value sub-matrix , any query sub-matrix, key sub-matrix and value sub-matrix form an attention head, and the attention weight between all elements is calculated for any attention head . The attention weights between all elements corresponding to all attention heads are spliced to obtain the fused feature , is a learnable parameter matrix.

[0040] In this embodiment, the attention weight between elements is calculated through the multi-head attention mechanism, and the interaction relationship between elements in different subspaces is captured in parallel, avoiding the limitations of single attention mechanism. The fused feature generated by the final weighting encodes both the local details of the elements and the global features of all elements, improving the accuracy of feature fusion and thus improving the quality of the generated poster.

[0041] In an embodiment, as shown in Figure 6 , a poster generation method is provided, and the fused feature is input into the graph neural network module of the target poster generation model in step S202, and a target relationship graph representing the relationship between all elements is output through the graph neural network module, including the following steps. Step S601: input the fused feature into the graph constructor in the graph neural network module, and output an initial relationship graph through the graph constructor.

[0042] Step S602: input the initial relationship graph into the graph attention subnetwork in the graph neural network module, update the initial feature represented by each node in the initial relationship graph through the graph attention subnetwork, and output a target relationship graph.

[0043] In this embodiment, the graph constructor can refer to a network module for modeling the relationship between all elements, the initial relationship graph can refer to a graph structure representing the relationship between all elements based on the graph constructor, each node in the initial relationship graph represents an element and the initial feature of the corresponding element, the initial feature can refer to the element feature of the corresponding element, each edge represents the spatial relationship between elements, the graph attention subnetwork can refer to a network module for updating the initial feature of the node in the initial relationship graph, and the target relationship graph can refer to a graph structure obtained by updating the initial feature of the node in the initial relationship graph based on the graph attention subnetwork.

[0044] Specifically, the fusion features are input into the graph constructor in the graph neural network module, and the initial relationship graph is output. , where the node Represents elements, edges Representing the spatial relationship between elements, the initial relationship graph is input into the graph attention sub-network, and the initial features of each node in the initial relationship graph are updated through the message passing mechanism to obtain the target relationship graph.

[0045] Optionally, in the process of updating the initial features representing each node in the initial relationship graph to obtain the target relationship graph, the initial relationship graph can be input into the input layer of the graph attention sub-network, and for any node in the initial relationship graph, the neighbor nodes of the node are determined, and the attention coefficient between the node and any neighbor node is calculated according to the initial features of the node and the initial features of the node's neighbor nodes. The updated features of the node are obtained according to the initial features of each neighbor node and the attention coefficient between the node and each neighbor node. The initial relationship graph is updated according to the updated features of all nodes to obtain an updated relationship graph, and the updated relationship graph is input into the next layer, and the updated features of all nodes are set to the initial features of the corresponding nodes. The updated relationship graph is the initial relationship graph, and the step of determining the neighbor nodes of the node for any node in the initial relationship graph is returned to execute until the output layer of the graph attention sub-network outputs the target relationship graph.

[0046] Among them, the updated feature may refer to the feature after the initial feature is updated based on the attention coefficient, and the updated relationship graph may refer to the graph structure obtained by updating the initial relationship graph based on the updated features of all nodes.

[0047] Specifically, any node in the initial relationship graph is , any neighbor node of this node is ,node The initial feature of any layer in the graph attention sub-network is ,node The initial features of this layer are , then calculate the node in this layer Its neighboring nodes The calculation formula of the attention coefficient between can be: ,in, is a learnable vector, T is vector transpose, || is vector concatenation, according to the node With neighboring nodes Attention coefficient and neighbor nodes The initial characteristics , computing nodes Update features in this layer The calculation formula can be: ,in, representing nodes of a neighbor set, is a learnable weight matrix, is an activation function, the initial relation graph is updated according to the updated features of all nodes in the layer, an updated relation graph is obtained, the updated relation graph is input into the next layer, the updated relation graph is taken as a new initial relation graph, the updated features of the nodes in the updated relation graph are initial features of the corresponding nodes, the initial features of the nodes are iteratively updated in the next layer according to the calculation formula of the attention coefficient and the calculation formula of the updated feature, and the output layer of the graph attention subnetwork outputs an updated target relation graph.

[0048] In the embodiment, the initial relation graph representing the relationship between elements is constructed based on the fused features by the graph constructor, the initial features of the nodes in the initial relation graph are updated to obtain a target relation graph based on a multi-layer message passing mechanism by the graph attention subnetwork, the local neighbor information and the global context information of the nodes are gradually aggregated, and the relationship between elements is adaptively captured and optimized, so that the overall quality of the generated poster is improved when the poster is generated based on the target relation graph.

[0049] In an embodiment, as shown in Figure 7 , a poster generation method is provided, and in step S203, the target relation graph is input into the first layout generator of the target poster generation model, and a first type of layout is generated by the first layout generator, including the following steps. Step S701: input the target relation graph into the first layout generator, and map the target relation graph into a mean vector and a variance vector by the encoder in the first layout generator.

[0050] Step S702: input all the mean vectors and variance vectors into the decoder in the first layout generator, output the layout coordinates of all elements by the decoder in the first layout generator, and obtain the first type of layout according to the layout coordinates of all elements.

[0051] In the embodiment, the layout coordinates can refer to the coordinates of the elements in the poster image predicted based on the first layout generator.

[0052] Specifically, the target relation graph is input into the first layout generator, the target relation graph is mapped into a mean vector and a variance vector of a Gaussian distribution by the encoder (such as a conditional variational autoencoder) of the first layout generator, wherein and are encoder networks, H is the updated feature of the node in the target relation graph, a latent variable is sampled from the distribution, and is input into the decoder, and the layout coordinates of all elements are decoded and output , according to the layout coordinates of all elements, the first type of layout is obtained.

[0053] Optionally, in the above step S203, the target relationship graph is input into the second layout generator of the target poster generation model. In the process of generating the second type of layout by the second layout generator, the target relationship graph can be input into the second layout generator, a first node is determined from the target relationship graph, the first node is input into the decoder in the second layout generator, and the layout coordinates of the element corresponding to the first node are output through the decoder in the second layout generator. All layout coordinates corresponding to the elements in the current target relationship graph are determined, a second node is determined from the target relationship graph, all layout coordinates corresponding to the second node and the elements in the current target relationship graph are input into the decoder in the second layout generator, and the layout coordinates of the element corresponding to the second node are output through the decoder in the second layout generator. Return to the step of determining all layout coordinates corresponding to the elements in the current target relationship graph, until all nodes in the target relationship graph are traversed, the layout coordinates of all elements in the target relationship graph are obtained, and the second type of layout is obtained based on the layout coordinates of all elements.

[0054] The first node may refer to any node in the target relationship graph, and the second node may refer to a node other than the first node for which layout coordinates have not been generated.

[0055] For example, if the target relationship graph contains three nodes, 、 and , input the target relationship graph into the second layout generator, and determine a first node from the target relationship graph , the node The node features represented are input into the decoder to obtain the node The layout coordinates of the corresponding element , determine a second node from the target relationship graph , the node in the current target relationship graph Layout coordinates of and nodes The node features represented are input into the decoder to obtain the node The layout coordinates of the corresponding element , determine a second node from the target relationship graph , the node in the current target relationship graph Layout coordinates of ,node Layout coordinates of and nodes The node features represented are input into the decoder to obtain the node The layout coordinates of the corresponding element , according to the node The layout coordinates of the corresponding element ,node The layout coordinates of the corresponding element and nodes The layout coordinates of the corresponding element , and obtain the second type of layout.

[0056] In this embodiment, a first layout generator is used to generate a first type of layout that conforms to aesthetic rules, and a second layout generator is used to generate a second type of layout that conforms to semantic coherence, so that a target poster image is obtained based on the fusion of the first type of layout and the second type of layout. Through the collaborative work of the two layout generators, dual optimization of aesthetics and semantics is achieved, ensuring that the generated poster not only conforms to the visual design but also can accurately convey information, thereby improving the quality of the generated poster.

[0057] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0058] In one embodiment, a poster generation device is provided, which corresponds to the poster generation method in the above embodiment. Figure 8 As shown, the poster generation device includes a feature encoding module 81, a graph construction module 82 and a poster generation module 83. The functional modules are described in detail as follows: A feature encoding module 81 is used to obtain elements of the poster image to be generated, input all elements into a feature encoder of a target poster generation model, output element features of each element through the feature encoder, input all element features into a feature fusion device of the target poster generation model, and output fusion features of all elements through the feature fusion device; A graph construction module 82 is configured to input the fusion features into a graph neural network module of the target poster generation model, and output a target relationship graph representing the relationships between all elements through the graph neural network module; The poster generation module 83 is used to input the target relationship diagram into the first layout generator and the second layout generator of the target poster generation model respectively, generate a first type of layout through the first layout generator, generate a second type of layout through the second layout generator, and perform layout fusion on the first type of layout and the second type of layout to obtain a target poster image.

[0059] Optionally, the feature encoding module 81 includes: A visual encoding unit, configured to input all visual elements into a visual feature encoder of the target poster generation model, and output a visual feature of each visual element through the visual feature encoder; a text encoding unit, configured to input all text elements into a text feature encoder of the target poster generation model, and output text features of each text element through the text feature encoder; a position encoding unit, configured to input all elements into a position feature encoder of the target poster generation model, and output position features of each element through the position feature encoder; a feature fusion unit, configured to input all visual features, text features, and position features into a feature fusioner of the target poster generation model, and output the fusion features through the feature fusioner.

[0060] Optionally, the feature fusion unit includes: a feature mapping subunit, configured to input all visual features, text features, and position features into the feature fusioner of the target poster generation model, map all visual features, text features, and position features to the same feature dimension, splice all visual features, text features, and position features in the same feature dimension to obtain joint features; a feature transformation subunit, configured to determine a number of attention heads according to the feature dimension, perform linear transformation on the joint features to obtain a query matrix, a key matrix, and a value matrix, and split the query matrix, the key matrix, and the value matrix according to the number of attention heads to obtain query submatrices, key submatrices, and value submatrices; a first calculation subunit, configured to form any query submatrix, key submatrix, and value submatrix into an attention head, calculate attention weights between all elements for any attention head, and splice the attention weights between all elements corresponding to all attention heads to obtain the fusion features.

[0061] Optionally, the graph construction module 82 includes: an initial construction unit, configured to input the fusion features into a graph constructor in the graph neural network module, and output an initial relation graph through the graph constructor, wherein each node in the initial relation graph represents an element and an initial feature of the corresponding element, and each edge represents a spatial relationship between elements; a graph updating unit, configured to input the initial relation graph into a graph attention subnetwork in the graph neural network module, update the initial feature represented by each node in the initial relation graph through the graph attention subnetwork, and output the target relation graph.

[0062] Optionally, the graph updating unit includes: The second calculation subunit is configured to input the initial relation graph into an input layer of the graph attention subnetwork, determine neighbor nodes of any node in the initial relation graph, and calculate an attention coefficient between the node and any neighbor node according to an initial feature of the node and initial features of the neighbor nodes of the node. The feature updating subunit is configured to obtain updated features of the node according to the initial features of each neighbor node and the attention coefficient between the node and each neighbor node, update the initial relation graph according to the updated features of all nodes to obtain an updated relation graph, and input the updated relation graph into a next layer. The first loop subunit is configured to return to execute the step of determining the neighbor nodes of the node in the initial relation graph until the output layer of the graph attention subnetwork outputs the target relation graph, with the updated features of all nodes being the initial features of the corresponding nodes and the updated relation graph being the initial relation graph.

[0063] Optionally, the poster generation module 83 includes: The graph mapping unit is configured to input the target relation graph into the first layout generator, and map the target relation graph into mean vectors and variance vectors through an encoder in the first layout generator. The first coordinate generation unit is configured to input all mean vectors and variance vectors into a decoder in the first layout generator, output layout coordinates of all elements through the decoder in the first layout generator, and obtain the first type of layout according to the layout coordinates of all elements.

[0064] Optionally, the poster generation module 83 includes: The second coordinate generation unit is configured to input the target relation graph into the second layout generator, determine a first node from the target relation graph, and input the first node into a decoder in the second layout generator to output layout coordinates of elements corresponding to the first node through the decoder in the second layout generator. The mask unit is configured to determine all layout coordinates corresponding to elements in the target relation graph, determine a second node from the target relation graph, and input the second node and all layout coordinates corresponding to elements in the target relation graph into the decoder in the second layout generator to output layout coordinates of elements corresponding to the second node through the decoder in the second layout generator. The second loop unit is configured to return to execute the step of determining all layout coordinates corresponding to elements in the target relation graph until all nodes in the target relation graph are traversed to obtain layout coordinates of all elements in the target relation graph, and obtain the second type of layout according to the layout coordinates of all elements.

[0065] The specific limitations of the poster generation apparatus can refer to the limitations of the poster generation method described above, which will not be repeated here. Each module in the above poster generation apparatus can be implemented by software, hardware, and a combination thereof, in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so that the processor can call and execute the operations corresponding to each of the above modules.

[0066] In one embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in Figure 9 The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is configured to store elements of a poster image to be generated. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement a poster generation method.

[0067] In one embodiment, a computer device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor implements the poster generation method in the above embodiments when executing the computer program, such as Figure 2 S201-S203, or Figures 3 to 7 The details will not be repeated here to avoid repetition. Alternatively, the processor implements the functions of the modules / units in the poster generation apparatus in this embodiment when executing the computer program, such as Figure 8 The functions of the feature encoding module 81, the graph construction module 82, and the poster generation module 83 shown in

[0068] In one embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by the processor to implement the poster generation method in the above embodiments, such as Figure 2 S201-S203, or Figures 3 to 7 The details will not be repeated here to avoid repetition. Alternatively, the computer program is executed by the processor to implement the functions of the modules / units in the above poster generation apparatus in this embodiment, such as Figure 8The functions of the feature encoding module 81, the graph construction module 82 and the poster generation module 83 shown are not repeated here for the sake of brevity.

[0069] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing relevant hardware through a computer program, the computer program can be stored in a non-volatile computer readable storage medium, and the computer program can include the processes of the above-mentioned embodiments when executed. Wherein, any reference to memory, storage, database or other medium used in each embodiment provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM) and the like.

[0070] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of functional units and modules is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0071] The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, but not to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A poster generation method, characterized in that: include: Obtain elements of a poster image to be generated, input all elements into a feature encoder of a target poster generation model, output element features of each element through the feature encoder, input all element features into a feature fusion device of the target poster generation model, and output fusion features of all elements through the feature fusion device; Inputting the fusion features into the graph neural network module of the target poster generation model, and outputting a target relationship graph representing the relationship between all elements through the graph neural network module; The target relationship graph is respectively input into the first layout generator and the second layout generator of the target poster generation model, a first type of layout is generated by the first layout generator, a second type of layout is generated by the second layout generator, and the first type of layout and the second type of layout are merged to obtain a target poster image.

2. The poster generation method according to claim 1, wherein: The elements include visual elements and text elements, inputting all elements into a feature encoder of a target poster generation model, outputting element features of each element through the feature encoder, inputting all element features into a feature fusion device of the target poster generation model, and outputting fusion features of all elements through the feature fusion device, including: Input all visual elements into the visual feature encoder of the target poster generation model, and output the visual feature of each visual element through the visual feature encoder; Input all text elements into the text feature encoder of the target poster generation model, and output the text feature of each text element through the text feature encoder; Input all elements into the position feature encoder of the target poster generation model, and output the position feature of each element through the position feature encoder; All visual features, text features and position features are input into the feature fusion device of the target poster generation model, and the fusion features are output through the feature fusion device.

3. The poster generation method according to claim 2, wherein: The step of inputting all visual features, text features, and position features into the feature fusion device of the target poster generation model and outputting the fused features through the feature fusion device includes: Inputting all visual features, text features, and position features into the feature fusion device of the target poster generation model, mapping all visual features, text features, and position features to the same feature dimension, and concatenating all visual features, text features, and position features of the same feature dimension to obtain a joint feature; Determining the number of attention heads according to the feature dimension, performing a linear transformation on the joint feature to obtain a query matrix, a key matrix, and a value matrix; and splitting the query matrix, the key matrix, and the value matrix according to the number of attention heads to obtain a query sub-matrix, a key sub-matrix, and a value sub-matrix. Any query sub-matrix, key sub-matrix and value sub-matrix form an attention head, for any attention head, calculate the attention weights between all elements, and splice the attention weights between all elements corresponding to all attention heads to obtain the fusion feature.

4. The poster generation method according to claim 1, wherein: Inputting the fusion features into the graph neural network module of the target poster generation model, and outputting a target relationship graph representing the relationship between all elements through the graph neural network module, includes: Inputting the fused features into a graph constructor in the graph neural network module, and outputting an initial relationship graph through the graph constructor, wherein each node in the initial relationship graph represents an element and the initial features of the corresponding element, and each edge represents the spatial relationship between the elements; The initial relationship graph is input into the graph attention subnetwork in the graph neural network module, and the initial features represented by each node in the initial relationship graph are updated through the graph attention subnetwork to output the target relationship graph.

5. The poster generation method according to claim 4, wherein: The step of inputting the initial relationship graph into the graph attention subnetwork in the graph neural network module, updating the initial features represented by each node in the initial relationship graph through the graph attention subnetwork, and outputting the target relationship graph includes: Inputting the initial relationship graph into the input layer of the graph attention subnetwork, determining the neighbor nodes of any node in the initial relationship graph, and calculating the attention coefficient between the node and any neighbor node based on the initial features of the node and the initial features of the neighbor nodes of the node; Obtain an updated feature of the node based on the initial feature of each neighbor node and the attention coefficient between the node and each neighbor node; update the initial relationship graph based on the updated features of all nodes to obtain an updated relationship graph; and input the updated relationship graph into the next layer; The updated features of all nodes are set to the initial features of the corresponding nodes, the updated relationship graph is set to the initial relationship graph, and the step of determining the neighbor nodes of any node in the initial relationship graph is returned to execute until the output layer of the graph attention sub-network outputs the target relationship graph.

6. The poster generation method according to claim 1, wherein: Inputting the target relationship graph into a first layout generator of the target poster generation model, and generating a first type of layout by the first layout generator, comprises: Inputting the target relationship graph into the first layout generator, and mapping the target relationship graph into a mean vector and a variance vector through an encoder in the first layout generator; All mean vectors and variance vectors are input into the decoder in the first layout generator, the layout coordinates of all elements are output through the decoder in the first layout generator, and the first type of layout is obtained according to the layout coordinates of all elements.

7. The poster generation method according to claim 1, wherein: Inputting the target relationship graph into a second layout generator of the target poster generation model, and generating a second type of layout by the second layout generator, comprises: Inputting the target relationship graph into the second layout generator, determining a first node from the target relationship graph, inputting the first node into a decoder in the second layout generator, and outputting layout coordinates of an element corresponding to the first node through the decoder in the second layout generator; Determining all layout coordinates corresponding to elements in the current target relationship graph, determining a second node from the target relationship graph, inputting all layout coordinates corresponding to the second node and the elements in the current target relationship graph into a decoder in the second layout generator, and outputting the layout coordinates of the elements corresponding to the second node through the decoder in the second layout generator; Return to the step of determining all layout coordinates corresponding to the elements in the current target relationship graph until all nodes in the target relationship graph are traversed to obtain the layout coordinates of all elements in the target relationship graph, and obtain the second type of layout based on the layout coordinates of all elements.

8. A poster generating device, characterized in that: include: A feature encoding module is used to obtain elements of the poster image to be generated, input all elements into a feature encoder of a target poster generation model, output element features of each element through the feature encoder, input all element features into a feature fusion device of the target poster generation model, and output fusion features of all elements through the feature fusion device; A graph construction module, configured to input the fusion features into a graph neural network module of the target poster generation model, and output a target relationship graph representing the relationships between all elements through the graph neural network module; A poster generation module is used to input the target relationship graph into the first layout generator and the second layout generator of the target poster generation model respectively, generate a first type of layout through the first layout generator, generate a second type of layout through the second layout generator, and perform layout fusion on the first type of layout and the second type of layout to obtain a target poster image.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the poster generating method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the poster generating method according to any one of claims 1 to 7 is implemented.

Images