Training method, device, equipment and medium for poster generation model

Abstract

The present application belongs to the field of artificial intelligence and relates to a method for training a poster generation model, comprising: obtaining an annotated poster image, the poster image including position annotation information of layout elements; using an image network to extract global image feature information of the poster image; obtaining feature representation information of the layout elements, using a layout network to perform self-attention encoding on the feature representation information to obtain initial position information; using an interaction layer to perform feature interaction on the global image feature information and the initial position information to obtain layout element position information; based on the difference between the layout element position information and the position annotation information, respectively training the parameters of the image network, layout network, and interaction layer to construct a poster generation model. The present application also provides a device, an apparatus, and a medium. In addition, the present application also relates to blockchain technology, and the poster image and target parameters can be stored in the blockchain. The present application can realize the automation of poster layout.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method, apparatus, equipment and medium for a poster generation model. Background Art

[0002] In the field of image processing and generation, Stable Diffusion, as an advanced image generation algorithm, can gradually generate corresponding image content through multiple iterations based on text prompts. However, current image generation algorithms mainly focus on the direct generation of image content. Although they can produce rich visual content, they still have significant shortcomings in meeting specific design needs, such as poster production. Figure 1 Generating a complete image at one time is not enough for complex scenes such as poster design.

[0003] Poster design typically involves the meticulous arrangement and combination of multiple layout elements, rather than simply generating a single piece of content. Existing image generation algorithms have significant limitations in automating poster layout, making it difficult to generate a complete image that meets poster design requirements in a single pass. Therefore, effectively automating poster layout has become a pressing issue. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to propose a training method, device, equipment and medium for a poster generation model to solve the existing problem of difficulty in automating poster layout.

[0005] In order to solve the above technical problems, the present application provides a training method for a poster generation model, which adopts the following technical solutions:

[0006] A labeled poster image is obtained, where the poster image includes multiple layout element images and position annotation information of the layout elements of each layout element image; a preset image network is used to extract image features of the poster image to obtain global image feature information of the poster image; feature representation information of the layout elements is obtained, and a preset layout network is used to perform self-attention encoding on the feature representation information to obtain initial position information of the layout elements; a preset interaction layer is used to perform feature interaction on the global image feature information and the initial position information to obtain layout element position information of the layout elements; based on the difference between the layout element position information and the position annotation information, the parameters of the image network, the parameters of the layout network and the parameters of the interaction layer are trained respectively to obtain target parameters after training, and a poster generation model is constructed based on the target parameters, the image network, the layout network and the interaction layer.

[0007] Furthermore, the steps of extracting image features of the poster image using a preset image network to obtain global image feature information of the poster image specifically include:

[0008] A preset image segmentation algorithm is used to segment the poster image to obtain multiple image blocks; a preset image network is used to extract local image features of each image block to obtain multiple local image feature information; and multiple local image feature information are fused to obtain global image feature information of the poster image.

[0009] Furthermore, the step of obtaining the feature representation information of the layout element specifically includes:

[0010] Obtain category information and size information of the layout elements; embed the category information and size information to obtain position embedding information of the layout elements; use a preset feature encoding network to feature encode the layout element image to obtain image feature encoding of the layout element; encode the position embedding information and image feature as feature representation information of the layout elements.

[0011] Furthermore, the size information includes width information and height information. The step of embedding the category information and the size information to obtain the position embedding information of the layout element specifically includes:

[0012] A preset category embedding dictionary is used to map the category information into the corresponding category embedding vector, and the width information and height information are normalized to obtain a width vector corresponding to the width information and a height vector corresponding to the height information; the category embedding vector, the width vector and the height vector are concatenated to obtain a feature embedding vector of the layout element; a nonlinear transformation is performed on the feature embedding vector to obtain a hidden feature vector of the layout element, and the hidden feature vector is used to represent the abstract information of the category information and size information of the layout element; according to the position information of the layout element in the poster image, a preset position encoding function is used to generate a position embedding vector of the layout element; the hidden feature vector and the position embedding vector are added to obtain a layout element embedding vector of the layout element, and the layout element embedding vector is used to represent the content information and position information of the layout element; the layout element embedding vector is subjected to encoding enhancement processing to obtain the position embedding information of the layout element.

[0013] Furthermore, a preset layout network is used to perform self-attention encoding on the feature representation information to obtain the initial position information of the layout elements, specifically including:

[0014] A preset layout network is used to perform information splicing on the position embedding information and image feature encoding to obtain the position enhanced feature vector of the layout element; the self-attention mechanism of the layout network is used to perform self-attention processing on the position enhanced feature vector to obtain the element association feature vector of the layout element; the feedforward neural network of the layout network is used to perform nonlinear transformation and fusion on the element association feature vector to obtain the deep semantic features of the layout element; a preset coordinate prediction model is obtained, and based on the deep semantic features, the coordinate prediction model is used to predict the coordinates of the layout elements to obtain the predicted position coordinate values ​​of the layout elements, and the initial position information of the layout elements is determined based on the predicted position coordinate values.

[0015] Furthermore, the step of using a preset interaction layer to perform feature interaction on the global image feature information and the initial position information to obtain the layout element position information of the layout element specifically includes:

[0016] A preset interaction layer is used to convert the initial position information into a query vector, and the global image feature information into a key vector and a value vector. A dot product operation is performed on the key vector and the query vector to obtain an attention weight value, which is used to characterize the correlation between the initial position information and the global image feature information in the layout element. A weighted sum operation is performed on the value vector and the attention weight to obtain the layout element position information of the layout element.

[0017] Furthermore, based on the difference between the layout element position information and the position annotation information, the parameters of the image network, the parameters of the layout network, and the parameters of the interaction layer are trained to obtain the trained target parameters, which specifically includes:

[0018] The Euclidean distance between the layout element position information and the position annotation information is calculated, and based on the Euclidean distance, a position regression loss function is constructed. The position regression loss function is used to measure the degree of deviation between the layout element position indicated by the layout element position information and the annotation position indicated by the position annotation information; the intersection-over-union ratio of the layout element position information and the position annotation information is calculated, and based on the intersection-over-union ratio, a position classification loss function is constructed. The position classification loss function is used to measure the degree of overlap between the layout element position indicated by the layout element position information and the annotation position indicated by the position annotation information; the position regression loss function and the position classification loss function are weightedly summed to obtain the total loss function; based on the total loss function, the back propagation algorithm is used to update the parameters of the image network, the parameters of the layout network, and the parameters of the interaction layer respectively to obtain the trained target parameters.

[0019] Furthermore, after the step of constructing a poster generation model based on target parameters, image network, layout network, and interaction layer, the following steps are also included:

[0020] Obtain material information of the poster to be generated, which includes images and text prompt information; use the image network of the poster generation model to perform image feature extraction processing on the image to obtain the target global image feature information of the image; based on the text prompt information, generate target feature representation information of the layout elements of the poster to be generated; use the layout network of the poster generation model to perform self-attention encoding on the target feature representation information to obtain the target initial position information of the layout elements of the poster to be generated; use the interaction layer of the poster generation model to perform feature interaction on the target global image feature information and the target initial position information to obtain the target layout element position information of the layout elements of the poster to be generated; generate the poster to be generated based on the target layout element position information, image and text prompt information.

[0021] In order to solve the above technical problems, the embodiment of the present application further provides a training device for a poster generation model, which adopts the following technical solutions:

[0022] An image acquisition module is used to acquire an annotated poster image, where the poster image includes a plurality of layout element images and position annotation information of the layout elements of each layout element image;

[0023] A feature extraction module is used to extract image features of the poster image using a preset image network to obtain global image feature information of the poster image;

[0024] The encoding module is used to obtain the feature representation information of the layout elements. It uses a preset layout network to perform self-attention encoding on the feature representation information to obtain the initial position information of the layout elements.

[0025] A feature interaction module, configured to perform feature interaction on the global image feature information and the initial position information using a preset interaction layer to obtain layout element position information of the layout element;

[0026] The parameter training module is used to train the parameters of the image network, the layout network, and the interaction layer based on the differences between the layout element position information and the position annotation information, obtain the trained target parameters, and build a poster generation model based on the target parameters, the image network, the layout network, and the interaction layer.

[0027] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, it implements the steps of the training method of the poster generation model as described above.

[0028] In order to solve the above technical problems, an embodiment of the present application also provides a computer-readable storage medium, which stores computer-readable instructions. The computer-readable instructions can be executed by at least one processor to enable at least one processor to perform the steps of the training method of the poster generation model as mentioned above.

[0029] Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects: through refined data processing and deep learning model construction, the performance of the poster generation model is significantly improved. First, the use of annotated poster images as training materials ensures that the model can learn accurate layout element position information. The global image features are extracted through the preset image network, providing a rich visual context for the model. Then, the layout network uses the self-attention encoding mechanism to effectively capture the interdependence between layout elements and generate more reasonable initial position information. The introduction of the interaction layer further promotes the deep fusion of global image features and initial position information, making the generated layout element position information more accurate. During the training process, the model parameters are dynamically adjusted according to the difference between the layout element position information and the position annotation information, ensuring that the model can gradually approach the real layout. Finally, the poster generation model constructed based on the trained target parameters can realize the automation of poster layout and efficiently generate posters with reasonable layout. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the solutions in this application, a brief introduction will be given below to the drawings required for use in the description of the embodiments of this application. Obviously, the drawings described below are some embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0031] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0032] Figure 2 This is a flowchart of a training method for a poster generation model provided in this application;

[0033] Figure 3 This is a schematic diagram of the structure of a training device for a poster generation model provided in this application;

[0034] Figure 4 This is a structural diagram of a computer device provided by this application. DETAILED DESCRIPTION

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of the application are for the purpose of describing specific embodiments only and are not intended to limit this application. The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0036] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0037] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0038] like Figure 1 As shown, system architecture 100 may include a terminal device 101, a network 102, and a server 103. Terminal device 101 may be a laptop computer 1011, a tablet computer 1012, or a mobile phone 1013. Network 102 is a medium for providing a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0039] The user can use the terminal device 101 to interact with the server 103 via the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0040] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing. In addition to the laptop computer 1011, tablet computer 1012 or mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer and a desktop computer, etc.

[0041] The server 103 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal device 101 .

[0042] It should be noted that the training method of the poster generation model provided in the embodiment of the present application is generally executed by a server, and accordingly, the training device of the poster generation model is generally set in the server.

[0043] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0044] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for training a poster generation model according to the present application. The method for training a poster generation model comprises the following steps:

[0045] Step S201 : obtaining a poster image with annotations, where the poster image includes a plurality of layout element images and position annotation information of the layout elements of each layout element image.

[0046] In this embodiment, the training method of the poster generation model is run on the electronic device (e.g. Figure 1 The server shown in FIG. 1 may obtain the labeled poster image via a wired connection or a wireless connection. It should be noted that the wireless connection may include, but is not limited to, 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other currently known or future developed wireless connection methods.

[0047] The annotated poster image refers to a poster image that includes position annotation information, which can detail the position and attributes of each layout element in the poster image.

[0048] The multiple layout element images refer to the individual image elements that make up the poster image, such as text, icons, pictures, etc. These elements have different functions and visual effects in the poster design, and together constitute the complete poster content.

[0049] Among them, layout elements, in poster design, refer to the independent parts or components that make up the entire poster, such as title, text, pictures, decorative elements, etc. Each layout element has its specific position, size and category.

[0050] The position annotation information is information that details the precise location of each layout element in the poster image. It is usually expressed in the form of coordinates and is used to guide the model to learn the correct layout of layout elements.

[0051] Step S202 : Using a preset image network, extract image features of the poster image to obtain global image feature information of the poster image.

[0052] The image network refers to a deep learning network (e.g., an image transformer network) that is used to extract image features from the poster image. The image network can automatically learn visual features in the image.

[0053] Image features refer to abstract representations of the visual content of a poster image, such as edges, textures, colors, etc. In poster images, these features help the model understand the overall structure and content of the image.

[0054] Global image feature information refers to the feature information extracted by the image network that represents the overall visual content of the poster image. It contains the global structure and content information of the poster image.

[0055] Step S203: Acquire feature representation information of the layout elements, use a preset layout network to perform self-attention encoding on the feature representation information, and obtain initial position information of the layout elements.

[0056] The feature representation information refers to information that combines the category information, size information, and image feature encoding of each layout element in the poster image, and is used to comprehensively describe the attributes and features of the layout elements.

[0057] The layout network is a deep learning network specifically designed to process layout element features (e.g., a layout Transformer network). Through mechanisms such as self-attention encoding, the layout network can learn the spatial relationships and interactions between layout elements.

[0058] Among them, self-attention encoding refers to a deep learning technology that automatically learns the spatial relationships and dependencies between layout elements by calculating the correlation scores between layout elements, thereby generating a more accurate layout representation.

[0059] The initial position information refers to the preliminary predicted position information obtained after the layout network performs self-attention encoding on the feature representation information of the layout elements. It is the basis for subsequent position refinement.

[0060] Step S204 : Using a preset interaction layer, perform feature interaction on the global image feature information and the initial position information to obtain layout element position information of the layout element.

[0061] The interaction layer is a deep learning layer (e.g., a Cross Attention layer) that fuses global image feature information with initial layout element position information. Through feature interaction, the interaction layer can further refine the position information of layout elements.

[0062] Feature interaction refers to the interaction between the global features of the poster image (such as overall color and texture) and the features of the layout elements (such as category, size, and position). This interaction helps the model capture the complex relationships between features, thereby improving the accuracy of predicting the positions of layout elements.

[0063] The layout element position information refers to the final layout element position information obtained after processing by the interaction layer. It represents the model's prediction result for the position of each layout element in the poster image.

[0064] In step S205, based on the difference between the layout element position information and the position annotation information, the parameters of the image network, the parameters of the layout network, and the parameters of the interaction layer are trained respectively to obtain the trained target parameters. Based on the target parameters, the image network, the layout network, and the interaction layer, a poster generation model is constructed.

[0065] The difference refers to the difference between the layout element position information and the position annotation information, which is used to evaluate the accuracy of the model prediction.

[0066] The target parameters are the final parameters obtained after training and used to build the poster generation model. These target parameters include the model's understanding of the poster layout and image features, and are able to generate high-quality poster images.

[0067] Among them, the poster generation model refers to the use of trained target parameters, image network, layout network and interaction layer to build a complete model that can automatically generate poster images based on the input material information.

[0068] The embodiments of the present application can significantly improve the performance of the poster generation model through refined data processing and deep learning model construction. First, the use of annotated poster images as training materials ensures that the model can learn accurate layout element position information. Global image features are extracted through a preset image network, providing a rich visual context for the model. Then, the layout network uses the self-attention encoding mechanism to effectively capture the interdependence between layout elements and generate more reasonable initial position information. The introduction of the interaction layer further promotes the deep fusion of global image features and initial position information, making the generated layout element position information more accurate. During the training process, the model parameters are dynamically adjusted according to the difference between the layout element position information and the position annotation information, ensuring that the model can gradually approach the real layout. Finally, the poster generation model constructed based on the trained target parameters can realize the automation of poster layout and efficiently generate posters with a reasonable layout.

[0069] In some optional implementations of this embodiment, step 201, using a preset image network to extract image features of the poster image to obtain global image feature information of the poster image, specifically includes the following steps:

[0070] A preset image segmentation algorithm is used to segment the poster image to obtain multiple image blocks; a preset image network is used to extract local image features of each image block to obtain multiple local image feature information; and multiple local image feature information are fused to obtain global image feature information of the poster image.

[0071] In one example, a preset image segmentation algorithm, such as a superpixel-based segmentation algorithm, can be used to segment a poster image, such as a financial poster image. The algorithm can segment the poster image into multiple image blocks with similar attributes based on features such as color and texture. These image blocks may contain key information such as pictures, text descriptions, and price tags of financial products. Next, a preset image network, such as an image Transformer model, is used to extract local image features from each image block. The image Transformer model can capture the global dependencies between image blocks and extract local features with rich semantic information. The multiple extracted local image feature information is then fused. The fusion process can use methods such as weighted averaging, maximum pooling, or an attention mechanism to obtain global image feature information of the poster image. Global image feature information can reflect the overall style of the financial poster, the distribution of key information, and the correlation between financial products.

[0072] The embodiments of the present application can segment a poster image into multiple image blocks using a preset image segmentation algorithm, facilitating the detailed capture of local features. Using a preset image network, local image features of each image block can be efficiently extracted, fully mining the image information. Finally, by fusing multiple local image feature information, global feature information of the poster image is obtained, helping to capture the poster's overall style and key information distribution.

[0073] In some optional implementations of this embodiment, in step S203, obtaining feature representation information of the layout element specifically includes the following steps:

[0074] Obtain category information and size information of the layout elements; embed the category information and size information to obtain position embedding information of the layout elements; use a preset feature encoding network to feature encode the layout element image to obtain image feature encoding of the layout element; encode the position embedding information and image feature as feature representation information of the layout elements.

[0075] In one example, the category information (type of layout element, such as text, picture, logo, etc.) and size information (width, height, etc. of layout element) of the layout element can be obtained. This information can be obtained through image processing algorithms or user input. Then, the category information and size information are embedded. The embedding process can use the embedding technology in deep learning to map the category information and size information into a high-dimensional space to obtain position embedding information. The position embedding information can reflect the position and size relationship of the layout element in the overall poster image layout. Then, a preset feature encoding network is used to feature encode the layout element image to generate the image feature encoding of the layout element. The feature encoding network can be constructed by multiple convolutional layers, pooling layers and activation functions. Finally, the position embedding information and image feature encoding are used as feature representation information of the layout element.

[0076] The embodiments of the present application embed category and size information to obtain positional embedding information that accurately reflects the position and size relationship of layout elements within the overall layout. Furthermore, a pre-set feature encoding network is used to encode layout element images, efficiently extracting local and global features of the image and generating image feature encodings rich in semantic information.

[0077] In some optional implementations of this embodiment, the size information includes width information and height information. The step of "embedding the category information and the size information to obtain the position embedding information of the layout element" specifically includes the following steps:

[0078] A preset category embedding dictionary is used to map the category information into the corresponding category embedding vector, and the width information and height information are normalized to obtain a width vector corresponding to the width information and a height vector corresponding to the height information; the category embedding vector, the width vector and the height vector are concatenated to obtain a feature embedding vector of the layout element; a nonlinear transformation is performed on the feature embedding vector to obtain a hidden feature vector of the layout element, and the hidden feature vector is used to represent the abstract information of the category information and size information of the layout element; according to the position information of the layout element in the poster image, a preset position encoding function is used to generate a position embedding vector of the layout element; the hidden feature vector and the position embedding vector are added to obtain a layout element embedding vector of the layout element, and the layout element embedding vector is used to represent the content information and position information of the layout element; the layout element embedding vector is subjected to encoding enhancement processing to obtain the position embedding information of the layout element.

[0079] In one example, a preset category embedding dictionary can be used to map the category information of a layout element to a corresponding category embedding vector. The category embedding dictionary is a pre-trained vector space in which each category corresponds to a unique vector representation. By mapping the category information to the category embedding vector, the semantic relationship between categories can be captured. Next, the width and height information of the layout element are normalized to obtain a width vector and a height vector. Normalization eliminates dimensional differences between the dimensions, making the dimensions of different layout elements comparable. The category embedding vector, width vector, and height vector are then concatenated to obtain a feature embedding vector for the layout element. The feature embedding vector combines the category and size information of the layout element. Next, a preset fully connected neural network can be used to perform a nonlinear transformation on the feature embedding vector to obtain a hidden feature vector. The fully connected neural network can learn the complex relationships between the feature embedding vectors and extract feature representations with abstract meaning. The hidden feature vector represents the abstract information of the category and size information of the layout element. Furthermore, a preset position encoding function is used to generate a position embedding vector for the layout element based on the position information of the layout element in the poster image. The position embedding vector captures the spatial positional relationships of layout elements within the poster image. Finally, the hidden feature vector and the position embedding vector are added together to obtain the layout element embedding vector for the layout element. This layout element embedding vector can be input into the encoder of the layout Transformer network. Through self-attention calculation and feature transformation, the final embedding vector for the layout element is obtained, ultimately yielding the layout element's positional embedding information. This positional embedding information includes the category, size, and position information of each layout element, as well as its spatial relationship with other layout elements.

[0080] The embodiment of the present application can map category information into category embedding vectors through a preset category embedding dictionary, effectively capturing the semantic relationship between categories. At the same time, the width and height information are normalized and spliced ​​into feature embedding vectors, and a hidden feature vector is obtained by combining nonlinear transformation to abstractly represent the category and size information of the layout elements. The position embedding vector generated by the position encoding function is combined to further incorporate the spatial position information of the layout elements. Finally, the layout element embedding vector obtained by the encoding enhancement process comprehensively represents the content information and position information of the layout elements, providing more comprehensive and accurate feature support for subsequent poster image analysis, recognition or processing tasks.

[0081] In some optional implementations of this embodiment, step S203 uses a preset layout network to perform self-attention encoding on the feature representation information to obtain initial position information of the layout elements, which specifically includes the following steps:

[0082] A preset layout network is used to perform information splicing on the position embedding information and image feature encoding to obtain the position enhanced feature vector of the layout element; the self-attention mechanism of the layout network is used to perform self-attention processing on the position enhanced feature vector to obtain the element association feature vector of the layout element; the feedforward neural network of the layout network is used to perform nonlinear transformation and fusion on the element association feature vector to obtain the deep semantic features of the layout element; a preset coordinate prediction model is obtained, and based on the deep semantic features, the coordinate prediction model is used to predict the coordinates of the layout elements to obtain the predicted position coordinate values ​​of the layout elements, and the initial position information of the layout elements is determined based on the predicted position coordinate values.

[0083] In one example, a preset layout network, such as a layout Transformer network, can be used to concatenate position embedding information and image feature encoding to obtain a position-enhanced feature vector. The layout Transformer network's self-attention mechanism is then used to perform self-attention processing on the position-enhanced feature vector. By calculating the correlations between different layout elements, an element-association feature vector is obtained. This vector not only reflects the characteristics of each layout element itself but also incorporates information about its associations with other layout elements, making the feature representation richer and more comprehensive. Next, the layout Transformer network's feedforward neural network is used to perform nonlinear transformations and fusion on the element-association feature vectors to obtain deep semantic features of the layout elements. These deep semantic features represent high-level, abstract information about the layout elements within the overall layout, including their position, relationships, context, and function. Finally, a preset coordinate prediction model is obtained and used to predict the coordinates of the layout elements based on the deep semantic features. The predicted position coordinates are calculated to determine the initial position information of the layout elements.

[0084] The embodiment of the present application can fuse the position embedding information with the image feature encoding through the layout network to generate a position enhanced feature vector, which comprehensively enriches the feature representation of the layout elements. Utilizing the self-attention mechanism, the association information between the layout elements is deeply mined to form an element association feature vector, thereby enhancing the information content of the feature vector. After the nonlinear transformation and fusion of the feedforward neural network, the deep semantic features of the layout elements are extracted to accurately reflect their position, relationship, context and function in the overall layout. Finally, the coordinate prediction model based on the deep semantic features accurately predicts the position of the layout elements, providing strong support for layout optimization and intelligent design.

[0085] In some optional implementations of this embodiment, step S204, using a preset interaction layer to perform feature interaction on the global image feature information and the initial position information to obtain layout element position information of the layout element, specifically includes the following steps:

[0086] A preset interaction layer is used to convert the initial position information into a query vector, and the global image feature information into a key vector and a value vector. A dot product operation is performed on the key vector and the query vector to obtain an attention weight value, which is used to characterize the correlation between the initial position information and the global image feature information in the layout element. A weighted sum operation is performed on the value vector and the attention weight to obtain the layout element position information of the layout element.

[0087] In one example, a preset interaction layer (such as Cross Attention) can be used to fuse the initial position information and the global image feature information. Specifically, the initial position information can be converted into a query vector (Query), and the global image feature information can be converted into a key vector (Key) and a value vector (Value). This process utilizes the embedding technology in deep learning to map different types of information into the same high-dimensional space, which is convenient for subsequent calculations and processing. In the CrossAttention interaction layer, the dot product operation between the query vector and the key vector is first calculated to obtain the attention weight value. The purpose of this step is to measure the correlation between the initial position information of the layout element and the global image feature information, that is, to determine which global features have an important influence on the position information of the current layout element. Subsequently, the obtained attention weight value is used to perform a weighted sum operation on the value vector to obtain the layout element position information of each layout element.

[0088] The embodiments of the present application can achieve a deep fusion of layout element initial position information and global image feature information through a preset interaction layer. Specifically, the transformation of query vectors, key vectors, and value vectors, as well as the attention weight values ​​obtained by dot product operations, accurately characterize the correlation between layout element position information and global features. Through weighted summation operations, the global feature information most relevant to the layout element position is further integrated, significantly improving the accuracy and completeness of layout element position information.

[0089] In some optional implementations of this embodiment, in step S205, based on the difference between the layout element position information and the position annotation information, the parameters of the image network, the parameters of the layout network, and the parameters of the interaction layer are trained to obtain trained target parameters, which specifically includes the following steps:

[0090] The Euclidean distance between the layout element position information and the position annotation information is calculated, and based on the Euclidean distance, a position regression loss function is constructed. The position regression loss function is used to measure the degree of deviation between the layout element position indicated by the layout element position information and the annotation position indicated by the position annotation information; the intersection-over-union ratio of the layout element position information and the position annotation information is calculated, and based on the intersection-over-union ratio, a position classification loss function is constructed. The position classification loss function is used to measure the degree of overlap between the layout element position indicated by the layout element position information and the annotation position indicated by the position annotation information; the position regression loss function and the position classification loss function are weightedly summed to obtain the total loss function; based on the total loss function, the back propagation algorithm is used to update the parameters of the image network, the parameters of the layout network, and the parameters of the interaction layer respectively to obtain the trained target parameters.

[0091] In one example, a position regression loss function can be constructed by calculating the Euclidean distance between the layout element position information and the position annotation information. This position regression loss function can quantify the degree of deviation between the layout element position information and the annotated position, thereby guiding the model to learn how to more accurately predict the layout element position. At the same time, the intersection over union (IoU) of the layout element position information and the position annotation information is calculated to construct a position classification loss function. IoU is an effective indicator of the degree of overlap between two regions. Through this position classification loss function, the model can learn the spatial relationship between the layout element position and the annotated position, further improving the accuracy of position prediction. Then, the position regression loss function and the position classification loss function are weightedly summed to obtain the total loss function. Finally, based on the total loss function, the backpropagation algorithm is used to update the parameters of the image network, layout network, and interaction layer. Through multiple iterative training, the image network, layout network, and interaction layer gradually learn how to extract effective features from the image and accurately predict the position of layout elements.

[0092] The embodiments of the present application can construct a position regression loss function by calculating the Euclidean distance between the layout element position information and the position annotation information to accurately quantify the position deviation. At the same time, the intersection-over-union ratio is used to construct a position classification loss function to effectively evaluate the degree of position overlap. The total loss function formed by the combination of the two comprehensively guides the optimization of the image network, layout network, and interaction layer parameters. The backpropagation algorithm is used to iteratively update the parameters, significantly improving the accuracy of layout element position prediction, making the predicted position closer to the actual annotation, optimizing the layout effect, and improving the intelligent level of image processing and layout optimization.

[0093] In some optional implementations of this embodiment, after constructing the poster generation model based on the target parameters, the image network, the layout network, and the interaction layer in step S205, the following steps are further included:

[0094] Obtain material information of the poster to be generated, which includes images and text prompt information; use the image network of the poster generation model to perform image feature extraction processing on the image to obtain the target global image feature information of the image; based on the text prompt information, generate target feature representation information of the layout elements of the poster to be generated; use the layout network of the poster generation model to perform self-attention encoding on the target feature representation information to obtain the target initial position information of the layout elements of the poster to be generated; use the interaction layer of the poster generation model to perform feature interaction on the target global image feature information and the target initial position information to obtain the target layout element position information of the layout elements of the poster to be generated; generate the poster to be generated based on the target layout element position information, image and text prompt information.

[0095] In one example, a landscape image and a descriptive text can be selected as the source information for the poster to be generated. First, the image network of the poster generation model extracts features from the landscape image to obtain target global image feature information. This feature information comprehensively reflects the key content and overall style of the image. Next, target feature representation information for layout elements is generated based on the descriptive text. This information accurately expresses the key elements and layout requirements mentioned in the text. The layout network of the poster generation model then performs self-attention encoding on the target feature representation information to obtain target initial positions for the layout elements. Through the self-attention mechanism, the model learns the spatial relationships and dependencies between layout elements, thereby obtaining a reasonable initial layout. Subsequently, the interaction layer of the poster generation model performs feature interaction between the target global image feature information and the target initial position information. This interaction process further adjusts the positions of layout elements to better match the image content and overall style. Ultimately, target layout element positions are obtained. Finally, the poster to be generated is generated based on the target layout element position information, the landscape image, and the descriptive text.

[0096] The embodiment of the present application can perform deep feature extraction on the image by adopting the image network of the poster generation model, accurately capturing the key elements and overall style of the image. The layout network performs self-attention encoding on the target feature representation information, intelligently predicts and optimizes the position of the layout elements, and improves the rationality and aesthetics of the poster layout. This process reduces manual intervention and improves the level of automation of poster generation. The introduction of the interactive layer enables the deep fusion of image features and layout information, further refines and adjusts the position of layout elements, ensures the perfect fit between the poster content and the image content, and enhances the visual appeal and information transmission effect of the poster.

[0097] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned annotated poster images and target parameters, the above-mentioned annotated poster images and target parameters can also be stored in a node of a blockchain.

[0098] The blockchain referred to in this application is a new application model for computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Blockchain is essentially a decentralized database, a series of data blocks generated using cryptographic methods. Each data block contains information about a batch of network transactions, which is used to verify the validity of this information (to prevent counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0099] The embodiments of the present application can build and optimize related models and networks based on artificial intelligence technology, such as image networks, layout networks, interaction layers, poster generation models, etc. Among them, artificial intelligence (AI) models are the crystallization of theory and practice that simulate the decision-making process of human intelligence through algorithms and data analysis to solve complex problems, predict future trends, or realize automated tasks. These models utilize large amounts of historical data and real-time information, and are trained and optimized through specific algorithmic frameworks to achieve efficient, accurate, and reliable performance.

[0100] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware via computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes in the above-described method embodiments. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0101] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0102] Further references Figure 3 , as a response to the above Figure 2 In order to realize the method shown in the figure, the present application provides an embodiment of a training device for a poster generation model. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0103] like Figure 3 As shown, the poster generation model training device 400 of this embodiment includes: an image acquisition module 401, a feature extraction module 402, an encoding module 403, a feature interaction module 404 and a parameter training module 405. Among them:

[0104] An image acquisition module 401 is configured to acquire an annotated poster image, where the poster image includes a plurality of layout element images and position annotation information of the layout elements of each layout element image.

[0105] A feature extraction module 402 is configured to extract image features of the poster image using a preset image network to obtain global image feature information of the poster image;

[0106] The encoding module 403 is used to obtain feature representation information of the layout elements, and use a preset layout network to perform self-attention encoding on the feature representation information to obtain initial position information of the layout elements;

[0107] A feature interaction module 404 is configured to perform feature interaction on the global image feature information and the initial position information using a preset interaction layer to obtain layout element position information of the layout element;

[0108] The parameter training module 405 is used to train the parameters of the image network, the layout network and the interaction layer respectively based on the difference between the layout element position information and the position annotation information, obtain the trained target parameters, and construct a poster generation model based on the target parameters, the image network, the layout network and the interaction layer.

[0109] The embodiments of the present application can significantly improve the performance of the poster generation model through refined data processing and deep learning model construction. First, the use of annotated poster images as training materials ensures that the model can learn accurate layout element position information. Global image features are extracted through a preset image network, providing a rich visual context for the model. Then, the layout network uses the self-attention encoding mechanism to effectively capture the interdependence between layout elements and generate more reasonable initial position information. The introduction of the interaction layer further promotes the deep fusion of global image features and initial position information, making the generated layout element position information more accurate. During the training process, the model parameters are dynamically adjusted according to the difference between the layout element position information and the position annotation information, ensuring that the model can gradually approach the real layout. Finally, the poster generation model constructed based on the trained target parameters can realize the automation of poster layout and efficiently generate posters with a reasonable layout.

[0110] In one embodiment, the feature extraction module 402 includes:

[0111] The segmentation submodule is used to segment the poster image using a preset image segmentation algorithm to obtain multiple image blocks;

[0112] The feature extraction submodule is used to extract local image features of each image block using a preset image network to obtain multiple local image feature information;

[0113] The fusion submodule is used to fuse multiple local image feature information to obtain the global image feature information of the poster image.

[0114] The embodiments of the present application can segment a poster image into multiple image blocks using a preset image segmentation algorithm, facilitating the detailed capture of local features. Using a preset image network, local image features of each image block can be efficiently extracted, fully mining the image information. Finally, by fusing multiple local image feature information, global feature information of the poster image is obtained, helping to capture the poster's overall style and key information distribution.

[0115] In one embodiment, the encoding module 403 includes:

[0116] Get submodule, used to obtain category information and size information of layout elements;

[0117] The embedding submodule is used to embed the category information and size information to obtain the position embedding information of the layout elements;

[0118] The encoding submodule is used to use a preset feature encoding network to perform feature encoding on the layout element image to obtain the image feature encoding of the layout element; and to encode the position embedding information and the image feature encoding as the feature representation information of the layout element.

[0119] The embodiments of the present application embed category and size information to obtain positional embedding information that accurately reflects the position and size relationship of layout elements within the overall layout. Furthermore, a pre-set feature encoding network is used to encode layout element images, efficiently extracting local and global features of the image and generating image feature encodings rich in semantic information.

[0120] In one embodiment, the embedding submodule is further used to use a preset category embedding dictionary to map the category information into a corresponding category embedding vector, normalize the width information and the height information to obtain a width vector corresponding to the width information, and a height vector corresponding to the height information; concatenate the category embedding vector, the width vector, and the height vector to obtain a feature embedding vector of the layout element; perform a nonlinear transformation on the feature embedding vector to obtain a hidden feature vector of the layout element, and the hidden feature vector is used to represent abstract information of the category information and size information of the layout element; based on the position information of the layout element in the poster image, a preset position encoding function is used to generate a position embedding vector of the layout element; add the hidden feature vector and the position embedding vector to obtain a layout element embedding vector of the layout element, and the layout element embedding vector is used to represent the content information and position information of the layout element; perform encoding enhancement processing on the layout element embedding vector to obtain the position embedding information of the layout element.

[0121] The embodiment of the present application can map category information into category embedding vectors through a preset category embedding dictionary, effectively capturing the semantic relationship between categories. At the same time, the width and height information are normalized and spliced ​​into feature embedding vectors, and a hidden feature vector is obtained by combining nonlinear transformation to abstractly represent the category and size information of the layout elements. The position embedding vector generated by the position encoding function is combined to further incorporate the spatial position information of the layout elements. Finally, the layout element embedding vector obtained by the encoding enhancement process comprehensively represents the content information and position information of the layout elements, providing more comprehensive and accurate feature support for subsequent poster image analysis, recognition or processing tasks.

[0122] In one embodiment, the encoding module 403 includes:

[0123] A splicing submodule is used to splice the position embedding information and the image feature encoding using a preset layout network to obtain a position enhanced feature vector of the layout element;

[0124] The self-attention processing submodule is used to use the self-attention mechanism of the layout network to perform self-attention processing on the position enhancement feature vector to obtain the element-related feature vector of the layout element;

[0125] The transformation fusion submodule is used to use the feedforward neural network of the layout network to perform nonlinear transformation and fusion on the element-related feature vectors to obtain the deep semantic features of the layout elements;

[0126] The coordinate prediction submodule is used to obtain a preset coordinate prediction model, and based on deep semantic features, use the coordinate prediction model to predict the coordinates of layout elements, obtain the predicted position coordinate values ​​of the layout elements, and determine the initial position information of the layout elements based on the predicted position coordinate values.

[0127] The embodiment of the present application can fuse the position embedding information with the image feature encoding through the layout network to generate a position enhanced feature vector, which comprehensively enriches the feature representation of the layout elements. Utilizing the self-attention mechanism, the association information between the layout elements is deeply mined to form an element association feature vector, thereby enhancing the information content of the feature vector. After the nonlinear transformation and fusion of the feedforward neural network, the deep semantic features of the layout elements are extracted to accurately reflect their position, relationship, context and function in the overall layout. Finally, the coordinate prediction model based on the deep semantic features accurately predicts the position of the layout elements, providing strong support for layout optimization and intelligent design.

[0128] In one embodiment, the feature interaction module 404 includes:

[0129] A conversion submodule, configured to convert the initial position information into a query vector and the global image feature information into a key vector and a value vector using a preset interaction layer;

[0130] The dot product operation submodule is used to perform a dot product operation on the key vector and the query vector to obtain an attention weight value. The attention weight value is used to represent the correlation between the initial position information of the layout element and the global image feature information;

[0131] The weighted summation operation submodule is used to perform a weighted summation operation on the value vector and the attention weight to obtain the layout element position information of the layout element.

[0132] The embodiments of the present application can achieve a deep fusion of layout element initial position information and global image feature information through a preset interaction layer. Specifically, the transformation of query vectors, key vectors, and value vectors, as well as the attention weight values ​​obtained by dot product operations, accurately characterize the correlation between layout element position information and global features. Through weighted summation operations, the global feature information most relevant to the layout element position is further integrated, significantly improving the accuracy and completeness of layout element position information.

[0133] In one embodiment, the parameter training module 405 includes:

[0134] A distance calculation submodule is used to calculate the Euclidean distance between the layout element position information and the position annotation information, and to construct a position regression loss function based on the Euclidean distance. The position regression loss function is used to measure the degree of deviation between the layout element position indicated by the layout element position information and the annotation position indicated by the position annotation information.

[0135] An IoU calculation submodule is used to calculate the IoU of the layout element position information and the position annotation information. Based on the IoU, a position classification loss function is constructed. The position classification loss function is used to measure the degree of overlap between the layout element position indicated by the layout element position information and the annotation position indicated by the position annotation information.

[0136] The weighted summation submodule is used to perform weighted summation of the position regression loss function and the position classification loss function to obtain the total loss function;

[0137] The update submodule is used to update the parameters of the image network, the layout network, and the interaction layer based on the total loss function and the back propagation algorithm to obtain the target parameters after training.

[0138] The embodiments of the present application can construct a position regression loss function by calculating the Euclidean distance between the layout element position information and the position annotation information to accurately quantify the position deviation. At the same time, the intersection-over-union ratio is used to construct a position classification loss function to effectively evaluate the degree of position overlap. The total loss function formed by the combination of the two comprehensively guides the optimization of the image network, layout network, and interaction layer parameters. The backpropagation algorithm is used to iteratively update the parameters, significantly improving the accuracy of layout element position prediction, making the predicted position closer to the actual annotation, optimizing the layout effect, and improving the intelligent level of image processing and layout optimization.

[0139] In one embodiment, the poster generation model training apparatus 400 further includes:

[0140] The information acquisition module is used to obtain the material information of the poster to be generated, including image and text prompt information;

[0141] An extraction module is used to extract image features from an image using the image network of the poster generation model to obtain target global image feature information of the image;

[0142] A first generating module is used to generate target feature representation information of layout elements of a poster to be generated based on the text prompt information;

[0143] A self-attention encoding module is used to use the layout network of the poster generation model to perform self-attention encoding on the target feature representation information to obtain the target initial position information of the layout elements of the poster to be generated;

[0144] An interaction module, configured to use the interaction layer of the poster generation model to perform feature interaction on the target global image feature information and the target initial position information to obtain target layout element position information of the layout elements of the poster to be generated;

[0145] The second generation module is used to generate a poster to be generated based on the target layout element position information, image and text prompt information.

[0146] The embodiment of the present application can perform deep feature extraction on the image by adopting the image network of the poster generation model, accurately capturing the key elements and overall style of the image. The layout network performs self-attention encoding on the target feature representation information, intelligently predicts and optimizes the position of the layout elements, and improves the rationality and aesthetics of the poster layout. This process reduces manual intervention and improves the level of automation of poster generation. The introduction of the interactive layer enables the deep fusion of image features and layout information, further refines and adjusts the position of layout elements, ensures the perfect fit between the poster content and the image content, and enhances the visual appeal and information transmission effect of the poster.

[0147] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0148] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 6 with a memory 61, a processor 62, and a network interface 63, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.

[0149] Computer devices can be desktop computers, laptops, PDAs, cloud servers, etc. Computer devices can interact with users through keyboards, mice, remote controls, touchpads, or voice-activated devices.

[0150] The memory 61 includes at least one type of readable storage medium, including flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, a magnetic disk, an optical disk, etc. In some embodiments, the memory 61 may be an internal storage unit of the computer device 6, such as the hard disk or memory of the computer device 6. In other embodiments, the memory 61 may also be an external storage device of the computer device 6, such as a plug-in hard disk equipped on the computer device 6, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, etc. Of course, the memory 61 may also include both the internal storage unit of the computer device 6 and its external storage devices. In this embodiment, the memory 61 is generally used to store the operating system and various application software installed on the computer device 6, such as computer-readable instructions for the training method of the poster generation model. In addition, the memory 61 may also be used to temporarily store various types of data that have been output or are about to be output.

[0151] In some embodiments, the processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 62 is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to execute computer-readable instructions or process data stored in the memory 61, such as computer-readable instructions for executing the training method of the poster generation model.

[0152] The network interface 63 may include a wireless network interface or a wired network interface. The network interface 63 is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0153] The embodiments of the present application can significantly improve the performance of the poster generation model through refined data processing and deep learning model construction. First, the use of annotated poster images as training materials ensures that the model can learn accurate layout element position information. Global image features are extracted through a preset image network, providing a rich visual context for the model. Then, the layout network uses the self-attention encoding mechanism to effectively capture the interdependence between layout elements and generate more reasonable initial position information. The introduction of the interaction layer further promotes the deep fusion of global image features and initial position information, making the generated layout element position information more accurate. During the training process, the model parameters are dynamically adjusted according to the difference between the layout element position information and the position annotation information, ensuring that the model can gradually approach the real layout. Finally, the poster generation model constructed based on the trained target parameters can realize the automation of poster layout and efficiently generate posters with a reasonable layout.

[0154] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions. The computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the training method of the poster generation model as described above.

[0155] The embodiments of the present application can significantly improve the performance of the poster generation model through refined data processing and deep learning model construction. First, the use of annotated poster images as training materials ensures that the model can learn accurate layout element position information. Global image features are extracted through a preset image network, providing a rich visual context for the model. Then, the layout network uses the self-attention encoding mechanism to effectively capture the interdependence between layout elements and generate more reasonable initial position information. The introduction of the interaction layer further promotes the deep fusion of global image features and initial position information, making the generated layout element position information more accurate. During the training process, the model parameters are dynamically adjusted according to the difference between the layout element position information and the position annotation information, ensuring that the model can gradually approach the real layout. Finally, the poster generation model constructed based on the trained target parameters can realize the automation of poster layout and efficiently generate posters with a reasonable layout.

[0156] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of each embodiment of the present application.

[0157] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

[0158] Obviously, the embodiments described above are only some of the embodiments of the present application, rather than all of the embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned specific embodiments, or to make equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the present application specification and the accompanying drawings, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present application.

Claims

1. A training method for a poster generation model, characterized in that: The steps include: Acquire a poster image with annotations, wherein the poster image includes a plurality of layout element images and position annotation information of the layout element of each layout element image; Using a preset image network, extracting image features of the poster image to obtain global image feature information of the poster image; Acquire the feature representation information of the layout element, use a preset layout network to perform self-attention encoding on the feature representation information, and obtain the initial position information of the layout element, specifically including: acquiring the category information and size information of the layout element, embedding the category information and the size information to obtain the position embedding information of the layout element, using a preset feature encoding network to feature encode the layout element image to obtain the image feature encoding of the layout element, using the position embedding information and the image feature encoding as the feature representation information of the layout element, using a preset layout network to embed the position embedding information and the image feature encoding Encoding and information splicing are performed to obtain a position-enhanced feature vector of the layout element; self-attention processing is performed on the position-enhanced feature vector using the self-attention mechanism of the layout network to obtain an element-association feature vector of the layout element; nonlinear transformation and fusion are performed on the element-association feature vector using the feedforward neural network of the layout network to obtain deep semantic features of the layout element; a preset coordinate prediction model is obtained; coordinate prediction is performed on the layout element using the coordinate prediction model based on the deep semantic features to obtain a predicted position coordinate value of the layout element; and initial position information of the layout element is determined based on the predicted position coordinate value; Using a preset interaction layer, performing feature interaction on the global image feature information and the initial position information to obtain layout element position information of the layout element; Based on the difference between the layout element position information and the position annotation information, the parameters of the image network, the parameters of the layout network and the parameters of the interaction layer are trained respectively to obtain the trained target parameters. Based on the target parameters, the image network, the layout network and the interaction layer, a poster generation model is constructed.

2. The method according to claim 1, characterized in that The size information includes width information and height information. The step of embedding the category information and the size information to obtain the position embedding information of the layout element specifically includes: Using a preset category embedding dictionary, mapping the category information into a corresponding category embedding vector, normalizing the width information and the height information to obtain a width vector corresponding to the width information and a height vector corresponding to the height information; Concatenating the category embedding vector, the width vector, and the height vector to obtain a feature embedding vector of the layout element; Performing a nonlinear transformation on the feature embedding vector to obtain a hidden feature vector of the layout element, where the hidden feature vector is used to represent abstract information of category information and size information of the layout element; Generate a position embedding vector of the layout element using a preset position encoding function according to the position information of the layout element in the poster image; Adding the hidden feature vector and the position embedding vector to obtain a layout element embedding vector of the layout element, wherein the layout element embedding vector is used to represent content information and position information of the layout element; Perform coding enhancement processing on the layout element embedding vector to obtain position embedding information of the layout element.

3. The method according to claim 1, characterized in that The step of using a preset interaction layer to perform feature interaction on the global image feature information and the initial position information to obtain the layout element position information of the layout element specifically includes: Using a preset interaction layer, converting the initial position information into a query vector, and converting the global image feature information into a key vector and a value vector; performing a dot product operation on the key vector and the query vector to obtain an attention weight value, where the attention weight value is used to represent the correlation between the initial position information in the layout element and the global image feature information; A weighted sum operation is performed on the value vector and the attention weight to obtain layout element position information of the layout element.

4. The method according to claim 1, wherein The step of respectively training the parameters of the image network, the parameters of the layout network, and the parameters of the interaction layer based on the difference between the layout element position information and the position annotation information to obtain trained target parameters specifically includes: calculating a Euclidean distance between the layout element position information and the position annotation information, and constructing a position regression loss function based on the Euclidean distance, wherein the position regression loss function is used to measure a degree of deviation between a layout element position indicated by the layout element position information and an annotation position indicated by the position annotation information; calculating an intersection-over-union (IoU) of the layout element position information and the position annotation information, and constructing a position classification loss function based on the IoU, where the position classification loss function is used to measure a degree of overlap between a layout element position indicated by the layout element position information and an annotation position indicated by the position annotation information; Performing a weighted summation on the position regression loss function and the position classification loss function to obtain a total loss function; Based on the total loss function, a back propagation algorithm is used to update the parameters of the image network, the parameters of the layout network, and the parameters of the interaction layer respectively to obtain the trained target parameters.

5. The method according to claim 1, wherein After the step of constructing a poster generation model based on the target parameters, the image network, the layout network, and the interaction layer, the method further includes: Obtaining material information of a poster to be generated, the material information including image and text prompt information; Using the image network of the poster generation model, performing image feature extraction processing on the image to obtain target global image feature information of the image; generating target feature representation information of the layout elements of the poster to be generated based on the text prompt information; Using the layout network of the poster generation model, self-attention encoding is performed on the target feature representation information to obtain target initial position information of the layout elements of the poster to be generated; Using the interactive layer of the poster generation model, performing feature interaction on the target global image feature information and the target initial position information to obtain target layout element position information of the layout elements of the poster to be generated; The poster to be generated is generated based on the target layout element position information, the image, and the text prompt information.

6. A training device for a poster generation model, characterized in that: include: An image acquisition module, configured to acquire an annotated poster image, wherein the poster image includes a plurality of layout element images and position annotation information of the layout element of each layout element image; a feature extraction module, configured to extract image features of the poster image using a preset image network to obtain global image feature information of the poster image; an encoding module, configured to obtain feature representation information of the layout element, and perform self-attention encoding on the feature representation information using a preset layout network to obtain initial position information of the layout element; The encoding module is further configured to obtain category information and size information of the layout element, embed the category information and the size information to obtain position embedding information of the layout element, perform feature encoding on the layout element image using a preset feature encoding network to obtain image feature encoding of the layout element, use the position embedding information and the image feature encoding as feature representation information of the layout element, concatenate the position embedding information and the image feature encoding using a preset layout network to obtain a position enhanced feature vector of the layout element, perform self-attention processing on the position enhanced feature vector using a self-attention mechanism of the layout network to obtain an element-associated feature vector of the layout element, perform nonlinear transformation and fusion on the element-associated feature vector using a feedforward neural network of the layout network to obtain deep semantic features of the layout element, obtain a preset coordinate prediction model, perform coordinate prediction on the layout element using the coordinate prediction model based on the deep semantic features to obtain predicted position coordinate values ​​of the layout element, and determine initial position information of the layout element based on the predicted position coordinate values; a feature interaction module, configured to perform feature interaction on the global image feature information and the initial position information using a preset interaction layer to obtain layout element position information of the layout element; A parameter training module is used to train the parameters of the image network, the parameters of the layout network, and the parameters of the interaction layer based on the difference between the layout element position information and the position annotation information, obtain the trained target parameters, and construct a poster generation model based on the target parameters, the image network, the layout network, and the interaction layer.

7. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the training method for the poster generation model according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the method for training a poster generation model according to any one of claims 1 to 5.

Images