Poster generation model training method and device, equipment and medium

Abstract

The invention belongs to the field of artificial intelligence, and relates to a poster generation model training method comprising the following steps: obtaining a poster image with a label, the poster image comprising position label information of layout elements; extracting global image feature information of the poster image by adopting an image network; obtaining feature representation information of the layout elements, and performing self-attention coding on the feature representation information by adopting a layout network to obtain initial position information; performing feature interaction on the global image feature information and the initial position information by adopting an interaction layer to obtain layout element position information; and based on the difference between the layout element position information and the position labeling information, respectively training parameters of the image network, the layout network and the interaction layer so as to construct a poster generation model. The invention further provides a device, equipment and a medium. In addition, the invention also relates to a block chain technology, and the poster image and the target parameter can be stored in a block chain. According to the invention, automation of poster layout can be realized.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method, device, equipment and medium for a poster generation model. Background Art

[0002] In the field of image processing and generation, Stable Diffusion is an advanced image generation algorithm that can gradually generate corresponding image content through multiple iterations based on text prompts. However, current image generation algorithms mainly focus on the direct generation of image content. Although they can generate rich visual content, they still have significant deficiencies in meeting specific design needs, such as poster production. Figure 1 Generating a complete image at one time is not enough in complex scenarios such as poster design.

[0003] Poster design usually involves the careful arrangement and combination of multiple layout elements, rather than just the generation of a single content. Existing image generation algorithms have obvious limitations in the automated layout of poster design, and it is difficult to generate a complete image that meets the requirements of poster design in one go. Therefore, how to effectively automate poster layout has become an urgent problem to be solved. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a training method, device, equipment and medium for a poster generation model to solve the existing problem of difficulty in automating poster layout.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a training method for a poster generation model, which adopts the following technical solutions:

[0006] A poster image with annotations is obtained, where the poster image includes multiple layout element images and position annotation information of the layout elements of each layout element image; a preset image network is used to extract image features of the poster image to obtain global image feature information of the poster image; feature representation information of the layout elements is obtained, and a preset layout network is used to perform self-attention encoding on the feature representation information to obtain initial position information of the layout elements; a preset interaction layer is used to perform feature interaction on the global image feature information and the initial position information to obtain layout element position information of the layout elements; based on the difference between the layout element position information and the position annotation information, the parameters of the image network, the parameters of the layout network and the parameters of the interaction layer are respectively trained to obtain the trained target parameters, and a poster generation model is constructed based on the target parameters, the image network, the layout network and the interaction layer.

[0007] Furthermore, the step of using a preset image network to extract image features of the poster image and obtain global image feature information of the poster image specifically includes:

[0008] The poster image is segmented by using a preset image segmentation algorithm to obtain multiple image blocks; the local image features of each image block are extracted by using a preset image network to obtain multiple local image feature information; the multiple local image feature information are fused to obtain the global image feature information of the poster image.

[0009] Furthermore, the step of obtaining the feature representation information of the layout element specifically includes:

[0010] Obtain category information and size information of layout elements; embed the category information and size information to obtain position embedding information of layout elements; use a preset feature encoding network to feature encode the layout element image to obtain image feature encoding of the layout element; encode the position embedding information and image feature as feature representation information of the layout element.

[0011] Furthermore, the size information includes width information and height information, and the step of embedding the category information and the size information to obtain the position embedding information of the layout element specifically includes:

[0012] Using a preset category embedding dictionary, the category information is mapped into a corresponding category embedding vector, the width information and the height information are normalized to obtain a width vector corresponding to the width information, and a height vector corresponding to the height information; the category embedding vector, the width vector and the height vector are concatenated to obtain a feature embedding vector of the layout element; a nonlinear transformation is performed on the feature embedding vector to obtain a hidden feature vector of the layout element, and the hidden feature vector is used to represent the abstract information of the category information and size information of the layout element; according to the position information of the layout element in the poster image, a preset position encoding function is used to generate a position embedding vector of the layout element; the hidden feature vector and the position embedding vector are added to obtain a layout element embedding vector of the layout element, and the layout element embedding vector is used to represent the content information and position information of the layout element; the layout element embedding vector is subjected to encoding enhancement processing to obtain the position embedding information of the layout element.

[0013] Furthermore, the step of using a preset layout network to perform self-attention encoding on the feature representation information to obtain the initial position information of the layout elements specifically includes:

[0014] A preset layout network is used to perform information splicing on position embedding information and image feature encoding to obtain a position enhanced feature vector of the layout element; the self-attention mechanism of the layout network is used to perform self-attention processing on the position enhanced feature vector to obtain an element association feature vector of the layout element; a feedforward neural network of the layout network is used to perform nonlinear transformation fusion on the element association feature vector to obtain the deep semantic features of the layout element; a preset coordinate prediction model is obtained, and based on the deep semantic features, the coordinate prediction model is used to perform coordinate prediction on the layout element to obtain the predicted position coordinate value of the layout element, and the initial position information of the layout element is determined based on the predicted position coordinate value.

[0015] Furthermore, the step of using a preset interaction layer to perform feature interaction on the global image feature information and the initial position information to obtain the layout element position information of the layout element specifically includes:

[0016] A preset interaction layer is used to convert the initial position information into a query vector, and the global image feature information is converted into a key vector and a value vector. A dot product operation is performed on the key vector and the query vector to obtain an attention weight value, which is used to characterize the correlation between the initial position information and the global image feature information in the layout element. A weighted sum operation is performed on the value vector and the attention weight to obtain the layout element position information of the layout element.

[0017] Further, based on the difference between the layout element position information and the position annotation information, the parameters of the image network, the parameters of the layout network, and the parameters of the interaction layer are trained respectively to obtain the trained target parameters, which specifically includes:

[0018] The Euclidean distance between the layout element position information and the position annotation information is calculated, and based on the Euclidean distance, a position regression loss function is constructed, which is used to measure the degree of deviation between the layout element position indicated by the layout element position information and the annotation position indicated by the position annotation information; the intersection-over-union ratio of the layout element position information and the position annotation information is calculated, and based on the intersection-over-union ratio, a position classification loss function is constructed, which is used to measure the degree of overlap between the layout element position indicated by the layout element position information and the annotation position indicated by the position annotation information; the position regression loss function and the position classification loss function are weightedly summed to obtain the total loss function; based on the total loss function, a back propagation algorithm is used to update the parameters of the image network, the parameters of the layout network and the parameters of the interaction layer respectively to obtain the trained target parameters.

[0019] Furthermore, after the step of constructing a poster generation model based on the target parameters, the image network, the layout network, and the interaction layer, it also includes:

[0020] Obtain material information of the poster to be generated, which includes images and text prompt information; use the image network of the poster generation model to perform image feature extraction processing on the image to obtain the target global image feature information of the image; based on the text prompt information, generate the target feature representation information of the layout elements of the poster to be generated; use the layout network of the poster generation model to perform self-attention encoding on the target feature representation information to obtain the target initial position information of the layout elements of the poster to be generated; use the interaction layer of the poster generation model to perform feature interaction on the target global image feature information and the target initial position information to obtain the target layout element position information of the layout elements of the poster to be generated; generate the poster to be generated based on the target layout element position information, the image and the text prompt information.

[0021] In order to solve the above technical problems, the embodiment of the present application also provides a training device for a poster generation model, which adopts the following technical solutions:

[0022] An image acquisition module, used to acquire a poster image with annotations, the poster image including a plurality of layout element images and position annotation information of the layout element of each layout element image;

[0023] A feature extraction module is used to extract image features of the poster image using a preset image network to obtain global image feature information of the poster image;

[0024] The encoding module is used to obtain the feature representation information of the layout elements, and uses a preset layout network to perform self-attention encoding on the feature representation information to obtain the initial position information of the layout elements;

[0025] A feature interaction module, used to perform feature interaction on the global image feature information and the initial position information by using a preset interaction layer, so as to obtain layout element position information of the layout element;

[0026] The parameter training module is used to train the parameters of the image network, the layout network and the interaction layer respectively based on the difference between the layout element position information and the position annotation information, obtain the trained target parameters, and build a poster generation model based on the target parameters, image network, layout network and interaction layer.

[0027] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the training method for the poster generation model as mentioned above are implemented.

[0028] In order to solve the above technical problems, an embodiment of the present application also provides a computer-readable storage medium, which stores computer-readable instructions. The computer-readable instructions can be executed by at least one processor to enable at least one processor to perform the steps of the training method of the poster generation model as mentioned above.

[0029] Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects: through refined data processing and deep learning model construction, the performance of the poster generation model is significantly improved. First, the use of annotated poster images as training materials ensures that the model can learn accurate layout element position information. The global image features are extracted through a preset image network, providing a rich visual context for the model. Then, the layout network uses the self-attention encoding mechanism to effectively capture the interdependence between layout elements and generate more reasonable initial position information. The introduction of the interaction layer further promotes the deep fusion of global image features and initial position information, making the generated layout element position information more accurate. During the training process, the model parameters are dynamically adjusted according to the difference between the layout element position information and the position annotation information, ensuring that the model can gradually approach the real layout. Finally, the poster generation model constructed based on the trained target parameters can realize the automation of poster layout and efficiently generate posters with reasonable layout. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0031] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0032] Figure 2 It is a flowchart of a training method for a poster generation model provided in this application;

[0033] Figure 3 It is a structural schematic diagram of a training device for a poster generation model provided in the present application;

[0034] Figure 4 It is a structural schematic diagram of a computer device provided by this application. DETAILED DESCRIPTION

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0036] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0037] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0038] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0039] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0040] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0041] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0042] It should be noted that the training method of the poster generation model provided in the embodiment of the present application is generally executed by a server, and accordingly, the training device of the poster generation model is generally set in the server.

[0043] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0044] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for training a poster generation model according to the present application. The method for training a poster generation model comprises the following steps:

[0045] Step S201 : acquiring a poster image with annotations, wherein the poster image includes a plurality of layout element images and position annotation information of the layout elements of each layout element image.

[0046] In this embodiment, the training method of the poster generation model is run on the electronic device (e.g. Figure 1 The server shown in the figure) can obtain the labeled poster image through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0047] The annotated poster image refers to a poster image containing position annotation information, which can describe the position and attributes of each layout element in the poster image in detail.

[0048] Among them, multiple layout element images refer to the individual image elements that constitute the poster image, such as text, icons, pictures, etc. These elements have different functions and visual effects in poster design, and together constitute the complete poster content.

[0049] Among them, layout elements, in poster design, refer to the independent parts or components that make up the whole poster, such as title, text, pictures, decorative elements, etc. Each layout element has its specific position, size and category.

[0050] The position annotation information refers to the information that records in detail the precise position of each layout element in the poster image, usually in the form of coordinates, and is used to guide the model to learn the correct layout of the layout elements.

[0051] Step S202: using a preset image network to extract image features of the poster image, and obtaining global image feature information of the poster image.

[0052] The image network refers to a deep learning network (e.g., an image Transformer network) for extracting image features of a poster image. The image network can automatically learn visual features in an image.

[0053] Among them, image features refer to abstract representations of the visual content of the poster image, such as edges, textures, colors, etc. In the poster image, these features help the model understand the overall structure and content of the image.

[0054] Among them, global image feature information refers to the feature information extracted by the image network, which represents the overall visual content of the poster image. It contains the global structure and content information of the poster image.

[0055] Step S203, acquiring feature representation information of the layout elements, using a preset layout network to perform self-attention encoding on the feature representation information, and obtaining initial position information of the layout elements.

[0056] The feature representation information refers to information that combines the category information, size information, and image feature encoding of each layout element in the poster image, and is used to fully describe the attributes and features of the layout elements.

[0057] The layout network refers to a deep learning network (e.g., layout Transformer network) that is specifically used to process layout element features. Through mechanisms such as self-attention encoding, the layout network can learn the spatial relationships and interactions between layout elements.

[0058] Among them, self-attention encoding refers to a deep learning technology that automatically learns the spatial relationship and dependency between layout elements by calculating the correlation scores between layout elements, thereby generating a more accurate layout representation.

[0059] The initial position information refers to the preliminary predicted position information obtained after the layout network performs self-attention encoding on the feature representation information of the layout elements. It is the basis for subsequent position refinement.

[0060] Step S204 : using a preset interaction layer to perform feature interaction on the global image feature information and the initial position information to obtain layout element position information of the layout element.

[0061] The interaction layer refers to a deep learning layer (e.g., Cross Attention layer) used to fuse global image feature information and initial position information of layout elements. Through feature interaction, the interaction layer can further refine the position information of layout elements.

[0062] Feature interaction refers to the interaction between the global features of the poster image (such as overall color, texture, etc.) and the features of the layout elements (such as category, size, position, etc.). This interaction helps the model capture the complex relationship between features, thereby improving the prediction accuracy of the layout element position.

[0063] Among them, the layout element position information refers to the final layout element position information obtained after being processed by the interaction layer. It represents the model's prediction result of the position of each layout element in the poster image.

[0064] Step S205, based on the difference between the layout element position information and the position annotation information, the parameters of the image network, the parameters of the layout network and the parameters of the interaction layer are trained respectively to obtain the trained target parameters, and based on the target parameters, the image network, the layout network and the interaction layer, a poster generation model is constructed.

[0065] The difference refers to the difference between the layout element position information and the position annotation information, which is used to evaluate the accuracy of the model prediction.

[0066] The target parameters are the final parameters obtained after training and used to build the poster generation model. These target parameters include the model's understanding of the poster layout and image features, and can generate high-quality poster images.

[0067] Among them, the poster generation model refers to the use of trained target parameters, image network, layout network and interaction layer to build a complete model that can automatically generate poster images based on the input material information.

[0068] The embodiments of the present application can significantly improve the performance of the poster generation model through refined data processing and deep learning model construction. First, the annotated poster images are used as training materials to ensure that the model can learn accurate layout element position information. The global image features are extracted through a preset image network to provide a rich visual context for the model. Then, the layout network uses the self-attention encoding mechanism to effectively capture the interdependence between layout elements and generate more reasonable initial position information. The introduction of the interaction layer further promotes the deep fusion of global image features and initial position information, making the generated layout element position information more accurate. During the training process, the model parameters are dynamically adjusted according to the difference between the layout element position information and the position annotation information, ensuring that the model can gradually approach the real layout. Finally, the poster generation model constructed based on the trained target parameters can realize the automation of poster layout and efficiently generate posters with reasonable layout.

[0069] In some optional implementations of this embodiment, step 201, using a preset image network to extract image features of the poster image to obtain global image feature information of the poster image, specifically includes the following steps:

[0070] The poster image is segmented by using a preset image segmentation algorithm to obtain multiple image blocks; the local image features of each image block are extracted by using a preset image network to obtain multiple local image feature information; the multiple local image feature information are fused to obtain the global image feature information of the poster image.

[0071] In one example, a preset image segmentation algorithm, such as a superpixel-based segmentation algorithm, can be used to segment a poster image, such as a financial poster image. The algorithm can segment the poster image into multiple image blocks with similar attributes based on its color, texture and other features. These image blocks may contain key information such as pictures, text descriptions, price tags, etc. of financial products. Then, a preset image network, such as an image Transformer model, is used to extract local image features from each image block. The image Transformer model can capture the global dependencies between image blocks and extract local features with rich semantic information. Then, the extracted multiple local image feature information is fused. The fusion process can use methods such as weighted averaging, maximum pooling or attention mechanism to obtain the global image feature information of the poster image. The global image feature information can reflect the overall style of the financial poster, the distribution of key information, and the correlation between financial products.

[0072] The embodiment of the present application can segment the poster image into multiple image blocks through a preset image segmentation algorithm, so as to capture the local features in detail. By using the preset image network, the local image features of each image block can be efficiently extracted to fully mine the image information. Finally, by fusing multiple local image feature information, the global feature information of the poster image is obtained, which helps to capture the overall style and key information distribution of the poster.

[0073] In some optional implementations of this embodiment, in step S203, obtaining feature representation information of the layout element specifically includes the following steps:

[0074] Obtain category information and size information of layout elements; embed the category information and size information to obtain position embedding information of layout elements; use a preset feature encoding network to feature encode the layout element image to obtain image feature encoding of the layout element; encode the position embedding information and image feature as feature representation information of the layout element.

[0075] In one example, the category information (type of layout element, such as text, picture, logo, etc.) and size information (width, height, etc. of layout element) of the layout element can be obtained. This information can be obtained through image processing algorithms or user input. Then, the category information and size information are embedded. The embedding process can use the embedding technology in deep learning to map the category information and size information into a high-dimensional space to obtain position embedding information. The position embedding information can reflect the position and size relationship of the layout element in the overall poster image layout. Then, a preset feature encoding network is used to feature encode the layout element image and generate the image feature encoding of the layout element. The feature encoding network can be constructed by multiple convolutional layers, pooling layers and activation functions. Finally, the position embedding information and image feature encoding are used as feature representation information of the layout element.

[0076] The embodiment of the present application can embed the category information and size information, and the position embedding information obtained can accurately reflect the position and size relationship of the layout element in the overall layout. At the same time, the layout element image is feature-encoded using a preset feature coding network, which can efficiently extract the local and global features of the image and generate image feature coding with rich semantic information.

[0077] In some optional implementations of this embodiment, the size information includes width information and height information, and the step of "embedding the category information and the size information to obtain the position embedding information of the layout element" specifically includes the following steps:

[0078] Using a preset category embedding dictionary, the category information is mapped into a corresponding category embedding vector, the width information and the height information are normalized to obtain a width vector corresponding to the width information, and a height vector corresponding to the height information; the category embedding vector, the width vector and the height vector are concatenated to obtain a feature embedding vector of the layout element; a nonlinear transformation is performed on the feature embedding vector to obtain a hidden feature vector of the layout element, and the hidden feature vector is used to represent the abstract information of the category information and size information of the layout element; according to the position information of the layout element in the poster image, a preset position encoding function is used to generate a position embedding vector of the layout element; the hidden feature vector and the position embedding vector are added to obtain a layout element embedding vector of the layout element, and the layout element embedding vector is used to represent the content information and position information of the layout element; the layout element embedding vector is subjected to encoding enhancement processing to obtain the position embedding information of the layout element.

[0079] In one example, a preset category embedding dictionary can be used to map the category information of the layout element to the corresponding category embedding vector. The category embedding dictionary is a pre-trained vector space in which each category corresponds to a unique vector representation. By mapping the category information to the category embedding vector, the semantic relationship between the categories can be captured. Next, the width information and height information of the layout element are normalized to obtain a width vector and a height vector. The normalization process can eliminate the dimensional difference between the size information and make the size information of different layout elements comparable. Then, the category embedding vector, the width vector and the height vector are concatenated to obtain a feature embedding vector of the layout element. The feature embedding vector integrates the category information and size information of the layout element. Next, a preset fully connected neural network can be used to perform a nonlinear transformation on the feature embedding vector to obtain a hidden feature vector. The fully connected neural network can learn the complex relationship between the feature embedding vectors and extract feature representations with abstract meanings. The hidden feature vector is used to represent the abstract information of the category information and size information of the layout element. In addition, according to the position information of the layout element in the poster image, a preset position encoding function is used to generate a position embedding vector of the layout element. The position embedding vector can capture the spatial position relationship of the layout elements in the poster image. Finally, the hidden feature vector and the position embedding vector are added to obtain the layout element embedding vector of the layout element. The layout element embedding vector can be input into the encoder of the layout Transformer network. Through self-attention calculation and feature transformation, the final embedding vector of the layout element is obtained, and finally the position embedding information of the layout element is obtained. The position embedding information contains not only the category, size information and position information of each layout element, but also the spatial relationship information with other layout elements.

[0080] The embodiment of the present application can map the category information to the category embedding vector through a preset category embedding dictionary, which effectively captures the semantic relationship between categories. At the same time, the width and height information are normalized and spliced ​​into a feature embedding vector, and the hidden feature vector is obtained by combining the nonlinear transformation, which abstractly represents the category and size information of the layout element. The position embedding vector generated by the position encoding function is further integrated into the spatial position information of the layout element. Finally, the layout element embedding vector obtained by the coding enhancement process comprehensively represents the content information and position information of the layout element, providing more comprehensive and accurate feature support for subsequent poster image analysis, recognition or processing tasks.

[0081] In some optional implementations of this embodiment, step S203, using a preset layout network to perform self-attention encoding on the feature representation information to obtain the initial position information of the layout element, specifically includes the following steps:

[0082] A preset layout network is used to perform information splicing on position embedding information and image feature encoding to obtain a position enhanced feature vector of the layout element; the self-attention mechanism of the layout network is used to perform self-attention processing on the position enhanced feature vector to obtain an element association feature vector of the layout element; a feedforward neural network of the layout network is used to perform nonlinear transformation fusion on the element association feature vector to obtain the deep semantic features of the layout element; a preset coordinate prediction model is obtained, and based on the deep semantic features, the coordinate prediction model is used to perform coordinate prediction on the layout element to obtain the predicted position coordinate value of the layout element, and the initial position information of the layout element is determined based on the predicted position coordinate value.

[0083] In one example, a preset layout network, such as a layout Transformer network, can be used to concatenate the position embedding information and the image feature encoding to obtain a position enhanced feature vector. Then, the self-attention mechanism of the layout Transformer network is used to perform self-attention processing on the position enhanced feature vector. By calculating the correlation between different layout elements, an element association feature vector is obtained. This vector not only reflects the characteristics of each layout element itself, but also incorporates the association information with other layout elements, making the feature representation richer and more comprehensive. Next, the feedforward neural network of the layout Transformer network is used to perform nonlinear transformation and fusion on the element association feature vector to obtain the deep semantic features of the layout elements. These deep semantic features represent the high-level abstract information of the layout elements in the overall layout, including position, relationship, context, and function. Finally, a preset coordinate prediction model is obtained, and the model is used to predict the coordinates of the layout elements based on the deep semantic features. The initial position information of the layout element is determined by calculating the predicted position coordinate value.

[0084] The embodiment of the present application can fuse the position embedding information with the image feature encoding through the layout network to generate a position enhanced feature vector, which comprehensively enriches the feature representation of the layout elements. Utilizing the self-attention mechanism, the association information between the layout elements is deeply mined to form an element association feature vector, thereby enhancing the information content of the feature vector. After nonlinear transformation and fusion through the feedforward neural network, the deep semantic features of the layout elements are extracted to accurately reflect their position, relationship, context and function in the overall layout. Finally, the coordinate prediction model based on the deep semantic features accurately predicts the position of the layout elements, providing strong support for layout optimization and intelligent design.

[0085] In some optional implementations of this embodiment, step S204, using a preset interaction layer to perform feature interaction on the global image feature information and the initial position information to obtain layout element position information of the layout element, specifically includes the following steps:

[0086] A preset interaction layer is used to convert the initial position information into a query vector, and the global image feature information is converted into a key vector and a value vector. A dot product operation is performed on the key vector and the query vector to obtain an attention weight value, which is used to characterize the correlation between the initial position information and the global image feature information in the layout element. A weighted sum operation is performed on the value vector and the attention weight to obtain the layout element position information of the layout element.

[0087] In one example, a preset interaction layer (such as Cross Attention) can be used to fuse the initial position information and the global image feature information. Specifically, the initial position information can be converted into a query vector (Query), and the global image feature information can be converted into a key vector (Key) and a value vector (Value). This process utilizes the embedding technology in deep learning to map different types of information into the same high-dimensional space, which is convenient for subsequent calculations and processing. In the CrossAttention interaction layer, the dot product operation between the query vector and the key vector is first calculated to obtain the attention weight value. The purpose of this step is to measure the correlation between the initial position information of the layout element and the global image feature information, that is, to determine which global features have an important influence on the position information of the current layout element. Subsequently, the value vector is weighted and summed using the obtained attention weight value to obtain the layout element position information of each layout element.

[0088] The embodiments of the present application can achieve deep fusion of the initial position information of the layout elements and the global image feature information through a preset interaction layer. Specifically, the conversion of the query vector, key vector and value vector, as well as the attention weight value obtained by the dot product operation are used to accurately characterize the correlation between the layout element position information and the global features. Through the weighted sum operation, the global feature information most relevant to the layout element position is further integrated, thereby significantly improving the accuracy and completeness of the layout element position information.

[0089] In some optional implementations of this embodiment, in step S205, based on the difference between the layout element position information and the position annotation information, the parameters of the image network, the parameters of the layout network, and the parameters of the interaction layer are trained respectively to obtain the trained target parameters, which specifically includes the following steps:

[0090] The Euclidean distance between the layout element position information and the position annotation information is calculated, and based on the Euclidean distance, a position regression loss function is constructed, which is used to measure the degree of deviation between the layout element position indicated by the layout element position information and the annotation position indicated by the position annotation information; the intersection-over-union ratio of the layout element position information and the position annotation information is calculated, and based on the intersection-over-union ratio, a position classification loss function is constructed, which is used to measure the degree of overlap between the layout element position indicated by the layout element position information and the annotation position indicated by the position annotation information; the position regression loss function and the position classification loss function are weightedly summed to obtain the total loss function; based on the total loss function, a back propagation algorithm is used to update the parameters of the image network, the parameters of the layout network and the parameters of the interaction layer respectively to obtain the trained target parameters.

[0091] In one example, a position regression loss function can be constructed by calculating the Euclidean distance between the layout element position information and the position annotation information. The position regression loss function can quantify the degree of deviation between the layout element position information and the annotation position, thereby guiding the model to learn how to more accurately predict the layout element position. At the same time, the intersection over Union (IoU) of the layout element position information and the position annotation information is calculated to construct a position classification loss function. IoU is an effective indicator to measure the degree of overlap between two regions. Through the position classification loss function, the model can learn the spatial relationship between the layout element position and the annotation position, further improving the accuracy of position prediction. Then, the position regression loss function and the position classification loss function are weighted and summed to obtain the total loss function. Finally, based on the total loss function, the back propagation algorithm is used to update the parameters of the image network, layout network, and interaction layer. Through multiple iterative training, the image network, layout network, and interaction layer gradually learn how to extract effective features from the image and accurately predict the position of the layout element.

[0092] The embodiment of the present application can construct a position regression loss function by calculating the Euclidean distance between the layout element position information and the position annotation information, accurately quantify the position deviation, and use the intersection-over-union ratio to construct a position classification loss function to effectively evaluate the degree of position overlap. The total loss function formed by the combination of the two comprehensively guides the optimization of image network, layout network and interaction layer parameters. The back propagation algorithm is used to iteratively update parameters, which significantly improves the accuracy of layout element position prediction, makes the predicted position closer to the actual annotation, optimizes the layout effect, and improves the intelligent level of image processing and layout optimization.

[0093] In some optional implementations of this embodiment, after building a poster generation model based on target parameters, image network, layout network and interaction layer in step S205, the following steps are specifically included:

[0094] Obtain material information of the poster to be generated, which includes images and text prompt information; use the image network of the poster generation model to perform image feature extraction processing on the image to obtain the target global image feature information of the image; based on the text prompt information, generate the target feature representation information of the layout elements of the poster to be generated; use the layout network of the poster generation model to perform self-attention encoding on the target feature representation information to obtain the target initial position information of the layout elements of the poster to be generated; use the interaction layer of the poster generation model to perform feature interaction on the target global image feature information and the target initial position information to obtain the target layout element position information of the layout elements of the poster to be generated; generate the poster to be generated based on the target layout element position information, the image and the text prompt information.

[0095] In one example, a landscape picture and a descriptive text can be selected as the material information of the poster to be generated. First, the image network of the poster generation model is used to extract the features of the landscape picture to obtain the target global image feature information. This feature information can fully reflect the key content and overall style in the picture. Then, the target feature representation information of the layout element can be generated based on the descriptive text. This information can accurately express the key elements and layout requirements mentioned in the text. Then, the layout network of the poster generation model is used to perform self-attention encoding on the target feature representation information to obtain the target initial position information of the layout element. Through the self-attention mechanism, the model can learn the spatial relationship and dependency between the layout elements, so as to obtain a reasonable initial layout. Subsequently, the interactive layer of the poster generation model can be used to perform feature interaction on the target global image feature information and the target initial position information. Through this interactive process, the position of the layout element can be further adjusted to make it more consistent with the content and overall style of the picture. Finally, the target layout element position information of the layout element can be obtained. Finally, the poster to be generated can be generated based on the target layout element position information, the landscape picture and the descriptive text.

[0096] The embodiment of the present application can perform deep feature extraction on the image by adopting the image network of the poster generation model, accurately capturing the key elements and overall style in the image. The layout network performs self-attention encoding on the target feature representation information, intelligently predicts and optimizes the position of the layout elements, and improves the rationality and aesthetics of the poster layout. This process reduces manual intervention and improves the level of automation of poster generation. The introduction of the interactive layer enables the deep fusion of image features and layout information, further refines and adjusts the position of layout elements, ensures the perfect fit between the poster content and the image content, and enhances the visual appeal and information transmission effect of the poster.

[0097] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned annotated poster images and target parameters, the above-mentioned annotated poster images and target parameters can also be stored in a node of a blockchain.

[0098] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0099] The embodiments of the present application can build and optimize related models and networks based on artificial intelligence technology, such as image networks, layout networks, interaction layers, poster generation models, etc. Among them, artificial intelligence (AI) models are the crystallization of theory and practice that simulates the decision-making process of human intelligence through algorithms and data analysis to solve complex problems, predict future trends, or realize automation tasks. These models use a large amount of historical data and real-time information, and are trained and optimized through a specific algorithm framework to achieve efficient, accurate and reliable performance.

[0100] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0101] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0102] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a training device for a poster generation model, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0103] like Figure 3 As shown, the poster generation model training device 400 of this embodiment includes: an image acquisition module 401, a feature extraction module 402, an encoding module 403, a feature interaction module 404 and a parameter training module 405. Among them:

[0104] An image acquisition module 401 is used to acquire a poster image with annotations, where the poster image includes a plurality of layout element images and position annotation information of the layout elements of each layout element image;

[0105] A feature extraction module 402 is used to extract image features of the poster image using a preset image network to obtain global image feature information of the poster image;

[0106] The encoding module 403 is used to obtain feature representation information of the layout element, and use a preset layout network to perform self-attention encoding on the feature representation information to obtain initial position information of the layout element;

[0107] A feature interaction module 404 is used to perform feature interaction on the global image feature information and the initial position information using a preset interaction layer to obtain layout element position information of the layout element;

[0108] The parameter training module 405 is used to train the parameters of the image network, the parameters of the layout network and the parameters of the interaction layer respectively based on the difference between the layout element position information and the position annotation information, obtain the trained target parameters, and construct a poster generation model based on the target parameters, the image network, the layout network and the interaction layer.

[0109] The embodiments of the present application can significantly improve the performance of the poster generation model through refined data processing and deep learning model construction. First, the annotated poster images are used as training materials to ensure that the model can learn accurate layout element position information. The global image features are extracted through a preset image network to provide a rich visual context for the model. Then, the layout network uses the self-attention encoding mechanism to effectively capture the interdependence between layout elements and generate more reasonable initial position information. The introduction of the interaction layer further promotes the deep fusion of global image features and initial position information, making the generated layout element position information more accurate. During the training process, the model parameters are dynamically adjusted according to the difference between the layout element position information and the position annotation information, ensuring that the model can gradually approach the real layout. Finally, the poster generation model constructed based on the trained target parameters can realize the automation of poster layout and efficiently generate posters with reasonable layout.

[0110] In one embodiment, the feature extraction module 402 includes:

[0111] The segmentation submodule is used to segment the poster image using a preset image segmentation algorithm to obtain multiple image blocks;

[0112] The feature extraction submodule is used to extract the local image features of each image block using a preset image network to obtain multiple local image feature information;

[0113] The fusion submodule is used to fuse multiple local image feature information to obtain the global image feature information of the poster image.

[0114] The embodiment of the present application can segment the poster image into multiple image blocks through a preset image segmentation algorithm, so as to capture the local features in detail. By using the preset image network, the local image features of each image block can be efficiently extracted to fully mine the image information. Finally, by fusing multiple local image feature information, the global feature information of the poster image is obtained, which helps to capture the overall style and key information distribution of the poster.

[0115] In one embodiment, the encoding module 403 includes:

[0116] Get submodule, used to get category information and size information of layout elements;

[0117] The embedding submodule is used to embed the category information and size information to obtain the position embedding information of the layout element;

[0118] The encoding submodule is used to use a preset feature encoding network to perform feature encoding on the layout element image to obtain the image feature encoding of the layout element; and to embed the position information and the image feature encoding as the feature representation information of the layout element.

[0119] The embodiment of the present application can embed the category information and size information, and the position embedding information obtained can accurately reflect the position and size relationship of the layout element in the overall layout. At the same time, the layout element image is feature-encoded using a preset feature coding network, which can efficiently extract the local and global features of the image and generate image feature coding with rich semantic information.

[0120] In one embodiment, the embedding submodule is further used to use a preset category embedding dictionary to map the category information into a corresponding category embedding vector, normalize the width information and the height information to obtain a width vector corresponding to the width information, and a height vector corresponding to the height information; concatenate the category embedding vector, the width vector and the height vector to obtain a feature embedding vector of the layout element; perform a nonlinear transformation on the feature embedding vector to obtain a hidden feature vector of the layout element, the hidden feature vector is used to represent abstract information of the category information and size information of the layout element; generate a position embedding vector of the layout element according to the position information of the layout element in the poster image using a preset position encoding function; add the hidden feature vector and the position embedding vector to obtain a layout element embedding vector of the layout element, the layout element embedding vector is used to represent the content information and position information of the layout element; perform encoding enhancement processing on the layout element embedding vector to obtain the position embedding information of the layout element.

[0121] The embodiment of the present application can map the category information to the category embedding vector through a preset category embedding dictionary, which effectively captures the semantic relationship between categories. At the same time, the width and height information are normalized and spliced ​​into a feature embedding vector, and the hidden feature vector is obtained by combining the nonlinear transformation, which abstractly represents the category and size information of the layout element. The position embedding vector generated by the position encoding function is further integrated into the spatial position information of the layout element. Finally, the layout element embedding vector obtained by the coding enhancement process comprehensively represents the content information and position information of the layout element, providing more comprehensive and accurate feature support for subsequent poster image analysis, recognition or processing tasks.

[0122] In one embodiment, the encoding module 403 includes:

[0123] A splicing submodule is used to use a preset layout network to perform information splicing on the position embedding information and the image feature encoding to obtain a position enhanced feature vector of the layout element;

[0124] A self-attention processing submodule, which is used to use the self-attention mechanism of the layout network to perform self-attention processing on the position enhancement feature vector to obtain the element-related feature vector of the layout element;

[0125] The transformation fusion submodule is used to use the feedforward neural network of the layout network to perform nonlinear transformation fusion on the element-related feature vectors to obtain the deep semantic features of the layout elements;

[0126] The coordinate prediction submodule is used to obtain a preset coordinate prediction model, and based on deep semantic features, use the coordinate prediction model to predict the coordinates of the layout elements, obtain the predicted position coordinate values ​​of the layout elements, and determine the initial position information of the layout elements based on the predicted position coordinate values.

[0127] The embodiment of the present application can fuse the position embedding information with the image feature encoding through the layout network to generate a position enhanced feature vector, which comprehensively enriches the feature representation of the layout elements. Utilizing the self-attention mechanism, the association information between the layout elements is deeply mined to form an element association feature vector, thereby enhancing the information content of the feature vector. After nonlinear transformation and fusion through the feedforward neural network, the deep semantic features of the layout elements are extracted to accurately reflect their position, relationship, context and function in the overall layout. Finally, the coordinate prediction model based on the deep semantic features accurately predicts the position of the layout elements, providing strong support for layout optimization and intelligent design.

[0128] In one embodiment, the feature interaction module 404 includes:

[0129] A conversion submodule, for converting the initial position information into a query vector and converting the global image feature information into a key vector and a value vector using a preset interaction layer;

[0130] A dot product operation submodule is used to perform a dot product operation on the key vector and the query vector to obtain an attention weight value, which is used to characterize the correlation between the initial position information in the layout element and the global image feature information;

[0131] The weighted sum operation submodule is used to perform a weighted sum operation on the value vector and the attention weight to obtain the layout element position information of the layout element.

[0132] The embodiments of the present application can achieve deep fusion of the initial position information of the layout elements and the global image feature information through a preset interaction layer. Specifically, the conversion of the query vector, key vector and value vector, as well as the attention weight value obtained by the dot product operation are used to accurately characterize the correlation between the layout element position information and the global features. Through the weighted sum operation, the global feature information most relevant to the layout element position is further integrated, thereby significantly improving the accuracy and completeness of the layout element position information.

[0133] In one embodiment, the parameter training module 405 includes:

[0134] A distance calculation submodule, used to calculate the Euclidean distance between the layout element position information and the position annotation information, and to construct a position regression loss function based on the Euclidean distance. The position regression loss function is used to measure the degree of deviation between the layout element position indicated by the layout element position information and the annotation position indicated by the position annotation information.

[0135] An I / O calculation submodule is used to calculate the I / O ratio of the layout element position information and the position annotation information, and based on the I / O ratio, construct a position classification loss function, which is used to measure the degree of overlap between the layout element position indicated by the layout element position information and the annotation position indicated by the position annotation information;

[0136] The weighted summation submodule is used to perform weighted summation of the position regression loss function and the position classification loss function to obtain the total loss function;

[0137] The update submodule is used to update the parameters of the image network, the layout network and the interaction layer respectively based on the total loss function and the back propagation algorithm to obtain the trained target parameters.

[0138] The embodiment of the present application can construct a position regression loss function by calculating the Euclidean distance between the layout element position information and the position annotation information, accurately quantify the position deviation, and use the intersection-over-union ratio to construct a position classification loss function to effectively evaluate the degree of position overlap. The total loss function formed by the combination of the two comprehensively guides the optimization of image network, layout network and interaction layer parameters. The back propagation algorithm is used to iteratively update parameters, which significantly improves the accuracy of layout element position prediction, makes the predicted position closer to the actual annotation, optimizes the layout effect, and improves the intelligent level of image processing and layout optimization.

[0139] In one embodiment, the poster generation model training device 400 further includes:

[0140] An information acquisition module is used to acquire material information of the poster to be generated, the material information includes image and text prompt information;

[0141] An extraction module is used to use the image network of the poster generation model to perform image feature extraction processing on the image to obtain target global image feature information of the image;

[0142] A first generating module, used to generate target feature representation information of layout elements of a poster to be generated based on the text prompt information;

[0143] A self-attention encoding module, which is used to use the layout network of the poster generation model to perform self-attention encoding on the target feature representation information to obtain the target initial position information of the layout elements of the poster to be generated;

[0144] An interaction module, used to use the interaction layer of the poster generation model to perform feature interaction on the target global image feature information and the target initial position information to obtain the target layout element position information of the layout element of the poster to be generated;

[0145] The second generation module is used to generate a poster to be generated based on the target layout element position information, image and text prompt information.

[0146] The embodiment of the present application can perform deep feature extraction on the image by adopting the image network of the poster generation model, accurately capturing the key elements and overall style in the image. The layout network performs self-attention encoding on the target feature representation information, intelligently predicts and optimizes the position of the layout elements, and improves the rationality and aesthetics of the poster layout. This process reduces manual intervention and improves the level of automation of poster generation. The introduction of the interactive layer enables the deep fusion of image features and layout information, further refines and adjusts the position of layout elements, ensures the perfect fit between the poster content and the image content, and enhances the visual appeal and information transmission effect of the poster.

[0147] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0148] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 6 having a memory 61, a processor 62, and a network interface 63, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASIC), programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc.

[0149] Computer devices can be computing devices such as desktop computers, notebooks, PDAs, and cloud servers. Computer devices can interact with users through keyboards, mice, remote controls, touch pads, or voice control devices.

[0150] The memory 61 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 61 can be an internal storage unit of the computer device 6, such as a hard disk or memory of the computer device 6. In other embodiments, the memory 61 can also be an external storage device of the computer device 6, such as a plug-in hard disk equipped on the computer device 6, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 61 can also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 61 is generally used to store the operating system and various application software installed on the computer device 6, such as computer-readable instructions of the training method of the poster generation model, etc. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or are to be output.

[0151] The processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 62 is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to run computer-readable instructions stored in the memory 61 or process data, such as computer-readable instructions for running a training method for a poster generation model.

[0152] The network interface 63 may include a wireless network interface or a wired network interface, and the network interface 63 is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0153] The embodiments of the present application can significantly improve the performance of the poster generation model through refined data processing and deep learning model construction. First, the annotated poster images are used as training materials to ensure that the model can learn accurate layout element position information. The global image features are extracted through a preset image network to provide a rich visual context for the model. Then, the layout network uses the self-attention encoding mechanism to effectively capture the interdependence between layout elements and generate more reasonable initial position information. The introduction of the interaction layer further promotes the deep fusion of global image features and initial position information, making the generated layout element position information more accurate. During the training process, the model parameters are dynamically adjusted according to the difference between the layout element position information and the position annotation information, ensuring that the model can gradually approach the real layout. Finally, the poster generation model constructed based on the trained target parameters can realize the automation of poster layout and efficiently generate posters with reasonable layout.

[0154] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the training method of the poster generation model as described above.

[0155] The embodiments of the present application can significantly improve the performance of the poster generation model through refined data processing and deep learning model construction. First, the annotated poster images are used as training materials to ensure that the model can learn accurate layout element position information. The global image features are extracted through a preset image network to provide a rich visual context for the model. Then, the layout network uses the self-attention encoding mechanism to effectively capture the interdependence between layout elements and generate more reasonable initial position information. The introduction of the interaction layer further promotes the deep fusion of global image features and initial position information, making the generated layout element position information more accurate. During the training process, the model parameters are dynamically adjusted according to the difference between the layout element position information and the position annotation information, ensuring that the model can gradually approach the real layout. Finally, the poster generation model constructed based on the trained target parameters can realize the automation of poster layout and efficiently generate posters with reasonable layout.

[0156] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0157] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

[0158] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A training method for a poster generation model, characterized in that: The steps include: Acquire a poster image with annotations, wherein the poster image includes a plurality of layout element images and position annotation information of the layout element of each layout element image; Using a preset image network, extracting image features of the poster image to obtain global image feature information of the poster image; Acquire feature representation information of the layout element, use a preset layout network to perform self-attention encoding on the feature representation information, and obtain initial position information of the layout element; Using a preset interaction layer, performing feature interaction on the global image feature information and the initial position information to obtain layout element position information of the layout element; Based on the difference between the layout element position information and the position annotation information, the parameters of the image network, the parameters of the layout network and the parameters of the interaction layer are trained respectively to obtain the trained target parameters, and a poster generation model is constructed based on the target parameters, the image network, the layout network and the interaction layer.

2. The method according to claim 1, characterized in that The step of obtaining the feature representation information of the layout element specifically includes: Obtaining category information and size information of the layout element; Embedding the category information and the size information to obtain position embedding information of the layout element; Using a preset feature coding network, feature coding the layout element image to obtain the image feature coding of the layout element; The position embedding information and the image features are encoded as feature representation information of the layout element.

3. The method according to claim 2, characterized in that The size information includes width information and height information, and the step of embedding the category information and the size information to obtain the position embedding information of the layout element specifically includes: Using a preset category embedding dictionary, mapping the category information into a corresponding category embedding vector, normalizing the width information and the height information to obtain a width vector corresponding to the width information and a height vector corresponding to the height information; Concatenating the category embedding vector, the width vector, and the height vector to obtain a feature embedding vector of the layout element; Performing a nonlinear transformation on the feature embedding vector to obtain a hidden feature vector of the layout element, wherein the hidden feature vector is used to represent abstract information of category information and size information of the layout element; According to the position information of the layout element in the poster image, a preset position encoding function is used to generate a position embedding vector of the layout element; Adding the hidden feature vector and the position embedding vector to obtain a layout element embedding vector of the layout element, wherein the layout element embedding vector is used to represent content information and position information of the layout element; The layout element embedding vector is subjected to coding enhancement processing to obtain position embedding information of the layout element.

4. The method according to claim 2, characterized in that: The step of using a preset layout network to perform self-attention encoding on the feature representation information to obtain the initial position information of the layout element specifically includes: Using a preset layout network, the position embedding information and the image feature code are concatenated to obtain a position enhancement feature vector of the layout element; Using the self-attention mechanism of the layout network, performing self-attention processing on the position enhancement feature vector to obtain an element association feature vector of the layout element; Using the feedforward neural network of the layout network, nonlinear transformation and fusion are performed on the element-related feature vectors to obtain deep semantic features of the layout elements; A preset coordinate prediction model is obtained, and based on the deep semantic features, the coordinate prediction model is used to perform coordinate prediction on the layout element to obtain a predicted position coordinate value of the layout element, and based on the predicted position coordinate value, initial position information of the layout element is determined.

5. The method according to claim 1, characterized in that The step of using a preset interaction layer to perform feature interaction on the global image feature information and the initial position information to obtain the layout element position information of the layout element specifically includes: Using a preset interaction layer, converting the initial position information into a query vector, and converting the global image feature information into a key vector and a value vector; Performing a dot product operation on the key vector and the query vector to obtain an attention weight value, where the attention weight value is used to characterize the correlation between the initial position information in the layout element and the global image feature information; A weighted sum operation is performed on the value vector and the attention weight to obtain layout element position information of the layout element.

6. The method according to claim 1, characterized in that The step of respectively training the parameters of the image network, the parameters of the layout network, and the parameters of the interaction layer based on the difference between the layout element position information and the position annotation information to obtain the trained target parameters specifically includes: Calculating the Euclidean distance between the layout element position information and the position annotation information, and constructing a position regression loss function based on the Euclidean distance, wherein the position regression loss function is used to measure the degree of deviation between the layout element position indicated by the layout element position information and the annotation position indicated by the position annotation information; Calculating an intersection-and-union ratio of the layout element position information and the position annotation information, and constructing a position classification loss function based on the intersection-and-union ratio, wherein the position classification loss function is used to measure the degree of overlap between the layout element position indicated by the layout element position information and the annotation position indicated by the position annotation information; Performing a weighted summation on the position regression loss function and the position classification loss function to obtain a total loss function; Based on the total loss function, a back propagation algorithm is used to update the parameters of the image network, the parameters of the layout network, and the parameters of the interaction layer respectively to obtain the trained target parameters.

7. The method according to claim 1, characterized in that After the step of constructing a poster generation model based on the target parameters, the image network, the layout network and the interaction layer, the step further includes: Obtaining material information of a poster to be generated, wherein the material information includes image and text prompt information; Using the image network of the poster generation model, performing image feature extraction processing on the image to obtain target global image feature information of the image; Based on the text prompt information, generating target feature representation information of the layout elements of the poster to be generated; Using the layout network of the poster generation model, self-attention encoding is performed on the target feature representation information to obtain the target initial position information of the layout elements of the poster to be generated; Using the interactive layer of the poster generation model, feature interaction is performed on the target global image feature information and the target initial position information to obtain target layout element position information of the layout element of the poster to be generated; The poster to be generated is generated based on the target layout element position information, the image and the text prompt information.

8. A training device for a poster generation model, characterized in that: include: An image acquisition module, used to acquire a poster image with annotations, wherein the poster image includes a plurality of layout element images and position annotation information of the layout element of each layout element image; A feature extraction module, used to extract image features of the poster image using a preset image network to obtain global image feature information of the poster image; An encoding module, used to obtain feature representation information of the layout element, and use a preset layout network to perform self-attention encoding on the feature representation information to obtain initial position information of the layout element; A feature interaction module, configured to use a preset interaction layer to perform feature interaction on the global image feature information and the initial position information to obtain layout element position information of the layout element; A parameter training module is used to train the parameters of the image network, the parameters of the layout network and the parameters of the interaction layer respectively based on the difference between the layout element position information and the position annotation information, obtain the trained target parameters, and construct a poster generation model based on the target parameters, the image network, the layout network and the interaction layer.

9. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the training method of the poster generation model as claimed in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the training method for the poster generation model as described in any one of claims 1 to 7 are implemented.

Images