Poster generation method and device, computer equipment and storage medium

Abstract

The embodiment of the invention belongs to the field of artificial intelligence, and relates to a poster generation method, which comprises the following steps: acquiring an initial poster image; inputting the initial poster image into an image model in a poster generation model to obtain poster image features and multi-modal information of layout elements in the initial poster image; inputting the multi-modal information into a layout model in the poster generation model to obtain layout information; inputting the poster image features and the layout information into a cross attention network in a poster generation model for feature interaction to obtain improved layout information; and mapping the improved layout information back to the image space to obtain a target poster image. The invention further provides a poster generation device, computer equipment and a storage medium. According to the invention, the accurate layout of each element in the poster is automatically realized.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a poster generation method, device, computer equipment and storage medium. Background Art

[0002] In the field of poster production, designers usually need to manually perform page layout and typesetting, which is a cumbersome and time-consuming process. This processing method is not only inefficient, but also difficult to meet the growing personalized needs. With the development of AI (Artificial Intelligence) technology, AI-controlled layout generation technology has emerged. However, this technology still has certain shortcomings. In terms of automatic poster layout, how to effectively extract image features and accurately layout each element is an urgent problem to be solved. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a poster generation method, apparatus, computer equipment and storage medium to achieve automatic layout and generation of posters.

[0004] In order to solve the above technical problems, the embodiment of the present application provides a poster generation method, which adopts the following technical solution:

[0005] Get the initial poster image;

[0006] Inputting the initial poster image into an image model in a poster generation model to obtain poster image features and multimodal information of layout elements in the initial poster image;

[0007] Inputting the multimodal information into a layout model in the poster generation model to obtain layout information;

[0008] Inputting the poster image features and the layout information into the cross attention network in the poster generation model for feature interaction to obtain improved layout information;

[0009] The improved layout information is mapped back to the image space to obtain a target poster image.

[0010] In order to solve the above technical problems, the embodiment of the present application further provides a poster generation device, which adopts the following technical solution:

[0011] An initial acquisition module, used to acquire an initial poster image;

[0012] An initial input module, used for inputting the initial poster image into an image model in a poster generation model to obtain poster image features and multimodal information of layout elements in the initial poster image;

[0013] An information input module, used for inputting the multimodal information into the layout model in the poster generation model to obtain layout information;

[0014] A feature interaction module, used for inputting the poster image features and the layout information into a cross attention network in the poster generation model to perform feature interaction, so as to obtain improved layout information;

[0015] The image mapping module is used to map the improved layout information back to the image space to obtain a target poster image.

[0016] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0017] Get the initial poster image;

[0018] Inputting the initial poster image into an image model in a poster generation model to obtain poster image features and multimodal information of layout elements in the initial poster image;

[0019] Inputting the multimodal information into a layout model in the poster generation model to obtain layout information;

[0020] Inputting the poster image features and the layout information into the cross attention network in the poster generation model for feature interaction to obtain improved layout information;

[0021] The improved layout information is mapped back to the image space to obtain a target poster image.

[0022] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0023] Get the initial poster image;

[0024] Inputting the initial poster image into an image model in a poster generation model to obtain poster image features and multimodal information of layout elements in the initial poster image;

[0025] Inputting the multimodal information into a layout model in the poster generation model to obtain layout information;

[0026] Inputting the poster image features and the layout information into the cross attention network in the poster generation model for feature interaction to obtain improved layout information;

[0027] The improved layout information is mapped back to the image space to obtain a target poster image.

[0028] Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects: an initial poster image is acquired and input into an image model in a poster generation model, poster image features and multimodal information of layout elements in the initial poster image are obtained, and the multimodal information is multidimensional description information of the layout elements; the multimodal information is input into a layout model in the poster generation model to obtain layout information, and the layout information describes the position of the layout elements in the image; the poster image features and the layout information are input into a cross-attention network in the poster generation model for feature interaction, so that the dynamic interaction between the image features and the layout information is strengthened, the position of the layout elements is optimized, the rationality of the layout and the visual appeal are ensured, and the integration of the layout elements is improved; the improved layout information is mapped back to the image space to obtain the target poster image, thereby improving the quality of the generated poster. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0030] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0031] Figure 2 is a flowchart of an embodiment of a poster generation method according to the present application;

[0032] Figure 3 is a structural schematic diagram of an embodiment of a poster generating device according to the present application;

[0033] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0035] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0036] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0037] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0038] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0039] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0040] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0041] It should be noted that the poster generation method provided in the embodiment of the present application is generally executed by a server, and accordingly, the poster generation device is generally set in the server.

[0042] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0043] Continue to refer Figure 2 , shows a flowchart of an embodiment of a poster generation method according to the present application. The poster generation method comprises the following steps:

[0044] Step S201, obtaining an initial poster image.

[0045] In this embodiment, the poster generation method is executed on the electronic device (eg Figure 1 The server shown in the figure) can communicate with the terminal device through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0046] Specifically, firstly, an initial poster image is obtained. The image may be uploaded by a user or stored in a database. It is an initialized poster image and needs to be further adjusted and optimized.

[0047] Step S202: input the initial poster image into the image model in the poster generation model to obtain poster image features and multimodal information of layout elements in the initial poster image.

[0048] Specifically, the initial poster image is input into the image model in the poster generation model. The image model extracts the features of the poster image. The poster image contains layout elements, which usually refer to the key visual components that constitute the overall structure of the poster, such as titles, images, or icons; the image model recognizes the multimodal information of the layout elements, including the category, width, and height of the elements.

[0049] In this application, the poster generation model can be a composite model, including an image model, a layout model, and a cross-attention network. The image model, the layout model, and the cross-attention network can all be built based on the transformer network.

[0050] Step S203, inputting the multimodal information into the layout model in the poster generation model to obtain layout information.

[0051] Specifically, the extracted multimodal information is input into the layout model, and the layout model generates layout information based on the multimodal information. The layout information can describe the position of each layout element in the image.

[0052] Step S204, input the poster image features and layout information into the cross-attention network in the poster generation model for feature interaction to obtain improved layout information.

[0053] Specifically, the poster image features and layout information are fed into the cross-attention network to achieve feature interaction. Through feature interaction, the image content and layout information can influence each other, thereby optimizing the position of layout elements and outputting improved layout information.

[0054] Step S205 , mapping the improved layout information back to the image space to obtain a target poster image.

[0055] Specifically, the improved layout information is mapped back to the image space to generate the final target poster image, which is more visually attractive and well laid out, with the element positions more consistent with the image content.

[0056] In this embodiment, an initial poster image is obtained and input into an image model in a poster generation model to obtain poster image features and multimodal information of layout elements in the initial poster image, where the multimodal information is a multidimensional description of the layout elements; the multimodal information is input into a layout model in the poster generation model to obtain layout information, where the layout information describes the positions of the layout elements in the image; the poster image features and the layout information are input into a cross-attention network in the poster generation model for feature interaction, so that the dynamic interaction between the image features and the layout information is enhanced, the positions of the layout elements are optimized, the rationality of the layout and the visual appeal are ensured, and the integration of the layout elements is improved; the improved layout information is mapped back to the image space to obtain the target poster image, thereby improving the quality of the generated poster.

[0057] Furthermore, the above-mentioned step S202 may include: inputting the initial poster image into the image model in the poster generation model, and performing image segmentation on the initial poster image through the image model to obtain multiple image blocks; marking the layout element image blocks and non-layout element image blocks in each image block through a mask strategy; performing feature extraction on the layout elements in each layout element image block respectively to obtain a first image block feature, and obtaining the category and size information of the layout elements in each layout element image block; inputting each non-layout element image block into the encoder in the image model to obtain a second image block feature; inputting the first image block feature and the second image block feature into the decoder in the image model to obtain the poster image feature of the initial poster image, and determining the first image block feature, category and size information of the layout element in each layout element image block as the multimodal information of the layout element in the initial poster image.

[0058] Specifically, the initial poster image is input into the image model in the poster generation model. Through image segmentation technology, the image model divides the initial poster image into multiple small image blocks. Different image blocks contain different visual information. Some image blocks contain layout elements and belong to layout element image blocks, while some image blocks do not contain layout elements and belong to non-layout element image blocks.

[0059] Each image block is marked through a mask strategy to distinguish layout element image blocks from non-layout element image blocks, allowing the model to focus on layout-related parts.

[0060] For the layout element image block, the image model performs feature extraction to generate the first image block feature, and at the same time obtains the category and size information (width and height) of the layout element in the image block. The category and size information constitute the multimodal information of the layout element and provide important features of the layout element in the design.

[0061] For non-layout element image blocks, they are input into the encoder in the image model to obtain the second image block features. The first image block features and the second image block features are input into the decoder in the image model to generate the overall poster image features of the initial poster image. Through this series of steps, the model not only extracts the multimodal information of the layout elements, but also fully understands and encodes the entire poster image.

[0062] In this embodiment, by applying image segmentation and masking strategies to the initial poster image, the distinction between layout elements and non-layout elements is achieved, so that the model can focus on extracting the features of key layout elements and accurately obtain their category and size information, laying the foundation for subsequent layout optimization; by extracting the features of non-layout elements, it is ensured that the overall information of the image is fully utilized.

[0063] Furthermore, the above step S203 may include: inputting the multimodal information into the layout model in the poster generation model to obtain the position, confidence and category probability of the layout elements in the initial poster image; and determining the obtained position, confidence and category probability as the layout information of the layout elements in the initial poster image.

[0064] Specifically, the extracted multimodal information (class and size information of layout elements) is input into the layout model in the poster generation model. The layout model focuses on processing and optimizing the position and relationship of layout elements in the image. By analyzing the input multimodal information, the layout model outputs the specific position, confidence and class probability of each layout element. Among them:

[0065] Position: refers to the exact coordinates of the layout element within the initial poster image.

[0066] Confidence: Indicates the model's confidence in the existence of each layout element. The higher the value, the more certain the model is about the correctness of the element's position.

[0067] Category probability: refers to the probability of each layout element belonging to a category, which is used to identify the element type (such as title, picture or text box).

[0068] These output information constitute the layout information of the layout elements in the initial poster image, providing basic data for further image generation and optimization.

[0069] In this embodiment, multimodal information is input into the layout model, which improves the ability to accurately locate and classify layout elements in the image; the position, confidence, and category probability output by the model ensure that the design information of each layout element is accurately captured, making the generated poster visually coordinated and unified, and the element layout more reasonable; the confidence enables the model to better handle uncertainty, thereby improving the stability and reliability of the overall layout.

[0070] Furthermore, the above step S204 may include: taking the poster image features as the query and the layout information as the key and value, respectively inputting them into the cross-attention network in the poster generation model; calculating the similarity between the query and the key through the cross-attention network to obtain the attention weight; multiplying the obtained attention weight with the value, and performing weighted summation on the multiplication result to obtain the weighted feature representation after interaction; integrating the weighted feature representation with the layout information to obtain improved layout information.

[0071] Specifically, the poster image features extracted from the image model are used as the query (Query), and the layout information obtained from the layout model is used as the key (Key) and the value (Value).

[0072] The cross-attention network calculates the similarity between the query and the key to determine the relevance of the poster image features and layout information, which can be achieved through inner product or other similarity calculation methods.

[0073] The similarity results are normalized (usually using the Softmax function) to obtain attention weights, which reflect the importance of layout information in feature interactions.

[0074] The attention weight is multiplied by the value, and the multiplication result is weighted summed to obtain the weighted feature representation after interaction. The weighted feature representation integrates the relationship between image features and layout information, which can enhance the final output.

[0075] The weighted feature representation is integrated with the layout information to generate improved layout information. The improved layout information provides a more accurate and optimized basis for the subsequent generation of the target poster image.

[0076] In this embodiment, the poster image features and layout information are integrated through the cross-attention network to achieve dynamic interaction between features; the similarity calculation between the query and the key enables the model to automatically identify the association between important layout elements and image content, thereby enhancing the relevance of information; through weighted summation, the model can highlight the key features in layout optimization and generate more reasonable new layout information.

[0077] Furthermore, before the above step S201, it can also include: obtaining a training poster image; inputting the training poster image into the initial image model in the poster generation model to obtain a reconstructed poster image; calculating the first loss of the model according to the training poster image and the reconstructed poster image; adjusting the initial image model according to the first loss until the first loss meets the training stop condition to obtain the image model.

[0078] Specifically, during the model training phase, training poster images are obtained. These images are the data set used to train the model, which contain a variety of design styles and layout elements, providing rich samples for model learning.

[0079] The training poster images are input to the initial image model in the poster generation model. The model will try to reconstruct these poster images and generate output images similar to the input images. The initial image model processes the training poster images to obtain poster image features, and image reconstruction can be performed based on the poster image features.

[0080] The first loss of the model is calculated based on the difference between the training poster image and the reconstructed poster image. The first loss reflects the performance of the model in the reconstruction task. The smaller the loss, the better the model reconstruction effect. In one embodiment, the difference between the reconstructed poster image and the training poster image is calculated using the mean square error (MSE) loss function in pixel space. The first loss can be calculated only on the masked image blocks, similar to the masked language model task in BERT.

[0081] Based on the calculated first loss, the model adjusts the parameters of the initial image model until the loss meets the preset training stop condition. This process ensures that the model is continuously optimized, and ultimately forms an image model that can accurately generate and reconstruct the poster image.

[0082] In this embodiment, the performance of the initial image model is improved by establishing an effective training process; the training poster image is obtained and reconstructed so that the model can learn rich visual features and layout information when processing images of different styles; by calculating the reconstruction loss, the model can make corresponding parameter adjustments in a timely manner, thereby continuously improving the generation quality.

[0083] Furthermore, the above-mentioned poster generation method may also include: obtaining a training poster image and its corresponding layout label; based on the training poster image, obtaining the training layout information output by the initial cross-attention network in the poster generation model; calculating the second loss according to the training layout information and the layout label; adjusting the initial layout model and the initial cross-attention network in the poster generation model according to the second loss until the second loss meets the training stop condition to obtain the layout model.

[0084] Specifically, during the model training phase, training poster images and their corresponding layout labels can also be obtained. The layout label is key data for identifying the exact location and type of each layout element.

[0085] Based on the training poster image, the training layout information is obtained from the initial cross-attention network in the poster generation model. Then, the second loss is calculated based on the obtained training layout information and layout label. The second loss reflects the performance of the model in positioning and classifying layout elements. The smaller the second loss, the closer the layout information predicted by the model is to the true label.

[0086] According to the calculated second loss, the parameters of the initial layout model and the initial cross-attention network in the poster generation model are adjusted until the second loss meets the preset training stop condition. This process ensures that the model is continuously optimized and eventually forms an accurate layout model that can effectively generate and optimize layout information.

[0087] In this embodiment, by obtaining training poster images and their layout labels and using a cross-attention network to learn the layout information, the accuracy and reliability of the layout model are improved; by calculating the second loss between the layout prediction and the true label, the model can adjust its parameters in a timely manner to enhance the ability to recognize and locate layout elements, which not only improves the generation quality of layout information, but also ensures that the generated poster has higher coordination and professionalism in visual effects.

[0088] Furthermore, the above-mentioned step of calculating the second loss based on the training layout information and layout labels may include: obtaining the position label, confidence label and category label in the layout label, and obtaining the predicted position, prediction confidence and predicted category probability in the training layout information; calculating the first sub-loss based on the position label and the predicted position; calculating the second sub-loss and the third sub-loss based on the confidence label and the predicted confidence; calculating the fourth sub-loss based on the category label and the predicted category probability; calculating the second loss based on the first sub-loss, the second sub-loss, the third sub-loss and the fourth sub-loss.

[0089] Specifically, the position label, confidence label and category label in the layout label are obtained. At the same time, the prediction results are obtained from the training layout information, including the predicted position, prediction confidence and prediction category probability.

[0090] The first sub-loss is calculated based on the position label and the predicted position to quantify the model's accuracy in locating layout elements. The second and third sub-losses are calculated based on the confidence label and the predicted confidence to evaluate the model's confidence in the element's existence. The fourth sub-loss is calculated using the category label and the predicted category probability to measure the model's accuracy in classifying layout elements.

[0091] All sub-losses (first sub-loss, second sub-loss, third sub-loss and fourth sub-loss) are combined to calculate the final second loss. This comprehensive loss provides comprehensive feedback to the model, guiding it to make corresponding parameter adjustments during training.

[0092] In one embodiment, the second loss is expressed as follows:

[0093]

[0094] Among them, loss is the second loss; For the first child loss; for the loss of the second child; For the loss of the third child; is the fourth child loss; box , obj , noobj , class are weight constants respectively; N1×N2 means dividing the initial poster image into N1×N2 image blocks / grids, each of which has N preset a Anchor; is the confidence label, indicating that if it is a positive example, the output is 1, otherwise the output is 0; is the confidence label, indicating that if it is a negative example, the output is 1, otherwise the output is 0; C i ' j represents the prediction confidence; t x and t y is the horizontal and vertical coordinates of the layout element in the position tag; t x ' and t y ' is the horizontal and vertical coordinates of the layout element in the predicted position; P ij (c) represents the category label, P ij '(c) represents the predicted class probability.

[0095] In this application, the construction and labeling of anchors is an important component to help train the layout model. The anchor label includes the position label t x and t y , confidence label C, category probability label P(c).

[0096] First, according to the width g of the layout elements in the training data set w and height g h , clustered by Kmeans clustering algorithm to get N a anchors of different shapes, assuming the width of the anchor is a w , height a h Then the entire image is divided into N1×N2 grids, with the midpoint of the grid as the center, and N is preset. a There are N1×N2×N anchors in total. a Anchor.

[0097] Next, construct a label for the anchor. Position label t x and t y Instead of using the groundtruth position directly, we use the difference between the groundtruth and the anchor, which makes subsequent regression training easier. Specifically, for any layout element, let the coordinates of the upper left corner be (g x , g y ), first determine the grid it belongs to based on its midpoint position, and then in the grid N a The anchor with the largest IOU ratio is selected as the positive example. The IOU is calculated by aligning the layout element with the upper left corner of the anchor. The formula is:

[0098]

[0099] Assume that the coordinates of the upper left corner of the anchor with the largest IOU are (a x , a y ), then the position label formula of the layout element is as follows:

[0100] t x =g x -a x

[0101] t y =g y -a y

[0102] In order to expand the positive examples, the anchors whose IOU with the layout elements in the grid is greater than the first threshold are taken as positive examples, the anchors whose IOU is less than the second threshold are taken as negative examples, and the other anchors between the first threshold and the second threshold are taken as irrelevant examples. The position label of the positive example is the t obtained above. x and t y , the confidence label C = 1, the category probability label P(c) corresponding to the category is 1, and the rest are 0; the confidence label C = 0 of the negative example.

[0103] In this embodiment, the training efficiency and accuracy of the layout model are enhanced by subdividing the loss calculation process; the precise measurement of each important element in the layout label enables the model to be comprehensively optimized in positioning, confidence assessment and category classification; the independent calculation and integration of each sub-loss provides the model with multi-dimensional performance feedback, thereby promoting continuous improvement of the model in all aspects, ensuring the high quality of the generated layout information, and improving the overall effect of poster generation.

[0104] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned target poster image, the above-mentioned target poster image can also be stored in a node of a blockchain.

[0105] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0106] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0107] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0108] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0109] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0110] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a poster generation device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0111] like Figure 3 As shown, the poster generation device 300 described in this embodiment includes: an initial acquisition module 301, an initial input module 302, an information input module 303, a feature interaction module 304 and an image mapping module 305, wherein:

[0112] The initial acquisition module 301 is used to acquire an initial poster image.

[0113] The initial input module 302 is used to input the initial poster image into the image model in the poster generation model to obtain poster image features and multimodal information of layout elements in the initial poster image.

[0114] The information input module 303 is used to input the multimodal information into the layout model in the poster generation model to obtain layout information.

[0115] The feature interaction module 304 is used to input the poster image features and layout information into the cross-attention network in the poster generation model for feature interaction to obtain improved layout information.

[0116] The image mapping module 305 is used to map the improved layout information back to the image space to obtain a target poster image.

[0117] In this embodiment, an initial poster image is obtained and input into an image model in a poster generation model to obtain poster image features and multimodal information of layout elements in the initial poster image, where the multimodal information is a multidimensional description of the layout elements; the multimodal information is input into a layout model in the poster generation model to obtain layout information, where the layout information describes the positions of the layout elements in the image; the poster image features and the layout information are input into a cross-attention network in the poster generation model for feature interaction, so that the dynamic interaction between the image features and the layout information is enhanced, the positions of the layout elements are optimized, the rationality of the layout and the visual appeal are ensured, and the integration of the layout elements is improved; the improved layout information is mapped back to the image space to obtain the target poster image, thereby improving the quality of the generated poster.

[0118] In some optional implementations of this embodiment, the initial input module 302 may also be used to:

[0119] The initial poster image is input into the image model in the poster generation model, and the image model is used to segment the initial poster image to obtain a plurality of image blocks.

[0120] The layout element image blocks and non-layout element image blocks in each image block are marked by a mask strategy.

[0121] Feature extraction is performed on the layout elements in each layout element image block to obtain first image block features, and category and size information of the layout elements in each layout element image block is obtained.

[0122] Each non-layout element image block is input into an encoder in the image model to obtain a second image block feature.

[0123] The first image block feature and the second image block feature are input into a decoder in the image model to obtain a poster image feature of the initial poster image, and the first image block feature, category and size information of the layout element in each layout element image block are determined as multimodal information of the layout element in the initial poster image.

[0124] In some optional implementations of this embodiment, the information input module 303 may also be used to:

[0125] The multimodal information is input into the layout model in the poster generation model to obtain the position, confidence and category probability of the layout elements in the initial poster image.

[0126] The obtained positions, confidences, and category probabilities are determined as layout information of the layout elements in the initial poster image.

[0127] In some optional implementations of this embodiment, the feature interaction module 304 may also be used to:

[0128] The poster image features are used as queries and the layout information is used as keys and values, respectively, and are input into the cross-attention network in the poster generation model.

[0129] The similarity between the query and the key is calculated through the cross-attention network to obtain the attention weight.

[0130] The obtained attention weight is multiplied by the value, and the multiplication results are weighted summed to obtain the weighted feature representation after interaction.

[0131] The weighted feature representation is integrated with the layout information to obtain improved layout information.

[0132] In some optional implementations of this embodiment, the poster generating device 300 may further include:

[0133] The training acquisition module is used to acquire training poster images.

[0134] The training reconstruction module is used to input the training poster image into the initial image model in the poster generation model to obtain the reconstructed poster image.

[0135] The first calculation module is used to calculate the first loss of the model according to the training poster image and the reconstructed poster image.

[0136] The first adjustment module is used to adjust the initial image model according to the first loss until the first loss meets the training stop condition to obtain the image model.

[0137] In some optional implementations of this embodiment, the poster generating device 300 may further include:

[0138] The acquisition module is used to obtain the training poster images and their corresponding layout labels.

[0139] The layout acquisition module is used to obtain the training layout information output by the initial cross-attention network in the poster generation model based on the training poster image.

[0140] The second calculation module is used to calculate the second loss according to the training layout information and the layout label.

[0141] The second adjustment module is used to adjust the initial layout model and the initial cross-attention network in the poster generation model according to the second loss until the second loss meets the training stop condition to obtain the layout model.

[0142] In some optional implementations of this embodiment, the second calculation module may also be used for:

[0143] Get the position label, confidence label, and category label in the layout label, and get the predicted position, predicted confidence, and predicted category probability in the training layout information.

[0144] The first sub-loss is calculated based on the position label and the predicted position.

[0145] The second and third sub-losses are calculated based on the confidence labels and prediction confidences.

[0146] The fourth sub-loss is calculated based on the class labels and the predicted class probabilities.

[0147] The second loss is calculated based on the first sub-loss, the second sub-loss, the third sub-loss and the fourth sub-loss.

[0148] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0149] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 4 with a memory 41, a processor 42, and a network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0150] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0151] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the poster generation method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0152] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the poster generation method.

[0153] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0154] The computer device provided in this embodiment can execute the above poster generation method. Here, the poster generation method can be the poster generation method of each of the above embodiments.

[0155] In this embodiment, an initial poster image is obtained and input into an image model in a poster generation model to obtain poster image features and multimodal information of layout elements in the initial poster image, where the multimodal information is a multidimensional description of the layout elements; the multimodal information is input into a layout model in the poster generation model to obtain layout information, where the layout information describes the positions of the layout elements in the image; the poster image features and the layout information are input into a cross-attention network in the poster generation model for feature interaction, so that the dynamic interaction between the image features and the layout information is enhanced, the positions of the layout elements are optimized, the rationality of the layout and the visual appeal are ensured, and the integration of the layout elements is improved; the improved layout information is mapped back to the image space to obtain the target poster image, thereby improving the quality of the generated poster.

[0156] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the poster generation method as described above.

[0157] In this embodiment, an initial poster image is obtained and input into an image model in a poster generation model to obtain poster image features and multimodal information of layout elements in the initial poster image, where the multimodal information is a multidimensional description of the layout elements; the multimodal information is input into a layout model in the poster generation model to obtain layout information, where the layout information describes the positions of the layout elements in the image; the poster image features and the layout information are input into a cross-attention network in the poster generation model for feature interaction, so that the dynamic interaction between the image features and the layout information is enhanced, the positions of the layout elements are optimized, the rationality of the layout and the visual appeal are ensured, and the integration of the layout elements is improved; the improved layout information is mapped back to the image space to obtain the target poster image, thereby improving the quality of the generated poster.

[0158] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0159] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A poster generation method, characterized in that: The steps include: Get the initial poster image; Inputting the initial poster image into an image model in a poster generation model to obtain poster image features and multimodal information of layout elements in the initial poster image; Inputting the multimodal information into a layout model in the poster generation model to obtain layout information; Inputting the poster image features and the layout information into the cross attention network in the poster generation model for feature interaction to obtain improved layout information; The improved layout information is mapped back to the image space to obtain a target poster image.

2. The poster generation method according to claim 1, characterized in that: The step of inputting the initial poster image into an image model in a poster generation model to obtain poster image features and multimodal information of layout elements in the initial poster image comprises: Inputting the initial poster image into an image model in a poster generation model, and performing image segmentation on the initial poster image using the image model to obtain a plurality of image blocks; Marking the layout element image blocks and non-layout element image blocks in each image block by using a mask strategy; Extracting features of the layout elements in each layout element image block respectively to obtain first image block features, and acquiring category and size information of the layout elements in each layout element image block; Inputting each non-layout element image block into an encoder in the image model to obtain a second image block feature; The first image block feature and the second image block feature are input into a decoder in the image model to obtain a poster image feature of the initial poster image, and the first image block feature, category and size information of the layout element in each layout element image block are determined as multimodal information of the layout element in the initial poster image.

3. The poster generation method according to claim 1, characterized in that: The step of inputting the multimodal information into the layout model in the poster generation model to obtain layout information comprises: Inputting the multimodal information into the layout model in the poster generation model to obtain the position, confidence and category probability of the layout elements in the initial poster image; The obtained position, confidence level and category probability are determined as layout information of the layout elements in the initial poster image.

4. The poster generation method according to claim 1, characterized in that: The step of inputting the poster image features and the layout information into the cross attention network in the poster generation model to perform feature interaction to obtain improved layout information comprises: Using the poster image feature as a query and the layout information as a key and a value, respectively inputting the poster image feature into a cross-attention network in the poster generation model; Calculate the similarity between the query and the key through the cross attention network to obtain an attention weight; Multiply the obtained attention weight by the value, and perform weighted summation on the multiplication results to obtain a weighted feature representation after interaction; The weighted feature representation is integrated with the layout information to obtain improved layout information.

5. The poster generation method according to claim 1, characterized in that: Before the step of obtaining the initial poster image, the method further includes: Get the training poster image; Inputting the training poster image into an initial image model in a poster generation model to obtain a reconstructed poster image; Calculate a first model loss based on the training poster image and the reconstructed poster image; The initial image model is adjusted according to the first loss until the first loss satisfies a training stop condition, thereby obtaining an image model.

6. The poster generation method according to claim 5, characterized in that: The method further comprises: Obtaining the training poster image and its corresponding layout label; Based on the training poster image, obtaining training layout information output by an initial cross-attention network in the poster generation model; Calculate a second loss according to the training layout information and the layout label; The initial layout model and the initial cross-attention network in the poster generation model are adjusted according to the second loss until the second loss meets the training stop condition to obtain a layout model.

7. The poster generation method according to claim 6, characterized in that: The step of calculating the second loss according to the training layout information and the layout label comprises: Obtaining the position label, confidence label and category label in the layout label, and obtaining the predicted position, predicted confidence and predicted category probability in the training layout information; Calculate a first sub-loss according to the position label and the predicted position; Calculating a second sub-loss and a third sub-loss according to the confidence label and the prediction confidence; Calculating a fourth sub-loss according to the category label and the predicted category probability; A second loss is calculated according to the first sub-loss, the second sub-loss, the third sub-loss, and the fourth sub-loss.

8. A poster generating device, characterized in that: include: An initial acquisition module, used to acquire an initial poster image; An initial input module, used for inputting the initial poster image into an image model in a poster generation model to obtain poster image features and multimodal information of layout elements in the initial poster image; An information input module, used for inputting the multimodal information into the layout model in the poster generation model to obtain layout information; A feature interaction module, used for inputting the poster image features and the layout information into a cross attention network in the poster generation model to perform feature interaction, so as to obtain improved layout information; The image mapping module is used to map the improved layout information back to the image space to obtain a target poster image.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the poster generation method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the poster generation method according to any one of claims 1 to 7 are implemented.

Images