Poster generation method, poster generation model training method, poster generation model training device and poster generation equipment

Abstract

The invention provides a poster generation method, a poster generation model training method, a poster generation model training device and poster generation equipment, and relates to the technical field of computers, in particular to the technical field of artificial intelligence such as computer vision, deep learning and large models. The method comprises the following steps: firstly, receiving a poster generation request, generating first prompt information based on the generation request, then determining second prompt information based on the first prompt information, inputting the second prompt information into a generated text encoder to obtain text features, and finally inputting the first prompt information and the text features into a poster generation model. And obtaining a poster output by the poster generation model.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the fields of artificial intelligence technology such as computer vision, deep learning, and large models. Specifically, it relates to a poster generation method, a training method for a poster generation model, an apparatus, and a device. Background Technology

[0002] Currently, there are generally two methods for generating Chinese posters: the first is using pre-made poster templates, which matches a suitable template from the user's request and fills in the text content to generate the final poster; the second is generating a poster background image based on the text description provided by the user, allowing the user to edit and adjust the background image multiple times to generate the final poster. However, the first method cannot guarantee that the template will meet all the user's needs, and the generated poster may not match the user's request, resulting in poor poster quality. The second method significantly increases the time and cost of poster generation, leading to low efficiency. How to provide a poster generation method that improves the quality and efficiency of Chinese poster generation is a pressing issue that needs to be addressed. Summary of the Invention

[0003] This disclosure aims to at least partially address one of the technical problems in the related art.

[0004] The first aspect of this disclosure provides a poster generation method, including:

[0005] Receive poster generation request;

[0006] Based on the generation request, a first prompt message is generated, wherein the first prompt message is used to generate a poster image;

[0007] A second prompt is determined based on the first prompt, wherein the second prompt is used to generate the text content in the poster image;

[0008] The second prompt information is input into the generated text encoder to obtain text features, wherein the text encoder is jointly trained with the poster generation model;

[0009] The first prompt information and the text features are input into the poster generation model to obtain the poster output by the poster generation model.

[0010] A second aspect of this disclosure proposes a method for training a poster generation model, comprising:

[0011] Multiple training data sets are obtained, wherein the training data sets include a target poster, raw image prompts for generating the target poster, and raw character prompts for generating text content in the target poster;

[0012] The new character prompt information is input into the initial text encoder to obtain the text features output by the initial text encoder;

[0013] The semantic features corresponding to the raw image prompt information and the image features corresponding to the target poster are determined respectively.

[0014] The text features, semantic features, and image features are respectively input into the initial poster generation model to obtain the poster generated by the initial poster generation model;

[0015] Based on the differences between the generated poster and the target poster, the initial poster generation model and the initial text encoder are modified to obtain the trained poster generation model and text encoder.

[0016] A third aspect of this disclosure provides a poster generation apparatus, comprising:

[0017] The receiving module is used to receive poster generation requests;

[0018] The generation module is used to generate a first prompt message based on the generation request, wherein the first prompt message is used to generate a poster image;

[0019] The first determining module is used to determine the second prompt information based on the first prompt information, wherein the second prompt information is used to generate the text content in the poster image;

[0020] The first input module is used to input the second prompt information into the generated text encoder to obtain text features, wherein the text encoder is jointly trained with the poster generation model;

[0021] The second output module is used to input the first prompt information and the text features into the poster generation model to obtain the poster output by the poster generation model.

[0022] A fourth aspect of this disclosure provides a training apparatus for a poster generation model, comprising:

[0023] The acquisition module is used to acquire multiple training data sets, wherein the training data sets include a target poster, raw image prompts for generating the target poster, and raw character prompts for generating text content in the target poster;

[0024] The third input module is used to input the new character prompt information into the initial text encoder to obtain the text features output by the initial text encoder;

[0025] The second determining module is used to determine the semantic features corresponding to the raw image prompt information and the image features corresponding to the target poster, respectively.

[0026] The fourth input module is used to input the text features, semantic features and image features into the initial poster generation model respectively, so as to obtain the poster generated by the initial poster generation model;

[0027] The correction module is used to correct the initial poster generation model and the initial text encoder based on the differences between the generated poster and the target poster, so as to obtain the trained poster generation model and text encoder.

[0028] A fifth aspect of this disclosure provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the poster generation method proposed in the first aspect of this disclosure and the poster generation model training method proposed in the second aspect of this disclosure.

[0029] A sixth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the poster generation method proposed in the first aspect of this disclosure and the training method for the poster generation model proposed in the second aspect of this disclosure.

[0030] A seventh aspect embodiment of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the poster generation method proposed in the first aspect embodiment of this disclosure and the poster generation model training method proposed in the second aspect embodiment of this disclosure.

[0031] The poster generation method, poster generation model training method, apparatus, and equipment disclosed herein have the following beneficial effects:

[0032] In this embodiment, a poster generation request is first received, and a first prompt message is generated based on the request. Then, a second prompt message is determined based on the first prompt message. The second prompt message is then input into a pre-generated text encoder to obtain text features. Finally, the first prompt message and the text features are input into a poster generation model to obtain the poster output by the model. Thus, by generating text-based image prompt messages based on a poster generation request, determining the text prompt messages in the generated poster based on these prompt messages, inputting the generated text prompt messages into a text encoder to obtain text features, and then inputting the text-based image prompt messages and text features into a poster generation model to generate a poster, the poster generation model can accurately understand and generate multilingual text, including Chinese, improving its processing capabilities in multilingual environments and enhancing the quality and efficiency of poster generation.

[0033] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0034] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0035] Figure 1 This is a flowchart illustrating a poster generation method provided in an embodiment of the present disclosure;

[0036] Figure 2 A schematic flowchart illustrating a poster generation method provided in an embodiment of this disclosure;

[0037] Figure 3 This is a flowchart illustrating the poster generation method proposed in this disclosure;

[0038] Figure 4 This is a flowchart illustrating a training method for a poster generation model provided in an embodiment of the present disclosure.

[0039] Figure 5 A schematic diagram illustrating the process of obtaining training data sets in the training method of the poster generation model proposed in this disclosure;

[0040] Figure 6 This is a flowchart illustrating a training method for a poster generation model provided in an embodiment of the present disclosure.

[0041] Figure 7 This is a flowchart illustrating the training method for the poster generation model proposed in this disclosure;

[0042] Figure 8 This is a schematic diagram of the poster generation apparatus provided in an embodiment of the present disclosure;

[0043] Figure 9 A schematic diagram of the structure of the training device for the poster generation model provided in the embodiments of this disclosure;

[0044] Figure 10 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present disclosure is shown. Detailed Implementation

[0045] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0046] This disclosure relates to the field of computer technology, and in particular to the fields of artificial intelligence, such as computer vision, deep learning, and large models.

[0047] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence.

[0048] Computer vision refers to machine vision that uses cameras and computers to identify, track, and measure targets instead of human eyes, and further processes the images to make them more suitable for human observation or transmission to instruments for detection.

[0049] Deep learning (DL) learns the inherent patterns and hierarchical representations of sample data. The information gained during this learning process greatly aids in interpreting data such as text, images, and sound. The ultimate goal of deep learning is to enable machines to possess analytical and learning capabilities similar to humans, allowing them to recognize data such as text, images, and sound.

[0050] Large models, also known as Foundation Models, are models that extract knowledge from hundreds of millions of corpora or images, learn, and then produce large models with hundreds of millions of parameters.

[0051] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0052] The poster generation method, poster generation model training method, apparatus, and equipment of the present disclosure are described below with reference to the accompanying drawings.

[0053] Figure 1 This is a schematic flowchart of a poster generation method provided in an embodiment of the present disclosure.

[0054] like Figure 1 As shown, the poster generation method may include the following steps:

[0055] Step 101: Receive poster generation request.

[0056] The specific content of the poster generation request can be determined according to the user's needs. For example, when a user needs to generate a movie poster for movie A, the poster generation request content can be "generate a movie poster for movie A", etc. This disclosure does not limit this.

[0057] In some possible implementations, poster generation requests can be received through a pre-defined large language model. The large language model can then output prompts for poster generation based on the request content, providing a basis for subsequent poster generation.

[0058] Step 102: Based on the generation request, generate first prompt information, wherein the first prompt information is used to generate a poster image.

[0059] It should be noted that since the first prompt information is used to generate the poster image, the first prompt information may include the overall style of the poster to be generated, the image background, the image content and the image text, etc., and this disclosure does not limit this.

[0060] In some possible implementations, when generating the first prompt information based on the generation request, the generation request can be input into a preset large language model to obtain the first prompt information output by the large language model. This disclosure does not limit this.

[0061] Step 103: Determine the second prompt information based on the first prompt information, wherein the second prompt information is used to generate the text content in the poster image.

[0062] In some possible implementations, since the second prompt information is used to generate the text content in the poster image, in order to accurately and reliably generate the text in the poster image and improve the quality of poster generation, the second prompt information may include at least one of the following: text content, text style, text color, text display position, and text layout information.

[0063] The text style may include the font, font size, font weight (thickness of strokes, etc.), alignment, special effects (such as italics, underlines, shadows, etc.), etc., and this disclosure does not limit it.

[0064] The display position and layout information of the text constitute the layout information of the text.

[0065] In some possible implementations, when determining the second prompt information based on the first prompt information, the second prompt information can be determined through a model. That is, the first prompt information can be input into the model to obtain the second prompt information output by the model; this disclosure does not limit this approach.

[0066] It should be noted that the specific type and structure of the model can be preset as needed. For example, the model can be a pre-trained introductory model, a generative model, etc., and this disclosure does not limit this.

[0067] In some possible implementation forms, when determining the second prompt information based on the first prompt information, the second prompt information can also be extracted from the first prompt information based on a preset rule. The preset rule can include the content and format included in the second prompt information, etc., and the present disclosure does not limit this.

[0068] In some possible implementation forms, after determining the second prompt information based on the first prompt information, the second prompt information can also be structurally processed. For example, first, the text content, color, font, and layout information in the second prompt information are extracted, and using a predefined color list and font list, these information are formatted into a unified standard format: <Text "{text content}" in {color}, {language-font}, {layout}>. This ensures that the model can accurately understand and apply the style features of different languages, as well as the specific position of the text and its relationship with other visual elements, ensuring the correctness of the glyph and layout when generating text, and further enhancing the compatibility of the model in a multilingual environment and improving its performance in processing complex text styles. The present disclosure does not limit this.

[0069] Step 104, input the second prompt information into the already generated text encoder to obtain text features.

[0070] Among them, the text encoder is jointly trained with the poster generation model.

[0071] It should be noted that the specific type and structure of the text encoder can be determined in advance according to needs. For example, the text encoder can be an encoder based on a recurrent neural network, an encoder based on a self-attention mechanism, etc. For example, it can be a Glyph-ByT5 encoder, a text encoder based on a self-attention mechanism, which can be used to process Chinese and multilingual tasks and can improve the accuracy of text rendering in image generation. Among them, Glyph refers to the glyph, and the visual structure information (such as strokes, etc.) of Chinese characters is incorporated into the encoder; ByT5 means using byte-level input and output, without pre-segmentation, and supporting multiple languages. The present disclosure does not limit this.

[0072] Among them, the text features can include features such as text content, glyph, font type, and color, etc., and the present disclosure does not limit this.

[0073] It should be noted that the specific type and structure of the poster generation model can be determined in advance according to needs. For example, the poster generation model can be a text-to-image generation model, etc. For example, it can be a FLUX model, a text-to-image generation model based on a diffusion model. The present disclosure does not limit this.

[0074] Step 105: Input the first prompt information and text features into the poster generation model to obtain the poster output by the poster generation model.

[0075] In this disclosure, after obtaining the text features, the first prompt information and the text features are input into the poster generation model to obtain the poster output by the poster generation model. This ensures that the poster generation model can accurately understand and generate multilingual text, including Chinese, when generating posters, thereby enhancing the compatibility of the poster generation model in multilingual environments and improving the quality of poster generation.

[0076] In this embodiment, a poster generation request is first received, and a first prompt message is generated based on the request. Then, a second prompt message is determined based on the first prompt message. The second prompt message is then input into a pre-generated text encoder to obtain text features. Finally, the first prompt message and the text features are input into a poster generation model to obtain the poster output by the model. Thus, by generating text-based image prompt messages based on a poster generation request, determining the text prompt messages in the generated poster based on these prompt messages, inputting the generated text prompt messages into a text encoder to obtain text features, and then inputting the text-based image prompt messages and text features into a poster generation model to generate a poster, the poster generation model can accurately understand and generate multilingual text, including Chinese, improving its processing capabilities in multilingual environments and enhancing the quality and efficiency of poster generation.

[0077] Figure 2 This is a schematic flowchart illustrating a poster generation method provided in one embodiment of the present disclosure.

[0078] like Figure 2 As shown, the poster generation method may include the following steps:

[0079] Step 201: Receive poster generation request.

[0080] The specific implementation of step 201 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0081] Step 202: Determine the reference template for the prompt message based on the language type corresponding to the generated request.

[0082] The language type can be Chinese, English, or other languages, etc., and this disclosure does not limit it.

[0083] It should be noted that the reference template for the prompt message may vary depending on the language of the poster generation request, and this disclosure does not impose any restrictions on this.

[0084] The prompt information reference template can be a prompt word template for a large language model, which can be used to assist the large language model in accurately and reliably generating the first prompt information. Its specific content and structure can be preset as needed. For example, the prompt information reference template may include the overall style, image content, color, text content, and layout of the poster to be generated, etc., which are not limited in this disclosure.

[0085] Step 203: Based on the prompt information reference template, expand the generation request to generate the first prompt information.

[0086] In this disclosure, after determining the reference template for the prompt information, the generation request is expanded based on the reference template to generate the first prompt information. This expansion of the generation request enriches and completes the content of the first prompt information, providing a basis for the subsequent accurate and reliable generation of the poster.

[0087] Step 204: Determine the second prompt information based on the first prompt information, wherein the second prompt information is used to generate the text content in the poster image.

[0088] In some possible implementations, the first prompt information can be displayed before determining the second prompt information based on the first prompt information. Upon receiving an update instruction for the first prompt information, the first prompt information is updated based on the update instruction. This makes the updated first prompt information more in line with the user's personalized needs and improves the accuracy of the first prompt information.

[0089] The update command can be an instruction to update information such as the text color, position, and style of the poster in the first prompt information. The specific content of the update command can be determined according to user needs, and this disclosure does not limit it.

[0090] Step 205: Input the second prompt information into the generated text encoder to obtain text features.

[0091] The text encoder was trained jointly with the poster generation model.

[0092] Step 206: Input the first prompt information and text features into the poster generation model to obtain the poster output by the poster generation model.

[0093] The specific implementation of steps 204 to 206 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0094] The following is combined with Figure 3 The process of the poster generation method proposed in this disclosure will be further illustrated with examples. Figure 3 This is a flowchart illustrating the poster generation method proposed in this disclosure.

[0095] Figure 3 In this example, we will use the generation of movie posters as an illustration.

[0096] like Figure 3 As shown, after the large language model receives a poster generation request with the content "Generate a movie poster for movie A", it can expand the poster generation request (the specific implementation of the expansion can be referred to steps 202 to 203, which will not be repeated here) to generate a first prompt message. Then, it extracts the text information in the poster from the first prompt message to generate a second prompt message (the specific implementation of determining the second prompt message based on the first prompt message can be referred to step 103, which is only an example and not a limitation). Then, it inputs the second prompt message into the text encoder to obtain the text features output by the text encoder. The first prompt message and the text features output by the text encoder are then input into the poster generation model to obtain the movie poster output by the poster generation model, thereby improving the quality and efficiency of poster generation.

[0097] In this embodiment, a poster generation request is first received. Based on the language type corresponding to the generation request, a prompt information reference template is determined. Then, based on the prompt information reference template, the generation request is expanded to generate first prompt information. A second prompt information is then determined based on the first prompt information. The second prompt information is then input into a pre-generated text encoder to obtain text features. Finally, the first prompt information and text features are input into a poster generation model to obtain the poster output by the model. Thus, by determining the corresponding prompt information reference template based on the language type of the poster generation request and expanding the generation request based on the reference template to obtain prompt information for generating the poster image, the content of the prompt information used to generate the poster image is enriched and improved, providing a foundation for improving the quality of poster generation. Next, prompt information for generating text in the image is determined based on this prompt information. This prompt information for generating text is then input into a text encoder to determine text features. Finally, the poster is generated based on the prompt information and text features of the generated image, improving the quality of poster generation.

[0098] Figure 4 This is a flowchart illustrating a training method for a poster generation model provided in one embodiment of the present disclosure.

[0099] like Figure 4 As shown, the training method for this poster generation model may include the following steps:

[0100] Step 401: Obtain multiple training data sets, including the target poster, raw image prompts for generating the target poster, and raw character prompts for generating the text content in the target poster.

[0101] Among them, the target poster can be a high-quality poster that is matched according to the rules.

[0102] The rules can be set based on multiple dimensions, such as aesthetic standards and textual accuracy. Aesthetic standards can include style, composition, image layering, and the degree of integration between text and images, but this disclosure does not limit these aspects.

[0103] This ensures the visual quality and textual standardization of the target posters in the training data set, resulting in high-quality target posters and improving the overall quality of the training data set.

[0104] The raw image prompt information may include information such as the overall style of the target poster, image background, image content, and image text, etc., which are not limited in this disclosure.

[0105] The information provided for new words may include text content, text style, text color, text display position, and text layout information, etc., which are not limited in this disclosure.

[0106] The display position and layout information of the text constitute its layout information. Based on this layout information, subsequent training of the poster generation model can help the model accurately understand the layout concept, precisely control the position of the generated text, and effectively avoid chaotic text layout in the generated poster, thus affecting the poster quality.

[0107] In some possible implementations, when acquiring multiple training data sets, the target poster can be acquired first. Then, the target poster and pre-defined prompts are input into a visual language model to obtain raw image prompts. Next, character prompts are determined based on the raw image prompts. Finally, the training data set is determined based on the target poster, the corresponding raw image prompts, and the character prompts. This approach improves the quality of the training data set by combining high-quality target posters, corresponding raw image prompts, and character prompts, providing reliable training data for subsequent model training.

[0108] The preset prompts can be used to generate prompts for the raw image, and their specific content can be preset as needed. For example, the prompts can include descriptions of the overall style of the image, the background atmosphere, the details and layout of elements in the image, and text in multiple languages ​​such as Chinese or English in the image. For example, the prompt could be "Based on the provided image, deduce the prompts used to generate this raw image, which should include descriptions of the overall style, image background, background atmosphere, image content, and image text," etc. This disclosure does not limit this.

[0109] The specific type and structure of the visual language model can be predetermined as needed. For example, the visual language model can be a contrastive model, a generative model, a pre-trained model, a multimodal large language model (MLM), etc., and this disclosure does not limit it.

[0110] The following is combined with Figure 5 The process of obtaining training data sets in the training method of the poster generation model proposed in this disclosure is illustrated with an example. Figure 5 This is a schematic diagram illustrating the process of obtaining training data sets in the training method of the poster generation model proposed in this disclosure. Wherein, Figure 5 This is just an example; no restrictions are imposed here.

[0111] like Figure 5 As shown, it is mainly divided into three parts: the first part is to obtain the target poster, the second part is to obtain the raw image prompt information, and the third part is to extract the raw character prompt information.

[0112] The first step is to obtain the target poster. Specifically, this involves acquiring a large number of posters from multiple relevant professional poster design platforms, and then filtering them based on aesthetic standards and text accuracy to obtain high-quality target posters.

[0113] Then, the second part obtains raw image prompt information. The specific process is as follows: input the target poster and the preset prompt words into the visual language model to obtain the raw image prompt information output by the visual language model.

[0114] The third part extracts the new character hints. The specific process is as follows: extract the text information from the image hints and generate new character hints based on the text information.

[0115] It should be noted that before generating new word hints, the extracted text information can be structured, and then new word hints can be generated based on the structured text information.

[0116] Finally, based on the acquired target poster, raw image prompts, and new character prompts, a training data set is generated, thereby obtaining a high-quality training data set.

[0117] Step 402: Input the new character prompt information into the initial text encoder to obtain the text features output by the initial text encoder.

[0118] In this disclosure, by inputting the unfamiliar word prompts into the initial text encoder to obtain the text features output by the initial text encoder, a data foundation can be provided for subsequent model training.

[0119] It should be noted that the specific implementation of obtaining the text features output by the initial text encoder by inputting the new character hints from the training data set is similar to that of step 104, and will not be repeated here.

[0120] Step 403: Determine the semantic features corresponding to the raw image prompt information and the image features corresponding to the target poster.

[0121] In this disclosure, after obtaining the text features output by the initial encoder based on the character prompt information, in order to generate a complete poster, it is also necessary to determine the semantic features corresponding to the character prompt information and the image features corresponding to the target poster.

[0122] Step 404: Input the text features, semantic features and image features into the initial poster generation model to obtain the poster generated by the initial poster generation model.

[0123] It should be noted that the specific implementation of the initial poster generation model is the same as that of the poster generation model in step 104, and will not be repeated here.

[0124] Since the word hints include layout information such as the display position and layout of the text in the poster, when training the model based on the text features obtained from the word hints, the model's ability to process the display position and layout information of the text in the poster image can be trained. This allows the trained model to accurately understand the layout concept and precisely control the position of the generated text in order to generate higher quality posters.

[0125] In this disclosure, determined text features, semantic features, and image features are respectively input into an initial poster generation model to obtain a poster generated by the initial poster generation model, thereby providing data for model training.

[0126] Step 405: Based on the difference between the generated poster and the target poster, the initial poster generation model and the initial text encoder are corrected to obtain the trained poster generation model and text encoder.

[0127] In this disclosure, after obtaining the poster generated by the initial poster generation model, the difference between the generated poster and the target poster in the training data group can be determined first. Then, based on the difference, the initial poster generation model and the initial text encoder are corrected to obtain the trained poster generation model and text encoder, thereby realizing the joint training of the poster generation model and the text encoder, improving the training effect and efficiency, enabling the poster generation model to accurately understand and generate multilingual text, and improving the model's adaptability.

[0128] In this embodiment, multiple training data sets are first acquired, and the character prompts are input into an initial text encoder to obtain the text features output by the initial text encoder. Then, the semantic features corresponding to the raw image prompts and the image features corresponding to the target poster are determined respectively. The text features, semantic features, and image features are then input into the initial poster generation model to obtain the poster generated by the initial poster generation model. Finally, based on the differences between the generated poster and the target poster, the initial poster generation model and the initial text encoder are corrected to obtain the trained poster generation model and text encoder. Thus, by acquiring high-quality target posters and corresponding raw image and character prompts, a high-quality training data set is constructed. The poster generation model and text encoder are then jointly trained using the training data set, improving the training quality and effect. The ability of the poster generation model to process multilingual text, including Chinese, is trained, enhancing the adaptability of the trained poster generation model in multilingual environments and improving the performance of the trained model.

[0129] Figure 6 This is a flowchart illustrating a training method for a poster generation model provided in one embodiment of the present disclosure.

[0130] like Figure 6 As shown, the training method for this poster generation model may include the following steps:

[0131] Step 601: Obtain multiple training data sets, wherein the training data sets include the target poster, raw image prompts for generating the target poster, and raw character prompts for generating the text content in the target poster.

[0132] Step 602: Input the new character prompt information into the initial text encoder to obtain the text features output by the initial text encoder.

[0133] The specific implementation of steps 601 to 602 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0134] Step 603: Input the raw image prompt information into the text semantic encoder and the cross-modal encoder respectively to obtain the semantic features corresponding to the raw image prompt information.

[0135] It should be noted that the specific type and structure of the text semantic encoder can be preset as needed. For example, the text semantic encoder can be an encoder based on a recurrent neural network, an encoder based on a self-attention mechanism, such as a T5 (Text-to-Text Transfer Transformer) encoder, an encoder based on a self-attention mechanism, etc. This disclosure does not limit it.

[0136] It should be noted that the specific type and structure of the cross-modal encoder can be preset as needed. For example, the cross-modal encoder can be a generative architecture, a two-stream architecture (i.e., two independent encoders process different modalities respectively, and achieve semantic alignment through contrastive learning or cross-attention), etc., such as a contrastive language-image pretraining (CLIP) encoder, a two-stream encoder, which is not limited in this disclosure.

[0137] Step 604: Input the target poster into the preset encoder to obtain the image features corresponding to the target poster.

[0138] It should be noted that the preset encoder can be a variational autoencoder decoder (VAE decoder), and this disclosure does not limit it.

[0139] In this disclosure, text semantic encoders and cross-modal encoders are used to process the raw image prompt information to obtain semantic features. The target poster is then processed by a preset encoder to obtain image features, providing a data foundation for the model to generate poster images.

[0140] Step 605: Input the text features, semantic features and image features into the initial poster generation model to obtain the poster generated by the initial poster generation model.

[0141] Step 606: Based on the difference between the generated poster and the target poster, the initial poster generation model and the initial text encoder are corrected to obtain the trained poster generation model and text encoder.

[0142] The specific implementation of steps 605 to 606 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0143] In this disclosure, taking the Glyph-ByT5 encoder as the text encoder and the FLUX model as the poster generation model as an example, since the encoder used by the FLUX model cannot understand the semantics and structure of Chinese characters, it has significant limitations in processing Chinese characters. All the Chinese characters generated in the poster are garbled, which greatly reduces the quality of the poster generation and affects the readability of the poster.

[0144] To address the aforementioned issues, this disclosure jointly trains a text encoder and a poster generation model, fine-tuning the process using Chinese data to align the features from Chinese text to images. The Glyph-ByT5 encoder can better process Chinese text and transform it into text features understandable by the FLUX model. In the two-stream and single-stream layers of the FLUX model, a cross-attention mechanism is used to interact text features with image and text features, significantly improving the FLUX model's ability to understand and generate Chinese text, enabling it to accurately represent Chinese text in posters.

[0145] The following is combined Figure 7 The training process of the poster generation model proposed in this disclosure is illustrated with an example. Figure 7 This is a flowchart illustrating the training method for the poster generation model proposed in this disclosure. Wherein, Figure 7 The process shown is merely an example and is not intended to be limiting.

[0146] Figure 7 In this example, we will use the Glyph-ByT5 encoder as the initial text encoder and the FLUX model as the initial poster generation model.

[0147] like Figure 7 The training process shown is divided into two stages. The first stage is the semantic feature alignment training stage from text to image, where the text prompts do not contain the display position and layout information of the text. The second stage is the spatial feature alignment training stage from text to image, where the text prompts contain the layout information of the text.

[0148] The first stage involves inputting the character prompts into the initial text encoder to obtain the text features output by the initial text encoder. The image prompts are then input into the text semantic encoder and the cross-modal encoder to obtain the semantic features corresponding to the image prompts. The target poster is input into the preset encoder to obtain the image features corresponding to the target poster. Then, in both the two-stream and single-stream layers of the initial poster generation model, a cross-attention mechanism is used to interact the text features, semantic features, and image features. In this way, the text features are injected into the initial poster generation model, enabling the model to generate multilingual text, including Chinese.

[0149] Then, in the second stage, layout information of the text is added to the word prompt information. Through the specific implementation of steps 602 to 606, the initial text encoder and the initial poster generation model are jointly trained so that the model can understand the layout concept and accurately control the position of the generated text in order to generate a higher quality poster.

[0150] It should be noted that in the training process of the poster generation model, the initial text encoder and the initial poster generation model can also be jointly trained directly based on the word hints, image hints containing layout information, and the target poster; that is, only the following steps are performed. Figure 7 The second stage shown in this disclosure is not limited thereto.

[0151] In this embodiment, multiple training data sets are first acquired, and the unfamiliar character prompts are input into an initial text encoder to obtain the text features output by the initial text encoder. Then, the unfamiliar image prompts are input into a text semantic encoder and a cross-modal encoder to obtain the semantic features corresponding to the unfamiliar image prompts. Next, the target poster is input into a preset encoder to obtain the image features corresponding to the target poster. The text features, semantic features, and image features are then input into an initial poster generation model to obtain the poster generated by the initial poster generation model. Finally, based on the differences between the generated poster and the target poster, the initial poster generation model and the initial text encoder are corrected to obtain the trained poster generation model and text encoder. Thus, by processing the unfamiliar character prompts with the initial text encoder to obtain the corresponding text features, processing the unfamiliar image prompts with the text semantic encoder and the cross-modal encoder to obtain the corresponding semantic features, and processing the target poster with the preset encoder to obtain the corresponding image features, the initial poster generation model and the initial text encoder are jointly trained using the text features, semantic features, and image features, thereby improving the effectiveness and efficiency of model training.

[0152] To achieve the above embodiments, this disclosure also proposes a poster generation apparatus.

[0153] Figure 8 This is a schematic diagram of the poster generation apparatus provided in an embodiment of the present disclosure.

[0154] like Figure 8 As shown, the poster generating device 800 includes: a receiving module 801, a generating module 802, a first determining module 803, a first input module 804, and a second input module 805.

[0155] Receiver module 801 is used to receive poster generation requests;

[0156] The generation module 802 is used to generate first prompt information based on the generation request, wherein the first prompt information is used to generate a poster image;

[0157] The first determining module 803 is used to determine the second prompt information based on the first prompt information, wherein the second prompt information is used to generate the text content in the poster image;

[0158] The first input module 804 is used to input the second prompt information into the generated text encoder to obtain text features, wherein the text encoder is jointly trained with the poster generation model;

[0159] The second input module 805 is used to input the first prompt information and text features into the poster generation model to obtain the poster output by the poster generation model.

[0160] In one possible implementation of this disclosure, the above-mentioned generation module 802 is specifically used for:

[0161] Based on the language type corresponding to the generated request, determine the reference template for the prompt message;

[0162] Based on the prompt message reference template, the generation request is expanded to generate the first prompt message.

[0163] In one possible implementation of this disclosure, the first determining module 803 is further configured to:

[0164] The first prompt message is displayed;

[0165] Upon receiving an update instruction for the first prompt information, the first prompt information is updated based on the update instruction.

[0166] In one possible implementation of this disclosure, the second prompt information includes at least one of the following: text content, text style, text color, text display position, and text layout information.

[0167] The functions and specific implementation principles of the modules described in this embodiment can be found in the above method embodiments, and will not be repeated here.

[0168] In this embodiment, a poster generation request is first received, and a first prompt message is generated based on the request. Then, a second prompt message is determined based on the first prompt message. The second prompt message is then input into a pre-generated text encoder to obtain text features. Finally, the first prompt message and the text features are input into a poster generation model to obtain the poster output by the model. Thus, by generating text-based image prompt messages based on a poster generation request, determining the text prompt messages in the generated poster based on these prompt messages, inputting the generated text prompt messages into a text encoder to obtain text features, and then inputting the text-based image prompt messages and text features into a poster generation model to generate a poster, the poster generation model can accurately understand and generate multilingual text, including Chinese, improving its processing capabilities in multilingual environments and enhancing the quality and efficiency of poster generation.

[0169] Figure 9 This is a schematic diagram of the structure of the training device for the poster generation model provided in an embodiment of this disclosure.

[0170] like Figure 9 As shown, the training device 900 for the poster generation model includes: an acquisition module 901, a third input module 902, a second determination module 903, a fourth input module 904, and a correction module 905.

[0171] The acquisition module 901 is used to acquire multiple training data sets, wherein the training data sets include the target poster, raw image prompts for generating the target poster, and raw character prompts for generating the text content in the target poster;

[0172] The third input module 902 is used to input the new character prompt information into the initial text encoder in order to obtain the text features output by the initial text encoder;

[0173] The second determining module 903 is used to determine the semantic features corresponding to the raw image prompt information and the image features corresponding to the target poster, respectively.

[0174] The fourth input module 904 is used to input text features, semantic features and image features into the initial poster generation model respectively, so as to obtain the poster generated by the initial poster generation model;

[0175] The correction module 905 is used to correct the initial poster generation model and the initial text encoder based on the difference between the generated poster and the target poster, so as to obtain the trained poster generation model and text encoder.

[0176] In one possible implementation of this disclosure, the second determining module 903 is specifically used for:

[0177] The raw image prompts are input into the text semantic encoder and the cross-modal encoder respectively to obtain the semantic features corresponding to the raw image prompts.

[0178] Input the target poster into the preset encoder to obtain the image features corresponding to the target poster.

[0179] In one possible implementation of this disclosure, the acquisition module 901 is further configured to:

[0180] Obtain the target poster;

[0181] Input the target poster and the preset prompts into the visual language model to obtain the raw image prompt information output by the visual language model;

[0182] The prompts for unfamiliar characters are determined based on the prompts in the raw image.

[0183] The functions and specific implementation principles of the modules described in this embodiment can be found in the above method embodiments, and will not be repeated here.

[0184] In this embodiment, multiple training data sets are first acquired, and the character prompts are input into an initial text encoder to obtain the text features output by the initial text encoder. Then, the semantic features corresponding to the raw image prompts and the image features corresponding to the target poster are determined respectively. The text features, semantic features, and image features are then input into the initial poster generation model to obtain the poster generated by the initial poster generation model. Finally, based on the differences between the generated poster and the target poster, the initial poster generation model and the initial text encoder are corrected to obtain the trained poster generation model and text encoder. Thus, by acquiring high-quality target posters and corresponding raw image and character prompts, a high-quality training data set is constructed. The poster generation model and text encoder are then jointly trained using the training data set, improving the training quality and effect. The ability of the poster generation model to process multilingual text, including Chinese, is trained, enhancing the adaptability of the trained poster generation model in multilingual environments and improving the performance of the trained model.

[0185] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0186] like Figure 10 As shown, device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1002 or a computer program loaded from storage unit 1008 into random access memory (RAM) 1003. The RAM 1003 may also store various programs and data required for the operation of device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0187] Multiple components in device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of monitors, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0188] The computing unit 1001 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as poster generation methods and poster generation model training methods. For example, in some embodiments, the poster generation methods and poster generation model training methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the poster generation method and one or more steps of the poster generation model training method described above can be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured in any other suitable manner (e.g., by means of firmware) to execute the poster generation method and the training method of the poster generation model.

[0189] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0190] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a general-purpose computer, a special-purpose computer, or other programmable processor or controller, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0191] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0192] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0193] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0194] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0195] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0196] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified. In the description of this disclosure, the words "if" and "suppose" as used may be interpreted as "when," "when," "in response to determination," or "in the circumstances."

[0197] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A poster generation method, characterized in that, include: Receive poster generation request; Based on the generation request, a first prompt message is generated, wherein the first prompt message is used to generate a poster image; A second prompt is determined based on the first prompt, wherein the second prompt is used to generate the text content in the poster image; The second prompt information is input into the generated text encoder to obtain text features, wherein the text encoder is jointly trained with the poster generation model; The first prompt information and the text features are input into the poster generation model to obtain the poster output by the poster generation model.

2. The method as described in claim 1, characterized in that, The generation of the first prompt information based on the generation request includes: Based on the language type corresponding to the generated request, a reference template for the prompt message is determined; Based on the aforementioned prompt information reference template, the generation request is expanded to generate the first prompt information.

3. The method as described in claim 1, characterized in that, Before determining the second prompt information based on the first prompt information, the method further includes: Display the first prompt message; Upon receiving an update instruction for the first prompt information, the first prompt information is updated based on the update instruction.

4. The method according to any one of claims 1-3, characterized in that, The second prompt message includes at least one of the following: text content, text style, text color, text display position, and text layout information.

5. A training method for a poster generation model, characterized in that, include: Multiple training data sets are obtained, wherein the training data sets include a target poster, raw image prompts for generating the target poster, and raw character prompts for generating text content in the target poster; The new character prompt information is input into the initial text encoder to obtain the text features output by the initial text encoder; The semantic features corresponding to the raw image prompt information and the image features corresponding to the target poster are determined respectively. The text features, semantic features, and image features are respectively input into the initial poster generation model to obtain the poster generated by the initial poster generation model; Based on the differences between the generated poster and the target poster, the initial poster generation model and the initial text encoder are modified to obtain the trained poster generation model and text encoder.

6. The method as described in claim 5, characterized in that, The step of determining the semantic features corresponding to the raw image prompt information and the image features corresponding to the target poster includes: The raw image prompt information is input into the text semantic encoder and the cross-modal encoder respectively to obtain the semantic features corresponding to the raw image prompt information; The target poster is input into a preset encoder to obtain the image features corresponding to the target poster.

7. The method as described in claim 5 or 6, characterized in that, The acquisition of multiple training data sets includes: Obtain the target poster; The target poster and the preset prompt words are input into the visual language model to obtain the raw image prompt information output by the visual language model. The new character prompt information is determined based on the image prompt information.

8. A poster generating device, characterized in that, The device includes: The receiving module is used to receive poster generation requests; The generation module is used to generate a first prompt message based on the generation request, wherein the first prompt message is used to generate a poster image; The first determining module is used to determine the second prompt information based on the first prompt information, wherein the second prompt information is used to generate the text content in the poster image; The first input module is used to input the second prompt information into the generated text encoder to obtain text features, wherein the text encoder is jointly trained with the poster generation model; The second output module is used to input the first prompt information and the text features into the poster generation model to obtain the poster output by the poster generation model.

9. A training device for a poster generation model, characterized in that, The device includes: The acquisition module is used to acquire multiple training data sets, wherein the training data sets include a target poster, raw image prompts for generating the target poster, and raw character prompts for generating text content in the target poster; The third input module is used to input the new character prompt information into the initial text encoder to obtain the text features output by the initial text encoder; The second determining module is used to determine the semantic features corresponding to the raw image prompt information and the image features corresponding to the target poster, respectively. The fourth input module is used to input the text features, semantic features and image features into the initial poster generation model respectively, so as to obtain the poster generated by the initial poster generation model; The correction module is used to correct the initial poster generation model and the initial text encoder based on the differences between the generated poster and the target poster, so as to obtain the trained poster generation model and text encoder.

10. An electronic device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that may be executed by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the poster generation method according to any one of claims 1-4, or the training method for the poster generation model according to any one of claims 5-7.

11. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the poster generation method according to any one of claims 1-4, or the training method of the poster generation model according to any one of claims 5-7.

12. A computer program product comprising a computer program that, when executed by a processor, implements the poster generation method according to any one of claims 1-4, or the training method for the poster generation model according to any one of claims 5-7.

Images