A template generation method, apparatus, device, medium and product

Abstract

The present disclosure provides a template generation method, device, equipment, medium and product. The method comprises: obtaining an original multimedia material, and obtaining a layout image; wherein the layout image is used to indicate a region in which a foreground object cannot be generated; determining description information of the original multimedia material according to the original multimedia material; wherein the description information is used to describe multimedia content of the original multimedia material; inputting the description information of the original multimedia material and the layout image into an image generation model to obtain a template image output by the image generation model; wherein the template image comprises a background picture, a foreground object and a placeholder region, and the placeholder region corresponds to the region in which the foreground object cannot be generated indicated by the layout image. The method can quickly and flexibly generate diversified template images for different original multimedia materials.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a template generation method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] In advertising and poster campaigns, original multimedia materials are packaged to highlight their unique features, enabling targeted advertising or poster campaigns. Typically, templates are used to package original multimedia materials.

[0003] However, the generated template images in related technologies are of low quality and lack flexibility. Summary of the Invention

[0004] This disclosure provides a template generation method. This method can quickly and flexibly generate diverse template images for different original multimedia materials. This disclosure also provides apparatus, electronic devices, computer-readable storage media, and computer program products corresponding to the above method.

[0005] Firstly, this disclosure provides a template generation method, which includes:

[0006] Acquire original multimedia materials, and acquire a layout image; wherein the layout image is used to indicate areas where foreground objects cannot be generated;

[0007] Based on the original multimedia material, determine the descriptive information of the original multimedia material; wherein the descriptive information is used to describe the multimedia content of the original multimedia material;

[0008] The description information of the original multimedia material and the layout image are input into the image generation model to obtain the template image output by the image generation model; wherein, the template image includes a background image, a foreground object and a placeholder area, and the placeholder area corresponds to the area indicated by the layout image where a foreground object cannot be generated.

[0009] Secondly, this disclosure provides a template generation apparatus, the apparatus comprising:

[0010] An acquisition module is used to acquire original multimedia materials and to acquire a layout image; wherein the layout image is used to indicate areas where foreground objects cannot be generated;

[0011] A determining module is used to determine the description information of the original multimedia material based on the original multimedia material; wherein the description information is used to describe the multimedia content of the original multimedia material;

[0012] The generation module is used to input the description information of the original multimedia material and the layout image into the image generation model to obtain a template image output by the image generation model; wherein, the template image includes a background image, a foreground object and a placeholder area, and the placeholder area corresponds to the area indicated by the layout image where a foreground object cannot be generated.

[0013] Thirdly, this disclosure provides an electronic device including a processor and a memory. The processor and the memory communicate with each other. The processor is configured to execute instructions stored in the memory to cause the electronic device to perform a template generation method as described in the first aspect or any implementation thereof.

[0014] Fourthly, this disclosure provides a computer-readable storage medium storing instructions that instruct an electronic device to perform the template generation method described in the first aspect or any implementation thereof.

[0015] Fifthly, this disclosure provides a computer program product containing instructions that, when run on an electronic device, causes the electronic device to execute the template generation method described in the first aspect or any implementation thereof.

[0016] Based on the implementation methods provided in the above aspects, this disclosure can be further combined to provide more implementation methods.

[0017] As can be seen from the above technical solutions, this disclosure has the following advantages:

[0018] This disclosure provides a template generation method. The method first obtains original multimedia material and a layout image, wherein the layout image is used to indicate areas where foreground objects cannot be generated. Based on the original multimedia material, descriptive information of the original multimedia material is determined, wherein the descriptive information is used to describe the multimedia content of the original multimedia material. Then, the descriptive information of the original multimedia material and the layout image are input into an image generation model to obtain a template image output by the image generation model. The template image includes a background image, a foreground object, and a placeholder area, wherein the placeholder area corresponds to the area where foreground objects cannot be generated as indicated by the layout image.

[0019] In this method, an image generation model is used to automatically generate template images. Since the input of the image generation model includes the descriptive information and layout image of the original multimedia material, on the one hand, the generated template image is strongly correlated with the multimedia content of the original multimedia material, which enhances the correlation between the template image and the original multimedia material. On the other hand, background images and foreground objects are generated at the same time, which enhances the overall integrity of the template image. Furthermore, the area where the foreground object is located in the template image meets the user's layout requirements, which can quickly and flexibly generate diverse template images for different original multimedia materials. Attached Figure Description

[0020] To more clearly illustrate the technical methods of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below.

[0021] Figure 1 A flowchart illustrating a template generation method provided in this embodiment of the present disclosure;

[0022] Figure 2 A schematic diagram of a layout image provided for an embodiment of this disclosure;

[0023] Figure 3 A schematic diagram of a layout image provided for an embodiment of this disclosure;

[0024] Figure 4 This is a schematic diagram of the structure of an image generation model provided in an embodiment of the present disclosure;

[0025] Figure 5 This is a schematic diagram of the structure of a template generation device provided in an embodiment of the present disclosure;

[0026] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0027] The terms "first" and "second" used in the embodiments of this disclosure are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0028] First, some technical terms and application scenarios involved in the embodiments of this disclosure will be introduced.

[0029] In advertising and poster campaigns, original multimedia materials are packaged to highlight their unique features, enabling targeted advertising or poster campaigns. To maximize efficiency, template images are typically generated beforehand. The original multimedia materials and corresponding text are then added to these template images to complete the packaging process.

[0030] In related technologies, since the advertising content or poster content formed after packaging the original multimedia materials usually includes background images and foreground objects, the background images and foreground objects are usually generated separately during the automatic template generation process. Then, based on layout rules, the background images and foreground objects are combined to generate template images.

[0031] However, the above method has the following problems: First, the correlation between the template image and the original multimedia material is low, making it difficult for the template image to reflect the multimedia content of the original material. Furthermore, the background image and foreground object are generated separately and are unrelated to each other, resulting in a low overall aesthetic appeal of the template image. In addition, when layout rules or business scenarios change, the layout rules need to be redefined, leading to poor controllability and limited richness of the template image, making it difficult to achieve flexible template generation.

[0032] In view of this, the present disclosure provides a template generation method. The method first obtains original multimedia material and a layout image, wherein the layout image is used to indicate areas where foreground objects cannot be generated. Based on the original multimedia material, the method determines the description information of the original multimedia material, wherein the description information is used to describe the multimedia content of the original multimedia material. Then, the description information of the original multimedia material and the layout image are input into an image generation model to obtain a template image output by the image generation model. The template image includes a background image, a foreground object, and a placeholder area, wherein the placeholder area corresponds to the area where foreground objects cannot be generated as indicated by the layout image.

[0033] In this method, an image generation model is used to automatically generate template images. Since the input of the image generation model includes the descriptive information and layout image of the original multimedia material, on the one hand, the generated template image is strongly correlated with the multimedia content of the original multimedia material, which enhances the correlation between the template image and the original multimedia material. On the other hand, background images and foreground objects are generated at the same time, which enhances the overall integrity of the template image. Furthermore, the area where the foreground object is located in the template image meets the user's layout requirements, which can quickly and flexibly generate diverse template images for different original multimedia materials.

[0034] To facilitate understanding of the technical solutions provided in the embodiments of this disclosure, a description will be given below in conjunction with the accompanying drawings. See also... Figure 1The diagram shown illustrates a template generation method, which specifically includes:

[0035] S101: Obtain the original multimedia materials and the layout image.

[0036] In this context, the original multimedia material can be understood as multimedia material to be packaged. In other words, after generating a template image using the template generation method provided in this embodiment, the template image can be used to package the original multimedia material.

[0037] This disclosure does not limit the source of the original multimedia materials. For example, the original multimedia materials may be locally stored multimedia materials, or they may be multimedia materials uploaded to the cloud for storage.

[0038] This disclosure does not limit the type of original multimedia material. For example, the original multimedia material can be video material, audio material, graphic material, etc.

[0039] A layout image can be understood as an image used to indicate the layout of a template image. Typically, a template image may include a background image and a foreground object. In embodiments of this disclosure, the layout image may be used to indicate areas where a foreground object cannot be generated.

[0040] In other words, users (such as business entities that require template generation) only need to provide a layout image to indicate the desired layout position of the template image, without needing to provide information describing the specific position, shape, color, etc. of the background image or foreground object.

[0041] This disclosure does not limit the source of the layout image. For example, the layout image can originate from a pre-made template image. By parsing the template image, multiple layout images can be obtained, and the user can select the desired layout image from these multiple images. Alternatively, the user can design the layout image based on actual image generation requirements.

[0042] S102: Determine the description information of the original multimedia material based on the original multimedia material.

[0043] The descriptive information can be used to describe the multimedia content of the original multimedia material. In this embodiment of the disclosure, the descriptive information of the original multimedia material can be described in natural language, representing the natural language content; that is, the descriptive information of the original multimedia material can describe the multimedia content of the original multimedia material in the form of natural language.

[0044] S103: Input the description information and layout image of the original multimedia material into the image generation model to obtain the template image output by the image generation model.

[0045] The image generation model can be understood as a model that generates images based on text; that is, the image generation model has the ability to generate images based on text. Simultaneously, the image generation model also has the ability to control the layout and position of the generated image; that is, the image generation model can control the position of elements (e.g., foreground objects) in the generated image. This disclosure does not limit the type of image generation model; for example, the image generation model can be a diffusion model.

[0046] Specifically, the template image output by the image generation model can include a background image, foreground objects, and placeholder areas. The background image can be understood as the image that serves as the background in the template image; its size can be the same as the template image. Foreground objects can be understood as objects in the template image other than the background image. Foreground objects can be located above the background image, and there can be multiple foreground objects. In advertising or poster promotion scenarios, foreground objects can be stickers, decorations, search boxes, organizational logos, etc.

[0047] Placeholder areas can be understood as areas used to place elements other than foreground objects. For example, placeholder areas can be used to place original multimedia materials or text content. In other words, in a template image, placeholder areas cannot be used to place foreground objects.

[0048] In this embodiment of the disclosure, the placeholder area corresponds to the area indicated by the layout image where a foreground object cannot be generated. In other words, the template image output by the image generation model satisfies the image generation requirements indicated by the layout image, thereby enabling the foreground object to be positioned controllably in the template image.

[0049] In this method, an image generation model is used to automatically generate template images. Since the input of the image generation model includes the descriptive information and layout image of the original multimedia material, on the one hand, the generated template image is strongly correlated with the multimedia content of the original multimedia material, which enhances the correlation between the template image and the original multimedia material. On the other hand, background images and foreground objects are generated at the same time, which enhances the overall integrity of the template image. Furthermore, the area where the foreground object is located in the template image meets the user's layout requirements, which can quickly and flexibly generate diverse template images for different original multimedia materials.

[0050] The template generation method provided in the embodiments of this disclosure has been introduced above. The specific content of the template generation process will be described below.

[0051] In this embodiment of the disclosure, a layout image is used to inform the image generation model of the image generation requirements for the desired template image. In some embodiments, the layout image may include a first region carrying a first identifier and a second region carrying a non-first identifier, wherein the first region is a region capable of generating foreground objects, and the second region is a region unable to generate foreground objects.

[0052] An identifier can be understood as a feature that can identify a region. This disclosure does not limit the first identifier. For example, the first identifier can be a first color, a first texture, a first grayscale, etc.

[0053] Let's take the first identifier being the first color as an example for explanation, such as... Figure 2 As shown, the layout image 20 includes a first region 201 of a first color (white) and a second region 202 of a non-first color (gray). The first region 201 is the region where foreground objects can be generated, and the second region 202 is the region where foreground objects cannot be generated. That is, the second region 202 can be used to indicate a placeholder region.

[0054] In this way, different regions are distinguished by different identifiers in the layout image, so that the image generation model can learn the regions that can generate foreground objects and the regions that cannot generate foreground objects by using the identifiers of each region in the layout image. Then, foreground objects are generated in the regions that can generate foreground objects, and not in the regions that cannot generate foreground objects. The placeholder region in the output template image corresponds to the second region in the layout image.

[0055] In this embodiment of the disclosure, the layout image is only used to inform the image generation model of the areas where the foreground object cannot be generated, without restricting the specific location of the foreground object. In this way, the image generation model can freely generate template images without restricting the specific location, outline, style, etc. of the foreground object.

[0056] The template generation method provided in this disclosure can be applied to scenarios such as advertising and poster promotion. In these scenarios, the advertising content or poster content formed by packaging the original multimedia materials can include text content. Therefore, the layout image can be used to indicate the area for placing multimedia materials and the area for placing text content.

[0057] Similarly, areas for multimedia materials and areas for text content can be distinguished by identifiers. For example, using color as the identifier... Figure 3As shown, the layout image 30 includes a first region 301 of a first color (white) and a second region of a non-first color (black and gray). The first region 301 is the region where foreground objects can be generated, and the second region is the region where foreground objects cannot be generated. Specifically, the second region is divided into a region 302 (black region) for placing text content and a region 303 (gray region) for placing multimedia materials.

[0058] In other words, the areas in the layout image used to indicate that foreground objects cannot be generated can be specifically divided into areas for placing multimedia materials and areas for placing text content. Understandably, since the area for multimedia materials needs to hold the original multimedia materials, it cannot generate foreground objects. Similarly, since the area for text content needs to hold the text content, it cannot generate foreground objects.

[0059] In this case, after generating a template image using an image generation model, the placeholder areas in the template image can include a third area and a fourth area. The third area corresponds to the area indicated by the layout image for placing multimedia materials, and the fourth area corresponds to the area indicated by the layout image for placing text content.

[0060] Thus, in scenarios such as advertising and poster promotion, by configuring separate areas for placing multimedia materials and text content in the layout image, space is reserved for the original multimedia materials and text content, making it easier to efficiently package the original multimedia materials later.

[0061] The following describes the specific process of determining the descriptive information of the original multimedia material. In some possible implementations, a model is used to determine the descriptive information of the original multimedia material. Specifically, a first prompt word is generated, sent to a first model, and the descriptive information of the original multimedia material returned by the first model is received.

[0062] The first model can be any model that extracts descriptive information from the original multimedia material. For example, the first model can be a multimodal model. A multimodal model is an artificial intelligence (AI) model with the ability to process multimodal information (such as text, image, audio, and video information).

[0063] The first model possesses natural language processing capabilities, enabling it to understand the meaning of natural language and handle different types of natural language tasks. In this embodiment, the first model determines the descriptive information of the original multimedia material based on prompt learning. The prompt words can be used to guide the first model to make specific outputs in generative tasks (e.g., text generation, question answering, and dialogue tasks). By configuring the prompt words, the first model understands the background and requirements of the task, enabling it to handle different types of natural language processing tasks without retraining, thus increasing its scalability and flexibility.

[0064] In this embodiment of the disclosure, the first prompt word may include the original multimedia material and descriptive information indicating the generation of the original multimedia material. Since the first prompt word includes the aforementioned information, the first model can analyze the original multimedia material based on the prompting capability of the first prompt word and output descriptive information of the original multimedia material.

[0065] In some embodiments, the descriptive information can also be augmented to enhance the inference performance of the subsequent image generation model. Specifically, based on the original multimedia material, initial descriptive information of the original multimedia material is determined, and this initial descriptive information is augmented to obtain the original multimedia material's descriptive information.

[0066] The amplification process includes at least one of the following: splitting the initial description information into information describing the background of the original multimedia material and information describing the foreground of the original multimedia material; adding information to the initial description information to characterize the multimedia style; and rewriting the initial description information.

[0067] In other words, amplification processing can be understood as the process of modifying and optimizing the initial description information of the original multimedia material. In this embodiment, amplification processing can include various different types of processing. By splitting the initial description information into information describing the background and information describing the foreground of the original multimedia material, the description information of the original multimedia material can be described hierarchically and logically, making it more structured. By adding information representing the multimedia style to the initial description information, the description information of the original multimedia material can represent the multimedia style, thereby generating a template image similar in style to the original multimedia material, improving the correlation between the template image and the original multimedia material. By rewriting the initial description information, perturbations are added to the description information of the original multimedia material. For example, if the initial description information of the original multimedia material is "a puppy wearing a yellow shirt sunbathing on the beach," after rewriting, the description information of the original multimedia material becomes "a kitten wearing a gray hat and a floral shirt skateboarding in the square." This results in a template image with more detail, improving the diversity of the template image.

[0068] In some possible implementations, augmenting the initial descriptive information of the original multimedia material can be achieved using a model. Specifically, a second prompt word is generated, sent to a second model, and the descriptive information of the original multimedia material returned by the second model is received.

[0069] The second model can be any model capable of augmenting the initial descriptive information; for example, it can be a multimodal model or a language model. The second model possesses natural language processing capabilities, enabling it to understand the meaning of natural language and handle different types of natural language tasks. In this embodiment, the second model can augment the initial descriptive information of the original multimedia material based on cue learning. The second cue word includes the initial descriptive information of the original multimedia material and content indicating the augmentation of the initial descriptive information. Since the second cue word includes the aforementioned information, the second model can analyze the initial descriptive information of the original multimedia material based on the cue word's cueing ability, augment the initial descriptive information of the original multimedia material, and output the augmented descriptive information of the original multimedia material. This achieves automatic augmentation of the descriptive information, optimizing the inference performance of the image generation model.

[0070] This disclosure does not limit the first model and the second model. For example, the first model and the second model can be the same model or different models. The first model and the second model can be multimodal models.

[0071] After the image generation model outputs a template image, the target content can be created using the template image. Specifically, the original multimedia material is placed in the placeholder area of ​​the template image to generate the target content.

[0072] The target content can differ across different business scenarios. For example, in an advertising scenario, the target content could be the advertisement itself, while in a poster promotion scenario, it could be the poster content. This is achieved by adding original multimedia materials to a template image, thus packaging the original multimedia materials.

[0073] Furthermore, considering that in advertising or poster promotion scenarios, the advertising content or poster content formed after packaging the original multimedia materials can include copywriting content, and the placeholder area of ​​the template image can be a third or fourth area, in this case, the copywriting content related to the original multimedia materials can also be determined based on the original multimedia materials.

[0074] In some possible implementations, candidate text content is generated from multiple dimensions. For example, candidate text content may include the original title content of the original multimedia material, the title content extracted from the comments of the original multimedia material, the title content extracted from the multimedia summary of the original multimedia material, etc. The candidate text content is scored from multiple dimensions, for example, by using an existing scoring model to score multiple candidate text content. Based on the scores of multiple candidate text content, the candidate text content with the highest score is selected as the text content related to the original multimedia material.

[0075] In this way, the generated copy content is related to the original multimedia material, the original multimedia material is placed in the third area, and the copy content related to the original multimedia material is placed in the fourth area to generate target content, thereby further realizing targeted packaging of the original multimedia material.

[0076] Furthermore, embodiments of this disclosure can also enhance the effect of the template image by adjusting the layer where the foreground object is located. Specifically, the foreground object in the template image is determined, and the foreground object is configured in the topmost layer of the template image.

[0077] In other words, by identifying foreground objects in a template image, the foreground objects are positioned on the top layer of the entire template image. Thus, in advertising or poster campaigns, when the original multimedia materials are packaged to create the advertising or poster content, the foreground objects can appear above the original multimedia materials, achieving a floating effect for the foreground objects (e.g., stickers), thereby enhancing the effectiveness of the advertising or poster campaign.

[0078] This disclosure does not limit the method of determining the foreground object in the template image. For example, the template image can be segmented using an existing segmentation model to extract the foreground object.

[0079] As mentioned above, in this embodiment of the disclosure, an image generation model is used to generate a template image. The training process of the image generation model will be described below.

[0080] The image generation model is trained as follows: training data is obtained, and the descriptive information and layout image of the training image are used as input to the initial image generation model. The training image is used as a label to train the initial image generation model, so that the initial image generation model generates an output image that matches the descriptive information and layout image of the training image, thus obtaining the image generation model.

[0081] The training data can be understood as the data used to train the image generation model. The training data may include training images, descriptive information of training images, and layout images of training images. The layout images of training images can be used to indicate regions in the training images that do not include foreground objects.

[0082] The training of the image generation model can be supervised, and the labels can be understood as the correct output results in supervised training. Specifically, the initial image generation model is fed with the descriptive information and layout image of the training image. The initial image generation model generates an output image that matches the descriptive information and layout image of the training image. The content of the output image is associated with the descriptive information of the training image, and the placeholder regions included in the output image correspond to the regions in the layout image of the training image that do not include foreground objects.

[0083] Next, based on the output image of the initial image generation model and the training image used as a label, the loss function value is calculated. This loss function value can characterize the degree of difference between the output image of the initial image generation model and the training image. The model parameters of the initial image generation model are adjusted to minimize the loss function value, thus completing one round of training for the initial image generation model.

[0084] Repeat the above steps to perform multiple rounds of training. The training images in each round are different. When the difference between the output image of the initial image generation model and the training image meets the training termination condition, the training of the initial image generation model ends, resulting in an image generation model that can generate background images and foreground objects together and can control the position of the foreground objects.

[0085] In some embodiments, considering that the image generation model needs to generate template images and control the position of foreground objects, the image generation model may include an image generation module and a layout control module. For example... Figure 4 As shown, the layout control module controls the layout of foreground objects in the image. Its inputs include an initial noisy image, descriptive information, and a layout image. Its output is a layout control feature. The image generation module generates an image. Its inputs include the initial noisy image, descriptive information, and layout control features. Its output is a template image.

[0086] The initial noise image can be understood as an image composed of initial noise (e.g., random numbers). Typically, the initial noise image has the same size as the layout image and template image. The image generation module can use the initial noise image as a reference to generate the template image. In this way, the layout control module can extract layout control features based on the initial noise image, combining descriptive information and the layout image, increasing the controllability of the template image. The image generation module can then generate a template image that satisfies the descriptive information and layout control features, thus achieving image generation.

[0087] It should be noted that the image generation model can include more modules in addition to the image generation module and the layout control module. For example, the image generation model can also include a variational autoencoder module, which includes an encoder and a decoder. The encoder is used to downsample the initial noisy image (e.g., compressing a 512×512 initial noisy image into a 16×16 image), and the decoder is used to upsample the image output by the image generation module (e.g., restoring the 512×512 template image from the 16×16 image output by the image generation module).

[0088] The above text combined Figures 1 to 4 The template generation method provided in the embodiments of this disclosure has been described in detail. The apparatus and equipment provided in the embodiments of this disclosure will be described below with reference to the accompanying drawings.

[0089] See Figure 5 The schematic diagram of the template generation device shown shows that the device 50 includes:

[0090] The acquisition module 501 is used to acquire original multimedia materials and to acquire a layout image; wherein the layout image is used to indicate areas where foreground objects cannot be generated.

[0091] The determining module 502 is used to determine the description information of the original multimedia material based on the original multimedia material; wherein the description information is used to describe the multimedia content of the original multimedia material;

[0092] The generation module 503 is used to input the description information of the original multimedia material and the layout image into the image generation model to obtain a template image output by the image generation model; wherein, the template image includes a background image, a foreground object and a placeholder area, and the placeholder area corresponds to the area indicated by the layout image where a foreground object cannot be generated.

[0093] In some possible implementations, the layout image includes a first region carrying a first identifier and a second region carrying a non-first identifier, wherein the first region is a region capable of generating foreground objects and the second region is a region unable to generate foreground objects.

[0094] In some possible implementations, the layout image is used to indicate areas for placing multimedia materials and areas for placing text content. The placeholder areas in the template image include a third area and a fourth area, wherein the third area corresponds to the area for placing multimedia materials indicated by the layout image, and the fourth area corresponds to the area for placing text content indicated by the layout image.

[0095] In some possible implementations, the determining module 502 is specifically used for:

[0096] Generate a first prompt word; wherein the first prompt word includes the original multimedia material and descriptive information indicating the generation of the original multimedia material;

[0097] The first prompt word is sent to the first model, and the description information of the original multimedia material returned by the first model is received.

[0098] In some possible implementations, the determining module 502 is specifically used for:

[0099] Based on the original multimedia material, determine the initial description information of the original multimedia material;

[0100] The initial description information of the original multimedia material is amplified to obtain the description information of the original multimedia material; wherein, the amplification process includes at least one of the following processes: splitting the initial description information into information describing the background of the original multimedia material and information describing the foreground of the original multimedia material; adding information for characterizing the multimedia style to the initial description information; and rewriting the initial description information.

[0101] In some possible implementations, the determining module 502 is specifically used for:

[0102] Generate a second prompt word; wherein the second prompt word includes the initial description information of the original multimedia material and content indicating that the initial description information should be amplified;

[0103] The second prompt word is sent to the second model, and the description information of the original multimedia material returned by the second model is received.

[0104] In some possible implementations, the device 50 further includes a configuration module, which is used for:

[0105] Determine the foreground object in the template image;

[0106] The foreground object is placed in the topmost layer of the template image.

[0107] In some possible implementations, the generation module 503 is further configured to:

[0108] The original multimedia material is placed in the placeholder area of ​​the template image to generate the target content.

[0109] In some possible implementations, the placeholder region of the template image includes a third region and a fourth region, and the determining module 502 is further configured to:

[0110] Based on the original multimedia materials, determine the text content related to the original multimedia materials;

[0111] The generation module 503 is specifically used for:

[0112] The original multimedia material is placed in the third area, and the text content related to the original multimedia material is placed in the fourth area to generate the target content.

[0113] In some possible implementations, the image generation model is trained in the following manner:

[0114] Acquire training data; wherein the training data includes training images, descriptive information of the training images, and layout images of the training images, the layout images of the training images being used to indicate regions in the training images that do not include foreground objects;

[0115] The descriptive information and layout image of the training image are used as input to an initial image generation model. The training image is used as a label to train the initial image generation model, so that the initial image generation model generates an output image that matches the descriptive information and layout image of the training image, thus obtaining the image generation model. The image content of the output image is associated with the descriptive information of the training image, and the placeholder region included in the output image corresponds to the region in the layout image of the training image that does not include foreground objects.

[0116] The template generation apparatus 50 according to the embodiments of this disclosure can correspond to the execution of the methods described in the embodiments of this disclosure, and the above and other operations and / or functions of each module / unit of the template generation apparatus 50 are respectively for implementing Figure 1 For the sake of brevity, the corresponding processes of each method in the illustrated embodiments will not be described in detail here.

[0117] This disclosure also provides an electronic device. This electronic device is specifically used to implement, as described above. Figure 5 The template generation device 50 in the illustrated embodiment has the following functions.

[0118] Figure 6 A structural schematic diagram of an electronic device 600 is provided, such as... Figure 6 As shown, the electronic device 600 includes a bus 601, a processor 602, a communication interface 603, and a memory 604. The processor 602, the memory 604, and the communication interface 603 communicate with each other via the bus 601.

[0119] Bus 601 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0120] The processor 602 can be any one or more of the following processors: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), or digital signal processor (DSP).

[0121] The communication interface 603 is used for external communication. For example, the communication interface 603 can be used to communicate with a terminal.

[0122] Memory 604 may include volatile memory, such as random access memory (RAM). Memory 604 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0123] The memory 604 stores executable code, and the processor 602 executes the executable code to perform the aforementioned template generation method.

[0124] Specifically, in achieving Figure 5 In the case of the illustrated embodiment, and Figure 5 When the modules or units of the template generation apparatus 50 described in the embodiment are implemented by software, the following steps are performed: Figure 5 The software or program code required for the functions of each module / unit can be partially or wholly stored in memory 604. Processor 602 executes the program code corresponding to each unit stored in memory 604 and performs the aforementioned template generation method.

[0125] This disclosure also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform the template generation method applied to the template generation apparatus 50 described above.

[0126] This disclosure also provides a computer program product comprising one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this disclosure are generated.

[0127] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another website, computer, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0128] When the computer program product is executed by a computer, the computer performs any of the aforementioned template generation methods. The computer program product can be a software installation package; when any of the aforementioned template generation methods is required, the computer program product can be downloaded and executed on the computer.

[0129] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. Each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0130] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units / modules do not necessarily limit the specific unit itself.

[0131] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0132] In the context of embodiments of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0133] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems or apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

[0134] It should be understood that in this disclosure, "at least one item" means one or more, and "more than one" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0135] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0136] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0137] The above description of the disclosed embodiments enables those skilled in the art to make or use this disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A template generation method, characterized in that, The method includes: Acquire original multimedia materials, and acquire a layout image; wherein the layout image is used to indicate areas where foreground objects cannot be generated; Based on the original multimedia material, determine the descriptive information of the original multimedia material; wherein the descriptive information is used to describe the multimedia content of the original multimedia material; The description information of the original multimedia material and the layout image are input into the image generation model to obtain the template image output by the image generation model; wherein, the template image includes a background image, a foreground object and a placeholder area, and the placeholder area corresponds to the area indicated by the layout image where a foreground object cannot be generated.

2. The method according to claim 1, characterized in that, The layout image includes a first region carrying a first identifier and a second region carrying a non-first identifier, wherein the first region is a region capable of generating foreground objects and the second region is a region unable to generate foreground objects.

3. The method according to claim 1, characterized in that, The layout image is used to indicate the area for placing multimedia materials and the area for placing text content. The placeholder area in the template image includes a third area and a fourth area. The third area corresponds to the area for placing multimedia materials indicated by the layout image, and the fourth area corresponds to the area for placing text content indicated by the layout image.

4. The method according to claim 1, characterized in that, The step of determining the description information of the original multimedia material based on the original multimedia material includes: Generate a first prompt word; wherein the first prompt word includes the original multimedia material and descriptive information indicating the generation of the original multimedia material; The first prompt word is sent to the first model, and the description information of the original multimedia material returned by the first model is received.

5. The method according to claim 1, characterized in that, The step of determining the description information of the original multimedia material based on the original multimedia material includes: Based on the original multimedia material, determine the initial description information of the original multimedia material; The initial description information of the original multimedia material is amplified to obtain the description information of the original multimedia material; wherein, the amplification process includes at least one of the following processes: splitting the initial description information into information describing the background of the original multimedia material and information describing the foreground of the original multimedia material; adding information for characterizing the multimedia style to the initial description information; and rewriting the initial description information.

6. The method according to claim 5, characterized in that, The step of amplifying the initial description information of the original multimedia material to obtain the description information of the original multimedia material includes: Generate a second prompt word; wherein the second prompt word includes the initial description information of the original multimedia material and content indicating that the initial description information should be amplified; The second prompt word is sent to the second model, and the description information of the original multimedia material returned by the second model is received.

7. The method according to claim 1, characterized in that, The method further includes: Determine the foreground object in the template image; The foreground object is placed in the topmost layer of the template image.

8. The method according to claim 1, characterized in that, The method further includes: The original multimedia material is placed in the placeholder area of ​​the template image to generate the target content.

9. The method according to claim 8, characterized in that, The placeholder region of the template image includes a third region and a fourth region, and the method further includes: Based on the original multimedia materials, determine the text content related to the original multimedia materials; The step of placing the original multimedia material in the placeholder area of ​​the template image to generate the target content includes: The original multimedia material is placed in the third area, and the text content related to the original multimedia material is placed in the fourth area to generate the target content.

10. The method according to any one of claims 1 to 9, characterized in that, The image generation model is trained in the following manner: Acquire training data; wherein the training data includes training images, descriptive information of the training images, and layout images of the training images, the layout images of the training images being used to indicate regions in the training images that do not include foreground objects; The descriptive information and layout image of the training image are used as input to an initial image generation model. The training image is used as a label to train the initial image generation model, so that the initial image generation model generates an output image that matches the descriptive information and layout image of the training image, thus obtaining the image generation model. The image content of the output image is associated with the descriptive information of the training image, and the placeholder region included in the output image corresponds to the region in the layout image of the training image that does not include foreground objects.

11. A template generation device, characterized in that, The device includes: An acquisition module is used to acquire original multimedia materials and to acquire a layout image; wherein the layout image is used to indicate areas where foreground objects cannot be generated; A determining module is used to determine the description information of the original multimedia material based on the original multimedia material; wherein the description information is used to describe the multimedia content of the original multimedia material; The generation module is used to input the description information of the original multimedia material and the layout image into the image generation model to obtain a template image output by the image generation model; wherein, the template image includes a background image, a foreground object and a placeholder area, and the placeholder area corresponds to the area indicated by the layout image where a foreground object cannot be generated.

12. An electronic device, characterized in that, The electronic device includes a processor and a memory; The processor is configured to execute instructions stored in the memory, causing the electronic device to perform the method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, Includes instructions that instruct an electronic device to perform the method as described in any one of claims 1 to 10.

14. A computer program product, characterized in that, The computer program product includes computer-readable instructions for implementing the method according to any one of claims 1 to 10.

Images