Template generation method, device and equipment, medium and product

Abstract

The invention provides a template generation method and device, equipment, a medium and a product, and the method comprises the steps: obtaining an original multimedia material, and obtaining a layout image; wherein the layout image is used for indicating an area in which the foreground object cannot be generated; determining description information of the original multimedia material according to the original multimedia material; wherein the description information is used for describing the multimedia content of the original multimedia material; inputting the description information and the layout image of the original multimedia material into an image generation model to obtain a template image output by the image generation model; wherein the template image comprises a background picture, a foreground object and a placeholder region, and the placeholder region corresponds to a region, which is indicated by the layout image and cannot generate the foreground object. According to the method, diversified template images can be quickly and flexibly generated for different original multimedia materials.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a template generation method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] In the scenarios of advertising and poster promotion, the original multimedia materials are packaged to highlight their characteristics and conduct targeted advertising or poster promotion. Usually, templates are used to package the original multimedia materials.

[0003] However, in the related art, the generated template image has low quality and poor flexibility. Summary of the invention

[0004] The present disclosure provides a template generation method. The method can quickly and flexibly generate diversified template images for different original multimedia materials. The present disclosure also provides a device, an electronic device, a computer-readable storage medium, and a computer program product corresponding to the above method.

[0005] In a first aspect, the present disclosure provides a template generation method, the method comprising:

[0006] Acquire original multimedia material, and acquire a layout image; wherein the layout image is used to indicate an area where a foreground object cannot be generated;

[0007] Determining description information of the original multimedia material according to the original multimedia material; wherein the description information is used to describe multimedia content of the original multimedia material;

[0008] The description information of the original multimedia material and the layout image are input into an image generation model to obtain a template image output by the image generation model; wherein the template image includes a background picture, a foreground object and a placeholder area, and the placeholder area corresponds to an area indicated by the layout image where a foreground object cannot be generated.

[0009] In a second aspect, the present disclosure provides a template generation device, the device comprising:

[0010] An acquisition module, used to acquire original multimedia materials and to acquire a layout image; wherein the layout image is used to indicate an area where a foreground object cannot be generated;

[0011] A determination module, configured to determine description information of the original multimedia material according to the original multimedia material; wherein the description information is used to describe multimedia content of the original multimedia material;

[0012] A generation module is used to input the description information of the original multimedia material and the layout image into an image generation model to obtain a template image output by the image generation model; wherein the template image includes a background image, a foreground object and a placeholder area, and the placeholder area corresponds to an area indicated by the layout image where a foreground object cannot be generated.

[0013] In a third aspect, the present disclosure provides an electronic device, the electronic device comprising a processor and a memory. The processor and the memory communicate with each other. The processor is used to execute instructions stored in the memory, so that the electronic device performs the template generation method in the first aspect or any implementation of the first aspect.

[0014] In a fourth aspect, the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, wherein the instructions instruct an electronic device to execute the template generation method described in the first aspect or any one of the implementations of the first aspect.

[0015] In a fifth aspect, the present disclosure provides a computer program product comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the template generation method described in the first aspect or any one of the implementations of the first aspect.

[0016] Based on the implementations provided in the above aspects, the present disclosure can be further combined to provide more implementations.

[0017] It can be seen from the above technical solutions that the present disclosure has the following advantages:

[0018] The present disclosure provides a template generation method, which first obtains original multimedia material and a layout image, wherein the layout image is used to indicate an area where a foreground object cannot be generated, determines description information of the original multimedia material based on the original multimedia material, wherein the description information is used to describe the multimedia content of the original multimedia material, and then inputs the description information of the original multimedia material and the layout image into an image generation model to obtain a template image output by the image generation model, wherein the template image includes a background image, a foreground object and a placeholder area, and the placeholder area corresponds to the area where the foreground object cannot be generated as indicated by the layout image.

[0019] In this method, an image generation model is used to realize automatic generation of template images. Since the input of the image generation model includes the description information and layout image of the original multimedia material, on the one hand, the generated template image is strongly correlated with the multimedia content of the original multimedia material, which improves the correlation between the template image and the original multimedia material. On the other hand, background images and foreground objects are generated at the same time to improve the integrity of the template image. Moreover, the area where the foreground object is located in the template image meets the layout requirements of the user, and a variety of template images can be generated quickly and flexibly for different original multimedia materials. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical method of the embodiments of the present disclosure, the drawings required for use in the embodiments are briefly introduced below.

[0021] Figure 1 A schematic diagram of a flow chart of a template generation method provided in an embodiment of the present disclosure;

[0022] Figure 2 A schematic diagram of a layout image provided by an embodiment of the present disclosure;

[0023] Figure 3 A schematic diagram of a layout image provided by an embodiment of the present disclosure;

[0024] Figure 4 A schematic diagram of the structure of an image generation model provided by an embodiment of the present disclosure;

[0025] Figure 5 A schematic diagram of the structure of a template generation device provided in an embodiment of the present disclosure;

[0026] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] The terms "first" and "second" in the embodiments of the present disclosure are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0028] First, some technical terms and application scenarios involved in the embodiments of the present disclosure are introduced.

[0029] In the scenarios of advertising, poster promotion, etc., the original multimedia materials are packaged to highlight the characteristics of the original multimedia materials and carry out targeted advertising or poster promotion. Considering the efficiency of packaging the original multimedia materials, usually, a template image is generated in advance, and the original multimedia materials and corresponding copy are added to the template image to complete the packaging of the original multimedia materials.

[0030] In the related art, since the advertising content or poster content formed after packaging the original multimedia materials usually includes background images and foreground objects, in the process of automatically generating templates, the background images and foreground objects are usually generated separately, and then based on the layout rules, the background images and foreground objects are combined to generate a template image.

[0031] However, the above method has the following problems: First, the correlation between the template image and the original multimedia material is low, and the template image is difficult to reflect the multimedia content of the original multimedia material. In addition, the background image and the foreground object are generated separately and have no correlation with each other, and the overall aesthetics of the template image is low. In addition, when the layout rules or business scenarios change, the layout rules need to be re-formulated, the controllability of the template image is poor, the richness is limited, and it is difficult to achieve flexible template generation.

[0032] In view of this, the present disclosure provides a template generation method, which first obtains original multimedia material and a layout image, wherein the layout image is used to indicate an area where a foreground object cannot be generated, determines description information of the original multimedia material based on the original multimedia material, wherein the description information is used to describe the multimedia content of the original multimedia material, and then inputs the description information of the original multimedia material and the layout image into an image generation model to obtain a template image output by the image generation model, wherein the template image includes a background picture, a foreground object and a placeholder area, and the placeholder area corresponds to the area where the foreground object cannot be generated as indicated by the layout image.

[0033] In this method, an image generation model is used to realize automatic generation of template images. Since the input of the image generation model includes the description information and layout image of the original multimedia material, on the one hand, the generated template image is strongly correlated with the multimedia content of the original multimedia material, which improves the correlation between the template image and the original multimedia material. On the other hand, background images and foreground objects are generated at the same time to improve the integrity of the template image. Moreover, the area where the foreground object is located in the template image meets the layout requirements of the user, and a variety of template images can be generated quickly and flexibly for different original multimedia materials.

[0034] To facilitate understanding of the technical solution provided by the embodiments of the present disclosure, the following is a description with reference to the accompanying drawings. Figure 1A schematic diagram of a process of generating a template is shown, the method specifically comprising:

[0035] S101: Acquire original multimedia materials, and acquire a layout image.

[0036] The original multimedia material can be understood as the multimedia material to be packaged. In other words, after the template image is generated by the template generation method provided by the embodiment of the present disclosure, the template image can be used to package the original multimedia material.

[0037] The embodiments of the present disclosure do not limit the source of the original multimedia material. For example, the original multimedia material may be a multimedia material stored locally, or the original multimedia material may be a multimedia material uploaded to a cloud storage.

[0038] The embodiments of the present disclosure do not limit the type of original multimedia materials. For example, the original multimedia materials may be video materials, audio materials, graphic materials, etc.

[0039] The layout image can be understood as an image used to indicate the layout of the template image. Generally, the template image may include a background image and a foreground object. In the embodiment of the present disclosure, the layout image may be used to indicate an area where the foreground object cannot be generated.

[0040] That is to say, the user (for example, a business party with template generation needs) only needs to provide a layout image to indicate the layout position of the desired template image, without providing information describing the specific position, shape, color, etc. of the background image or foreground object.

[0041] The disclosed embodiments do not limit the source of the layout image. For example, the layout image can be derived from a pre-made template image. By performing template parsing on the template image, multiple layout images are obtained, and the user can select the desired layout image from the multiple layout images. For another example, the user can also design the layout image based on the actual image generation requirements.

[0042] S102: Determine description information of the original multimedia material according to the original multimedia material.

[0043] The description information can be used to describe the multimedia content of the original multimedia material. In the embodiment of the present disclosure, the description information of the original multimedia material can be described in natural language to represent the natural language content, that is, the description information of the original multimedia material can describe the multimedia content of the original multimedia material in the form of natural language.

[0044] S103: Input the description information and layout image of the original multimedia material into the image generation model to obtain a template image output by the image generation model.

[0045] The image generation model can be understood as a model for generating images based on text, that is, the image generation model has the ability to generate images based on text. At the same time, the image generation model also has the ability to control the layout position of the generated image, that is, the image generation model can control the position of the elements (such as foreground objects) in the generated image. The disclosed embodiment does not limit the type of the image generation model, for example, the image generation model can be a diffusion model.

[0046] Specifically, the template image output by the image generation model may include a background image, a foreground object, and a placeholder area. The background image may be understood as an image that serves as the background in the template image, and the size of the background image may be the same as that of the template image. The foreground object may be understood as an object other than the background image in the template image. The foreground object may be located above the background image, and may include multiple foreground objects. In advertising or poster promotion scenarios, the foreground object may be a sticker, decoration, search box, organization logo, etc.

[0047] The placeholder area can be understood as an area used to place elements other than the foreground object. For example, the placeholder area can be used to place original multimedia materials or text content. That is, in the template image, the placeholder area cannot be used to place the foreground object.

[0048] In the disclosed embodiment, the placeholder area corresponds to the area indicated by the layout image where the foreground object cannot be generated. In other words, the template image output by the image generation model meets the image generation requirements indicated by the layout image, so that the position of the foreground object in the template image can be controlled.

[0049] In this method, an image generation model is used to realize automatic generation of template images. Since the input of the image generation model includes the description information and layout image of the original multimedia material, on the one hand, the generated template image is strongly correlated with the multimedia content of the original multimedia material, which improves the correlation between the template image and the original multimedia material. On the other hand, background images and foreground objects are generated at the same time to improve the integrity of the template image. Moreover, the area where the foreground object is located in the template image meets the layout requirements of the user, and a variety of template images can be generated quickly and flexibly for different original multimedia materials.

[0050] The foregoing describes the template generation method provided in the embodiment of the present disclosure. The following describes the specific contents of the template generation process.

[0051] In the disclosed embodiment, the layout image is used to inform the image generation model of the image generation requirements of the expected template image. In some embodiments, the layout image may include a first area carrying a first identifier and a second area carrying a non-first identifier, the first area being an area where a foreground object can be generated, and the second area being an area where a foreground object cannot be generated.

[0052] The identifier can be understood as a feature that can identify a region. The embodiment of the present disclosure does not limit the first identifier. For example, the first identifier can be a first color, a first texture, a first grayscale, etc.

[0053] Take the first identifier as the first color as an example for explanation. Figure 2 As shown, the layout image 20 includes a first area 201 of a first color (white) and a second area 202 of a non-first color (gray), the first area 201 is an area where a foreground object can be generated, and the second area 202 is an area where a foreground object cannot be generated. That is, the second area 202 can be used to indicate a placeholder area.

[0054] In this way, different areas in the layout image are distinguished by different identifiers, so that the image generation model learns the areas that can generate foreground objects and the areas that cannot generate foreground objects through the identifiers of each area in the layout image, and then generates foreground objects in the areas that can generate foreground objects, and does not generate foreground objects in the areas that cannot generate foreground objects. The placeholder area in the output template image corresponds to the second area in the layout image.

[0055] In the disclosed embodiment, the layout image is only used to inform the image generation model of the area where the foreground object cannot be generated, without restricting the specific position of the foreground object. In this way, the image generation model can freely generate a template image without any restrictions on the specific position, outline, style, and pattern of the foreground object.

[0056] The template generation method provided by the embodiment of the present disclosure can be applied to scenes such as advertising placement and poster promotion. In the above scenes, the advertisement content or poster content formed after packaging the original multimedia material can include copy content. Therefore, the layout image can be used to indicate the area for placing the multimedia material and the area for placing the copy content.

[0057] Similarly, the area for placing multimedia materials and the area for placing copy content can be distinguished by identifiers. For example, the identifier is color. Figure 3As shown, the layout image 30 includes a first area 301 of a first color (white) and a second area of ​​a non-first color (black and gray), the first area 301 is an area where a foreground object can be generated, and the second area is an area where a foreground object cannot be generated. The second area is specifically divided into an area 302 (black area) for placing text content and an area 303 (gray area) for placing multimedia materials.

[0058] That is, the area in the layout image indicating that the foreground object cannot be generated can be specifically divided into an area for placing multimedia materials and an area for placing text content. It can be understood that since the area for placing multimedia materials needs to place the original multimedia materials, the foreground object cannot be generated. Since the area for placing text content needs to place the text content, the foreground object cannot be generated.

[0059] In this case, after generating the template image using the image generation model, the placeholder area in the template image may include a third area and a fourth area, wherein the third area corresponds to the area indicated by the layout image for placing multimedia materials, and the fourth area corresponds to the area indicated by the layout image for placing copy content.

[0060] In this way, in scenarios such as advertising and poster promotion, by respectively configuring an area for placing multimedia materials and an area for placing copy content in the layout image, space is reserved for the original multimedia materials and copy content, making it easier to efficiently package the original multimedia materials later.

[0061] The specific process of determining the description information of the original multimedia material is described below. In some possible implementations, the description information of the original multimedia material is determined with the help of a model. Specifically, a first prompt word is generated, the first prompt word is sent to the first model, and the description information of the original multimedia material returned by the first model is received.

[0062] The first model may be any model capable of extracting descriptive information of the original multimedia material, for example, the first model may be a multimodal model, wherein the multimodal model is an artificial intelligence (AI) model capable of processing multimodal information (for example, text information, image information, audio information, video information, etc.).

[0063] The first model has natural language processing capabilities, can understand the meaning of natural language, and handle different types of natural language tasks. In the disclosed embodiment, the first model determines the description information of the original multimedia material based on prompt learning. Among them, prompt words can be used to guide the first model to perform specific outputs in generative tasks (such as text generation tasks, question-answering tasks, and dialogue tasks). By configuring prompt words, the first model is helped to understand the background and requirements of the task, so that the first model can handle different types of natural language processing tasks without retraining the first model, thereby increasing the scalability and flexibility of the first model.

[0064] In the embodiment of the present disclosure, the first prompt word may include the original multimedia material and the content indicating the generation of the description information of the original multimedia material. Since the first prompt word includes the above information, the first model can analyze the original multimedia material based on the prompt capability of the first prompt word and output the description information of the original multimedia material.

[0065] In some embodiments, the description information may be augmented to enhance the reasoning performance of the subsequent image generation model. In specific implementation, the initial description information of the original multimedia material is determined based on the original multimedia material, and the initial description information of the original multimedia material is augmented to obtain the description information of the original multimedia material.

[0066] Among them, the amplification processing includes at least one of the following processing: splitting the initial description information into information describing the background of the original multimedia material and information describing the foreground of the original multimedia material, adding information used to characterize the multimedia style to the initial description information, and rewriting the initial description information.

[0067] That is, the augmentation process can be understood as a process of modifying and optimizing the initial description information of the original multimedia material. In the embodiment of the present disclosure, the augmentation process can include a variety of different types of processing. By splitting the initial description information into information describing the background of the original multimedia material and information describing the foreground of the original multimedia material, the description information of the original multimedia material can be described in a hierarchical and logical manner for the original multimedia material, and is more structured. By adding information used to characterize the multimedia style to the initial description information, the description information of the original multimedia material can characterize the multimedia style, and the generated template image is similar to the style of the original multimedia material, thereby improving the relevance of the template image to the original multimedia material. By rewriting the initial description information, disturbances are added to the description information of the original multimedia material. For example, the initial description information of the original multimedia material is "a puppy wearing yellow clothes basking in the sun on the beach". After rewriting the initial description information of the original multimedia material, the description information of the original multimedia material is "a kitten wearing a gray hat and a flowered shirt skateboarding in the square". In this way, the generated template image has more details, and the diversity of the template image is improved.

[0068] In some possible implementations, the augmentation process of the initial description information of the original multimedia material can be implemented with the help of a model. Specifically, a second prompt word is generated, the second prompt word is sent to the second model, and the description information of the original multimedia material returned by the second model is received.

[0069] The second model can be any model that can amplify the initial description information. For example, the second model can be a multimodal model or a language model. The second model has natural language processing capabilities, can understand the meaning of natural language, and can process different types of natural language tasks. In the embodiment of the present disclosure, the second model can amplify the initial description information of the original multimedia material based on prompt learning. The second prompt word includes the initial description information of the original multimedia material and the content indicating the amplification processing of the initial description information. Since the second prompt word includes the above information, the second model can analyze the initial description information of the original multimedia material based on the prompt capability of the second prompt word, and amplify the initial description information of the original multimedia material, and output the amplified description information of the original multimedia material. In this way, automatic amplification processing of the description information is achieved, and the reasoning effect of the image generation model is optimized.

[0070] The embodiment of the present disclosure does not limit the first model and the second model. For example, the first model and the second model may be the same model or different models. The first model and the second model may be multi-modal models.

[0071] After the image generation model outputs the template image, the template image can also be used to create target content. Specifically, the original multimedia material is placed in the placeholder area of ​​the template image to generate the target content.

[0072] The target content in different business scenarios may be different. For example, in an advertising scenario, the target content may be advertising content, and in a poster promotion scenario, the target content may be poster content. By adding original multimedia materials on the basis of the template image, the original multimedia materials can be packaged.

[0073] Furthermore, considering that in an advertising or poster promotion scenario, the advertising content or poster content formed after packaging the original multimedia material may include copy content, the placeholder area of ​​the template image may be the third area and the fourth area. In this case, the copy content related to the original multimedia material can also be determined based on the original multimedia material.

[0074] In some possible implementations, candidate copy content is generated from multiple dimensions. For example, the candidate copy content may include the original title content of the original multimedia material, the title content extracted from the comments of the original multimedia material, the title content extracted from the multimedia summary of the original multimedia material, etc. The candidate copy content of multiple dimensions is scored, for example, multiple candidate copy content is scored using an existing scoring model, and based on the scores of the multiple candidate copy content, the candidate copy content with the highest score is selected as the copy content related to the original multimedia material.

[0075] In this way, the generated copy content is related to the original multimedia material, the original multimedia material is placed in the third area, and the copy content related to the original multimedia material is placed in the fourth area to generate target content, thereby further realizing targeted packaging of the original multimedia material.

[0076] Furthermore, the disclosed embodiment can also improve the effect of the template image by adjusting the layer where the foreground object is located. Specifically, the foreground object in the template image is determined, and the foreground object is configured in the topmost layer in the template image.

[0077] That is, by identifying the foreground object in the template image, the foreground object is configured in the topmost layer of the entire template image. In this way, in the scene of advertising or poster promotion, in the advertising content or poster content formed after packaging the original multimedia material, the foreground object can be displayed above the original multimedia material, achieving the effect of the foreground object (such as a sticker) floating, thereby improving the effect of advertising or poster promotion.

[0078] The embodiments of the present disclosure do not limit the method of determining the foreground object in the template image. For example, the template image can be segmented using an existing segmentation model to extract the foreground object in the template image.

[0079] As described above, in the embodiments of the present disclosure, an image generation model is used to generate a template image. The following describes the training process of the image generation model.

[0080] The image generation model is trained in the following manner: obtaining training data, taking the description information of the training image and the layout image of the training image as input of the initial image generation model, taking the training image as a label, and training the initial image generation model so that the initial image generation model generates an output image that matches the description information of the training image and the layout image of the training image, thereby obtaining the image generation model.

[0081] The training data may be understood as data for training an image generation model, and the training data may include training images, description information of the training images, and layout images of the training images. The layout images of the training images may be used to indicate areas in the training images that do not include foreground objects.

[0082] The training of the image generation model can be supervised training, and the label can be understood as the correct output result in the supervised training. In specific implementation, the description information of the training image and the layout image of the training image are input to the initial image generation model, and the initial image generation model generates an output image that matches the description information of the training image and the layout image of the training image, wherein the image content of the output image is associated with the description information of the training image, and the placeholder area included in the output image corresponds to the area of ​​the training image that does not include the foreground object indicated in the layout image of the training image.

[0083] Next, based on the output image of the initial image generation model and the training image as a label, the loss function value is calculated. The loss function value can characterize the degree of difference between the output image of the initial image generation model and the training image. The model parameters of the initial image generation model are adjusted to minimize the loss function value, thereby completing a round of training process for the initial image generation model.

[0084] Repeat the above steps and perform multiple rounds of training. The training images in each round of training are different. When the degree of difference between the output image of the initial image generation model and the training image meets the training end condition, the training of the initial image generation model is terminated to obtain an image generation model that can generate background images and foreground objects at the same time and can control the position of the foreground object.

[0085] In some embodiments, considering that the image generation model needs to generate a template image and perform position control on a foreground object, the image generation model may include an image generation module and a layout control module. Figure 4 As shown, the layout control module is used to control the layout of the foreground objects in the image. The input of the layout control module includes the initial noise image, description information and layout image, and the output of the layout control module is the layout control feature. The image generation module is used to generate an image. The input of the image generation module includes the initial noise image, description information and layout control feature, and the output of the image generation module is the template image.

[0086] Among them, the initial noise image can be understood as an image composed of initial noise (for example, a random number). Usually, the initial noise image has the same size as the layout image and the template image. The image generation module can use the initial noise image as a reference to generate the template image based on the initial noise image. In this way, the layout control module can extract layout control features based on the initial noise image in combination with the description information and the layout image to increase the controllability of the template image. The image generation module can generate a template image that meets the description information and the layout control features based on the initial noise image in combination with the description information and the layout control features to achieve image generation.

[0087] It should be noted that the image generation model may include more modules in addition to the image generation module and the layout control module. For example, the image generation model may also include a variational autoencoder module, which includes an encoder and a decoder, wherein the encoder is used to downsample the initial noise image (for example, compressing the 512×512 initial noise image into a 16×16 image), and the decoder is used to upsample the image output by the image generation module (for example, restoring the 16×16 image output by the image generation module into a 512×512 template image).

[0088] Combination of the above Figures 1 to 4 The template generation method provided in the embodiment of the present disclosure is introduced in detail. The device and equipment provided in the embodiment of the present disclosure will be introduced below with reference to the accompanying drawings.

[0089] See also Figure 5 The schematic diagram of the structure of the template generating device shown in FIG. 50 includes:

[0090] The acquisition module 501 is used to acquire original multimedia materials and to acquire a layout image; wherein the layout image is used to indicate an area where a foreground object cannot be generated;

[0091] The determination module 502 is used to determine description information of the original multimedia material according to the original multimedia material; wherein the description information is used to describe the multimedia content of the original multimedia material;

[0092] The generation module 503 is used to input the description information of the original multimedia material and the layout image into the image generation model to obtain the template image output by the image generation model; wherein the template image includes a background picture, a foreground object and a placeholder area, and the placeholder area corresponds to the area indicated by the layout image where the foreground object cannot be generated.

[0093] In some possible implementations, the layout image includes a first area carrying a first identifier and a second area carrying a first identifier other than the first identifier, the first area being an area capable of generating a foreground object, and the second area being an area incapable of generating a foreground object.

[0094] In some possible implementations, the layout image is used to indicate an area for placing multimedia materials and an area for placing text content, and the placeholder area in the template image includes a third area and a fourth area, the third area corresponds to the area indicated by the layout image for placing multimedia materials, and the fourth area corresponds to the area indicated by the layout image for placing text content.

[0095] In some possible implementations, the determining module 502 is specifically configured to:

[0096] Generate a first prompt word; wherein the first prompt word includes the original multimedia material and the description information indicating the generation of the original multimedia material;

[0097] The first prompt word is sent to the first model, and the description information of the original multimedia material returned by the first model is received.

[0098] In some possible implementations, the determining module 502 is specifically configured to:

[0099] Determining initial description information of the original multimedia material according to the original multimedia material;

[0100] The initial description information of the original multimedia material is amplified to obtain the description information of the original multimedia material; wherein the amplification includes at least one of the following: splitting the initial description information into information describing the background of the original multimedia material and information describing the foreground of the original multimedia material; adding information for characterizing the multimedia style to the initial description information; and rewriting the initial description information.

[0101] In some possible implementations, the determining module 502 is specifically configured to:

[0102] Generate a second prompt word; wherein the second prompt word includes initial description information of the original multimedia material and content indicating that an amplification process is to be performed on the initial description information;

[0103] The second prompt word is sent to the second model, and the description information of the original multimedia material returned by the second model is received.

[0104] In some possible implementations, the apparatus 50 further includes a configuration module, and the configuration module is configured to:

[0105] determining the foreground object in the template image;

[0106] The foreground object is configured as the topmost layer in the template image.

[0107] In some possible implementations, the generating module 503 is further configured to:

[0108] The original multimedia material is placed in the placeholder area of ​​the template image to generate target content.

[0109] In some possible implementations, the placeholder area of ​​the template image includes a third area and a fourth area, and the determining module 502 is further configured to:

[0110] Determining, based on the original multimedia material, text content related to the original multimedia material;

[0111] The generating module 503 is specifically used for:

[0112] The original multimedia material is placed in the third area, and the text content related to the original multimedia material is placed in the fourth area to generate target content.

[0113] In some possible implementations, the image generation model is trained in the following manner:

[0114] Acquire training data; wherein the training data includes a training image, description information of the training image, and a layout image of the training image, wherein the layout image of the training image is used to indicate an area in the training image that does not include a foreground object;

[0115] The description information of the training image and the layout image of the training image are used as inputs of an initial image generation model, and the training image is used as a label to train the initial image generation model so that the initial image generation model generates an output image that matches the description information of the training image and the layout image of the training image, thereby obtaining the image generation model; wherein the image content of the output image is associated with the description information of the training image, and a placeholder area included in the output image corresponds to an area in the training image that does not include a foreground object and is indicated in the layout image of the training image.

[0116] The template generating device 50 according to the embodiment of the present disclosure may correspond to executing the method described in the embodiment of the present disclosure, and the above and other operations and / or functions of each module / unit of the template generating device 50 are respectively to implement Figure 1 For the sake of brevity, the corresponding processes of each method in the illustrated embodiment are not described in detail here.

[0117] The present disclosure also provides an electronic device. The electronic device is specifically used to implement Figure 5 The functions of the template generating device 50 in the illustrated embodiment.

[0118] Figure 6 A schematic diagram of the structure of an electronic device 600 is provided. Figure 6 As shown, the electronic device 600 includes a bus 601, a processor 602, a communication interface 603 and a memory 604. The processor 602, the memory 604 and the communication interface 603 communicate with each other via the bus 601.

[0119] The bus 601 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0120] The processor 602 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0121] The communication interface 603 is used for communicating with the outside. For example, the communication interface 603 can be used for communicating with a terminal.

[0122] The memory 604 may include a volatile memory, such as a random access memory (RAM). The memory 604 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0123] The memory 604 stores executable codes, and the processor 602 executes the executable codes to perform the above-mentioned template generation method.

[0124] Specifically, in implementing Figure 5 In the case of the embodiment shown, and Figure 5 When each module or unit of the template generation device 50 described in the embodiment is implemented by software, the execution Figure 5 The software or program code required for the functions of each module / unit in the system may be partially or completely stored in the memory 604. The processor 602 executes the program code corresponding to each unit stored in the memory 604 to perform the above-mentioned template generation method.

[0125] The embodiment of the present disclosure also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the template generation method applied to the template generation device 50.

[0126] The embodiment of the present disclosure also provides a computer program product, which includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, the process or function described in the embodiment of the present disclosure is generated in whole or in part.

[0127] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer or data center to another website, computer or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0128] When the computer program product is executed by a computer, the computer executes any of the aforementioned template generation methods. The computer program product may be a software installation package, and when any of the aforementioned template generation methods is required, the computer program product may be downloaded and executed on a computer.

[0129] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to each embodiment of the present disclosure. Each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0130] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit / module does not, in some cases, constitute a limitation on the unit itself.

[0131] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0132] In the context of the disclosed embodiments, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0133] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0134] It should be understood that in the present disclosure, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0135] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0136] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0137] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A template generation method, characterized in that: The method comprises: Acquire original multimedia material, and acquire a layout image; wherein the layout image is used to indicate an area where a foreground object cannot be generated; Determining description information of the original multimedia material according to the original multimedia material; wherein the description information is used to describe multimedia content of the original multimedia material; The description information of the original multimedia material and the layout image are input into an image generation model to obtain a template image output by the image generation model; wherein the template image includes a background picture, a foreground object and a placeholder area, and the placeholder area corresponds to an area indicated by the layout image where a foreground object cannot be generated.

2. The method according to claim 1, characterized in that The layout image includes a first area carrying a first identifier and a second area carrying an identifier other than the first identifier, the first area is an area where a foreground object can be generated, and the second area is an area where a foreground object cannot be generated.

3. The method according to claim 1, characterized in that The layout image is used to indicate an area for placing multimedia materials and an area for placing text content. The placeholder area in the template image includes a third area and a fourth area. The third area corresponds to the area indicated by the layout image for placing multimedia materials, and the fourth area corresponds to the area indicated by the layout image for placing text content.

4. The method according to claim 1, characterized in that: The determining, according to the original multimedia material, description information of the original multimedia material comprises: Generate a first prompt word; wherein the first prompt word includes the original multimedia material and the description information indicating the generation of the original multimedia material; The first prompt word is sent to the first model, and the description information of the original multimedia material returned by the first model is received.

5. The method according to claim 1, characterized in that The determining, according to the original multimedia material, description information of the original multimedia material comprises: Determining initial description information of the original multimedia material according to the original multimedia material; The initial description information of the original multimedia material is amplified to obtain the description information of the original multimedia material; wherein the amplification includes at least one of the following: splitting the initial description information into information describing the background of the original multimedia material and information describing the foreground of the original multimedia material; adding information for characterizing the multimedia style to the initial description information; and rewriting the initial description information.

6. The method according to claim 5, characterized in that The amplifying the initial description information of the original multimedia material to obtain the description information of the original multimedia material includes: Generate a second prompt word; wherein the second prompt word includes initial description information of the original multimedia material and content indicating that an amplification process is to be performed on the initial description information; The second prompt word is sent to the second model, and the description information of the original multimedia material returned by the second model is received.

7. The method according to claim 1, characterized in that The method further comprises: determining the foreground object in the template image; The foreground object is configured as the topmost layer in the template image.

8. The method according to claim 1, characterized in that: The method further comprises: The original multimedia material is placed in the placeholder area of ​​the template image to generate target content.

9. The method according to claim 8, characterized in that The placeholder area of ​​the template image includes a third area and a fourth area, and the method further includes: Determining, based on the original multimedia material, text content related to the original multimedia material; Placing the original multimedia material in the placeholder area of ​​the template image to generate target content includes: The original multimedia material is placed in the third area, and the text content related to the original multimedia material is placed in the fourth area to generate target content.

10. The method according to any one of claims 1 to 9, characterized in that: The image generation model is trained in the following way: Acquire training data; wherein the training data includes a training image, description information of the training image, and a layout image of the training image, wherein the layout image of the training image is used to indicate an area in the training image that does not include a foreground object; The description information of the training image and the layout image of the training image are used as inputs of an initial image generation model, and the training image is used as a label to train the initial image generation model so that the initial image generation model generates an output image that matches the description information of the training image and the layout image of the training image, thereby obtaining the image generation model; wherein the image content of the output image is associated with the description information of the training image, and a placeholder area included in the output image corresponds to an area in the training image that does not include a foreground object and is indicated in the layout image of the training image.

11. A template generating device, characterized in that: The device comprises: An acquisition module, used to acquire original multimedia materials and to acquire a layout image; wherein the layout image is used to indicate an area where a foreground object cannot be generated; A determination module, configured to determine description information of the original multimedia material according to the original multimedia material; wherein the description information is used to describe multimedia content of the original multimedia material; A generation module is used to input the description information of the original multimedia material and the layout image into an image generation model to obtain a template image output by the image generation model; wherein the template image includes a background image, a foreground object and a placeholder area, and the placeholder area corresponds to an area indicated by the layout image where a foreground object cannot be generated.

12. An electronic device, characterized in that: The electronic device comprises a processor and a memory; The processor is configured to execute instructions stored in the memory, so that the electronic device performs the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The method comprises instructions, wherein the instructions instruct an electronic device to execute the method according to any one of claims 1 to 10.

14. A computer program product, characterized in that The computer program product comprises computer readable instructions for implementing the method according to any one of claims 1 to 10.

Images