Media file generation method and system, electronic equipment and storage medium

Abstract

The invention discloses a media file generation method and system, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring a foreground image comprising at least one object, and acquiring at least one piece of background feature information matched with the object and to-be-rendered text description information; the foreground image and the background feature information are input into an image processing model for analysis to obtain image features, the text description information is input into a text processing model for analysis to obtain text features, and the image processing model and the text processing model are obtained based on training of a generative model; and fusing the image features and the text features by using the generative model to obtain a media file of the object, the media file at least comprising a background image and a rendering result presented on the background image. The technical problem of low generation efficiency of the media file is solved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology. Specifically, it relates to a method, system, electronic device, and storage medium for generating media files. Background Art

[0002] Currently, when generating media files, it is usually necessary for professional photographers to take foreground images including objects in a set scene. After shooting, designers also need to perform post-processing on the foreground images, including but not limited to cropping, color correction, background addition, and text design.

[0003] However, the above method requires manually adding appropriate text, design elements, and color schemes according to the style of the object to enhance the conveying power of the media file, which is not only time-consuming but also costly, thus resulting in the problem of low efficiency in generating media files.

[0004] In view of the above problems, no effective solution has been proposed yet. Summary of the Invention

[0005] Embodiments of this application provide a method, system, electronic device, and storage medium for generating media files to at least solve the technical problem of low efficiency in generating media files.

[0006] According to one aspect of the embodiments of this application, a method for generating a media file is provided. The method may include: obtaining a foreground image including at least one object, and obtaining at least one piece of background feature information matching the object, as well as text description information to be rendered; inputting the foreground image and the background feature information into an image processing model for analysis to obtain image features, and inputting the text description information into a text processing model for analysis to obtain text features, where the image processing model and the text processing model are trained based on a generation model; using the generation model to fuse the image features and the text features to obtain a media file of the object, where the media file at least includes a background image and a rendering result presented on the background image.

[0007] According to another aspect of the embodiments of the present application, a method for determining a model is provided. The method may include: obtaining a foreground image sample including at least one object sample, and obtaining at least one background feature information sample and a text description information sample that match the object sample; using the foreground image sample, the background feature information sample, and the text description information sample to train a generation model to obtain an image processing model and a text processing model, where the image processing model is used to analyze a foreground image including at least one object and at least one background feature information that matches the object to obtain image features, the text processing model is used to analyze the text description information to be rendered to obtain text features, and the generation model is used to fuse the image features and the text features to obtain a media file of the object, and the media file includes at least a background image and a rendering result presented on the background image.

[0008] According to another aspect of the embodiments of the present application, a method for generating a media file is provided. The method may include: obtaining, from an e-commerce platform, a foreground image including at least one object, and obtaining at least one background feature information that matches the object, and text description information to be rendered; inputting the foreground image and the background feature information into an image processing model for analysis to obtain image features, and inputting the text description information into a text processing model for analysis to obtain text features, where the image processing model and the text processing model are obtained by training based on a generation model; using the generation model to fuse the image features and the text features to obtain a media file of the object, where the media file includes at least a background image and a rendering result presented on the background image; and returning the media file to the e-commerce platform for display.

[0009] According to another aspect of the embodiments of the present application, a method for generating a media file is provided. The method may include: responding to an input operation on an operation interface, displaying, on the operation interface, a foreground image including at least one object, and displaying at least one background feature information that matches the object, and text description information to be rendered; responding to a file generation operation on the operation interface, displaying, on the operation interface, a media file of the object, where the media file is obtained by fusing image features and text features based on a generation model, and the media file includes at least a background image and a rendering result presented on the background image, the image features are obtained by analyzing the foreground image and the background feature information by an image processing model, the text features are obtained by analyzing the text description information by a text processing model, and the image processing model and the text processing model are obtained by training based on a generation model.

[0010] According to another aspect of the embodiments of the present application, a method for generating a media file is provided. The method may include: obtaining a foreground image including at least one object by invoking a first interface, and obtaining at least one piece of background feature information matching the object, as well as text description information to be rendered, where the first interface includes a first parameter, and the parameter value of the first parameter includes the foreground image, the background feature information, and the text description information; inputting the foreground image and the background feature information into an image processing model for analysis to obtain image features, and inputting the text description information into a text processing model for analysis to obtain text features, where the image processing model and the text processing model are trained based on a generation model; using the generation model to fuse the image features and the text features to obtain a media file of the object, where the media file at least includes a background image and a rendering result presented on the background image; outputting the media file by invoking a second interface, where the second interface includes a second parameter, and the parameter value of the second parameter includes the media file.

[0011] According to another aspect of the embodiments of the present application, a system for generating a media file is provided. The system may include: a client for transmitting a foreground image including at least one object, and obtaining at least one piece of background feature information matching the object, as well as text description information to be rendered; a server for inputting the foreground image and the background feature information into an image processing model for analysis to obtain image features, and inputting the text description information into a text processing model for analysis to obtain text features, where the image processing model and the text processing model are trained based on a generation model; generating a media file of the object based on the image features and the text features, where the media file at least includes a background image and a rendering result presented on the background image; and returning the media file to the client.

[0012] According to another aspect of the embodiments of the present application, a computing device is further provided, including: a memory storing an executable program; a processor for running the program, where when the program runs, it executes the methods in the various embodiments of the present application.

[0013] According to another aspect of the embodiments of the present application, an electronic device is further provided, including: a memory storing an executable program; a processor connected to the memory through a bus for running the program, where when the program runs, it executes the methods in the various embodiments of the present application.

[0014] According to another aspect of the embodiments of the present application, a computer-readable storage medium is further provided. The computer-readable storage medium includes a stored executable program, where when the executable program runs, it controls the device where the computer-readable storage medium is located to execute the methods in the various embodiments of the present application.

[0015] According to another aspect of the embodiments of the present application, there is also provided a computer program product, including a computer program, which implements the methods in various embodiments of the present application when executed by a processor.

[0016] According to another aspect of the embodiments of the present application, there is also provided a computer program product, including a non-volatile computer-readable storage medium, where the non-volatile computer-readable storage medium stores a computer program, and the computer program implements the methods in various embodiments of the present application when executed by a processor.

[0017] According to another aspect of the embodiments of the present application, there is also provided a computer program, which implements the methods in various embodiments of the present application when executed by a processor.

[0018] In the embodiments of the present application, a foreground image including at least one object is obtained, and at least one piece of background feature information matching the object, as well as text description information to be rendered, are obtained; the foreground image and the background feature information are input into an image processing model for analysis to obtain image features, and the text description information is input into a text processing model for analysis to obtain text features, where the image processing model and the text processing model are trained based on a generation model; the generation model is used to fuse the image features and the text features to obtain a media file of the object, where the media file at least includes a background image and a rendering result presented on the background image. That is to say, in the embodiments of the present application, the image processing model and the text processing model are trained based on the generation model, the image processing model is used to analyze the foreground image and the background feature information to obtain image features, and the text processing model is used to analyze the text description information to obtain text features, and then the generation model is used to directly fuse the image features and the text features to obtain a media file, avoiding generating a media file relying on a template, resulting in a strong sense of template, or relying on manual addition of text information to the background image, resulting in low efficiency of generating a media file, thereby achieving the technical effect of improving the generation efficiency of the media file and solving the technical problem of low generation efficiency of the media file.

[0019] It is easy to note that the above general description and the following detailed description are only for exemplifying and explaining the present application and do not constitute a limitation to the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present application and form a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation to the present application. In the drawings:

[0021] Figure 1 is a schematic diagram of an application scenario of a method for generating a media file according to an embodiment of the present application;

[0022] Figure 2 It is a flowchart of a method for generating a media file according to an embodiment of the present application;

[0023] Figure 3 It is a flowchart of a method for determining a model according to an embodiment of the present application;

[0024] Figure 4 It is a flowchart of another method for generating a media file according to an embodiment of the present application;

[0025] Figure 5 It is a flowchart of yet another method for generating a media file according to an embodiment of the present application;

[0026] Figure 6 It is a flowchart of yet another method for generating a media file according to an embodiment of the present application;

[0027] Figure 7 It is a schematic diagram of a system for generating a media file according to an embodiment of the present application;

[0028] Figure 8 It is a schematic diagram of a text-to-image algorithm model framework for automated advertising poster design according to an embodiment of the present application;

[0029] Figure 9 It is a schematic diagram of a device for generating a media file according to an embodiment of the present application;

[0030] Figure 10 It is a schematic diagram of a device for determining a model according to an embodiment of the present application;

[0031] Figure 11 It is a schematic diagram of another device for generating a media file according to an embodiment of the present application;

[0032] Figure 12 It is a schematic diagram of yet another device for generating a media file according to an embodiment of the present application;

[0033] Figure 13 It is a schematic diagram of yet another device for generating a media file according to an embodiment of the present application;

[0034] Figure 14 It is a structural block diagram of a computing device according to an embodiment of the present application;

[0035] Figure 15 It is a structural block diagram of an electronic device according to an embodiment of the present application. Detailed implementation manners

[0036] To enable those skilled in the art to better understand the solution of this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this application.

[0037] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned accompanying drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of this application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0038] The technical solution provided by this application is mainly implemented by a deep learning model. The deep learning model can be widely applied to fields such as natural language processing (abbreviated as NLP), computer vision, and speech processing. Specifically, it can be applied to tasks in the field of computer vision such as visual question answering (abbreviated as VQA), image caption (abbreviated as IC), and image generation, and can also be widely applied to tasks in the field of natural language processing such as text-based sentiment classification, text summary generation, and machine translation. Therefore, the main application scenarios of this application include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0039] First, some nouns or terms that appear in the process of describing the embodiments of this application are applicable to the following explanations:

[0040] Variational Auto-Encoder (abbreviated as VAE), which is widely used in unsupervised learning and generation tasks of data such as images, sounds, and texts. The design idea is to learn the low-dimensional representation of the data (i.e., the encoding process), and then reconstruct the original data from this low-dimensional representation (i.e., the decoding process), so as to achieve data compression and feature extraction;

[0041] An image generation framework based on a diffusion model, including an image reconstruction VAE module and a cascaded image generation framework module based on a diffusion model, which realizes high-quality image generation through a step-by-step denoising process;

[0042] The Control Network, an architecture expansion module of a neural network based on conditional control, belongs to the field of computer vision and generative artificial intelligence. The Control Network realizes multi-dimensional guidance of the generation process by an external control signal through introducing a trainable network copy to perform weight coupling with the original model parameters. That is to say, the Control Network realizes progressive fusion of the control signal input by the user with the feature space of the basic generation model (such as a diffusion model) by establishing a differentiable architecture including a zero-initialized convolutional layer, and realizes fine-grained control of the geometric structure, spatial layout, and semantic features of the generation result while maintaining the generation ability of the original model. The Control Network is embodied as a system combination including a conditional encoder, a feature fusion module, and a gradient stabilization mechanism, and is applicable to scenarios where precise control of the generated content is required in image generation.

[0043] According to an embodiment of the present application, a method for generating a media file is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0044] The deep learning model involved in the embodiments of the present application can be an artificial intelligence-based language model (LM for short) or a multimodal model (MM for short), etc. The embodiments of the present application do not limit the number of model parameters supported by the model, aiming to meet actual needs. If the number of model parameters is relatively large, the scale of the model will be relatively large and the model performance will be relatively better. Of course, more time and resources will be consumed during inference or training; if the number of model parameters is relatively small, the scale of the model will be relatively small, and the model is more lightweight when the performance meets the requirements, and relatively less time and resources are consumed during inference or training.

[0045] Considering the limited computing resources of mobile terminals, the above method provided by the embodiments of the present application can be applied to, for example, Figure 1 the application scenarios shown, but not limited thereto. Figure 1 is a schematic diagram of an application scenario of a method for generating a media file according to an embodiment of the present application. As shown in Figure 1In the application scenario shown, the deep learning model is deployed in the server 10. The server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. Here, the client devices 20 can include, but are not limited to: smartphones, tablets, laptops, palmtop computers, personal computers, smart home devices, in-vehicle devices, etc. An operation interface for obtaining a foreground image including at least one object can be deployed on the graphical user interface of the client device. The client device 20 can interact with the user through the graphical user interface to implement the invocation of the deep learning model, and further implement the method provided by the embodiments of the present application.

[0046] In the embodiments of the present application, the system composed of the client device and the server can perform the following steps: corresponding operations can be performed in the operation interface on the client device 20 to obtain a foreground image including at least one object. The client device can obtain the foreground image including at least one object and send it to the server through the network. After receiving the foreground image including at least one object, the server can perform the following steps: Step S102, obtain the foreground image including at least one object, and obtain at least one piece of background feature information matching the object, as well as the text description information to be rendered; Step S104, input the foreground image and the background feature information into an image processing model for analysis to obtain image features, and input the text description information into a text processing model for analysis to obtain text features, where the image processing model and the text processing model are trained based on a generation model; Step S106, use the generation model to fuse the image features and the text features to obtain a media file of the object, where the media file at least includes a background image and a rendering result presented on the background image.

[0047] It should be noted that with the rapid development of high-performance computing units, in other application scenarios, the above method provided by the embodiments of the present application can also be applied to a model all-in-one machine. In an alternative embodiment, multiple models are built into the model all-in-one machine, and one model can be selected for adjustment as needed. Thus, the high-performance computing unit built into the model all-in-one machine can directly call the adjusted model to execute the above method provided by the embodiments of the present application. In another alternative embodiment, the above model all-in-one machine can be a deep learning model all-in-one machine, and a trained model is built into the deep learning model all-in-one machine. Thus, the high-performance computing unit built into the model all-in-one machine can directly call the model to execute the above method provided by the embodiments of the present application.

[0048] Further, when it is necessary to train a model in a target scenario, a dataset can also be uploaded through the client. The dataset is sent from the client to the server, enabling the server to adjust the pre-trained model with this dataset to obtain the model in the target scenario, and then deploy it to the production environment. To facilitate the adjustment of the model, the server can provide a complete set of adjustment tools, development frameworks, and processes, which can support multiple adjustment strategies, enabling the adjusted model to better adapt to different domain applications and achieve high customization.

[0049] Under the above operating environment, the present application provides a method for generating a media file as Figure 2 shown. Figure 2 FIG. is a flowchart of a method for generating a media file according to an embodiment of the present application. As Figure 2 shown, the method may include the following steps:

[0050] Step S202, obtain a foreground image including at least one object, obtain at least one piece of background feature information matching the object, and caption information to be rendered.

[0051] In the technical solution provided in step S202 of the present application, a foreground image including at least one object can be obtained, and at least one piece of background feature information matching the object, and caption information to be rendered can be obtained. Among them, the object can be an entity object to be displayed on the media file. For example, when the media file is an advertising poster file, the object can be the product displayed in the advertising poster file, and the type of the product can be clothing (such as shoes), snacks (such as biscuits), furniture (such as sofas), daily necessities (such as laundry detergent), etc. This is only for illustrative purposes and does not specifically limit the type of the object.

[0052] In this embodiment, the foreground image can be an image including one or more objects. For example, in advertising poster design, the foreground image can be a high-definition image of the product, which can be placed in the main position (conspicuous position) of the advertising poster file to attract the attention of users. For example, an image showing a new mobile phone can be determined as a foreground image including the object of the mobile phone, and an image of a pile of round biscuits can be determined as a foreground image including the object of round biscuits.

[0053] Optionally, the background feature information can be information describing the background features of the media file, and can also be referred to as background text description information (prompt). The background feature information can include the color, texture, style, scene type (such as natural landscape, urban street view, indoor environment, etc.), spatial layout, etc. of the media file background. The background feature information can be used to generate a background image that matches the object. For example, in the design of an advertising poster, when the object is a sofa, the background feature information can be "a soft warm-color gradient background with some installations such as bookshelves".

[0054] Optionally, the text description information to be rendered can be the text content to be displayed on the media file and the presentation method of the text content. The presentation method can at least include the font, size, color, position, etc. of the text content. For example, based on the promotional application requirements of the advertising poster design scenario, in addition to the size and position of the text content, the presentation method of the text content can also include the font of the text content, the color of the text content, etc. to enhance the promotional effect of the advertising poster. It should be noted that the above is only an example, and other presentation methods of text content are within the protection scope of this patent. The text description information to be rendered can be used to convey the features, promotional information or other important information of the object to the user. For example, in the design of an advertising poster, when the object is a biscuit, the text description information to be rendered can be "This is a nutritious whole wheat biscuit with a limited-time discount", and at the same time, the position (such as the top of the advertising poster) and style (such as using green font) where the text content should be placed are attached.

[0055] Optionally, the foreground image can be obtained through an image library (such as obtained by professional photographers), online scraping, user upload, etc. The background feature information can be obtained through a preset background template, text description (such as the merchant or user describes the required background features through text), etc. The text description information to be rendered can be obtained through user input, automated generation, etc. It should be noted that the above is only an example, and there is no specific limitation on the acquisition methods of the foreground image, background feature information, and text description information to be rendered.

[0056] Step S204: Input the foreground image and the background feature information into an image processing model for analysis to obtain image features, and input the text description information into a text processing model for analysis to obtain text features.

[0057] In the technical solution provided in step S204 of the present application above, the image processing model and the text processing model are trained based on a generative model.

[0058] In this embodiment, after obtaining a foreground image including at least one object, obtaining at least one piece of background feature information matching the object, and obtaining the text description information to be rendered, the obtained foreground image and background feature information can be input into an image processing model for analysis to obtain image features, and the obtained text description information can be input into a text processing model for analysis to obtain text features.

[0059] In this embodiment, the image processing model and the text processing model can be trained based on a generative model. Although both the image processing model and the text processing model are trained based on the generative model, their training samples are different, so the functions of the trained models are different. For example, the image processing model can be trained based on foreground image samples. The image processing model can be used to output corresponding image features based on the foreground image and background feature information, and the image processing model can also be called a background generation network model. The text processing model can be trained based on foreground image samples, background feature information samples, and text description information samples. The text processing model can be used to output corresponding text features based on the text description information, and the text processing model can also be called a text rendering network model.

[0060] For example, in the design of an advertising poster, the product foreground image and background text description information can be input into the background generation network model for analysis to obtain corresponding image features.

[0061] Optionally, in the process of inputting the text description information into the text processing model for analysis, the text strings included in the text description information can be subjected to feature extraction through a pre-trained visual encoder (which can also be called a character image encoder) to obtain character features. The text positions and character orders included in the text description information can be encoded respectively to obtain text position encoding and character order encoding. Further, the obtained character features, text position encoding, and character order encoding can be concatenated to obtain a character-level text representation, and then the obtained character-level text representation can be used as a control signal to be input into the text rendering network model after passing through a projection layer to realize precise generation control of the text content and position, so as to obtain corresponding text features.

[0062] Step S206: Use a generative model to fuse the image features and text features to obtain a media file of the object.

[0063] In the technical solution provided in step S206 of the present application above, the media file includes at least a background image and a rendering result presented on the background image.

[0064] In this embodiment, after inputting the foreground image and background feature information into an image processing model for analysis to obtain image features, and inputting the caption information into a text processing model for analysis to obtain text features, a generation model can be used to fuse the image features and text features to obtain a media file of an object. The generation model can be used to generate media files and can also be referred to as a basic generation model or a basic model.

[0065] In this embodiment, the media file can at least include a background image and a rendering result presented on the background image. The media file can be an output result generated by fusing image features and text features through a generation model. For example, the media file can be an advertising poster file, a graphic advertising poster file, etc. that includes an object (such as a product), a background image, and a text description. This is only an example here, and the type of the media file is not specifically limited. The background image can be an image other than the foreground of the object in the media file. The rendering result can be the final visual effect of fusing the foreground image and the caption information to be rendered onto the background image.

[0066] Optionally, after obtaining the image features output by the image processing model and the text features output by the text processing model, the image features and text features can be added layer by layer and then input into the corresponding network layer of the basic model, and then the media file generated by the basic model can be obtained.

[0067] Through steps S202 to S206 of the present application above, a foreground image including at least one object is obtained, and at least one piece of background feature information matching the object and caption information to be rendered are obtained; the foreground image and the background feature information are input into an image processing model for analysis to obtain image features, and the caption information is input into a text processing model for analysis to obtain text features, where the image processing model and the text processing model are trained based on a generation model; a generation model is used to fuse the image features and text features to obtain a media file of the object, where the media file at least includes a background image and a rendering result presented on the background image. That is to say, in the embodiment of the present application, the image processing model and the text processing model are trained based on the generation model, the image processing model is used to analyze the foreground image and the background feature information to obtain image features, and the text processing model is used to analyze the caption information to obtain text features, and then the generation model is used to fuse the image features and text features directly to obtain a media file, avoiding generating a media file relying on a template, resulting in a strong sense of template, or relying on manual addition of text information to the background image, resulting in low efficiency of generating a media file, thereby achieving the technical effect of improving the generation efficiency of the media file and solving the technical problem of low generation efficiency of the media file.

[0068] The method of inputting the foreground image and background feature information into the image processing model for analysis to obtain image features, and inputting the text description information into the text processing model for analysis to obtain text features in the above embodiment will be further introduced below.

[0069] As an alternative implementation, in step S204, inputting the foreground image and background feature information into the image processing model for analysis to obtain image features, and inputting the text description information into the text processing model for analysis to obtain text features, including: inputting the foreground image, background feature information, and object position information into the image processing model for analysis to obtain image features, where the object position information is used to represent the position where the object is to be displayed on the background image; inputting the text description content and text position information in the text description information into the text processing model for analysis to obtain text features, where the text position information is used to represent the position where the text description information is to be rendered on the background image.

[0070] In this embodiment, after obtaining the foreground image including at least one object, at least one background feature information matching the object, and the text description information to be rendered, the obtained foreground image, background feature information, and object position information can be input into the image processing model for analysis to obtain image features. The text description content and text position information in the text description information can be input into the text processing model for analysis to obtain text features.

[0071] In this embodiment, the object position information can be used to represent the position where the object is to be displayed on the background image, and the object position information can be used to represent the position where the object is expected to be placed on the background image. The text description content can be used to represent the content of the text (i.e., the copywriting) to be rendered. The text position information can be used to represent the position where the text description information is to be rendered on the background image, and the text position information can be the corresponding position information of the text content to be rendered, and can also be called the geometric information of the text box.

[0072] For example, the product foreground image, the expected placement position, and the background text description prompt can be input into the background generation network model for analysis to obtain the corresponding image features. The text content to be rendered and the corresponding position information can be input into the text rendering network model for analysis to obtain the corresponding text features.

[0073] By adding the object position information and text position information in this embodiment, the generated media file can accurately control the positions of the object and the text on the background, avoiding the problem of low accuracy in manual layout, ensuring the symmetry and visual balance of the advertising poster design, and thus improving the professionalism and aesthetics of the generated media file.

[0074] The method of inputting the foreground image, background feature information, and object position information into an image processing model for analysis to obtain image features in the above embodiment will be further introduced below.

[0075] As an alternative implementation, inputting the foreground image, background feature information, and object position information into an image processing model for analysis to obtain image features includes: inputting the foreground image, background feature information, and object position information into the image processing model; using the image processing model to determine the foreground area of the foreground image on the background according to the object position information, and using the background feature information as guiding information to guide the image processing model to generate the background image of the background; using the image processing model to generate image features from the foreground area, background image, and foreground image.

[0076] In this embodiment, after obtaining the foreground image including at least one object and obtaining at least one piece of background feature information matching the object, the obtained foreground image, background feature information, and object position information can be input into the image processing model. Further, the image processing model can be used to determine the foreground area of the foreground image on the background according to the object position information, and the background feature information can be used as guiding information to guide the image processing model to generate the background image of the background. Furthermore, the image processing model can be used to generate image features from the foreground image, the determined foreground area, and the generated background image. Among them, the foreground area can be the placement area of the foreground image on the background.

[0077] Optionally, the image processing model is used to determine the foreground area (i.e., the placement area) of the foreground image on the background according to the object position information. Among them, the foreground area can be the image space occupied by the main object displayed in the media file. The object position information can be coordinates, center points, dimensions, or the relative position of the object in the image. That is, the image processing model can identify and retain the foreground area according to the object position information to ensure that the object is in a prominent position in the finally generated media file.

[0078] Optionally, the background feature information can include descriptive information about the background style, color, texture, etc. Based on the background feature information, the image processing model can be guided to generate a matching background image. For example, when the object is outdoor sports equipment, the background image can be a mountain forest or a sports field to set off the characteristics of the product.

[0079] Optionally, while the image processing model generates the background image, the foreground image can be integrated into the background to ensure seamless connection between the object and the background. After the foreground and background are synthesized, the image processing model will further analyze and extract image features to ensure that the generated media file not only contains visual elements but also has a sense of design.

[0080] By inputting the object position information, this embodiment can ensure that the foreground image is fused onto the background image at the expected position, avoiding the errors caused by manual positioning and achieving precise spatial layout and visual element positioning. Further, by using the background feature information as guiding information, the image processing model can generate a highly customized background image, which not only conforms to the characteristics of the object but also can be flexibly changed according to different scenes, themes, and objectives, thereby enhancing the diversity of the generated media files.

[0081] The following further introduces the method of using the image processing model in this embodiment to generate image features from the foreground region, background image, and foreground image.

[0082] As an optional implementation, using the image processing model to generate image features from the foreground region, background image, and foreground image includes: using the control network in the image processing model to fuse the foreground region, background image, and foreground image into the feature space of the target generation model in the image processing model to obtain a first fusion result, where the model parameters of the target generation model in the image processing model in the initialization stage are copied from the generation model; using the convolutional layer corresponding to the target generation model in the image processing model to perform convolutional processing on the first fusion result to obtain image features.

[0083] In this embodiment, after using the image processing model to determine the foreground region of the foreground image on the background according to the object position information and using the background feature information as guiding information to guide the image processing model to generate the background image, the control network in the image processing model can be used to fuse the foreground region, background image, and foreground image into the feature space of the target generation model in the image processing model to obtain a first fusion result. Further, the convolutional layer corresponding to the target generation model in the image processing model can be used to perform convolutional processing on the first fusion result to obtain image features.

[0084] In this embodiment, the model parameters of the target generation model in the image processing model in the initialization stage are copied from the generation model, and the model parameters can at least include weights. The target generation model can also be called the basic generation model. For example, the target generation model can be a diffusion model. This is only an example for illustration, and the type of the target generation model is not specifically limited.

[0085] It should be noted that the background generation network model is based on a similar architecture to the control network and includes several cascaded image generation framework modules based on the diffusion model. The weights are copied from the basic model during initialization, and the output of each image generation framework module based on the diffusion model is added to the corresponding module of the basic model after passing through a zero convolutional layer.

[0086] Optionally, after obtaining the foreground region, the background image, and the foreground image, a control network can be used to progressively fuse the obtained foreground region, background image, and foreground image as control signals with the feature space of a base generation model (such as a diffusion model) to obtain a first fusion result. The first fusion result can be a preliminary generation result after feature space fusion under the action of the control network, which can contain the comprehensive information of the foreground region, background image, and foreground image, but further optimization is required to generate the final image features.

[0087] Optionally, after obtaining the first fusion result, a convolutional layer in the target generation model is used to perform convolutional processing on the first fusion result to further extract and optimize the image features. Among them, the convolutional layer performs local processing on the input features through a sliding window, which can capture spatial relationships, such as edges, shapes, textures, etc. This is crucial for generating visually appealing and semantically coherent media files. The features after convolutional processing, that is, the image features, contain the key information required to generate the media file and can be used to guide subsequent generation steps, thereby ensuring that the output media file is both beautiful and accurately conveys the intended information.

[0088] This embodiment uses a control network to accurately locate the position of foreground elements in the background based on the input foreground image and background information, thereby ensuring their natural visual fusion. During the process of generating the first fusion result, image processing can accurately place the foreground image at a specific position in the background while keeping the shape and texture of the foreground elements unchanged, thereby improving the visual effect and professionalism of the media file.

[0089] Next, a further introduction will be given to the method of inputting the text description content and text position information in the above text description information of this embodiment into a text processing model for analysis to obtain text features.

[0090] As an optional implementation manner, the text description information includes at least one text unit and the sorting information of the text unit in the text description information. Inputting the text description content and text position information in the text description information into a text processing model for analysis to obtain text features includes: encoding the text unit to obtain text unit features; encoding the sorting information to obtain a sorting encoding result; encoding the text position information to obtain a position encoding result; and inputting the position encoding result, the sorting encoding result, and the text unit features into the text processing model for analysis to obtain text features.

[0091] In this embodiment, the literal description information may include at least one literal unit and the sorting information of the literal unit in the literal description information. After obtaining the literal description information to be rendered, the literal unit can be encoded to obtain the literal unit feature; the sorting information can be encoded to obtain the sorting encoding result; and the literal position information can be encoded to obtain the position encoding result. Further, the obtained position encoding result, sorting encoding result, and literal unit feature can be input into a text processing model for analysis to obtain the text feature.

[0092] In this embodiment, the literal unit can be a character in a text string, and each character can be precisely represented by a token. The literal unit feature obtained by encoding the literal unit can also be called the character feature. The sorting information of the literal unit in the literal description information can be the character order. The sorting encoding result obtained by encoding the sorting information can also be called the character order encoding. The literal position information can be the position corresponding to the text content to be rendered, and can also be called the text position or text box position. The position encoding result obtained by encoding the literal position information can also be called the text box position encoding.

[0093] Optionally, after obtaining the literal description information to be rendered, the literal unit can be encoded by a pre-trained visual encoder (character image encoder) to obtain the literal unit feature. The sorting information and the literal position information can also be encoded respectively to obtain the sorting encoding result and the position encoding result. Further, the obtained literal unit feature, sorting encoding result, and position encoding result can be concatenated and then input into a text processing model for analysis to obtain the text feature.

[0094] Optionally, after obtaining the text string (literal description information) to be rendered, for each character in the text string, the feature is extracted separately by a pre-trained visual encoder to obtain the character feature. On this basis, the character order encoding (sorting encoding result) and the text box position encoding (position encoding result) are further introduced, and the character order encoding, text box position encoding, and character feature are concatenated. After passing through a projection layer, it is used as a control signal to be input into the text rendering network model to achieve precise generation control of the text content and position, thereby obtaining the text feature.

[0095] It should be noted that the character order encoding and the text box position encoding are vector representations obtained by encoding the order information of characters and the geometric information of text boxes respectively. The dimensions of the character order encoding and the text box position encoding are not necessarily the same as those of the character features. When the character order encoding, the text box position encoding and the character features are concatenated, they are concatenated in the feature dimension. For example, assuming that the dimension of the character order encoding is 50, the dimension of the text box position encoding is 100, and the dimension of the character features is 1000, the concatenated features will be 1150 dimensions.

[0096] Through the character-level representation method, this embodiment enables the text rendering network model to accurately control the position and layout of each character in the image, ensuring that the typesetting of the text in the media file is highly consistent with the design requirements, that is, allowing the text rendering network model to maintain precise control over the text content and position during the generation process, avoiding problems such as misalignment of the position between the text and the image background or low visual coordination.

[0097] Next, a further introduction will be given to the method of inputting the position encoding result, the sorting encoding result, and the text unit features into the text processing model in this embodiment to obtain text features.

[0098] As an optional implementation manner, inputting the position encoding result, the sorting encoding result, and the text unit features into the text processing model for analysis to obtain text features includes: concatenating the position encoding result, the sorting encoding result, and the text unit features to obtain a concatenated result; converting the concatenated result into a format matching the text processing model; and inputting the converted concatenated result into the text processing model for analysis to obtain text features.

[0099] In this embodiment, after encoding the text unit to obtain the text unit features, encoding the sorting information to obtain the sorting encoding result, and encoding the text position information to obtain the position encoding result, the obtained position encoding result, sorting encoding result, and text unit features can be concatenated to obtain a concatenated result. Further, the obtained concatenated result can be converted into a format matching the text processing model, and then the converted concatenated result is input into the text processing model for analysis to obtain text features.

[0100] Optionally, after encoding the text unit to obtain the text unit features, encoding the sorting information to obtain the sorting encoding result, and encoding the text position information to obtain the position encoding result, the obtained position encoding result, sorting encoding result, and text unit features can be concatenated, and the obtained concatenated result is the character-level text representation.

[0101] Further, the obtained character-level text representation can be converted into a format that matches the text processing model. That is, the obtained character-level text representation passes through a projection layer to obtain a converted character-level text representation, and the converted character-level text representation is used as a control signal to input into the text rendering network model to achieve precise generation control of the text content and position, thereby obtaining text features.

[0102] In this embodiment, by concatenating the position encoding result, the sorting encoding result, and the text unit feature, the text rendering network model can consider the position information, order information, and visual features of the characters simultaneously. As a result, the generated text content can not only accurately reflect the input text information but also be correctly positioned and arranged in order in the image, improving the visual consistency and information accuracy of the generated media file. In addition, the role of the projection layer is to convert the character-level text representation into a format that can match the text rendering network model. The converted character-level text representation is used as a control signal to input into the text rendering network model, realizing precise control of text elements in the media file and avoiding the problem of low generation quality caused by incorrect positions or poor visual effects.

[0103] Next, the method of inputting the converted concatenation result into the text processing model in this embodiment to analyze and obtain text features will be further introduced.

[0104] As an optional implementation manner, inputting the converted concatenation result into the text processing model for analysis to obtain text features includes: inputting the converted concatenation result into the text processing model; using the control network in the text processing model to fuse the converted concatenation result into the feature space of the target generation model in the text processing model to obtain a second fusion result, where the model parameters of the target generation model in the text processing model in the initialization stage are copied from the generation model; using the convolutional layer corresponding to the generation model in the text processing model to perform convolutional processing on the second fusion result to obtain text features.

[0105] In this embodiment, after concatenating the position encoding result, the sorting encoding result, and the text unit feature to obtain a concatenation result and converting the concatenation result into a format that matches the text processing model, the converted concatenation result can be input into the text processing model, and the control network in the text processing model is used to fuse the converted concatenation result into the feature space of the target generation model in the text processing model to obtain a second fusion result. Further, the convolutional layer corresponding to the generation model in the text processing model can be used to perform convolutional processing on the obtained second fusion result to obtain text features.

[0106] In this embodiment, the model parameters of the target generative model in the text processing model during the initialization phase are copied from the generative model. These model parameters may include at least weights. It should be noted that the text rendering network model is based on an architecture similar to the control network, comprising several cascaded diffusion-based image generation framework modules. During initialization, the weights are copied from the base model. The output of each diffusion-based image generation framework module is added to the corresponding module in the base model after passing through a zero-convolution layer.

[0107] Optionally, after converting the concatenation result into a format compatible with the text processing model, the control network can be used to progressively fuse the converted concatenation result with the feature space of a basic generative model (e.g., a diffusion model) using the control signal to obtain a second fusion result. This second fusion result can be a preliminary result generated after feature space fusion processing under the control network. It can include comprehensive information such as position encoding results, sorting encoding results, and character unit features, but further optimization is required to generate the final text features.

[0108] Optionally, after obtaining the second fusion result, the convolutional layer in the target generation model is used to convolve the second fusion result to further extract and optimize text features. The convolutional features, i.e., the text features, contain the key text information required to generate the media file and can be used to guide subsequent generation steps, thereby improving the lighting and shadow effects of the rendered text and background in the output media file, as well as the aesthetics of the spatial layout.

[0109] This embodiment fuses the control signal into the feature space of the target generation model and performs convolution processing to obtain text features, which can significantly improve the accuracy, quality and controllability of the generated media files, achieve a deep combination of semantics and vision, and solve the inconsistency between text rendering and the lighting effects and spatial layout of the background.

[0110] The following further describes the method of using the generative model to fuse image features and text features to obtain the media file of the object in this embodiment.

[0111] As an optional implementation, step S206 uses a generative model to fuse image features and text features to obtain a media file of the object, including: fusing image features and text features into the feature space of the corresponding network layer in the generative model to obtain a media file.

[0112] In this embodiment, after obtaining the image features output by the image processing model and the text features output by the text processing model, the image features and the text features can be fused into the feature space of the corresponding network layer in the generation model to obtain a media file.

[0113] Optionally, after obtaining the image features and text features, a corresponding network layer in the generation model can be selected as the fusion point to fuse the extracted image features and text features. This fusion can be achieved in various ways, including but not limited to concatenation, addition, attention mechanism, etc. This is only an example here and does not specifically limit the fusion method. The purpose of the fusion can be to enable the generation model to simultaneously learn and understand the association between the image and the text, so as to generate content that contains both image information and conforms to the text description.

[0114] Optionally, the feature space can be a multi-dimensional vector space representing features within the generation model. In the feature space, each vector represents a certain feature of the input data. Fusing the image features and text features into the feature space indicates that the generation model can analyze the image features and text features in this high-dimensional space to achieve more complex and creative synthesis tasks.

[0115] Optionally, after fusing the image features and text features in the feature space, the generation model can use the fused features for the next generation process, that is, generate the final output, namely the media file, based on the fused features.

[0116] In this embodiment, by fusing the image features and text features in the feature space, the generation model can understand and comprehensively process visual elements and semantic information, and the text and images in the generated media file can be naturally integrated, avoiding the disconnection between text and image elements, and improving the consistency and coordination of the overall visual effect.

[0117] The method of fusing the image features and text features into the feature space of the corresponding network layer in the generation model to obtain the media file in the above embodiment will be further introduced below.

[0118] As an optional implementation manner, fusing the image features and text features into the feature space of the corresponding network layer in the generation model to obtain the media file includes: performing an addition operation on the image features and text features to obtain an added feature; fusing the added feature into the feature space of the corresponding network layer in the generation model to obtain the media file, where the rendering result in the media file includes the rendering result of the object and the rendering result of the text description information.

[0119] In this embodiment, after obtaining the image features output by the image processing model and the text features output by the text processing model, an addition operation can be performed on the image features and text features to obtain an added feature. Further, the obtained added feature can be fused into the feature space of the corresponding network layer in the generation model to obtain the media file. Wherein, the rendering result in the media file can include the rendering result of the object and the rendering result of the text description information.

[0120] Optionally, after obtaining the image features output by the image processing model and the text features output by the text processing model, the image features and the text features can be added layer by layer and then input into the corresponding network layer of the base model, so as to obtain the media file generated by the base model.

[0121] In this embodiment, by adding the image features and the text features layer by layer and inputting them into the base model, it is possible to improve the controllability and personalization ability of the base model while maintaining high generation quality, providing support for media file generation tasks such as automated advertising poster design.

[0122] In the embodiment of the present application, a foreground image including at least one object is obtained, and at least one piece of background feature information matching the object and text description information to be rendered are obtained; the foreground image and the background feature information are input into an image processing model for analysis to obtain image features, and the text description information is input into a text processing model for analysis to obtain text features, where the image processing model and the text processing model are trained based on a generation model; the generation model is used to fuse the image features and the text features to obtain the media file of the object, where the media file at least includes a background image and a rendering result presented on the background image. That is to say, in the embodiment of the present application, the image processing model and the text processing model are trained based on the generation model, the image processing model is used to analyze the foreground image and the background feature information to obtain image features, and the text processing model is used to analyze the text description information to obtain text features, and then the generation model is used to directly fuse the image features and the text features to obtain the media file, avoiding generating the media file by relying on templates, resulting in a strong sense of templates, or relying on manual addition of text information to the background image, resulting in low efficiency of generating the media file, thereby achieving the technical effect of improving the generation efficiency of the media file and solving the technical problem of low generation efficiency of the media file.

[0123] The embodiment of the present application also provides a method for determining a model, Figure 3 which is a flowchart of a method for determining a model according to the embodiment of the present application, as Figure 3 shown, the method may include the following steps:

[0124] Step S302, obtain a foreground image sample including at least one object sample, and obtain at least one background feature information sample matching the object sample and a text description information sample.

[0125] In the technical solution provided in step S302 of the present application, a foreground image sample including at least one object sample can be obtained, and at least one background feature information sample matching the object sample and a text description information sample can be obtained.

[0126] Optionally, the foreground image samples can be obtained through an image library, online scraping, user upload, etc. The background feature information samples can be obtained through a preset background template, text description (for example, a merchant or user describes the required background features through text), etc. The text description information samples to be rendered can be obtained through user input, automatic generation, etc. It should be noted that the above are only examples and do not specifically limit the acquisition methods of the foreground image samples, background feature information samples, and text description information samples to be rendered.

[0127] Step S304: Use the foreground image samples, background feature information samples, and text description information samples to train the generation model to obtain an image processing model and a text processing model.

[0128] In the technical solution provided in step S304 of the present application, the image processing model is used to analyze a foreground image including at least one object and at least one background feature information matching the object to obtain image features. The text processing model is used to analyze the text description information to be rendered to obtain text features. The generation model is used to fuse the image features and text features to obtain a media file of the object. The media file includes at least a background image and a rendering result presented on the background image.

[0129] In this embodiment, after obtaining the foreground image samples including at least one object sample, at least one background feature information sample matching the object sample, and the text description information samples, the obtained foreground image samples, background feature information samples, and text description information samples can be used to train the generation model to obtain an image processing model and a text processing model.

[0130] For example, after training the image processing model, the obtained image processing model can be used to analyze the commodity foreground image and background text description information to obtain corresponding image features.

[0131] Optionally, after training the text processing model, the text strings included in the text description information can be subjected to feature extraction through a pre-trained visual encoder (character image encoder) to obtain character features. The text positions and character orders included in the text description information can be encoded respectively to obtain text position encoding and character order encoding. Further, the obtained character features, text position encoding, and character order encoding can be concatenated to obtain a character-level text representation, and the obtained character-level text representation is used as a control signal after passing through a projection layer. Then, the obtained text processing model is used to analyze the control signal to obtain text features.

[0132] Optionally, after obtaining the image features using the image processing model and the text features using the text processing model, the image features and the text features can be added layer by layer and then input into the corresponding network layer of the base model, so as to obtain the media file generated by the base model.

[0133] Through the above steps S302 to S304 of this application, a foreground image sample including at least one object sample is obtained, and at least one background feature information sample and a text description information sample that match the object sample are obtained; the generation model is trained using the foreground image sample, the background feature information sample, and the text description information sample to obtain an image processing model and a text processing model, where the image processing model is used to analyze a foreground image including at least one object and at least one background feature information that matches the object to obtain image features, the text processing model is used to analyze the text description information to be rendered to obtain text features, and the generation model is used to fuse the image features and the text features to obtain the media file of the object. The media file includes at least a background image and a rendering result presented on the background image. That is to say, in the embodiment of this application, the generation model is trained using the foreground image sample, the background feature information sample, and the text description information sample to obtain an image processing model and a text processing model. The image processing model is used to analyze the foreground image and the background feature information to obtain image features, and the text processing model is used to analyze the text description information to obtain text features. Then, the generation model is used to fuse the image features and the text features to obtain the media file, avoiding generating the media file relying on templates, resulting in a strong sense of templates, or relying on manual addition of text information to the background image, resulting in low efficiency of generating the media file. Furthermore, the technical effect of improving the generation efficiency of the media file is achieved, and the technical problem of low generation efficiency of the media file is solved.

[0134] The following further introduces the method of training the generation model using the foreground image sample, the background feature information sample, and the text description information sample to obtain the image processing model and the text processing model in the above embodiment.

[0135] As an optional implementation manner, step S304, training the generation model using the foreground image sample, the background feature information sample, and the text description information sample to obtain an image processing model and a text processing model includes: training the generation model using the foreground image sample to obtain an image processing model; training the generation model using the foreground image sample, the background feature information sample, and the text description information sample to obtain a text processing model.

[0136] In this embodiment, after obtaining a foreground image sample including at least one object sample, at least one background feature information sample matching the object sample, and a text description information sample, the obtained foreground image sample can be used to train a generation model to obtain an image processing model. The obtained foreground image sample, background feature information sample, and text description information sample can be used to train the generation model to obtain a text processing model.

[0137] Optionally, the training stage of the text processing model can be the first stage, and the training stage of the image processing model can be the second stage. That is, this embodiment proposes a two-stage training strategy, aiming to separate the learning of text rendering and background image generation.

[0138] In this embodiment, the image processing model is trained through the foreground image sample, so that the image processing model can more accurately capture and reconstruct object details, achieve high-fidelity generation of the object foreground, and thus make the generated media file more visually realistic and improve the effect of object display. The text processing model is trained through the foreground image sample, background feature information sample, and text description information sample, so that the text processing model can generate a matching background and layout according to the provided text information, realizing the intelligent fusion of text and images.

[0139] Next, the method of training the generation model with the foreground image sample, background feature information sample, and text description information sample to obtain a text processing model in this embodiment will be further introduced.

[0140] As an optional implementation manner, training the generation model with the foreground image sample, background feature information sample, and text description information sample to obtain a text processing model includes: determining a text mask of the text description information sample, and determining a mask image of the text description information sample based on the foreground image sample and the background feature information sample, where the text mask of the text description information sample is used to represent the position where the text description information sample is rendered, and the mask image of the text description information sample is used to represent the position of the text description information sample in the background image sample corresponding to the background feature information sample relative to the foreground image sample; adjusting the model parameters of the copied generation model based on the text mask of the text description information sample and the mask image of the text description information sample; using the adjusted model parameters to train the generation model with the structure of the control network as the network structure to obtain a text processing model.

[0141] In this embodiment, after obtaining a foreground image sample including at least one object sample, at least one background feature information sample matching the object sample, and a text description information sample, a text mask of the text description information sample can be determined, and a masked image of the text description information sample can be determined based on the foreground image sample and the background feature information sample. Further, based on the text mask and the masked image, the model parameters of the copied generation model are adjusted, and then, with the structure of the control network as the network structure, the generation model is trained using the adjusted model parameters to obtain a text processing model.

[0142] In this embodiment, the text mask of the text description information sample can be used to represent the position where the text description information sample is rendered. The masked image of the text description information sample can be used to represent the position of the text description information sample in the background image sample corresponding to the background feature information sample relative to the foreground image sample.

[0143] Optionally, the text mask can be used to indicate where the text should appear in the generated image. During the process of training the text processing model, determining the text mask of the text description information sample is to ensure that the text processing model learns how to place the text at the target position. The text mask is usually represented in the form of a binary image, where 1 represents the position where the text should be covered, and 0 represents the blank or the area not covered by the text. For example, if it is desired that the text appears in the center of the top of the media file, the text mask will be marked as 1 at this position and 0 at other parts.

[0144] Optionally, the masked image combines the foreground image and the image of the background feature information, and further marks the position where the text should appear through the text mask. In the generation model, the masked image, as part of the input, can enable the model to understand the spatial relationship between the text and other parts of the image.

[0145] Optionally, based on the text mask and the masked image, the model parameters of the copied generation model are adjusted. For example, the weights in the generation model are updated to obtain the adjusted model parameters, and then the generation model is trained using the adjusted model parameters so that the generation model learns to accurately render the text according to the given position information.

[0146] Optionally, the model parameters of the above-mentioned adjusted copied generation model can be implemented in a way that minimizes the error between the predicted text position and the actual expected position. For example, the generation model can be made to generate a prediction result that fuses text and images based on the input image and text information. The text position in this prediction result can be compared with the actual expected position defined by the text mask and the masked image, and the error between the two can be calculated. This error can be calculated using various metrics, such as mean squared error, cross-entropy loss, etc. This is only for illustration purposes here, and there is no specific limitation on the way to calculate the error. Further, to minimize the above error, that is, the error between the predicted text position and the actual expected position, the gradient of the model parameters (weights) can be calculated according to the backpropagation of the error. This gradient is used to indicate the direction of weight adjustment so that the generation model generates text closer to the actual expected position in the next iteration. By continuously updating the weights, the generation model gradually learns to accurately render text at the specified position without deviating from the actual expected position.

[0147] It should be noted that the above is only for illustration purposes, and there is no specific limitation on the way to adjust the model parameters of the copied generation model. That is, any way that enables the generation model to learn to accurately render text according to the given position information is within the scope of protection of this patent.

[0148] Optionally, a control network can be used to introduce additional signals to control the generation process and make the generation model more controllable. In the above process, the control network structure can be used to guide the generation model to generate text according to the instructions of the text mask and the masked image.

[0149] Optionally, before starting the training, the model parameters can be copied from a pre-trained base model. The model parameters of the copied generation model will then be adjusted, and through the above-mentioned mask-based training method, the model's perception and generation ability of text positions will be gradually optimized. This mask-based training ensures that the model accurately places the text at the specified position when generating images, while maintaining the clarity and readability of the text, and finally a text processing model is trained.

[0150] Based on the text mask and the masked image, this embodiment enables the generation model to accurately understand the position of the text in the final image and its layout relationship relative to the object foreground and background. Therefore, when generating a media file, the text can be placed at a preset precise position, avoiding conflicts between the text and image elements, and improving the professionalism and aesthetics of media file design.

[0151] Next, a further introduction will be made to the method of using the structure of the control network as the network structure in this embodiment, training the generation model with the adjusted model parameters, and obtaining the text processing model.

[0152] As an alternative implementation, with the structure of the control network as the network structure, the generation model is trained using the adjusted model parameters to obtain a text processing model, including: during the process of freezing the image processing model, with the structure of the control network as the network structure, the generation model is trained using the adjusted model parameters to obtain a text processing model.

[0153] In this embodiment, after adjusting the model parameters of the copied generation model based on the text mask of the text description information sample and the masked image of the text description information sample, during the process of freezing the image processing model, with the structure of the control network as the network structure, the generation model is trained using the adjusted model parameters to obtain a text processing model.

[0154] Optionally, the training stage of the text processing model can be the first stage, and the training task of the first stage can be local text editing. In the first stage, the background generation network can be frozen, and only the text rendering network is optimized. The network structure parameters pre-trained in the image editing task can be used to initialize the background generation network, so that the background generation network can well fill the local background, enabling the text rendering network to focus on learning text rendering.

[0155] This embodiment decomposes the training task and first focuses on the optimization of the text rendering network, which helps the generation model to finely learn and master text rendering techniques, including character shape, texture, position, and lighting effects, without being interfered by background generation, thus significantly improving the generation accuracy and aesthetics of the text on the graph.

[0156] Next, the method of training the generation model using the foreground image sample to obtain the image processing model in this embodiment is further introduced.

[0157] As an alternative implementation, using the foreground image sample to train the generation model to obtain an image processing model, including: determining the main mask of the foreground image sample and the masked image of the foreground image sample, where the main mask of the foreground image sample is used to represent the position where the foreground image sample is rendered, and the masked image of the foreground image sample is used to represent the features of the object sample; adjusting the model parameters of the copied generation model based on the main mask of the foreground image sample and the masked image of the foreground image sample; with the structure of the control network as the network structure, training the generation model using the adjusted model parameters to obtain an image processing model.

[0158] In this embodiment, after obtaining the foreground image sample, the main mask of the foreground image sample and the mask image of the foreground image sample can be determined. Further, based on the determined main mask and mask image, the model parameters of the copied generation model can be adjusted. Then, using the structure of the control network as the network structure and the adjusted model parameters to train the generation model, an image processing model can be obtained. Among them, the main mask of the foreground image sample can be used to represent the position where the foreground image sample is rendered. The mask image of the foreground image sample can be used to represent the features of the object sample.

[0159] Optionally, the main mask can be used to mark the position of the main object (such as a commodity) in the foreground image. In the main mask, the pixel points where the main object is located are marked as 1, and the remaining background part is marked as 0. The use of the main mask enables the model to determine which areas are foreground objects that need to be retained and which areas are backgrounds that can be freely generated during the generation process. The mask image not only contains the shape information of the object (through the main mask) but also carries information about the object's features, including texture, shape, etc. The mask image enables the model to understand the visual attributes of the object itself during the training process, so that these attributes can be more accurately reproduced when generating new images, maintaining visual consistency.

[0160] Optionally, based on the main mask and mask image of the foreground image sample, the parameters of the generation model are adjusted. Since the mask information provides additional guiding signals, such as which areas should generate similar textures and which areas should be regarded as backgrounds for free generation, the model parameters of the copied generation model can be adjusted based on the main mask and mask image of the foreground image sample.

[0161] Optionally, after adjusting the model parameters of the copied generation model, a network structure similar to that of the control network can be adopted. The control network is a neural network architecture that can receive additional control signals (such as the main mask and mask image), and the generation behavior of the model can be guided by introducing additional inputs. Training with the adjusted model parameters can obtain an image processing model.

[0162] In this embodiment, the use of the main mask enables the model to accurately identify and locate the main object in the foreground image, ensuring the integrity and position information of the object are retained during the generation process, avoiding interference or damage to the object by the background generation process, and thus maintaining the clarity and attractiveness of the object image in the media file.

[0163] Next, a further introduction is made to the method of using the structure of the control network as the network structure, adjusting the model parameters of the generation model, and obtaining an image processing model in the above embodiment.

[0164] As an alternative implementation, taking the structure of the control network as the network structure, training the generation model with the adjusted model parameters to obtain an image processing model, including: during the process of freezing the text processing model, taking the structure of the control network as the network structure, training the generation model with the adjusted model parameters to obtain an image processing model.

[0165] In this embodiment, after adjusting the model parameters of the copied generation model based on the body mask of the foreground image sample and the mask image of the foreground image sample, during the process of freezing the text processing model, taking the structure of the control network as the network structure, training the generation model with the adjusted model parameters to obtain an image processing model.

[0166] Optionally, the training stage of the image processing model can be the second stage, and the training task of the second stage is image generation from the text description of the object foreground. In the second stage, the text rendering network can be frozen, and only the background generation network is trained.

[0167] In this embodiment, by only training the background generation network in the second stage, the background generation network can focus more on understanding and generating background content that highly matches the text description, ensuring that the generated background image is accurately generated according to the environment, emotion, or style mentioned in the text description, thereby increasing the semantic consistency between the background image, the object foreground, and the text content, and improving the overall communication effect of the media file.

[0168] As an alternative implementation, the method further includes: obtaining a reward signal for the image processing model, where the reward signal is used to represent the feedback result of the image processing model; in response to the reward signal indicating a negative feedback result, adjusting the image processing model based on the reward signal.

[0169] In this embodiment, a reward signal for the image processing model can be obtained. When the obtained reward signal indicates a negative feedback result, the image processing model is adjusted based on the obtained reward signal. Wherein, the reward signal can be used to represent the feedback result of the image processing model.

[0170] Optionally, the reward signal can be used to represent the feedback of the environment on the model behavior (i.e., the output or generation result of the model). In the scenario of advertising poster design, the reward signal can be an index for evaluating the quality of the poster generated by the model, such as the clarity of the foreground product, the integration of text and background, and the overall aesthetic feeling of the poster. The reward signal can be positive feedback (indicating a good generation result) or negative feedback (indicating a poor generation result).

[0171] Optionally, when the reward signal represents a negative feedback result, the model adjusts its parameters according to this feedback signal to obtain better results in the next generation. This adjustment process can be achieved in various ways. For example, adjusting the weights of the generation model, modifying certain parameters of the generation algorithm, or introducing additional training data to enrich the model's learning experience, etc. This is only for illustrative purposes and does not specifically limit the ways to adjust the image processing model based on the reward signal.

[0172] Optionally, in order to suppress the problem of the model generating something long in the generation of the input subject, a reward learning module can be introduced as a supervision module for further optimization after the background is generated. At this stage, one can focus on learning the scenes and creative designs of media files from the training data.

[0173] This embodiment can apply the concept of reinforcement learning to model training by introducing a reward signal, enabling the model to adjust based on positive and negative feedback results and continuously optimize the quality of the generated images. When the reward signal represents a negative feedback result, the model can identify problems existing in the generation process, such as distortion of product features, poor coordination between the background and the foreground, etc., and adjust the parameters accordingly to improve the above problems.

[0174] In the embodiment of the present application, a foreground image sample including at least one object sample is obtained, and at least one background feature information sample and a text description information sample that match the object sample are obtained; the generation model is trained using the foreground image sample, the background feature information sample, and the text description information sample to obtain an image processing model and a text processing model. Among them, the image processing model is used to analyze a foreground image including at least one object and at least one background feature information that matches the object to obtain image features, the text processing model is used to analyze the text description information to be rendered to obtain text features, and the generation model is used to fuse the image features and the text features to obtain a media file of the object. The media file includes at least a background image and a rendering result presented on the background image. That is to say, the embodiment of the present application uses the foreground image sample, the background feature information sample, and the text description information sample to train the generation model to obtain an image processing model and a text processing model, uses the image processing model to analyze the foreground image and the background feature information to obtain image features, and uses the text processing model to analyze the text description information to obtain text features, and then uses the generation model to fuse the image features and the text features to obtain a media file, avoiding generating media files relying on templates, resulting in a strong sense of templates, or relying on manual addition of text information to the background image, resulting in low efficiency in generating media files, and thus achieving the technical effect of improving the generation efficiency of media files and solving the technical problem of low generation efficiency of media files.

[0175] The embodiment of the present application also provides a method for generating a media file.Figure 4 is a flowchart of another method for generating a media file according to an embodiment of the present application. As Figure 4 shown, the method may include the following steps:

[0176] Step S402: Obtain a foreground image including at least one object from an e-commerce platform, obtain at least one piece of background feature information matching the object, and obtain text description information to be rendered.

[0177] In the technical solution provided in step S402 of the present application, a foreground image including at least one object can be obtained from an e-commerce platform, at least one piece of background feature information matching the object can be obtained, and text description information to be rendered can be obtained.

[0178] Optionally, the foreground image can be obtained through an image library, online scraping, user upload, etc. The background feature information can be obtained through a preset background template, text description (for example, a merchant or user describes the required background features through text), etc. The text description information to be rendered can be obtained through user input, automatic generation, etc. It should be noted that the above are only examples and do not specifically limit the acquisition methods of the foreground image, background feature information, and text description information to be rendered.

[0179] Step S404: Input the foreground image and the background feature information into an image processing model for analysis to obtain image features, and input the text description information into a text processing model for analysis to obtain text features.

[0180] In the technical solution provided in step S404 of the present application, the image processing model and the text processing model are trained based on a generative model.

[0181] In this embodiment, after obtaining a foreground image including at least one object from an e-commerce platform, obtaining at least one piece of background feature information matching the object, and obtaining text description information to be rendered, the obtained foreground image and background feature information can be input into an image processing model for analysis to obtain image features, and the text description information can be input into a text processing model for analysis to obtain text features.

[0182] For example, in the design of an advertising poster, a product foreground image and background text description information can be input into a background generation network model for analysis to obtain corresponding image features.

[0183] Optionally, during the process of inputting the literal description information into the text processing model for analysis, the text strings included in the literal description information can be subjected to feature extraction through a pre-trained visual encoder (which can also be referred to as a character image encoder) to obtain character features. The text positions and character orders included in the literal description information can be encoded respectively to obtain text position encoding and character order encoding. Further, the obtained character features, text position encoding, and character order encoding can be concatenated to obtain a character-level text representation, and then the obtained character-level text representation can be used as a control signal to be input into the text rendering network model after passing through a projection layer, so as to achieve precise generation control of the text content and position, thereby obtaining the corresponding text features.

[0184] Step S406: Use a generation model to fuse the image features and text features to obtain the media file of the object.

[0185] In the technical solution provided in step S406 of the present application above, the media file at least includes a background image and a rendering result presented on the background image.

[0186] In this embodiment, after inputting the foreground image and background feature information into the image processing model for analysis to obtain image features, and inputting the literal description information into the text processing model for analysis to obtain text features, a generation model can be used to fuse the image features and text features to obtain the media file of the object.

[0187] Optionally, after obtaining the image features output by the image processing model and the text features output by the text processing model, the image features and text features can be added layer by layer and then input into the corresponding network layer of the basic model, and then the media file generated by the basic model can be obtained.

[0188] Step S408: Return the media file to the e-commerce platform for display.

[0189] In the technical solution provided in step S408 of the present application above, after using a generation model to fuse the image features and text features to obtain the media file of the object, the media file can be returned to the e-commerce platform for display.

[0190] Through the above steps S402 to S408 of this application, a foreground image including at least one object is obtained from an e-commerce platform, and at least one background feature information matching the object and the text description information to be rendered are obtained; the foreground image and the background feature information are input into an image processing model for analysis to obtain image features, and the text description information is input into a text processing model for analysis to obtain text features, where the image processing model and the text processing model are trained based on a generation model; the generation model is used to fuse the image features and the text features to obtain a media file of the object, where the media file at least includes a background image and a rendering result presented on the background image; the media file is returned to the e-commerce platform for display. That is to say, in the embodiment of this application, the image processing model and the text processing model are trained based on the generation model, the image processing model is used to analyze the foreground image and the background feature information to obtain image features, and the text processing model is used to analyze the text description information to obtain text features. Furthermore, the generation model is used to fuse the image features and the text features to directly obtain a media file, and finally the media file is returned to the e-commerce platform for display, avoiding relying on templates to generate media files, resulting in a strong sense of templates, or relying on manual addition of text information to the background image, resulting in low efficiency of generating media files. Furthermore, the technical effect of improving the generation efficiency of media files is achieved, and the technical problem of low generation efficiency of media files is solved.

[0191] The embodiment of this application also provides a method for generating a media file. Figure 5 It is a flowchart of another method for generating a media file according to the embodiment of this application, as Figure 5 shown. The method may include the following steps:

[0192] Step S502, in response to an input operation on the operation interface, a foreground image including at least one object is displayed on the operation interface, and at least one background feature information matching the object and the text description information to be rendered are displayed.

[0193] In the technical solution provided in step S502 of this application above, in response to an input operation on the operation interface, a foreground image including at least one object can be displayed on the operation interface, and at least one background feature information matching the object and the text description information to be rendered can be displayed. Among them, the operation interface can be an operation interface displayed on a client device. For example, it can be an interface for interaction between a user and a computer or software system, and can be a graphical user interface, a command line interface, etc. This is only for illustration and does not specifically limit the form of the operation interface.

[0194] In this embodiment, the input operation may be an operation of controlling the operation interface to display a foreground image including at least one object, and display at least one piece of background feature information matching the object, as well as the text description information to be rendered. For example, the user may input text through a keyboard, or click a button or option on the operation interface through a mouse, or drag the above information to a specified area to perform the input operation.

[0195] In an alternative embodiment, the user may open the operation interface deployed on the client device and input a foreground image including at least one object, at least one piece of background feature information matching the object, and the text description information to be rendered on the operation interface, and the input operation may be performed. In response to the input operation on the operation interface, a foreground image including at least one object may be displayed on the operation interface, and at least one piece of background feature information matching the object, as well as the text description information to be rendered, may be displayed.

[0196] Step S504, in response to the file generation operation on the operation interface, display the media file of the object on the operation interface.

[0197] In the technical solution provided in step S504 of the present application above, the media file is obtained by fusing the image features and text features based on the generation model, and the media file at least includes a background image and the rendering result presented on the background image. The image features are obtained by the image processing model analyzing the foreground image and the background feature information, the text features are obtained by the text processing model analyzing the text description information, and the image processing model and the text processing model are trained based on the generation model.

[0198] In this embodiment, after displaying a foreground image including at least one object, and at least one piece of background feature information matching the object, as well as the text description information to be rendered on the operation interface, in response to the file generation operation on the operation interface, the media file of the object may be displayed on the operation interface. Among them, the file generation operation may be an operation of generating a media file based on the foreground image including at least one object, at least one piece of background feature information matching the object, and the text description information to be rendered displayed on the operation interface. The user may perform the file generation operation by clicking on the above information displayed on the operation interface. This is only an example here and does not specifically limit the file generation operation.

[0199] Optionally, after displaying a foreground image including at least one object, at least one background feature information matching the object, and text description information to be rendered on the operation interface, in response to a file generation operation on the operation interface, the text string included in the text description information can be subjected to feature extraction through a pre-trained visual encoder (which can also be called a character image encoder) to obtain character features. The text position and character order included in the text description information can be encoded respectively to obtain a text position encoding and a character order encoding. Further, the obtained character features, text position encoding, and character order encoding can be concatenated to obtain a character-level text representation, and then the obtained character-level text representation can be used as a control signal to be input into a text rendering network model after passing through a projection layer, so as to realize precise generation control of the text content and position, thereby obtaining corresponding text features.

[0200] For example, in the design of an advertising poster, the foreground image of the product and the background text description information can be input into a background generation network model for analysis to obtain corresponding image features.

[0201] Optionally, after obtaining the image features output by the image processing model and the text features output by the text processing model, the image features and text features can be added layer by layer and then input into the corresponding network layer of the basic model, and then the media file generated by the basic model can be obtained.

[0202] Through the above steps S502 to S504 of the present application, in response to an input operation on the operation interface, a foreground image including at least one object is displayed on the operation interface, and at least one background feature information matching the object and text description information to be rendered are displayed; in response to a file generation operation on the operation interface, a media file of the object is displayed on the operation interface, where the media file is obtained by fusing image features and text features based on a generation model, and the media file at least includes a background image and a rendering result presented on the background image. The image features are obtained by an image processing model analyzing the foreground image and the background feature information, and the text features are obtained by a text processing model analyzing the text description information. The image processing model and the text processing model are trained based on the generation model. That is to say, in the embodiment of the present application, in response to a file generation operation on the operation interface, a media file of the object is displayed on the operation interface. The media file can be obtained by fusing image features and text features using a generation model. The image features can be obtained by an image processing model analyzing the foreground image and the background feature information, and the text features can be obtained by using a text processing model analyzing the text description information. This avoids relying on templates to generate media files, resulting in a strong sense of templates, or relying on manual addition of text information to the background image, resulting in low efficiency in generating media files. Furthermore, the technical effect of improving the generation efficiency of media files is achieved, and the technical problem of low generation efficiency of media files is solved.

[0203] The embodiment of the present application also provides a method for generating a media file. Figure 6 It is a flowchart of another method for generating a media file according to the embodiment of the present application, as Figure 6 shown. The method may include the following steps:

[0204] Step S602, by calling a first interface, obtain a foreground image including at least one object, and obtain at least one background feature information matching the object and text description information to be rendered.

[0205] In the technical solution provided in step S602 of the present application above, the first interface includes a first parameter, and the parameter value of the first parameter includes the foreground image, the background feature information, and the text description information.

[0206] In this embodiment, a foreground image including at least one object can be obtained by calling the first interface, and at least one background feature information matching the object and text description information to be rendered can be obtained. Among them, the first interface may include a first parameter, and the parameter value of the first parameter may include the foreground image, the background feature information, and the text description information.

[0207] Optionally, the foreground image can be obtained through an image library (such as those taken by professional photographers), online scraping, user uploads, etc. The background feature information can be obtained through preset background templates, text descriptions (such as merchants or users describing the required background features through text). The text description information to be rendered can be obtained through user input, automated generation, etc. It should be noted that the above are only examples and do not specifically limit the acquisition methods of the foreground image, background feature information, and text description information to be rendered.

[0208] Step S604: Input the foreground image and background feature information into an image processing model for analysis to obtain image features, and input the text description information into a text processing model for analysis to obtain text features.

[0209] In the technical solution provided in step S604 of the present application, the image processing model and the text processing model are trained based on a generative model.

[0210] In this embodiment, after obtaining the foreground image including at least one object, at least one background feature information matching the object, and the text description information to be rendered by calling the first interface, the obtained foreground image and background feature information can be input into an image processing model for analysis to obtain image features, and the text description information can be input into a text processing model for analysis to obtain text features.

[0211] For example, in the design of an advertising poster, the product foreground image and background text description information can be input into a background generation network model for analysis to obtain corresponding image features.

[0212] Optionally, during the process of inputting the text description information into the text processing model for analysis, the text strings included in the text description information can be subjected to feature extraction through a pre-trained visual encoder (which can also be called a character image encoder) to obtain character features. The text positions and character orders included in the text description information can be encoded respectively to obtain text position encoding and character order encoding. Further, the obtained character features, text position encoding, and character order encoding can be concatenated to obtain a character-level text representation, and then the obtained character-level text representation can be used as a control signal to be input into a text rendering network model after passing through a projection layer for realizing precise generation control of the text content and position, thereby obtaining corresponding text features.

[0213] Step S606: Use the generative model to fuse the image features and text features to obtain the media file of the object.

[0214] In the technical solution provided in step S606 of the present application, the media file at least includes a background image and a rendering result presented on the background image.

[0215] In this embodiment, after inputting the foreground image and background feature information into the image processing model for analysis to obtain image features, and inputting the caption information into the text processing model for analysis to obtain text features, a generation model can be used to fuse the image features and text features to obtain the media file of the object.

[0216] Optionally, after obtaining the image features output by the image processing model and the text features output by the text processing model, the image features and text features can be added layer by layer and then input into the corresponding network layer of the basic model, so as to obtain the media file generated by the basic model.

[0217] Step S608, output the media file by calling the second interface.

[0218] In the technical solution provided in step S608 of the present application above, the second interface includes a second parameter, and the parameter value of the second parameter includes the media file.

[0219] In this embodiment, after using the generation model to fuse the image features and text features to obtain the media file of the object, the media file can be output by calling the second interface. Among them, the second interface may include a second parameter, and the parameter value of the second parameter may include the media file.

[0220] Through the above steps S602 to S608 of this application, by invoking the first interface, a foreground image including at least one object is obtained, and at least one background feature information matching the object and the text description information to be rendered are obtained. The first interface includes a first parameter, and the parameter value of the first parameter includes the foreground image, the background feature information, and the text description information. The foreground image and the background feature information are input into an image processing model for analysis to obtain image features, and the text description information is input into a text processing model for analysis to obtain text features. The image processing model and the text processing model are trained based on a generative model. Using the generative model, the image features and the text features are fused to obtain a media file of the object. The media file at least includes a background image and a rendering result presented on the background image. The media file is output by invoking a second interface. The second interface includes a second parameter, and the parameter value of the second parameter includes the media file. That is to say, the embodiments of this application train an image processing model and a text processing model based on a generative model, use the image processing model to analyze the foreground image and the background feature information to obtain image features, and use the text processing model to analyze the text description information to obtain text features. Then, the generative model is used to directly fuse the image features and the text features to obtain a media file, avoiding generating a media file relying on a template, resulting in a strong sense of template, or relying on manual addition of text information to the background image, resulting in low efficiency of generating a media file. Furthermore, the technical effect of improving the generation efficiency of the media file is achieved, and the technical problem of low generation efficiency of the media file is solved.

[0221] The embodiments of this application also provide a media file generation system. Figure 7 It is a schematic diagram of a media file generation system according to an embodiment of this application, as Figure 7 shown. The media file generation system 700 may include:

[0222] A client 702, configured to transmit a foreground image including at least one object, and obtain at least one background feature information matching the object and the text description information to be rendered.

[0223] A server 704, configured to input the foreground image and the background feature information into an image processing model for analysis to obtain image features, and input the text description information into a text processing model for analysis to obtain text features. The image processing model and the text processing model are trained based on a generative model. Based on the image features and the text features, generate a media file of the object. The media file at least includes a background image and a rendering result presented on the background image. Return the media file to the client.

[0224] The media file generation system 700 transmits a foreground image including at least one object through the client 702, obtains at least one piece of background feature information matching the object, and text description information to be rendered. The foreground image and the background feature information are input into an image processing model through the server 704 for analysis to obtain image features, and the text description information is input into a text processing model for analysis to obtain text features, where the image processing model and the text processing model are trained based on a generation model; based on the image features and the text features, a media file of the object is generated, where the media file at least includes a background image and a rendering result presented on the background image; the media file is returned to the client. That is to say, the media file generation system trains an image processing model and a text processing model based on a generation model, uses the image processing model to analyze the foreground image and the background feature information to obtain image features, and uses the text processing model to analyze the text description information to obtain text features, and then uses the generation model to directly fuse the image features and the text features to obtain a media file, avoiding generating a media file relying on a template, resulting in a strong sense of template, or relying on manual addition of text information to the background image, resulting in low efficiency of generating a media file, thereby achieving the technical effect of improving the generation efficiency of the media file and solving the technical problem of low generation efficiency of the media file.

[0225] The technical solutions of the embodiments of the present disclosure will be further introduced by way of preferred embodiments below.

[0226] At present, with the increasing competition in the e-commerce industry, the visual presentation effect of advertising posters directly affects the click-through conversion rate of products. However, the traditional poster design process has a relatively high professional threshold and requires a large amount of manpower for scene construction, graphic layout, and visual beautification, which poses a significant cost pressure on small and medium-sized merchants that need to update materials frequently. Although the Text-to-Image (T2I) technology provides new possibilities for automated poster generation, the existing implementation still has the following problems.

[0227] In an alternative example, when generating a poster, usually, the background is first generated according to the product image and text description through a text-to-image algorithm, and then text elements are mechanically superimposed through a rule engine. Although this method ensures the diversity of the background, the text rendering based on a fixed template leads to a single style, and the light and shadow effects and spatial layouts of the text and the background often show inconsistent phenomena, and the overall aesthetic degree cannot reach the professional design level.

[0228] In another alternative example, an end-to-end text-to-image model can be used to directly output a full-element poster containing products, backgrounds, and text. However, the controllability of this method is low, resulting in problems such as distorted product features and deviation in promotional text content. Therefore, in the field of automated advertising poster generation technology, there is an urgent need to build a T2I technology system that can not only maintain the accuracy of generated elements but also achieve intelligent coordination between text and background.

[0229] In summary, the existing graphic and text creative production process is as follows: First, a professional photographer needs to take product pictures in a set-up scene. Then, a designer manually adds appropriate design elements such as text and backgrounds to the base image to form a complete graphic and text creation. However, this process relies on a large amount of manual input and has a certain professional threshold, which is too costly for small and medium-sized merchants. If merchants make the pictures themselves, the quality may be relatively low, affecting the placement effect, and they also need to spend a lot of time and energy costs. In addition, professional designers can batch-produce graphic and text templates for merchants to use. However, the produced graphic and text creative templates are too template-like, and the integration of products and backgrounds and the coordination with the copywriting cannot be guaranteed, and the quality of the produced creativity is also low.

[0230] Therefore, creating exquisite advertising graphic and text creativity has a certain threshold and requires merchants to invest a lot of energy. To help merchants quickly and low-costly produce high-quality creativity and improve the advertising placement effect, an artificial intelligence (AI) model can be used to directly generate graphic and text creativity using product foreground information and copywriting information. However, on the existing generated posters, the accuracy of Chinese text is relatively low, resulting in a low generation efficiency of media files.

[0231] To solve the above problems, this application proposes a text-to-image algorithm model for automated advertising poster design. This embodiment proposes a graphic and text fusion architecture with dual-network collaboration. By introducing a character-level vision and space fusion coding method, accurate text generation and high-quality graphic and text poster creativity generation can be achieved.

[0232] This embodiment proposes a text-to-image algorithm model for automated advertising poster design. The input of this model includes a product foreground image and the expected placement position, a background text description prompt, and the text content to be rendered and the corresponding position. Based on the above input information, this model can directly generate a graphic and text advertising poster.

[0233] Figure 8 is a schematic diagram of a text-to-image algorithm model framework for automated advertising poster design according to an embodiment of the present application, as Figure 8As shown, this embodiment is developed based on a basic model, which includes a VAE for reconstructing image details, such as text strokes, etc. On this basis, the algorithm proposes two modules, namely the text rendering network and the background generation network, to solve the poster generation task. Among them, the text rendering network is designed specifically for learning visual text rendering, taking character-level visual text features as input to achieve precise and controllable text rendering. The background generation network receives a foreground image of a commodity and a text description, and learns to generate the commodity foreground into the scene corresponding to the text description. The text rendering network and the background generation network are based on a control network-like architecture, including several cascaded image generation framework modules based on the diffusion model, and the weights are copied from the basic model during initialization. The output of each image generation framework module based on the diffusion model is added to the corresponding module of the basic model after passing through a zero convolutional layer.

[0234] Optionally, the difference between the text rendering network and the background generation network lies in the input. Among them, the background generation network receives the text description and the foreground image of the commodity as input in the text conditional branch to retain the foreground area and generate the corresponding background according to the text description. The text rendering network receives the character-level visual features of the text in the text conditional branch for controllable text rendering.

[0235] In the embodiment of this application, a plug-and-play and robust character-level text representation method is proposed, in which each character is accurately represented by a token. Optionally, the text string to be rendered has n characters. For each character, features are separately extracted through a pre-trained visual encoder (character image encoder) to obtain character features. On this basis, character order encoding (encoded from the character order, such as Figure 8 the character order 1, 2, 3, 8 shown) and text box position encoding (encoded from the text position) are further introduced. The character order encoding and the text box position encoding are concatenated with the character features, and then used as a control signal to be input into the text rendering network after passing through a projection layer, so as to achieve precise generation control of the text content and position.

[0236] Optionally, as Figure 8 shown, Z t can be used to represent the noisy input at time t. can be used to represent the denoising loss function used during the training of the diffusion model. This denoising loss function can have different forms and definitions, all within the scope of this patent. One definition method is: Denote the noise added at the t-th step of the denoising process during training as ∈, and the prediction of the model as ∈ t , then the loss function is The text description received by the background generation network can be "The picture shows a pile of round cookies with a spiral pattern on top. In the background, there is a soft warm - color gradient. There is also Chinese text on top, indicating that the cookies are very nutritious". The first training stage (masked image, text mask) and the second training stage (masked image, object mask) are directly used as the input of the background generation network.

[0237] To effectively train the above - mentioned model, this embodiment proposes a two - stage training strategy, aiming to separate the learning of text rendering and background image generation. The training task in the first stage is local text editing. The background generation network can be frozen, and only the text rendering network is optimized. Since the network structure parameters pre - trained in image editing tasks are used to initialize the background generation network, it can well fill the local background, enabling the text rendering network to focus on learning text rendering. The training task in the second stage is image generation from the text description based on the commodity foreground. The text rendering network can be frozen, and the background generation network is further trained. At the same time, to suppress the problem that the model generates something extra related to the input subject, a reward learning module is introduced as a supervision module to further optimize the generated background. In this stage, the model focuses on learning poster scenes and creative designs from the training data. Through the above two - stage training strategy, the model is effectively optimized.

[0238] This embodiment can directly generate corresponding graphic and text creativity according to the input commodity foreground, copywriting content, and background description information, without the need for merchants to take real - scene commodity photos or manually determine the text, reducing the cost of graphic and text creativity production. In addition, a graphic - text fusion architecture of dual - network collaboration is proposed, that is, a dual - branch architecture of the text rendering network and the background generation network, which is collaboratively controlled based on the diffusion model. The dual - network is based on the control network structure, shares the basic model weights but has separate inputs, and can achieve end - to - end graphic and text creativity generation.

[0239] This embodiment proposes a character - level visual and spatial joint encoding method, a plug - and - play character - level text representation system, which can use the joint representation of visual information, character - order encoding, and text - box position encoding to achieve accurate text generation on the graph. Further, a two - stage decoupled training paradigm is also proposed to improve the convergence effect of multi - task learning through a phased parameter optimization strategy. Finally, considering the problem of commodity subject distortion, a reward learning module is introduced in the background generation stage, and a foreground retention degree evaluation model is constructed as the reward function of reinforcement learning to solve the distortion caused by something growing around the commodity.

[0240] In an embodiment of the present application, an image processing model and a text processing model are trained based on a generation model. The image processing model is used to analyze a foreground image and background feature information to obtain image features, and the text processing model is used to analyze caption information to obtain text features. Then, the generation model is used to fuse the image features and text features directly to obtain a media file, avoiding generating a media file relying on a template, which results in a strong sense of template, or relying on manual addition of text information to a background image, which leads to low efficiency in generating a media file. Thus, the technical effect of improving the generation efficiency of the media file is achieved, and the technical problem of low generation efficiency of the media file is solved.

[0241] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or fully authorized by all parties. And the collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards of relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or reject.

[0242] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0243] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc), including several instructions for causing a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present application.

[0244] According to an embodiment of the present application, there is also provided a media file generation device for implementing the above Figure 2 shown media file generation method.

[0245] Figure 9 is a schematic diagram of a media file generation device according to an embodiment of the present application, as Figure 9As shown, the generating device 900 of the media file may include: a first obtaining unit 902, a first analyzing unit 904, and a first fusing unit 906.

[0246] The first obtaining unit 902 is configured to obtain a foreground image including at least one object, obtain at least one piece of background feature information matching the object, and text description information to be rendered.

[0247] The first analyzing unit 904 is configured to input the foreground image and the background feature information into an image processing model for analysis to obtain image features, and input the text description information into a text processing model for analysis to obtain text features, where the image processing model and the text processing model are trained based on a generating model.

[0248] The first fusing unit 906 is configured to fuse the image features and the text features by using the generating model to obtain a media file of the object, where the media file includes at least a background image and a rendering result presented on the background image.

[0249] It should be noted here that the above-mentioned first obtaining unit 902, first analyzing unit 904, and first fusing unit 906 correspond to steps S202 to S206 in the above embodiment. The instances and application scenarios implemented by the three modules and the corresponding steps are the same, but are not limited to the content disclosed in the above embodiment. It should be noted that the above-mentioned module or unit may be a hardware component or a software component stored in a memory and processed by one or more processors. The above-mentioned module may also be a part of the device and may run in the server 10 provided in the above embodiment.

[0250] In the generating device of the media file, a foreground image including at least one object is obtained by a first obtaining unit 902, and at least one piece of background feature information matching the object and the text description information to be rendered are obtained. The foreground image and the background feature information are input into an image processing model by a first analyzing unit 904 for analysis to obtain image features, and the text description information is input into a text processing model for analysis to obtain text features, where the image processing model and the text processing model are trained based on a generating model. A generating model is used by a first fusing unit 906 to fuse the image features and the text features to obtain the media file of the object, where the media file at least includes a background image and a rendering result presented on the background image. That is to say, the generating device of the media file trains the image processing model and the text processing model based on the generating model, uses the image processing model to analyze the foreground image and the background feature information to obtain image features, and uses the text processing model to analyze the text description information to obtain text features, and then directly obtains the media file by using the generating model to fuse the image features and the text features, avoiding generating the media file relying on a template, resulting in a strong sense of template, or relying on manual addition of text information to the background image, resulting in low efficiency of generating the media file, and thus achieving the technical effect of improving the generating efficiency of the media file and solving the technical problem of low generating efficiency of the media file.

[0251] According to an embodiment of the present application, there is also provided a model determining device for implementing the Figure 3 model determining method shown above.

[0252] Figure 10 is a schematic diagram of a model determining device according to an embodiment of the present application, as Figure 10 shown, the model determining device 1000 may include: a second obtaining unit 1002 and a training unit 1004.

[0253] The second obtaining unit 1002 is configured to obtain a foreground image sample including at least one object sample, and at least one background feature information sample matching the object sample and a text description information sample.

[0254] The training unit 1004 is configured to use the foreground image sample, the background feature information sample and the text description information sample to train a generating model to obtain an image processing model and a text processing model, where the image processing model is used to analyze a foreground image including at least one object and at least one piece of background feature information matching the object to obtain image features, the text processing model is used to analyze the text description information to be rendered to obtain text features, and the generating model is used to fuse the image features and the text features to obtain the media file of the object, and the media file at least includes a background image and a rendering result presented on the background image.

[0255] It should be noted here that the above-mentioned second acquisition unit 1002 and training unit 1004 correspond to steps S302 to S304 in the above-mentioned embodiments. The instances and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the content disclosed in the above-mentioned embodiments. It should be noted that the above-mentioned module or unit may be a hardware component or a software component stored in a memory and processed by one or more processors. The above-mentioned module may also be a part of the device and may run in the server 10 provided in the above-mentioned embodiments.

[0256] In the device for determining the model, the second acquisition unit 1002 acquires a foreground image sample including at least one object sample, and acquires at least one background feature information sample and a text description information sample that match the object sample. The training unit 1004 uses the foreground image sample, the background feature information sample, and the text description information sample to train the generation model, and obtains an image processing model and a text processing model. Among them, the image processing model is used to analyze a foreground image including at least one object and at least one background feature information that matches the object to obtain image features, and the text processing model is used to analyze the text description information to be rendered to obtain text features. The generation model is used to fuse the image features and the text features to obtain a media file of the object. The media file includes at least a background image and a rendering result presented on the background image. That is to say, the device for determining the model uses the foreground image sample, the background feature information sample, and the text description information sample to train the generation model to obtain an image processing model and a text processing model, uses the image processing model to analyze the foreground image and the background feature information to obtain image features, and uses the text processing model to analyze the text description information to obtain text features. Furthermore, the generation model is used to fuse the image features and the text features to obtain a media file, avoiding generating a media file relying on a template, resulting in a strong sense of template, or relying on manual addition of text information to the background image, resulting in low efficiency of generating the media file. Furthermore, the technical effect of improving the generation efficiency of the media file is achieved, and the technical problem of low generation efficiency of the media file is solved.

[0257] According to an embodiment of the present application, there is also provided a Figure 4 media file generation device for implementing the media file generation method shown above.

[0258] Figure 11 is a schematic diagram of another media file generation device according to an embodiment of the present application. As Figure 11 shown, the media file generation device 1100 may include: a third acquisition unit 1102, a second analysis unit 1104, a second fusion unit 1106, and a first display unit 1108.

[0259] A third acquisition unit 1102, configured to acquire a foreground image including at least one object from an e-commerce platform, and acquire at least one piece of background feature information matching the object, as well as text description information to be rendered.

[0260] A second analysis unit 1104, configured to input the foreground image and the background feature information into an image processing model for analysis to obtain image features, and input the text description information into a text processing model for analysis to obtain text features, where the image processing model and the text processing model are trained based on a generation model.

[0261] A second fusion unit 1106, configured to use the generation model to fuse the image features and the text features to obtain a media file of the object, where the media file at least includes a background image and a rendering result presented on the background image.

[0262] A first display unit 1108, configured to return the media file to the e-commerce platform for display.

[0263] It should be noted here that the above-mentioned third acquisition unit 1102, second analysis unit 1104, second fusion unit 1106, and first display unit 1108 correspond to steps S402 to S408 in the above-mentioned embodiment. The implementation examples and application scenarios of the four modules and the corresponding steps are the same, but are not limited to the content disclosed in the above-mentioned embodiment. It should be noted that the above-mentioned module or unit may be a hardware component or a software component stored in a memory and processed by one or more processors, and the above-mentioned module may also be a part of the device and can run in the server 10 provided in the above-mentioned embodiment.

[0264] In the generating device of the media file, a third obtaining unit 1102 is configured to obtain, from an e-commerce platform, a foreground image including at least one object, at least one piece of background feature information matching the object, and text description information to be rendered. The second analysis unit 1104 inputs the foreground image and the background feature information into an image processing model for analysis to obtain image features, and inputs the text description information into a text processing model for analysis to obtain text features, where the image processing model and the text processing model are trained based on a generating model. The second fusion unit 1106 uses the generating model to fuse the image features and the text features to obtain a media file of the object, where the media file at least includes a background image and a rendering result presented on the background image. The first display unit 1108 returns the media file to the e-commerce platform for display. That is to say, the generating device of the media file trains the image processing model and the text processing model based on the generating model, uses the image processing model to analyze the foreground image and the background feature information to obtain image features, and uses the text processing model to analyze the text description information to obtain text features, and then uses the generating model to directly fuse the image features and the text features to obtain the media file, and finally returns the media file to the e-commerce platform for display, avoiding generating the media file relying on a template, resulting in a strong sense of template, or relying on manual addition of text information to the background image, resulting in low efficiency of generating the media file, thereby achieving the technical effect of improving the generation efficiency of the media file and solving the technical problem of low generation efficiency of the media file.

[0265] According to an embodiment of the present application, there is also provided a generating device of a media file for implementing the Figure 5 generating method of the media file shown above.

[0266] Figure 12 is a schematic diagram of another generating device of a media file according to an embodiment of the present application, as Figure 12 shown, the generating device 1200 of the media file may include: a second display unit 1202 and a third display unit 1204.

[0267] The second display unit 1202 is configured to, in response to an input operation on the operation interface, display a foreground image including at least one object on the operation interface, and display at least one piece of background feature information matching the object, and text description information to be rendered.

[0268] A third display unit 1204, configured to respond to a file generation operation on the operation interface and display a media file of an object on the operation interface, where the media file is obtained by fusing an image feature and a text feature based on a generation model, and the media file at least includes a background image and a rendering result presented on the background image. The image feature is obtained by an image processing model analyzing a foreground image and background feature information, and the text feature is obtained by a text processing model analyzing caption information. The image processing model and the text processing model are trained based on the generation model.

[0269] It should be noted here that the above second display unit 1202 and third display unit 1204 correspond to steps S502 to S504 in the above embodiment. The examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the content disclosed in the above embodiment. It should be noted that the above module or unit may be a hardware component or a software component stored in a memory and processed by one or more processors, and the above module may also be part of a device and may run in the server 10 provided in the above embodiment.

[0270] In the media file generation device, the second display unit 1202 responds to an input operation on the operation interface, displays a foreground image including at least one object on the operation interface, and displays at least one piece of background feature information matching the object and caption information to be rendered. The third display unit 1204 responds to a file generation operation on the operation interface and displays a media file of an object on the operation interface, where the media file is obtained by fusing an image feature and a text feature based on a generation model, and the media file at least includes a background image and a rendering result presented on the background image. The image feature is obtained by an image processing model analyzing a foreground image and background feature information, and the text feature is obtained by a text processing model analyzing caption information. The image processing model and the text processing model are trained based on the generation model. That is to say, the media file generation device responds to a file generation operation on the operation interface and displays a media file of an object on the operation interface. The media file can be obtained by fusing an image feature and a text feature using a generation model. The image feature can be obtained by an image processing model analyzing a foreground image and background feature information, and the text feature can be obtained by using a text processing model to analyze caption information, avoiding generating a media file relying on a template, resulting in a strong sense of template, or relying on manual addition of text information to the background image, resulting in low efficiency of generating a media file. Furthermore, the technical effect of improving the generation efficiency of the media file is achieved, and the technical problem of low generation efficiency of the media file is solved.

[0271] According to an embodiment of the present application, there is also provided a method for implementing the above Figure 6A generating device for a media file of the generating method of the media file shown.

[0272] Figure 13 It is a schematic diagram of another generating device for a media file according to an embodiment of the present application. As Figure 13 shown, the generating device 1300 of the media file may include: a fourth obtaining unit 1302, a third analyzing unit 1304, a third fusing unit 1306, and an output unit 1308.

[0273] The fourth obtaining unit 1302 is configured to obtain a foreground image including at least one object, at least one piece of background feature information matching the object, and a text description information to be rendered by calling a first interface, where the first interface includes a first parameter, and the parameter value of the first parameter includes the foreground image, the background feature information, and the text description information.

[0274] The third analyzing unit 1304 is configured to input the foreground image and the background feature information into an image processing model for analysis to obtain image features, and input the text description information into a text processing model for analysis to obtain text features, where the image processing model and the text processing model are trained based on a generating model.

[0275] The third fusing unit 1306 is configured to fuse the image features and the text features by using the generating model to obtain a media file of the object, where the media file includes at least a background image and a rendering result presented on the background image.

[0276] The output unit 1308 is configured to output the media file by calling a second interface, where the second interface includes a second parameter, and the parameter value of the second parameter includes the media file.

[0277] It should be noted here that the above fourth obtaining unit 1302, third analyzing unit 1304, third fusing unit 1306, and output unit 1308 correspond to steps S602 to S608 in the above embodiment. The instances and application scenarios implemented by the four modules and the corresponding steps are the same, but are not limited to the content disclosed in the above embodiment. It should be noted that the above modules or units may be hardware components or software components stored in a memory and processed by one or more processors, and the above modules may also be part of a device and may run in the server 10 provided in the above embodiment.

[0278] In the generating device of the media file, a fourth obtaining unit 1302 obtains a foreground image including at least one object by invoking a first interface, and obtains at least one piece of background feature information matching the object and caption information to be rendered. The first interface includes a first parameter, and the parameter value of the first parameter includes the foreground image, the background feature information, and the caption information. A third analyzing unit 1304 inputs the foreground image and the background feature information into an image processing model for analysis to obtain image features, and inputs the caption information into a text processing model for analysis to obtain text features. The image processing model and the text processing model are trained based on a generating model. A third fusing unit 1306 uses the generating model to fuse the image features and the text features to obtain a media file of the object. The media file at least includes a background image and a rendering result presented on the background image. An output unit 1308 outputs the media file by invoking a second interface. The second interface includes a second parameter, and the parameter value of the second parameter includes the media file. That is to say, the generating device of the media file trains the image processing model and the text processing model based on the generating model, uses the image processing model to analyze the foreground image and the background feature information to obtain image features, and uses the text processing model to analyze the caption information to obtain text features, and then directly fuses the image features and the text features by using the generating model to obtain the media file, avoiding generating the media file depending on a template, resulting in a strong sense of template, or depending on manual addition of text information to the background image, resulting in low efficiency of generating the media file, thereby achieving the technical effect of improving the generation efficiency of the media file and solving the technical problem of low generation efficiency of the media file.

[0279] It should be noted that the preferred implementation schemes involved in the above embodiments of the present application are the same as the schemes, application scenarios, and implementation processes provided in the above embodiments, but are not limited to the schemes provided in the above embodiments.

[0280] An embodiment of the present application may provide a computing device. Figure 14 is a structural block diagram of a computing device according to an embodiment of the present application. As Figure 14 shown, the computing device 1400 may include: one or more (only one is shown in the figure) processors 1402, a memory 1404, a storage controller, and a peripheral interface.

[0281] The above computing device may be understood as an integrated intelligent terminal, including but not limited to a server, a desktop computer, a personal computer (abbreviated as a PC), a model all-in-one machine, etc. And the above model described in the above embodiments of the present application may be preset in the computing device.

[0282] Specifically, the computing device can be pre - installed with various types of models, including but not limited to models in the fields of natural language processing, visual processing, speech processing, code processing, multi - modal task processing, etc., so as to provide diverse model selection. In different product forms, the computing device can support one or more model usage methods, including but not limited to model training, model invocation, model fine - tuning, model deployment, model inference and application, etc. In some product forms, the computing device also supports model management, including but not limited to multi - type model management (supporting the management of various types of models such as discriminative and generative models), model version control (supporting the control of different model versions), model evaluation (evaluating the performance and effectiveness of models based on model evaluation tools), etc. In other product forms, the computing device can also create applications based on models, provide API invocation capabilities, and can call models into the created applications through API interfaces. At the same time, application management tools are provided to realize the control of applications.

[0283] Furthermore, the computing device can also include data management (supporting the creation and management of model - tuning data sets), a training center (providing rich training resources to help users learn and master AI technologies), and basic control capabilities (providing enterprise - level basic control capabilities to ensure the security and efficient operation of the system). Through the above functions, a comprehensive and integrated AI development, training, deployment, and application device is provided.

[0284] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and devices in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, to implement the methods in the above - mentioned embodiments. The memory can include high - speed random access memory, and can also include non - volatile memory, such as one or more magnetic storage devices, flash memory, or other non - volatile solid - state memories. In some instances, the memory can further include a memory remotely set relative to the processor, and these remote memories can be connected to terminal A through a network. Examples of the above - mentioned network include but are not limited to the Internet, enterprise intranets, local area networks, mobile communication networks, and their combinations.

[0285] The processor can call the executable program stored in the memory through the transmission device to execute the method described in any one of the above - mentioned embodiments.

[0286] Embodiments of the present application can provide an electronic device. Figure 15 It is a structural block diagram of an electronic device according to an embodiment of the present application, as Figure 15As shown, the electronic device may include: an input / output device 1502; a memory 1504; and a processor 1506. The processor 1506 is connected to the input / output device 1502 and the memory 1504 via a bus 1508.

[0287] The memory may be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatuses in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implements the methods in the above embodiments. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some instances, the memory may further include a memory remotely disposed relative to the processor, and these remote memories may be connected to the terminal A through a network. Examples of the above network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0288] The processor may call the executable program stored in the memory through a transmission device to execute the method described in any one of the above embodiments.

[0289] Those of ordinary skill in the art can understand that the structure shown in the figure is only illustrative, and the computing device may also be a terminal device such as a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, and a Mobile Internet Device (MID), a PAD, etc. The figure does not limit the structure of the above computing device. For example, the computing device 1000 may further include more or fewer components (such as a network interface, a display device, etc.) than those shown in the figure, or have a different configuration from that shown in the figure.

[0290] Those of ordinary skill in the art can understand that all or part of the steps in the various methods in the above embodiments can be completed by instructing the relevant hardware of the terminal device through a program, and the program can be stored in a computer-readable storage medium. The storage medium may include: a flash drive, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disc, etc.

[0291] The embodiments of the present application further provide a computer-readable storage medium. Optionally, in this embodiment, the above computer-readable storage medium may be used to store the program code executed by the method provided in the above embodiments.

[0292] Optionally, in this embodiment, the above storage medium may be located in the computing device.

[0293] Optionally, in this embodiment, the computer-readable storage medium is configured to store an executable program, and when the executable program runs, it controls the device where the computer-readable storage medium is located to execute the method described in any one of the above embodiments.

[0294] An embodiment of the present application also provides a computer program product. Optionally, in this embodiment, the above computer program product may include a computer program, and when the computer program is executed by a processor, it implements the method provided in the above embodiment.

[0295] An embodiment of the present application also provides a computer program product. Optionally, the above computer program product may include a non-volatile computer-readable storage medium, and the non-volatile computer-readable storage medium may be used to store a computer program, and when the computer program is executed by a processor, it implements the method provided in the above embodiment.

[0296] An embodiment of the present application also provides a computer program. Optionally, in this embodiment, when the above computer program is executed by a processor, it implements the method provided in the above embodiment.

[0297] In the above embodiments of the present application, the descriptions of each embodiment have their own emphases. For parts not described in detail in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0298] In the several embodiments provided by the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces, and the indirect coupling or communication connection of units or modules can be in an electrical or other form.

[0299] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place, or can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0300] In addition, the functional units in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0301] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage medium includes: various media such as USB flash drives, read-only memory ROM, random access memory RAM, mobile hard disks, magnetic disks, or optical discs that can store program codes.

[0302] The above are only the preferred embodiments of this application. It should be noted that for those of ordinary skill in the art, without departing from the principle of this application, several improvements and refinements can still be made, and these improvements and refinements should also be regarded as the protection scope of this application.

Claims

1. A method for generating a media file, characterized in that, Including: Obtain a foreground image including at least one object, obtain at least one piece of background feature information matching the object, and text description information to be rendered; Input the foreground image and the background feature information into an image processing model for analysis to obtain image features, and input the text description information into a text processing model for analysis to obtain text features, wherein the image processing model and the text processing model are trained based on a generative model; Use the generative model to fuse the image features and the text features to obtain a media file of the object, wherein the media file at least includes a background image and a rendering result presented on the background image.

2. The method according to claim 1, wherein Input the foreground image and the background feature information into an image processing model for analysis to obtain image features, and input the text description information into a text processing model for analysis to obtain text features, including: Input the foreground image, the background feature information, and object position information into the image processing model for analysis to obtain the image features, wherein the object position information is used to represent the position where the object is to be displayed on the background image; Input the text description content and text position information in the text description information into the text processing model for analysis to obtain the text features, wherein the text position information is used to represent the position where the text description information is to be rendered on the background image.

3. The method according to claim 2, wherein Input the foreground image, the background feature information, and object position information into the image processing model for analysis to obtain the image features, including: Input the foreground image, the background feature information, and object position information into the image processing model; Use the image processing model to determine a foreground area of the foreground image on the background according to the object position information, and use the background feature information as guiding information to guide the image processing model to generate a background image of the background; Use the image processing model to generate the image features from the foreground area, the background image, and the foreground image.

4. The method according to claim 3, characterized in that, Use the image processing model to generate the image features from the foreground area, the background image, and the foreground image, including: Use a control network in the image processing model to fuse the foreground area, the background image, and the foreground image into a feature space of a target generation model in the image processing model to obtain a first fusion result, wherein model parameters in an initialization stage of the target generation model in the image processing model are copied from the generative model; Use a convolutional layer corresponding to the target generation model in the image processing model to perform convolutional processing on the first fusion result to obtain the image features.

5. The method according to claim 2, wherein The text description information includes at least one text unit and sorting information of the text unit in the text description information. Input the text description content and text position information in the text description information into the text processing model for analysis to obtain the text features, including: Encode the text unit to obtain the text unit feature; Encode the sorting information to obtain the sorting encoding result; Encode the text position information to obtain the position encoding result; Input the position encoding result, the sorting encoding result, and the text unit feature into the text processing model for analysis to obtain the text feature.

6. The method according to claim 5, characterized in that, Input the position encoding result, the sorting encoding result, and the text unit feature into the text processing model for analysis to obtain the text feature, including: Concatenate the position encoding result, the sorting encoding result, and the text unit feature to obtain a concatenation result; Convert the concatenation result into a format matching the text processing model; Input the converted concatenation result into the text processing model for analysis to obtain the text feature.

7. The method according to claim 6, wherein Input the converted concatenation result into the text processing model for analysis to obtain the text feature, including: Input the converted concatenation result into the text processing model; Use the control network in the text processing model to fuse the converted concatenation result into the feature space of the target generation model in the text processing model to obtain a second fusion result, where the model parameters in the initialization stage of the target generation model in the text processing model are copied from the generation model; Use the convolutional layer corresponding to the generation model in the text processing model to perform convolutional processing on the second fusion result to obtain the text feature.

8. The method according to any one of claims 1 to 7, characterized in that, Use the generation model to fuse the image feature and the text feature to obtain the media file of the object, including: Fuse the image feature and the text feature into the feature space of the corresponding network layer in the generation model to obtain the media file.

9. The method according to claim 8, characterized in that Fuse the image feature and the text feature into the feature space of the corresponding network layer in the generation model to obtain the media file, including: Perform an addition operation on the image feature and the text feature to obtain an addition feature; Fuse the addition feature into the feature space of the corresponding network layer in the generation model to obtain the media file, where the rendering result in the media file includes the rendering result of the object and the rendering result of the text description information.

10. A method for determining a model, characterized in that, Including: Obtain a foreground image sample including at least one object sample, and obtain at least one background feature information sample and text description information sample matching the object sample; Using the foreground image sample, the background feature information sample, and the caption information sample, train a generation model to obtain an image processing model and a text processing model, where the image processing model is used to analyze a foreground image including at least one object and at least one background feature information matching the object to obtain image features, the text processing model is used to analyze the caption information to be rendered to obtain text features, and the generation model is used to fuse the image features and the text features to obtain a media file of the object, and the media file includes at least a background image and a rendering result presented on the background image.

11. The method according to claim 10, wherein Using the foreground image sample, the background feature information sample, and the caption information sample, training a generation model to obtain an image processing model and a text processing model includes: Using the foreground image sample, train the generation model to obtain the image processing model; Using the foreground image sample, the background feature information sample, and the caption information sample, train the generation model to obtain the text processing model.

12. The method according to claim 11, wherein, Using the foreground image sample, the background feature information sample, and the caption information sample, training the generation model to obtain the text processing model includes: Determine a text mask of the caption information sample, and based on the foreground image sample and the background feature information sample, determine a mask image of the caption information sample, where the text mask of the caption information sample is used to represent the position where the caption information sample is rendered, and the mask image of the caption information sample is used to represent the position of the caption information sample in the background image sample corresponding to the background feature information sample relative to the foreground image sample; Based on the text mask of the caption information sample and the mask image of the caption information sample, adjust the model parameters of the copied generation model; Using the adjusted model parameters, train the generation model with the structure of the control network as the network structure to obtain the text processing model.

13. The method according to claim 12, wherein Using the adjusted model parameters, training the generation model with the structure of the control network as the network structure to obtain the text processing model includes: During the process of freezing the image processing model, using the adjusted model parameters, train the generation model with the structure of the control network as the network structure to obtain the text processing model.

14. The method according to claim 11, wherein Using the foreground image sample, training the generation model to obtain the image processing model includes: Determine a main mask of the foreground image sample and a mask image of the foreground image sample, where the main mask of the foreground image sample is used to represent the position where the foreground image sample is rendered, and the mask image of the foreground image sample is used to represent the features of the object sample; Based on the main mask of the foreground image sample and the mask image of the foreground image sample, adjust the model parameters of the copied generation model; Using the structure of the control network as the network structure, training the generative model with the adjusted model parameters to obtain the image processing model.

15. The method according to claim 14, wherein Using the structure of the control network as the network structure, training the generative model with the adjusted model parameters to obtain the image processing model, including: During the process of freezing the text processing model, using the structure of the control network as the network structure, training the generative model with the adjusted model parameters to obtain the image processing model.

16. The method according to claim 14, characterized in that, The method further includes: Obtaining a reward signal for the image processing model, where the reward signal is used to represent the feedback result of the image processing model; In response to the reward signal indicating a negative feedback result, adjusting the image processing model based on the reward signal.

17. A method for generating a media file, characterized in that Including: Obtaining a foreground image including at least one object from an e-commerce platform, and obtaining at least one background feature information matching the object, as well as text description information to be rendered; Inputting the foreground image and the background feature information into the image processing model for analysis to obtain image features, and inputting the text description information into the text processing model for analysis to obtain text features, where the image processing model and the text processing model are trained based on a generative model; Using the generative model to fuse the image features and the text features to obtain a media file of the object, where the media file at least includes a background image and a rendering result presented on the background image; Returning the media file to the e-commerce platform for display.

18. A method for generating a media file, characterized in that, Including: Responding to an input operation on the operation interface, displaying a foreground image including at least one object on the operation interface, and displaying at least one background feature information matching the object, as well as text description information to be rendered; Responding to a file generation operation on the operation interface, displaying the media file of the object on the operation interface, where the media file is obtained by fusing image features and text features based on a generative model, and the media file at least includes a background image and a rendering result presented on the background image, the image features are obtained by the image processing model analyzing the foreground image and the background feature information, the text features are obtained by the text processing model analyzing the text description information, and the image processing model and the text processing model are trained based on the generative model.

19. A generation system for media files, characterized in that, Including: A client for transmitting a foreground image including at least one object, and obtaining at least one background feature information matching the object, as well as text description information to be rendered; A server, configured to input the foreground image and the background feature information into an image processing model for analysis to obtain image features, and input the text description information into a text processing model for analysis to obtain text features, wherein the image processing model and the text processing model are trained based on a generative model; generate a media file of the object based on the image features and the text features, wherein the media file at least includes a background image and a rendering result presented on the background image; and return the media file to the client.

20. A computing device, characterized in that, Comprising: A memory storing an executable program; A processor configured to run the program, wherein when the program runs, it executes the method according to any one of claims 1 to 18.

21. An electronic device, characterized in that, Comprising: A memory storing an executable program; A processor connected to the memory through a bus and configured to run the program, wherein when the program runs, it executes the method according to any one of claims 1 to 18.

22. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein when the executable program runs, it controls the device where the computer-readable storage medium is located to execute the method according to any one of claims 1 to 18.

23. A computer program product, characterized in that, Comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 18.

Images