Image generation method and device, model training method and device and electronic equipment

Abstract

The invention discloses an image generation method and device, a model training method and device, electronic equipment and a computer readable storage medium. The method comprises the steps of obtaining product description information and a product pattern corresponding to a target product of a to-be-generated display graph; generating background image description information corresponding to the product description information through a pre-trained text generation model based on the product description information; and based on the product pattern, through a pre-trained text guide image generation model, and by taking the background image description information as a guide text of the text guide image generation model, generating a display image for adding a background to the product pattern. Through the scheme provided by the invention, the high-quality display graph with the background matched with the product is more efficiently generated, and meanwhile, the labor cost required by background generation is saved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image generation method, a model training method, a device, an electronic device, and a computer-readable storage medium. Background Art

[0002] In e-commerce platforms, visual content plays a key role in attracting and maintaining audience attention. A high-quality, beautifully designed product display can quickly capture consumers' attention and increase customer purchase rates. Product display images include product images (such as physical product images) and background images. Product images are usually physical product images provided by merchants, and background images are usually generated by the platform or system. Therefore, generating a matching background for the product has a great impact on the quality of the product display image.

[0003] In the related art, a text-guided image generation model is usually used to generate product display images with matching backgrounds for products. However, the text-guided image generation model relies on carefully designed and accurate prompts. Therefore, each product requires manually designed accurate and detailed prompts to make the background of the product display image generated by the text-guided image generation model more matching with the product and with higher image quality. This is a huge challenge in the case of a large number of diverse products and is very labor-intensive. In addition, the manually designed prompts may have low accuracy, resulting in low matching degree and quality of the generated product background. Summary of the invention

[0004] The present application provides an image generation method, an image generation model training method, an apparatus, an electronic device, and a computer-readable storage medium, which can more efficiently generate a high-quality product display image with a background that matches the product, while saving the labor cost required for background generation. The specific scheme is as follows:

[0005] In a first aspect, the present application provides an image generation method, the method comprising:

[0006] Obtain product description information and product images corresponding to the target product to be displayed;

[0007] Based on the product description information, generating background image description information corresponding to the product description information through a pre-trained text generation model;

[0008] Based on the product pattern, a pre-trained text-guided image generation model is used, and the background image description information is used as the guide text of the text-guided image generation model to generate a display image that adds a background to the product pattern.

[0009] Optionally, the text generation model is used to generate corresponding description information according to the indication information;

[0010] The generating, based on the product description information, background image description information corresponding to the product description information by using a pre-trained text generation model includes:

[0011] The background image description information corresponding to the product description information is generated by a pre-trained text generation model using the product description information and the editing background indication information as indication information of the text generation model.

[0012] Optionally, before generating a display image for adding a background to the product image based on the product image by using a pre-trained text-guided image generation model and using the background image description information as a guide text of the text-guided image generation model, the method further includes:

[0013] Generate display image description information corresponding to the product description information by using a pre-trained text generation model, taking the product description information and the display image editing instruction information as instruction information of the text generation model;

[0014] The method of generating a display image with a background added to the product image based on the product image by using a pre-trained text-guided image generation model and using the background image description information as a guide text of the text-guided image generation model comprises:

[0015] Based on the product pattern, a pre-trained text-guided image generation model is used to generate a display image that adds a background to the product pattern using the background image description information and the display image description information as guide texts of the text-guided image generation model.

[0016] Optionally, the product description information includes at least one of the following: product name, product introduction, and product title.

[0017] Optionally, the generating of a display image with a background added to the product image based on the product image by using a pre-trained text-guided image generation model and using the background image description information as a guide text of the text-guided image generation model includes:

[0018] Determine product area marking information corresponding to the product pattern, wherein the product area marking information marks the area where the target product is located in the product pattern;

[0019] Based on the product pattern and the product area annotation information, a pre-trained text-guided image generation model is used, and the background image description information is used as the guide text of the text-guided image generation model to generate a display image that adds a background to the product pattern.

[0020] Optionally, the text-guided image generation model is a stable diffusion model;

[0021] The method of generating a display image with a background added to the product image based on the product image and the product area annotation information by using a pre-trained text-guided image generation model and using the background image description information as a guide text of the text-guided image generation model comprises:

[0022] Get the noise to be added;

[0023] Based on the product pattern, the product area annotation information and the noise to be added, a pre-trained text-guided image generation model is used, and the background image description information is used as the guiding text of the text-guided image generation model to generate a display image for adding a background to the product pattern.

[0024] Optionally, the generating of a display image for adding a background to the product image based on the product image, the product area annotation information and the noise to be added, using a pre-trained text-guided image generation model, and using the background image description information as a guide text of the text-guided image generation model, comprises:

[0025] Inputting the product image into a pre-trained dimension reduction encoder to obtain reduced-dimensional image encoding data corresponding to the product image;

[0026] The pattern encoding data, the background image description information, the product area annotation information and the noise to be added are input into a pre-trained text-guided image generation model to generate a display image for adding a background to the product pattern.

[0027] In a second aspect, the present application also provides a model training method, the method comprising:

[0028] Acquire a training sample, wherein the training sample includes sample product description information of a sample product and a pre-designed sample product display image with a background;

[0029] Based on the sample product description information, obtaining output background image description information corresponding to the sample product description information through a to-be-trained text generation model;

[0030] Determining sample background features corresponding to the sample product display picture according to the sample product display picture;

[0031] According to the difference between the output background image description information and the sample background features, the model parameters of the text generation model to be trained are adjusted to obtain a trained text generation model, and the text generation model is used to generate corresponding background image description information according to the product description information.

[0032] Optionally, the obtaining, based on the sample product description information, output background image description information corresponding to the sample product description information by generating a model for training text includes:

[0033] Based on the sample product description information and the editing background indication information, the output background image description information corresponding to the sample product description information is obtained through the to-be-trained text generation model.

[0034] Optionally, before adjusting the model parameters of the to-be-trained text generation model according to the difference between the output background image description information and the sample background features, the method further comprises:

[0035] Based on the sample product description information and the display image editing instruction information, obtaining output display image description information corresponding to the sample product description information through the to-be-trained text generation model;

[0036] Determining, according to the sample product display image, a sample display image feature corresponding to the sample product display image;

[0037] The adjusting the model parameters of the to-be-trained text generation model according to the difference between the output background image description information and the sample background features includes:

[0038] According to the difference between the output background image description information and the sample background features, and the difference between the output display image description information and the sample display image features, the model parameters of the text generation model to be trained are adjusted, wherein the text generation model is specifically used to generate corresponding background image description information and product display image description information according to the product description information.

[0039] Optionally, determining the sample background feature corresponding to the sample product display picture according to the sample product display picture includes:

[0040] Deducting a sample background area image from the sample display image;

[0041] Inputting the sample background area image into a pre-trained image encoder to obtain sample background features corresponding to the sample product display image;

[0042] The step of determining, according to the sample product display picture, a sample display picture feature corresponding to the sample product display picture includes:

[0043] The sample product display image is input into the image encoder to obtain a sample display image feature corresponding to the sample product display image.

[0044] Optionally, the training sample further includes a sample product image corresponding to the sample product, and the sample product image is a product area image deducted from the sample product display image;

[0045] The method further comprises:

[0046] Based on the sample product pattern, the output display image is obtained by using the to-be-trained text-guided image generation model and taking the output background image description information and the output display image description information as guide texts of the to-be-trained text-guided image generation model;

[0047] According to the difference between the output display image and the sample display image, the model parameters of the to-be-trained text-guided image generation model and the to-be-trained text generation model are adjusted to obtain a trained text-guided image generation model and a trained text generation model.

[0048] Optionally, the text-guided image generation model to be trained is a stable diffusion model;

[0049] The method of obtaining an output display image based on the sample product pattern by guiding the image generation model with the text to be trained, using the output background image description information and the output display image description information as guide texts of the image generation model with the text to be trained, comprises:

[0050] Based on the sample product pattern and the preset sample noise, the output display image and the predicted noise are obtained by guiding the image generation model with the output background image description information and the output display image description information as the guiding text of the image generation model with the output background image description information and the output display image description information;

[0051] The method further comprises:

[0052] According to the difference between the predicted noise and the preset sample noise, the model parameters of the to-be-trained text-guided image generation model and the to-be-trained text generation model are adjusted.

[0053] Optionally, the step of obtaining the output display image and the predicted noise based on the sample product image and the preset sample noise by guiding the image generation model through the text to be trained, using the output background image description information and the output display image description information as guide texts of the image generation model to be trained, comprises:

[0054] Determine sample product area marking information corresponding to the sample product pattern, wherein the sample product area marking information marks the area where the sample product in the sample product pattern is located;

[0055] The output background image description information, the output display image description information, the sample product image, the sample product area annotation information and the preset sample noise are input into the diffusion model to be trained to obtain the output display image and the predicted noise.

[0056] Optionally, obtaining a training sample includes:

[0057] Obtain a candidate training sample set;

[0058] Obtaining a screening prompt, wherein the screening prompt is used to instruct the large model to screen out training samples that meet preset screening rules;

[0059] Inputting each candidate training sample in the candidate training sample set and the screening prompt into a pre-trained large model to obtain a screening training sample;

[0060] The training samples used for model training are determined according to the screened training samples.

[0061] Optionally, the method further comprises:

[0062] The trained text generation model and text-guided image generation model are verified and evaluated through the verification sample set to obtain the evaluation results;

[0063] Adjust and update the screening prompt according to the evaluation result;

[0064] When the expected training condition is not met, continuing to perform the steps of obtaining the training sample to adjusting the model parameters of the to-be-trained text guiding image generation model and the to-be-trained text generation model;

[0065] The obtaining of the screening prompt includes:

[0066] Get the most recently updated filter prompt;

[0067] The step of obtaining candidate training samples comprises:

[0068] The training samples that have been screened out by the large model in the initial training sample set are deleted to obtain a deleted training sample set, and the deleted training sample set is determined as a candidate training sample set.

[0069] Optionally, the expected training condition includes at least one of the following:

[0070] The evaluation result meets the preset result;

[0071] The number of training iterations reaches the preset number;

[0072] The change in the loss function corresponding to each training round is less than the set threshold.

[0073] In a third aspect, the present application further provides an image generating device, the device comprising:

[0074] An acquisition unit, used to acquire product description information and product drawings corresponding to the target product for which a display drawing is to be generated;

[0075] A first generating unit, configured to generate background image description information corresponding to the product description information based on the product description information by using a pre-trained text generation model;

[0076] The second generation unit is used to generate a display image with a background added to the product image based on the product image through a pre-trained text-guided image generation model and using the background image description information as the guide text of the text-guided image generation model.

[0077] In a fourth aspect, the present application also provides a model training device, the device comprising:

[0078] A sample acquisition unit, used to acquire training samples, wherein the training samples include sample product description information of sample products and a pre-designed sample product display image with a background;

[0079] A third generating unit is used to obtain output background image description information corresponding to the sample product description information through a to-be-trained text generation model based on the sample product description information;

[0080] A determination unit, configured to determine, based on the sample product display picture, sample background features corresponding to the sample product display picture;

[0081] A training unit is used to adjust the model parameters of the text generation model to be trained according to the difference between the output background image description information and the sample background features to obtain a trained text generation model, wherein the text generation model is used to generate corresponding background image description information according to the product description information.

[0082] In a fifth aspect, the present application provides an electronic device comprising: a processor, a memory, and computer program instructions stored in the memory and executable on the processor; when the processor executes the computer program instructions, it implements the method described in any one of the first aspect or the second aspect.

[0083] In a sixth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in any one of the first aspect or the second aspect.

[0084] In a sixth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method described in any one of the first aspect or the second aspect.

[0085] Compared with the prior art, this application has the following advantages:

[0086] The image generation method provided in the embodiment of the present application obtains product description information and product pictures corresponding to the target product of the display picture to be generated, and then generates background picture description information corresponding to the above product description information through a pre-trained text generation model based on the product description information. That is, the text generation model can generate the corresponding background picture description information through the product description information. Since the text generation model is a machine learning model trained with a large number of designed sample product display pictures as training samples, the text generation model can quickly and accurately generate background picture description information corresponding to the above product description information and meeting the image aesthetic requirements, that is, it can generate background picture description information corresponding to the target product. The background picture description information can be used as the guiding text (i.e., prompt) of the text-guided image generation model. The product picture can be understood as a product picture without a background. In this way, the pre-trained text-guided image generation model can generate a display picture with a background added to the product picture based on the above product image and with the above background picture description information as the guiding text, and the generated display picture is more matched with the product and has a higher quality.

[0087] It can be seen that the solution provided by the present application does not require manual design of accurate prompts for the text-guided image generation model, but can efficiently generate background image description information corresponding to the product description information and with higher quality through a pre-trained text generation model, that is, the text generation model can align the hidden features between the product description information and its image background, so as to quickly and efficiently obtain high-quality background image description information that is strongly related to the product as the prompt of the text-guided image generation model, so that the text-guided image generation model can generate display images that are more matched with the product and with higher quality, thereby more efficiently generating high-quality display images with backgrounds that match the products, while saving the manpower cost required for background generation.

[0088] The model training method provided in the present application obtains training samples, and the training samples include sample product description information of the sample product and a pre-designed sample product display image with a background. Based on the sample product description information and through the text generation model to be trained, the output background image description information corresponding to the sample product description information is obtained, and then the sample background features corresponding to the sample product display image are determined. According to the difference between the output background image description information and the sample background features, the model parameters of the text generation model to be trained are adjusted to obtain a trained text generation model. The present application trains the text generation model to be trained by comparing the output background image description information with the sample background features, which can enhance the text generation model's understanding of background elements. The model training method provided in the present application can train a text generation model for generating corresponding background image description information according to the product description information. The trained text generation model can efficiently and accurately generate high-quality background image description information that is strongly related to the product, so that the background image description information can be used as a prompt for the text-guided image generation model to efficiently generate a high-quality display image with a background that matches the product. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 It is a schematic diagram of an application scenario of the image generation solution provided by this application;

[0090] Figure 2 is a flowchart of an example of an image generation method provided in an embodiment of the present application;

[0091] Figure 3 is a flowchart of another example of the image generation method provided in an embodiment of the present application;

[0092] Figure 4 It is a flowchart of an example of a model training method provided in an embodiment of the present application;

[0093] Figure 5 It is a flowchart of another example of the model training method provided in the embodiment of the present application;

[0094] Figure 6 It is a flowchart of an example of model training through data screening provided in an embodiment of the present application;

[0095] Figure 7 is a structural block diagram of an example of an image generating device provided in an embodiment of the present application;

[0096] Figure 8 It is a structural block diagram of the electronic device provided in this application. DETAILED DESCRIPTION

[0097] In order to enable those skilled in the art to better understand the technical solution of the present application, the present application is described clearly and completely below in conjunction with the drawings in the embodiments of the present application. However, the present application can be implemented in many other ways different from the following description. Therefore, based on the embodiments provided by the present application, all other embodiments obtained by those of ordinary skill in the art without creative work should fall within the scope of protection of the present application.

[0098] It should be noted that the terms "first", "source domain", "third", etc. in the claims, description and drawings of the present application are used to distinguish similar objects and are not used to describe a specific order or sequence. The data used in this way are interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including", "having" and their variants are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0099] In order to facilitate understanding of the various embodiments of the present application, the application background of the embodiments is described.

[0100] In e-commerce platforms, visual content plays a key role in attracting and maintaining audience attention. A high-quality, beautifully designed product display can quickly capture consumers' attention and increase customer purchase rates. Product display images include product images (such as physical product images) and background images. Product images are usually physical product images provided by merchants, and background images are usually generated by the platform or system. Therefore, generating a matching background for the product has a great impact on the quality of the product display image.

[0101] In the related art, a text-guided image generation model is usually used to generate product display images with matching backgrounds for products. However, the text-guided image generation model relies on carefully designed and accurate prompts. Therefore, each product requires manually designed accurate and detailed prompts to make the background of the product display image generated by the text-guided image generation model more matching with the product and with higher image quality. This is a huge challenge in the case of a large number of diverse products and is very labor-intensive. In addition, the manually designed prompts may have low accuracy, resulting in low matching degree and quality of the generated product background.

[0102] In order to solve the above problems, the embodiments of the present application provide an image generation method, a model training method, an apparatus, an electronic device, and a computer-readable storage medium, which are intended to more efficiently generate high-quality product display images with a background that matches the product, while saving the labor cost required for background generation.

[0103] The image generation method provided in this application can be used to generate product display images, specifically, to quickly generate corresponding product display images for products based on the product images and product description information. The solution provided in this application can be used to generate product display images in e-commerce platforms, and can also be used to generate product display images in other scenarios, such as generating product display images on advertising platforms, generating product display images in website design, etc. This application does not specifically limit the specific application scenarios of generating product display images.

[0104] In order to facilitate understanding of the method embodiments of the present application, the application scenarios are introduced. Figure 1 , Figure 1 Schematic diagram of the application scenario of the solution provided in the embodiment of the present application. For the convenience of description, the following describes the embodiment of the present application by taking the e-commerce platform as an example. The application scenario is a schematic illustration and is not intended to be a specific description of the application scenario. Figure 1 As shown, in this application scenario, a server 102, a merchant 103 and a client 101 are provided. In this embodiment, a connection is established between the client 101 and the server 102, and between the merchant 103 and the server 102 via network communication.

[0105] The merchant terminal 103 can be an electronic device with display and data processing functions, such as a mobile phone, a tablet computer (pad), a smart watch, a desktop computer, a smart TV, a VR device, a vehicle-mounted device, a wearable device, a laptop computer, etc. The merchant terminal 103 is used to upload product pictures to the server terminal 102, and obtain and display the generated product display pictures from the server terminal 102. The merchant terminal 103 can also be used to upload product information, upload order processing information, upload promotional information, obtain order information from the server terminal 102, etc., but is not limited to this.

[0106] A specific communication connection needs to be established between the merchant end 103 and the server end 102 to perform data transmission.

[0107] The client 101 may be a mobile phone, tablet computer (pad), smart watch, desktop computer, smart TV, VR device, vehicle-mounted device, wearable device, laptop computer or other electronic device with display function and data processing function. The client 101 is used to obtain and display the product display page of the e-commerce platform from the server 102, and the product display page includes the product display picture generated by the server 102. The client 101 may also be used to send order information, browse information, add-to-purchase information to the server 102, obtain and display the merchant's promotion information from the server 102, etc., but is not limited thereto.

[0108] A specific communication connection needs to be established between the client 101 and the server 102 to perform data transmission.

[0109] The server 102 has high computing power. The server 102 can be a server, which has high-speed processor (central processing unit, CPU) computing power, long-term reliable operation, powerful input / output (input / output, I / O) external data throughput capacity and better scalability. The server 102 can be a single server or a server cluster. The server 102 is used to receive product patterns from the merchant 103 and generate corresponding product display pictures based on the product patterns. The server 102 can also provide other specific services for the client 101 and the merchant 103, such as user information access, website access, application access, etc., which are not specifically limited in this application.

[0110] The client 101 and the server 102, and the merchant 103 and the server 102 can communicate with each other using various communication systems, such as a wired communication system or a wireless communication system. The wireless communication system may be, for example, a global system for mobile communications (GSM) system, a code division multiple access (CDMA) system, a wideband code division multiple access (WCDMA) system, a general packet radio service (GPRS), a long term evolution (LTE) system, a LTE frequency division duplex (FDD) system, a LTE time division duplex (TDD) system, a universal mobile telecommunication system (UMTS), a worldwide interoperability for microwave access (WiMAX) communication system, a future fifth generation (5G) system or a new radio (NR), a satellite communication system, etc.

[0111] Embodiment 1

[0112] The first embodiment of the present application provides an image generation method, which is applied to an electronic device, which may be a server, a laptop computer, a tablet computer, a desktop computer, or other electronic device with a data processing function.

[0113] Pre-trained text generation models and text-guided image generation models may be deployed on the electronic device, or pre-trained text generation models and text-guided image generation models may not be deployed on the electronic device. In this case, the electronic device can call the pre-trained text generation models and text-guided image generation models deployed on other devices and perform image generation through the models deployed on other devices.

[0114] like Figure 2 , Figure 3 As shown, the image generating method provided in the first embodiment of the present application includes the following steps S110 to S130.

[0115] Step S110: Obtain product description information and product drawings corresponding to the target product for which a display drawing is to be generated.

[0116] The target product can be a product sold on an e-commerce platform, such as clothing, food, dishes, meals, etc. It can also be a product sold on other platforms, such as virtual equipment and virtual characters in the gaming field, or it can be any non-commodity product that requires a display image to be generated.

[0117] The above-mentioned product description information can be information used to describe the product, such as product name, product introduction, product details, and product title. In the field of e-commerce, merchants usually set product description information about the product, such as the corresponding title, product name, product introduction, etc., and the electronic device can directly obtain the description information set by the merchant. Among them, the product name or product title can accurately express the product through concise and brief text, and the setting is simple and convenient. Therefore, many merchants will accurately set the product name or product title. Therefore, in the embodiment of the present application, the product description information can be the product name or product title, so that the electronic device can easily and conveniently obtain the product description information of each product, and the obtained product description information of each product is usually relatively simple, which is more convenient for each product to be applicable and better to obtain the product display picture. The product description information can be in Chinese, English, German, etc., or it can be description information in other languages, which is not specifically limited in this application. For example, such as Figure 3 The product description information shown is the English product name "XX UV Shield EssentialSunscreen Gel SPF 35PA++".

[0118] The above-mentioned product pattern refers to an image containing a physical image of the product. The background in the product pattern can be a preset background, such as a background of a preset color, a transparent background, etc., that is, the background in the product pattern is an initial background that has not yet been designed. The physical image included in the product pattern can be an actual product image. If the product is a virtual product, such as virtual equipment, virtual characters, etc., the physical image included in the product pattern is the appearance image of the virtual product. The product pattern can include a product image to be generated as a display image, such as Figure 3 The product image shown may be an image containing a real product image with a black background.

[0119] In one embodiment, the product pattern can be obtained in step S110 according to the following steps: obtain the product image uploaded by the merchant, deduct the product area where the product is located from the product image, and add a preset color background to the product area to obtain the product pattern. Since the product images uploaded by the merchant may be diverse, the backgrounds of some product images may be messy and not conducive to the subsequent generation of display images. Therefore, in this embodiment, by deducting the product area and adding a preset color background, the boundary between the product area and the background in the obtained product pattern is clearer and more distinguishable, which is more convenient for the subsequent generation of the background and thus the generation of a display image with a corresponding background.

[0120] Step S120: Based on the product description information, background image description information corresponding to the product description information is generated through a pre-trained text generation model.

[0121] The above-mentioned text generation model is used to generate corresponding background image description information through product description information. Specifically, the above-mentioned product description information can be input into a pre-trained text generation model to obtain background image description information corresponding to the above-mentioned product description information. Alternatively, the above-mentioned product description information can be subjected to text processing to obtain processed product description information, and then the processed product description information can be input into a pre-trained text generation model to obtain background image description information corresponding to the above-mentioned product description information. The above-mentioned text processing can be at least one of text expansion, text deduplication, and key information extraction, but is not limited thereto, so as to make the text description information input into the text generation model more accurate.

[0122] In a specific embodiment, the above-mentioned text generation model can generate corresponding description information according to the indication information, and the indication information is used to indicate what kind of description information the text generation model generates. In this case, the above-mentioned product description information is the indication information of the text generation model, so that the text generation model can generate a description text that is consistent with the product description information. In order to allow the text generation model to generate description information about the background, the indication information can also include editing background indication information to instruct the text generation model to generate relevant descriptions about the background. In this case, step S120 can be implemented according to the following steps S121.

[0123] Step S121: using a pre-trained text generation model, using the product description information and the editing background indication information as indication information of the text generation model, instructing the text generation model to generate background image description information corresponding to the product description information.

[0124] Specifically, the editing background indication information may be a prompt word for the editing background, such as "generate background text", Figure 3The word "background" shown in the figure may also be other prompt words that can indicate the editing background. In step S121, the product description information and the editing background indication information may be input into the text generation model to generate background image description information corresponding to the above product description information; or the product description information and the editing background indication information may be processed in a certain manner, such as splicing processing, encoding processing, etc. to obtain processed information, and the processed information may be input into the text generation model to obtain background image description information corresponding to the above product description information.

[0125] By setting the editing background indication information, this embodiment enables the text generation model to accurately and efficiently generate the background image description information corresponding to the display product to be generated under the instruction of the editing background indication information.

[0126] The text generation model is pre-trained and generated, and the specific training process will be described in detail in the embodiments below. The pre-trained text generation model is a machine learning model that can be trained to generate accurate background image description information. The background image description information is used to accurately describe the background image of the product in the form of text to guide the text-guided image generation model to generate a display image with a corresponding background. The text generation model can be a neural network model, a generative adversarial network model, a decision tree model, etc., or it can be other machine learning models.

[0127] Step S130: Based on the product image, a pre-trained text-guided image generation model is used, and the background image description information is used as the guide text of the text-guided image generation model to generate a display image with a background added to the product image.

[0128] Among them, the display image with the background added to the above product image is the display image of the generated target product.

[0129] The above-mentioned text-guided image generation model can be a Stable Diffusion model, a Midjourney model, a DALL·E model, etc., or can be other machine learning models that can generate images corresponding to the images described by the text through text guidance. The pre-trained text-guided image generation model can accurately generate corresponding images according to the text description information, and can specifically generate corresponding background-added display images for product images according to the description of the background image description information.

[0130] The text-guided image generation model is pre-trained and generated, and the specific training process will be described in detail in the embodiments below. The pre-trained text-guided image generation model is a machine learning model that can be trained to generate an accurate corresponding image based on the text description. In other words, the text-guided image generation model can generate a display image that adds a background to the product image under the guidance of the background image description information, and the background of the generated display image conforms to the background described by the background image description information.

[0131] Step S130 can specifically input the product image and background image description information into a pre-trained text-guided image generation model, so that the text-guided image generation model uses the background image description information as the guide text to generate a display image that adds a background to the product image. Alternatively, the product image, background image description information and other information used to generate the display image can be input into the text-guided image generation model to generate a display image that adds a background to the above-mentioned product image. Alternatively, the product image or background image description information can be processed to obtain processed information, such as encoding processing, text extraction processing, etc., and the processed information can be input into the text-guided image generation model to generate a display image that adds a background to the product image. The present application does not specifically limit the specific way in which the text-guided image generation model generates the display image.

[0132] In a specific embodiment, the following step S130a may be included before step S130.

[0133] Step S130a: Generate display image description information corresponding to the product description information by using the pre-trained text generation model, taking the product description information and the display image editing instruction information as instruction information of the text generation model.

[0134] The above-mentioned display image description information is the description information of the entire display image corresponding to the target product. The display image of the product can be described as a whole through the display image description information.

[0135] The display image editing instruction information may be a prompt word for editing the display image, for example, it may be "generate display image text", such as Figure 3 The "eos" shown may also be other prompt words that can indicate editing the display image.

[0136] Step S130a can be performed simultaneously with step S121. In this case, step S130a and step S121 can be combined into the following steps: by using a pre-trained text generation model, the above-mentioned product description information, editing background indication information, and editing display image indication information are used as indication information of the above-mentioned text generation model to generate display image description information and background image description information corresponding to the above-mentioned product description information. Specifically, the product description information, editing background indication information, and editing display image indication information can be input into the text generation model to generate the display image description information and background image description information, for example, Figure 3 As shown, P <background> <eos>Input the text generation model to generate the display image description information and the background image description information, where P represents the product name, background represents the editing background instruction information, and eos represents the editing display image instruction information. In this way, the display image description information and the background image description information can be generated at the same time through a single model call, which improves the efficiency of image generation.

[0137] The above-mentioned display image description information is used to guide the text generation model to generate description information of the display image corresponding to the product description information.

[0138] Correspondingly, step S130 can be implemented according to the following step S131.

[0139] Step S131: Based on the product pattern, a pre-trained text-guided image generation model is used to generate a display image that adds a background to the product pattern using the background image description information and the display image description information as guide texts of the text-guided image generation model.

[0140] Specifically, the product image, the background image description information and the display image description information may be input into a text-guided image generation model to generate a display image that adds a background to the product image.

[0141] This embodiment adds a guiding text of display image description information to the text-guided image generation model. Since the display image description information can describe the image from the display image as a whole, the display image generated by the text-guided image generation model can have a better overall image effect.

[0142] In one implementation, step S130 may be implemented by following steps S132 to S133.

[0143] Step S132: determining product area marking information corresponding to the product pattern, wherein the product area marking information marks the area where the target product is located in the product pattern.

[0144] The above-mentioned product area annotation information may be an image mask corresponding to the above-mentioned product pattern. The image mask can specify the product area in the product pattern that needs to be paid attention to or processed. It can isolate or highlight the product area in the product pattern. Specifically, the image mask can be a binary image of the same size as the product pattern, wherein the white or non-zero area in the mask represents the part of the product area that needs attention, and the black or zero area can represent the ignored part. The image mask can clearly mark the area where the product is located. The above-mentioned product area annotation information may also be the outline of the area where the product is located, or other information that can mark the product area, which is not specifically limited in this application.

[0145] In this step, the electronic device can use image annotation tools such as Labelme, VGG Image Annotator (VIA), COCOAnnotator, etc. to determine the corresponding image mask or product contour line and other product area annotation information from the product sample image. This application does not limit the specific method for determining the product area annotation information.

[0146] Step S133: Based on the product image and the product area annotation information, a pre-trained text-guided image generation model is used, and the background image description information is used as the guide text of the text-guided image generation model to generate a display image that adds a background to the product image.

[0147] Specifically, the product image, the product area annotation information, and the background image description information may be input into the text-guided image generation model, so that the text-guided image generation model generates a display image that adds a background to the product image under the guidance of the background image description information.

[0148] This embodiment assists the text-guided image generation model in generating a display image by inputting product area annotation information into the text-guided image generation model, which enables the text-guided image generation model to accurately distinguish between the background area and the product area, thereby being able to more accurately and appropriately integrate the background with the product area, thereby making the final generated display image higher in quality.

[0149] In a specific embodiment, the above-mentioned text-guided image generation model can be a Stable Diffusion model. The Stable Diffusion model is a generation model that can generate a desired image through the processes of forward diffusion and reverse diffusion. Forward diffusion refers to gradually adding noise to the original image until it becomes pure noise, and reverse diffusion starts from pure noise and gradually removes noise through a learned model to eventually generate a clear desired image. The way in which the Stable Diffusion model generates the desired image by adding noise can refer to the relevant technology and will not be described in detail in this application.

[0150] Correspondingly, the above step S133 can be implemented according to the following steps S133a to S133b.

[0151] Step S133a: Obtain the noise to be added.

[0152] Specifically, a preset noise may be determined as the noise to be added, or the noise to be added may be randomly generated based on a Gaussian distribution.

[0153] Step S133b: Based on the product pattern, the product area annotation information and the noise to be added, a pre-trained text-guided image generation model is used, and the background image description information is used as the guide text of the text-guided image generation model to generate a display image for adding a background to the product pattern.

[0154] In this step, the product image, the product area annotation information, the noise to be added, and the background image description information can be input into the trained Stable Diffusion model to generate a display image with a background added to the product image. Other guiding texts and input information can also be added to allow the text to guide the image generation model to generate a display image. In the process of generating the display image, the Stable Diffusion model will gradually add noise to the product image using the noise to be added through the forward diffusion process, and then reversely denoise the noisy image through the reverse diffusion process and based on the product area annotation information and the background image description information to obtain a denoised display image that meets the requirements.

[0155] The image generation process of the Stable Diffusion model is relatively stable, can generate high-quality images, and has strong controllability. This embodiment uses the Stable Diffusion model to generate display images, which can make the quality of the generated display images more stable and reliable.

[0156] In a specific embodiment, Figure 3 As shown, step S133b can be implemented according to the following steps A to B.

[0157] Step A: Input the product image into a pre-trained dimension reduction encoder to obtain reduced-dimensional image encoding data corresponding to the product image.

[0158] The above-mentioned dimensionality reduction encoder can be any one of a variational autoencoder (VAE), a principal component analysis (PCA) model, and a t-distributed stochastic neighbor embedding (t-SNE) model, but is not limited thereto.

[0159] The dimensionality reduction encoder can be trained using supervised training, unsupervised training, semi-supervised training or other training methods. This application does not specifically limit the specific training method of the dimensionality reduction encoder.

[0160] Step B: Input the above-mentioned pattern encoding data, the above-mentioned background image description information, the above-mentioned product area annotation information and the above-mentioned noise to be added into a pre-trained text-guided image generation model to generate a display image for adding a background to the above-mentioned product pattern.

[0161] Specifically, Figure 3 As shown, the pattern encoding data output by the dimensionality reduction encoder, the above-mentioned product area annotation information (such as the above-mentioned mask) and the above-mentioned noise to be added can be spliced ​​to obtain splicing features, and then the splicing features and the background image description information generated by the text generation model are input into the text-guided image generation model, and a display image with a background added to the above-mentioned product pattern is generated through multiple denoising and denoising processes.

[0162] After the product sample image is subjected to dimensionality reduction processing in this embodiment, the dimension of the image data can be reduced, thereby reducing the computing and storage requirements, making the image generation process more efficient. In addition, dimensionality reduction can also remove redundant information, further improving the efficiency and accuracy of image generation.

[0163] The image generation method provided in the embodiment of the present application obtains product description information and product pictures corresponding to the target product of the display picture to be generated, and then generates background picture description information corresponding to the above product description information through a pre-trained text generation model based on the product description information. That is, the text generation model can generate the corresponding background picture description information through the product description information. Since the text generation model is a machine learning model trained with a large number of designed product display pictures as training samples, the text generation model can quickly and accurately generate background picture description information corresponding to the above product description information and meeting the image aesthetic requirements, that is, it can generate background picture description information corresponding to the target product. The background picture description information can be used as the guiding text (i.e., prompt) of the text-guided image generation model. The product picture can be understood as a product picture without a background. In this way, the pre-trained text-guided image generation model can generate a display picture with a background added to the product picture based on the above product image and with the above background picture description information as the guiding text, and the generated display picture is more matched with the product and has higher quality.

[0164] It can be seen that the solution provided by the present application does not require manual design of accurate prompts for the text-guided image generation model, but can efficiently generate background image description information corresponding to the product description information and with higher quality through a pre-trained text generation model, that is, the text generation model can align the hidden features between the product description information and its image background, so as to quickly and efficiently obtain high-quality background image description information that is strongly related to the product as the prompt of the text-guided image generation model, so that the text-guided image generation model can generate display images that are more matched with the product and with higher quality, thereby more efficiently generating high-quality display images with backgrounds that match the products, while saving the manpower cost required for background generation.

[0165] Embodiment 2

[0166] The second embodiment of the present application also provides a model training method, which is applied to electronic devices, which can be electronic devices with data processing capabilities such as desktop computers, laptops, servers, mobile terminals, gateways, etc. This embodiment is at least used to train the text generation model required for image generation in the first embodiment. Other models in the first embodiment (such as text-guided image generation model, dimensionality reduction encoder) can be trained during the text generation model training process, or can be pre-trained. The model training method of this embodiment will also synchronously train the training process of the text-guided image generation model and the dimensionality reduction encoder.

[0167] like Figure 4 , Figure 5 As shown, the model training method provided in this embodiment includes the following steps S210 to S240.

[0168] Step S210: Obtain training samples.

[0169] The training samples include sample product description information of sample products and pre-designed sample product display pictures with backgrounds. The sample product display pictures in the training samples are display pictures of products with backgrounds pre-designed by designers, and each sample product display picture is pre-set with corresponding sample product description information.

[0170] The sample product description information may include at least one of the sample product name, the sample product introduction, and the sample product title, and may also include other information that can describe the product.

[0171] Step S220: Based on the sample product description information, the output background image description information corresponding to the sample product description information is obtained through the trained text generation model.

[0172] The text generation model to be trained can be a neural network model to be trained, a generative adversarial network model, a decision tree model, etc., or other machine learning models to be trained. The initial parameters of the text generation model to be trained can be randomly generated, or a model that has been trained and used can be used as the text generation model to be trained in this application, thereby training and adjusting the parameters of the model that has been trained and used, or the initial parameters of the text generation model to be trained can be determined by other means, which is not specifically limited in this application.

[0173] In step S220, the sample product description information may be input into the training text generation model to obtain the output background image description information corresponding to the sample product description information. Alternatively, the sample product description information may be encoded or otherwise processed and then input into the training text generation model to obtain the output background image description information corresponding to the sample product description information.

[0174] In one implementation, step S220 may be implemented according to the following step S221.

[0175] Step S221: Based on the sample product description information and the editing background indication information, the output background image description information corresponding to the sample product description information is obtained through the to-be-trained text generation model.

[0176] The editing context indication information in this embodiment is similar to the editing context indication information in the first embodiment, and will not be described in detail here.

[0177] Specifically, the sample product description information and the editing background indication information can be input into the to-be-trained text generation model to obtain the output background image description information corresponding to the sample product description information. The sample product description information and the editing background indication information can also be processed in a certain manner, such as splicing processing, encoding processing, etc. to obtain sample processed information, and the sample processed information is input into the to-be-trained text generation model to obtain the output background image description information corresponding to the above sample product description information.

[0178] By setting the editing background indication information, this embodiment can enable the text generation model to be trained to generate background image description information corresponding to the display product to be generated under the instruction of the editing background indication information. In this way, the trained text generation model can accurately and efficiently generate the background image description information under the instruction of the editing background indication information, and the trained text generation model is not limited to generating background image description information, but can also be used to generate other text information, thereby improving the applicability of the text generation model.

[0179] Step S230: According to the sample product display picture, determine the sample background features corresponding to the sample product display picture.

[0180] In a specific embodiment, the sample background features corresponding to the sample product display image may be determined according to the following steps S231 to S232.

[0181] Step S231: subtracting a sample background area image from the sample display image.

[0182] Specifically, you can use image annotation tools such as Labelme, VGG Image Annotator (VIA), COCO Annotator, etc. to deduct the sample background area from the sample display image.

[0183] Step S232: input the sample background area image into a pre-trained image encoder to obtain sample background features corresponding to the sample product display image.

[0184] The pre-trained image encoder is used to encode the image to obtain image features. The image encoder can be pre-trained by supervised training, unsupervised training, semi-supervised training or other training methods. This application does not specifically limit the specific training method of the image encoder.

[0185] In this embodiment, the sample background area image is first obtained, so that the corresponding sample background features can be quickly encoded from the sample background area image through the image encoder. This method enables the image encoder to quickly and accurately obtain the corresponding sample background features, and the image encoder does not need to have a background recognition function, but only a feature encoding function, which makes the training of the image encoder simpler.

[0186] Optionally, the corresponding sample background features may be determined from the sample display image by a pre-trained background feature extraction model, which can directly extract background features from the image. This embodiment can conveniently and quickly extract sample background features through the model.

[0187] Step S240: According to the difference between the output background image description information and the sample background features, the model parameters of the to-be-trained text generation model are adjusted to obtain a trained text generation model.

[0188] The above text generation model is used to generate corresponding background image description information according to product description information.

[0189] Step S240 is to train the text generation model by contrastive learning.

[0190] Optionally, the model parameters of the above-mentioned text generation model to be trained can be adjusted based on the principle that the difference between the output background image description information and the sample background characteristics is less than a second preset threshold, and the text generation model to be trained is trained with a large number of training samples until the difference between the output background image description information and the sample background characteristics is less than the second preset threshold.

[0191] Alternatively, the model parameters of the above-mentioned text generation model to be trained can be adjusted based on the principle of minimizing the difference between the output background image description information and the sample background features, and the text generation model to be trained can be trained with a large number of training samples until the difference between the output background image description information and the sample background features converges in multiple iterations of training, that is, the difference between the background image description information and the sample background features corresponding to multiple iterations is the same or less than the preset difference.

[0192] In a specific embodiment, when adjusting the model parameters of the above-mentioned text generation model to be trained based on the principle of minimizing the difference between the output background image description information and the sample background features, the parameters can be adjusted through the loss function of the following formula (1).

[0193]

[0194] In formula (1), L cl (b, B) represents the target loss (i.e., the difference between the output background image description information and the sample background features), Represents the current sample b and its positive sample The similarity between them is in exponential form after normalization by temperature parameter τ. It represents the exponential sum of the similarities between the current sample b) and all other samples (including positive and negative samples) normalized by the temperature parameter τ.

[0195] The loss function of formula (1) can minimize the distance between the current sample and its positive sample, while maximizing the distance with other negative samples. In this way, the model can learn a representation that can distinguish between positive and negative samples. During the training process, by minimizing this loss function, the model will adjust its parameters so that the similarity score of the positive sample is high and the similarity score of the negative sample is low, thus obtaining a text generation model with adjusted parameters.

[0196] When other training processes are adopted, the loss function is also different. Those skilled in the art can set the loss function according to the actual training process, and this application does not specifically limit it.

[0197] In a specific embodiment, before step S240, the following steps S230a and S230b may also be included.

[0198] Step S230a: Based on the sample product description information and the display image editing instruction information, the output display image description information corresponding to the sample product description information is obtained through the to-be-trained text generation model.

[0199] The editing display image indication information in this embodiment is similar to the editing display image indication information in the first embodiment, and will not be described in detail here.

[0200] Specifically, the sample product description information and the editing display image instruction information can be input into the to-be-trained text generation model to obtain the output display image description information corresponding to the sample product description information. The sample product description information and the editing display image instruction information can also be processed in a certain way, such as splicing processing, encoding processing, etc. to obtain sample processed information, and the sample processed information is input into the to-be-trained text generation model to obtain the output display image description information corresponding to the above sample product description information.

[0201] Step S230a can be performed simultaneously with step S221. In this case, step S230a and step S221 can be combined into the following steps: Based on the above sample product description information, the display image editing instruction information, and the background image editing instruction information, the output display image description information and the output background image description information corresponding to the above sample product description information are obtained through the text generation model to be trained. Specifically, the sample product description information, the display image editing instruction information, and the background image editing instruction information can be input into the text generation model to be trained to obtain the output display image description information and the output background image description information corresponding to the above sample product description information. For example, Figure 5 As shown, the sample product description information P and the edit display diagram indication information <eos>, Edit background image instructions <background>Input the text generation model to be trained and get the output display graph description information <eos>Pooling and output background image description information <background>pooling.

[0202] In this way, the output display image description information and the output background image description information can be generated simultaneously through a single model call, thereby improving the efficiency of model training and also enabling the text generation model generated by training to improve the efficiency of data output.

[0203] The above-mentioned output display image description information is used to guide the text generation model to be trained to generate description information of the sample product display image corresponding to the sample product description information.

[0204] Step S230b: According to the sample product display image, determine the sample display image feature corresponding to the sample product display image.

[0205] Specifically, Figure 5 As shown, the sample product display image can be input into the image encoder to obtain the sample display image features corresponding to the sample product display image.

[0206] Correspondingly, step S240 can adjust the model parameters of the text generation model to be trained according to the following step S241.

[0207] Step S241: adjusting the model parameters of the text generation model to be trained according to the difference between the output background image description information and the sample background features, and the difference between the output display image description information and the sample display image features.

[0208] Among them, the text generation model generated by training is specifically used to generate corresponding background image description information and product display image description information according to the product description information.

[0209] The process of adjusting the model parameters of the text generation model to be trained according to the difference between the output display image description information and the sample display image features in step S241 is similar to the process of adjusting the parameters according to the difference between the output background image description information and the sample background features in step S240, and will not be described in detail here.

[0210] In step S241, the parameter adjustment process can comprehensively refer to the difference between the output background image description information and the above-mentioned sample background features, and the difference between the above-mentioned output display image description information and the above-mentioned sample display image features, so as to quickly train a text generation model that can simultaneously output background image description information and product display image description information.

[0211] The text generation model trained in this embodiment can simultaneously generate corresponding background image description information and product display image description information based on the product description information, that is, it can generate a more comprehensive type of description information for the product display image, thereby being able to generate more comprehensive prompts for generating product display images. Therefore, the subsequent text-guided image generation model can generate higher quality product display images based on more comprehensive information.

[0212] In a specific embodiment, the training samples may also include sample product images corresponding to the sample products. The sample product images are product area images extracted from the sample product display images. The sample product images may also be understood as images obtained by removing the background area from the sample product display images.

[0213] Correspondingly, the above method may further include the following steps S250 to S260.

[0214] Step S250: Based on the sample product pattern, the output background image description information and the output display image description information are used as guide texts of the text-guided image generation model to be trained to obtain the output display image.

[0215] Specifically, the sample product image, the output background image description information, and the output display image description information may be input into the to-be-trained text-guided image generation model, so that the to-be-trained text-guided image generation model uses the output background image description information and the output display image description information as guide text to generate an output display image that adds a background to the sample product image. Alternatively, the sample product image, the sample background image description information, the sample display image description information, and other information used to generate a sample product display image may be input into the to-be-trained text-guided image generation model to generate a sample product display image.

[0216] The text-guided image generation model to be trained may be a Stable Diffusion model, a Midjourney model, a DALL·E model, etc., or may be other machine learning models that can generate images corresponding to images described by texts through text guidance.

[0217] The initial parameters of the text-guided image generation model to be trained can be randomly generated, or a model that has been trained and used can be used as the text-guided image generation model to be trained in this application, thereby training and adjusting the parameters of the text-guided image generation model that has been trained and used. Alternatively, the initial parameters of the text-guided image generation model to be trained can be determined by other means, which is not specifically limited in this application.

[0218] Step S260: According to the difference between the output display image and the sample display image, the model parameters of the to-be-trained text guidance image generation model and the to-be-trained text generation model are adjusted to obtain a trained text guidance image generation model and a trained text generation model.

[0219] Specifically, the model parameters of the text generation model to be trained and the text guidance image generation model to be trained can be adjusted based on the principle that the difference between the output display image and the sample display image is less than the third preset threshold. Alternatively, the model parameters of the text generation model to be trained and the text guidance image generation model to be trained can also be adjusted based on the principle of minimizing the difference between the output display image and the sample display image.

[0220] In the embodiment of the present application, step S260 and the above-mentioned step S240 both adjust the model parameters of the text generation model to be trained so that the generated text generation model can more accurately generate corresponding background image description information and product display image description information, so as to better guide the subsequent generation of better quality product display images.

[0221] This embodiment can simultaneously train a text-guided image generation model during the process of training a text generation model, thereby improving the training efficiency of multiple models. The trained text-guided image generation model can be used directly subsequently, and the parameters of the text generation model can be further adjusted through the output display diagram of the text-guided image generation model to be trained, so that the prediction accuracy of the trained text generation model is higher.

[0222] In a specific embodiment, the text-guided image generation model to be trained may be a stable diffusion model. Accordingly, the step S250 may be implemented according to the following step S251.

[0223] Step S251: Based on the sample product pattern and preset sample noise, the output display image and predicted noise are obtained by using the text-to-be-trained guided image generation model, with the output background image description information and the output display image description information as the guiding text of the text-to-be-trained guided image generation model.

[0224] Specifically, the sample product image, the preset sample noise, the output background image description information and the output display image description information can be input into the text-guided image generation model to be trained to obtain the output display image and the predicted noise. Alternatively, at least one of the sample product image, the preset sample noise, the output background image description information and the output display image description information can be encoded, spliced ​​or otherwise processed and then input into the text-guided image generation model to be trained to obtain the output display image and the predicted noise.

[0225] Since the Stable Diffusion model can gradually add noise to the original image and then gradually remove the noise, eventually generating a clear desired image and obtaining the predicted noise, the Stable Diffusion model can obtain an output display image and predicted noise.

[0226] Accordingly, the above model training method may further include the following steps S270:

[0227] Step S270: adjusting the model parameters of the to-be-trained text guidance image generation model and the to-be-trained text generation model according to the difference between the above-mentioned predicted noise and the above-mentioned preset sample noise.

[0228] Specifically, the model parameters of the text generation model to be trained and the text-guided image generation model to be trained can be adjusted based on the principle that the difference between the predicted noise and the preset sample noise is less than the fourth preset threshold. Alternatively, the model parameters of the text generation model to be trained and the text-guided image generation model to be trained can also be adjusted based on the principle of minimizing the difference between the predicted noise and the preset sample noise.

[0229] This embodiment uses a stable diffusion model as a text-guided image generation model and considers both the preset noise and the accuracy of the output product display image during the training process, thereby obtaining a more accurate text-guided image generation model.

[0230] In a specific embodiment, Figure 5 As shown, step S251 can be implemented according to the following steps S251a to S251b.

[0231] Step S251a: Determine the sample product area labeling information corresponding to the sample product pattern.

[0232] The sample product area marking information marks the area where the sample product is located in the sample product pattern. The relevant content of the sample product area marking information is similar to the product area marking information in the first embodiment, and will not be described in detail here.

[0233] Step S251b: input the output background image description information, the output display image description information, the sample product image, the sample product area annotation information and the preset sample noise into the diffusion model to be trained to obtain the output display image and the predicted noise.

[0234] This embodiment enables the trained text-guided image generation model to more accurately distinguish between the background area and the product area under the auxiliary guidance of the product area annotation information, so that the background can be more accurately and appropriately integrated with the product area, thereby making the final generated display image higher in quality.

[0235] In a specific embodiment, when the model parameters of the above-mentioned text generation model to be trained and the text-guided image generation model to be trained are adjusted based on the principle of minimizing the difference between the predicted noise and the preset sample noise and minimizing the difference between the output display image and the sample display image, the parameters can be adjusted through the loss function of the following formula (2).

[0236]

[0237] In formula (2), L inpaint represents the target loss (i.e., the combined value of the difference between the predicted noise and the preset sample noise, and the difference between the output display graph and the sample display graph), Indicates the conditions (including product sample image c, product area annotation information m, display image I, product description information p, background image B, time step T, noise level ε t ), ε t represents the preset sample noise at the current time step t, ε θ represents the prediction noise, x' represents the sample display graph, Γ θ (P, b) represents the output display diagram. Formula (2) is a commonly used loss function in the stable diffusion model training process. The specific interpretation is not described in detail in this application. Those skilled in the art can also use other loss functions for model training.

[0238] In a specific embodiment, when adjusting parameters of each model, the embodiment of the present application can adjust the parameters through the comprehensive loss function of the following formula (3).

[0239] L=L inpaint +λL cl (b,B)+λL cl (eos,I) (3)

[0240] Among them, λ is the weight coefficient, L cl (eos,I) is the difference between the output display graph description information and the sample display graph features.

[0241] In one implementation, step S210 may be implemented according to the following steps S211 to S214.

[0242] Step S211: Obtain a candidate training sample set.

[0243] The candidate training sample set includes various candidate training samples.

[0244] Step S212: Obtain a screening prompt, where the screening prompt is used to instruct the large model to screen out training samples that meet preset screening rules.

[0245] The screening prompts can be pre-designed manually, and the screening prompts can include the conditions that the screened training samples need to meet and the characteristics of unqualified training samples. Those skilled in the art can flexibly set the screening prompts to screen out training samples that meet the requirements. For example, the screening prompts can include that the image meets aesthetic characteristics, that the image contains products and backgrounds, etc., but are not limited thereto.

[0246] Step S213: input each candidate training sample in the candidate training sample set and the screening prompt into a pre-trained large model to obtain screening training samples.

[0247] The above-mentioned large model can be a large-scale language model (Large Language Models, LLMs for short), which can be a GPT model, a BERT model, etc., but is not limited to this.

[0248] Steps S211 to S213 are Figure 6 The data screening phase.

[0249] Step S214: determining training samples for model training according to the screened training samples.

[0250] Specifically, samples in the screened training samples may be determined as training samples for model training.

[0251] This implementation uses a large language model to screen each candidate training sample in the candidate training sample set, which can make the quality of the samples used to train the model higher, thereby training a model with higher accuracy and higher image generation quality.

[0252] In a specific implementation, the above-mentioned model training method may also include the following steps S280 to S2100.

[0253] Step S280: verify and evaluate the trained text generation model and text-guided image generation model using a verification sample set to obtain evaluation results.

[0254] The evaluation result includes verification passed or verification failed; or, the evaluation result may also be a verification score, which is not specifically limited in this application.

[0255] Step S290: updating the screening prompt according to the evaluation result.

[0256] Specifically, when the evaluation result is that the evaluation is passed, it means that the accuracy of the trained model is relatively high, which means that the training samples used in the current training meet the requirements, and the adjustment and update of the screening prompts are not required. When the evaluation result is that the evaluation is not passed, the above screening prompts are updated so that training samples that meet the requirements are screened out later. Exemplarily, when adjusting the screening prompts, if the original prompt is a long sentence, the screening prompts can be shortened to short sentences, or other adjustments can be made, which are not specifically limited in this application. When updating the screening prompts, adjustments can be made through the large language model to improve efficiency.

[0257] Steps S280 to S290 are Figure 6 The model evaluation and feedback improvement prompt phase.

[0258] Step S2100: When the expected training conditions are not met, continue to execute the process of step S210 to step S270 to perform model training.

[0259] The expected training conditions may be at least one of the following: the number of training iterations reaches a preset number, the loss function converges, verification is passed on the verification sample, the above evaluation results meet the preset results, and the change in the loss function corresponding to each training round is less than a set threshold. Other training conditions may also be included, which are not specifically limited in this application.

[0260] Correspondingly, step S211 may obtain the screening prompt in the following steps: obtaining the most recently updated screening prompt, that is, the screening prompt obtained each time is the most recently updated screening prompt.

[0261] This embodiment updates the screening prompts according to the evaluation results of the model, which can make the screened training samples more in line with the model training requirements, so that the model can be trained accurately and quickly in the subsequent iterative training process, thereby enabling a high-accuracy model to be trained more efficiently and quickly.

[0262] Optionally, step S210 can be implemented according to the following step S211.

[0263] Step S211: Delete the training samples that have been screened out by the large model in the initial training sample set to obtain a deleted training sample set, and determine the deleted training sample set as a candidate training sample set.

[0264] Through step S211, unqualified training samples can be deleted and then screened with the large model, thereby improving the screening efficiency of the large model.

[0265] The model training method provided in the present application obtains training samples, and the training samples include sample product description information of a sample product and a pre-designed sample product display image with a background. Based on the sample product description information and through the text generation model to be trained, the output background image description information corresponding to the sample product description information is obtained, and then the sample background features corresponding to the sample product display image are determined. According to the difference between the output background image description information and the sample background features, the model parameters of the text generation model to be trained are adjusted to obtain a trained text generation model. The present application trains the text generation model to be trained by comparing the output background image description information with the sample background features, which can enhance the text generation model's understanding of background elements. The model training method provided in the present application can train a text generation model for generating corresponding background image description information according to the product description information. The trained text generation model can efficiently and accurately generate high-quality background image description information that is strongly related to the product, so that the background image description information can be used as a prompt for the text-guided image generation model to efficiently generate high-quality product display images with backgrounds matching the products.

[0266] This embodiment introduces the training process of the estimation model from the perspective of model training. The processing process of the training samples by the to-be-trained model during the training process is similar to the processing process of the estimation model on the estimation information during the estimation process. Therefore, the relevant content and beneficial effects of the specific implementation process of this embodiment may refer to the first embodiment, and the similarities will not be described in detail in this embodiment.

[0267] Embodiment 3

[0268] The third embodiment of the present application also provides an image generation device corresponding to the image generation method embodiment provided in the first embodiment. Since the device embodiment is basically similar to the method embodiment, the description is relatively simple. For details of the relevant technical features and the effects achieved, please refer to the corresponding description of the above-mentioned image generation method embodiment. The image generation device provided in this embodiment includes:

[0269] An acquisition unit 301 is used to acquire product description information and product images corresponding to a target product for which a display image is to be generated;

[0270] A first generating unit 302 is used to generate background image description information corresponding to the product description information based on the product description information through a pre-trained text generation model;

[0271] The second generating unit 303 is used to generate a display image with a background added to the product image based on the product image through a pre-trained text-guided image generation model and using the background image description information as a guide text of the text-guided image generation model.

[0272] Optionally, the text generation model is used to generate corresponding description information according to the indication information;

[0273] The first generating unit is specifically used to: generate background image description information corresponding to the product description information through a pre-trained text generation model, using the product description information and the editing background indication information as indication information of the text generation model.

[0274] Optionally, the first generating unit is further used to: generate display image description information corresponding to the product description information by using a pre-trained text generation model, taking the product description information and the display image editing instruction information as instruction information of the text generation model;

[0275] The second generation unit is specifically used to: based on the product pattern, through a pre-trained text-guided image generation model, use the background image description information and the display image description information as the guiding text of the text-guided image generation model to generate a display image that adds a background to the product pattern.

[0276] Optionally, the product description information includes at least one of the following: product name, product introduction, and product title.

[0277] Optionally, the second generation unit is specifically used to: determine product area annotation information corresponding to the product pattern, the product area annotation information marks the area where the target product is located in the product pattern; based on the product pattern and the product area annotation information, through a pre-trained text-guided image generation model, and using the background image description information as the guiding text of the text-guided image generation model, generate a display image that adds a background to the product pattern.

[0278] Optionally, the text-guided image generation model is a stable diffusion model;

[0279] The second generation unit is specifically used to: obtain the noise to be added; based on the product pattern, the product area annotation information and the noise to be added, generate a display image for adding a background to the product pattern through a pre-trained text-guided image generation model, and use the background image description information as the guiding text of the text-guided image generation model.

[0280] Optionally, the second generation unit is specifically used to: input the product pattern into a pre-trained dimensionality reduction encoder to obtain reduced-dimensional pattern encoding data corresponding to the product pattern; input the pattern encoding data, the background image description information, the product area annotation information and the noise to be added into a pre-trained text-guided image generation model to generate a display image with a background added to the product pattern.

[0281] Embodiment 4

[0282] The fourth embodiment of the present application also provides a model training device corresponding to the model training method embodiment provided in the second embodiment. Since the device embodiment is basically similar to the method embodiment, the description is relatively simple. For the details of the relevant technical features and the effects achieved, please refer to the corresponding description of the model training method embodiment provided above. The image generation device provided in this embodiment includes:

[0283] A sample acquisition unit, used to acquire training samples, wherein the training samples include sample product description information of sample products and a pre-designed sample product display image with a background;

[0284] A third generating unit is used to obtain output background image description information corresponding to the sample product description information through a to-be-trained text generation model based on the sample product description information;

[0285] A determination unit, configured to determine, based on the sample product display picture, sample background features corresponding to the sample product display picture;

[0286] A training unit is used to adjust the model parameters of the text generation model to be trained according to the difference between the output background image description information and the sample background features to obtain a trained text generation model, wherein the text generation model is used to generate corresponding background image description information according to the product description information.

[0287] The fifth embodiment of the present application also provides an electronic device embodiment corresponding to the image generation method provided in the first embodiment. The following description of the electronic device embodiment is only illustrative. The electronic device embodiment is as follows:

[0288] Please refer to Figure 8 Understand the above electronic devices, Figure 8 The electronic device provided in this embodiment includes: a processor 1001, a memory 1002, a communication bus 1003, and a communication interface 1004;

[0289] The memory 1002 is used to store computer instructions for data processing. When the computer instructions are read and executed by the processor 1001, the following steps are performed:

[0290] Obtain product description information and product images corresponding to the target product to be displayed;

[0291] Based on the product description information, generating background image description information corresponding to the product description information through a pre-trained text generation model;

[0292] Based on the product pattern, a pre-trained text-guided image generation model is used, and the background image description information is used as the guide text of the text-guided image generation model to generate a display image that adds a background to the product pattern.

[0293] The sixth embodiment of the present application also provides an electronic device embodiment corresponding to the model training method provided in the second embodiment, and the electronic device embodiment is as follows:

[0294] The electronic device provided in this embodiment includes: a processor, a memory, a communication bus, and a communication interface;

[0295] The memory is used to store computer instructions for data processing, and when the computer instructions are read and executed by the processor, the following steps are performed:

[0296] Acquire a training sample, wherein the training sample includes sample product description information of a sample product and a pre-designed sample product display image with a background;

[0297] Based on the sample product description information, obtaining output background image description information corresponding to the sample product description information through a to-be-trained text generation model;

[0298] Determining sample background features corresponding to the sample product display picture according to the sample product display picture;

[0299] According to the difference between the output background image description information and the sample background features, the model parameters of the text generation model to be trained are adjusted to obtain a trained text generation model, and the text generation model is used to generate corresponding background image description information according to the product description information.

[0300] The seventh embodiment of the present application also provides a computer-readable storage medium for implementing the method described in the first embodiment. The computer-readable storage medium embodiment provided in this application is described relatively simply. For the relevant parts, please refer to the corresponding description of the above method embodiment. The embodiment described below is only illustrative.

[0301] The computer readable storage medium provided in this embodiment stores computer instructions, and when the instructions are executed by a processor, the following steps are implemented:

[0302] Obtain product description information and product images corresponding to the target product to be displayed;

[0303] Based on the product description information, generating background image description information corresponding to the product description information through a pre-trained text generation model;

[0304] Based on the product pattern, a pre-trained text-guided image generation model is used, and the background image description information is used as the guide text of the text-guided image generation model to generate a display image that adds a background to the product pattern.

[0305] The seventh embodiment of the present application also provides a computer-readable storage medium for implementing the method described in the second embodiment. The computer-readable storage medium embodiment provided in this application is described relatively simply. For the relevant parts, please refer to the corresponding description of the above method embodiment. The following described embodiment is only illustrative.

[0306] The computer readable storage medium provided in this embodiment stores computer instructions, and when the instructions are executed by a processor, the following steps are implemented:

[0307] Acquire a training sample, wherein the training sample includes sample product description information of a sample product and a pre-designed sample product display image with a background;

[0308] Based on the sample product description information, obtaining output background image description information corresponding to the sample product description information through a to-be-trained text generation model;

[0309] Determining sample background features corresponding to the sample product display picture according to the sample product display picture;

[0310] According to the difference between the output background image description information and the sample background features, the model parameters of the text generation model to be trained are adjusted to obtain a trained text generation model, and the text generation model is used to generate corresponding background image description information according to the product description information.

[0311] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0312] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0313] 1. Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0314] 2. Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0315] Although the present application is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.< / background> < / eos> < / background> < / eos> < / eos> < / background>

Claims

1. An image generation method, characterized in that: The method comprises: Obtain product description information and product images corresponding to the target product to be displayed; Based on the product description information, generating background image description information corresponding to the product description information through a pre-trained text generation model; Based on the product pattern, a pre-trained text-guided image generation model is used, and the background image description information is used as the guide text of the text-guided image generation model to generate a display image that adds a background to the product pattern.

2. The image generation method according to claim 1, characterized in that: The text generation model is used to generate corresponding description information according to the indication information; The generating, based on the product description information, background image description information corresponding to the product description information by using a pre-trained text generation model includes: The background image description information corresponding to the product description information is generated by a pre-trained text generation model using the product description information and the editing background indication information as indication information of the text generation model.

3. The image generation method according to claim 2, characterized in that: Before generating a display image with a background added to the product image based on the product image by using a pre-trained text-guided image generation model and using the background image description information as a guide text of the text-guided image generation model, the method further includes: Generate display image description information corresponding to the product description information by using a pre-trained text generation model, taking the product description information and the display image editing instruction information as instruction information of the text generation model; The method of generating a display image with a background added to the product image based on the product image by using a pre-trained text-guided image generation model and using the background image description information as a guide text of the text-guided image generation model comprises: Based on the product pattern, a pre-trained text-guided image generation model is used to generate a display image that adds a background to the product pattern using the background image description information and the display image description information as guide texts of the text-guided image generation model.

4. The image generation method according to claim 1, characterized in that: The product description information includes at least one of the following: product name, product introduction, and product title.

5. The image generation method according to claim 1, characterized in that: The method of generating a display image with a background added to the product image based on the product image by using a pre-trained text-guided image generation model and using the background image description information as a guide text of the text-guided image generation model comprises: Determine product area marking information corresponding to the product pattern, wherein the product area marking information marks the area where the target product is located in the product pattern; Based on the product pattern and the product area annotation information, a pre-trained text-guided image generation model is used, and the background image description information is used as the guide text of the text-guided image generation model to generate a display image that adds a background to the product pattern.

6. The image generation method according to claim 5, characterized in that: The text-guided image generation model is a stable diffusion model; The method of generating a display image with a background added to the product image based on the product image and the product area annotation information by using a pre-trained text-guided image generation model and using the background image description information as a guide text of the text-guided image generation model comprises: Get the noise to be added; Based on the product pattern, the product area annotation information and the noise to be added, a pre-trained text-guided image generation model is used, and the background image description information is used as the guiding text of the text-guided image generation model to generate a display image for adding a background to the product pattern.

7. The image generation method according to claim 6, characterized in that: The method of generating a display image for adding a background to the product image based on the product image, the product area annotation information, and the noise to be added by using a pre-trained text-guided image generation model, and using the background image description information as a guide text of the text-guided image generation model, comprises: Inputting the product pattern into a pre-trained dimension reduction encoder to obtain reduced-dimensional pattern encoding data corresponding to the product pattern; The pattern encoding data, the background image description information, the product area annotation information and the noise to be added are input into a pre-trained text-guided image generation model to generate a display image for adding a background to the product pattern.

8. A model training method, characterized in that: The method comprises: Acquire a training sample, wherein the training sample includes sample product description information of a sample product and a pre-designed sample product display image with a background; Based on the sample product description information, obtaining output background image description information corresponding to the sample product description information through a to-be-trained text generation model; According to the sample product display picture, determining the sample background features corresponding to the sample product display picture; According to the difference between the output background image description information and the sample background features, the model parameters of the text generation model to be trained are adjusted to obtain a trained text generation model, and the text generation model is used to generate corresponding background image description information according to the product description information.

9. The model training method according to claim 8, characterized in that: The step of obtaining output background image description information corresponding to the sample product description information by generating a model for training text based on the sample product description information includes: Based on the sample product description information and the editing background indication information, the output background image description information corresponding to the sample product description information is obtained through the to-be-trained text generation model.

10. The model training method according to claim 9, characterized in that: Before adjusting the model parameters of the to-be-trained text generation model according to the difference between the output background image description information and the sample background features, the method further includes: Based on the sample product description information and the display image editing instruction information, obtaining output display image description information corresponding to the sample product description information through the to-be-trained text generation model; Determining, according to the sample product display image, a sample display image feature corresponding to the sample product display image; The adjusting the model parameters of the to-be-trained text generation model according to the difference between the output background image description information and the sample background features includes: According to the difference between the output background image description information and the sample background features, and the difference between the output display image description information and the sample display image features, the model parameters of the text generation model to be trained are adjusted, wherein the text generation model is specifically used to generate corresponding background image description information and product display image description information according to the product description information.

11. The model training method according to claim 10, characterized in that: The determining, according to the sample product display picture, the sample background feature corresponding to the sample product display picture includes: Deducting a sample background area image from the sample display image; Inputting the sample background area image into a pre-trained image encoder to obtain sample background features corresponding to the sample product display image; The step of determining, according to the sample product display picture, a sample display picture feature corresponding to the sample product display picture includes: The sample product display image is input into the image encoder to obtain a sample display image feature corresponding to the sample product display image.

12. The model training method according to claim 10, characterized in that: The training sample also includes a sample product image corresponding to the sample product, wherein the sample product image is a product area image deducted from the sample product display image; The method further comprises: Based on the sample product pattern, the output display image is obtained by using the to-be-trained text-guided image generation model and taking the output background image description information and the output display image description information as guide texts of the to-be-trained text-guided image generation model; According to the difference between the output display image and the sample display image, the model parameters of the to-be-trained text-guided image generation model and the to-be-trained text generation model are adjusted to obtain a trained text-guided image generation model and a trained text generation model.

13. The model training method according to claim 12, characterized in that: The text-guided image generation model to be trained is a stable diffusion model; The method of obtaining an output display image based on the sample product pattern by guiding the image generation model with the text to be trained, using the output background image description information and the output display image description information as guide texts of the image generation model with the text to be trained, comprises: Based on the sample product pattern and the preset sample noise, the output display image and the predicted noise are obtained by guiding the image generation model with the output background image description information and the output display image description information as the guiding text of the image generation model with the output background image description information and the output display image description information; The method further comprises: According to the difference between the predicted noise and the preset sample noise, the model parameters of the to-be-trained text-guided image generation model and the to-be-trained text generation model are adjusted.

14. The model training method according to claim 13, characterized in that: The method of obtaining an output display image and predicted noise by guiding an image generation model through a text to be trained based on the sample product pattern and the preset sample noise, using the output background image description information and the output display image description information as guide texts of the text to be trained to guide the image generation model, comprises: Determine sample product area marking information corresponding to the sample product pattern, wherein the sample product area marking information marks the area where the sample product is located in the sample product pattern; The output background image description information, the output display image description information, the sample product image, the sample product area annotation information and the preset sample noise are input into the diffusion model to be trained to obtain the output display image and the predicted noise.

15. The model training method according to any one of claims 8 to 14, characterized in that: The obtaining of training samples comprises: Obtain a candidate training sample set; Obtaining a screening prompt, wherein the screening prompt is used to instruct the large model to screen out training samples that meet preset screening rules; Inputting each candidate training sample in the candidate training sample set and the screening prompt into a pre-trained large model to obtain a screening training sample; The training samples used for model training are determined according to the screened training samples.

16. The model training method according to claim 15, characterized in that: The method further comprises: The trained text generation model and text-guided image generation model are verified and evaluated through the verification sample set to obtain the evaluation results; Adjust and update the screening prompt according to the evaluation result; When the expected training condition is not met, continuing to perform the steps of obtaining the training sample to adjusting the model parameters of the to-be-trained text guiding image generation model and the to-be-trained text generation model; The obtaining of the screening prompt includes: Get the most recently updated filter prompt; The step of obtaining candidate training samples comprises: The training samples that have been screened out by the large model in the initial training sample set are deleted to obtain a deleted training sample set, and the deleted training sample set is determined as a candidate training sample set.

17. The model training method according to claim 16, characterized in that: The expected training conditions include at least one of the following: The evaluation result meets the preset result; The number of training iterations reaches the preset number; The change in the loss function corresponding to each training round is less than the set threshold.

18. An image generating device, characterized in that: The device comprises: An acquisition unit, used to acquire product description information and product drawings corresponding to the target product for which a display drawing is to be generated; A first generating unit, configured to generate background image description information corresponding to the product description information based on the product description information by using a pre-trained text generation model; The second generation unit is used to generate a display image with a background added to the product image based on the product image through a pre-trained text-guided image generation model and using the background image description information as the guide text of the text-guided image generation model.

19. A model training device, characterized in that: The device comprises: A sample acquisition unit, used to acquire training samples, wherein the training samples include sample product description information of sample products and a pre-designed sample product display image with a background; A third generating unit is used to obtain output background image description information corresponding to the sample product description information through a to-be-trained text generation model based on the sample product description information; A determination unit, configured to determine, based on the sample product display picture, sample background features corresponding to the sample product display picture; A training unit is used to adjust the model parameters of the text generation model to be trained according to the difference between the output background image description information and the sample background features to obtain a trained text generation model, wherein the text generation model is used to generate corresponding background image description information according to the product description information.

20. An electronic device, characterized in that: include: A processor, a memory, and computer program instructions stored on the memory and executable on the processor; When the processor executes the computer program instructions, the method according to any one of claims 1 to 17 is implemented.

21. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method as described in any one of claims 1 to 17 above.

Images