Image generation method and device, storage medium and product

Abstract

The invention provides an image generation method and device, a storage medium and a product, and the method comprises the steps: generating a to-be-processed image containing a first image element and a second image element based on the editing operation of the first image element and the second image element on an editing canvas, obtaining a first cue word and a second cue word corresponding to the second image element, the first cue word and the second cue word are used for describing the expected presentation effect of the corresponding image element. And according to the position information of the second image element on the to-be-processed image, performing region segmentation on the to-be-processed image to determine an image region corresponding to the second image element. And generating a target cue word by taking the second cue word as a region cue word of an image region corresponding to the second image element and taking the first cue word as a global cue word corresponding to the to-be-processed image. And inputting the to-be-processed image and the target cue word into the image generation model to obtain the first target image, so that the image generation quality can be improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image generation method, device, storage medium, and product. Background Technology

[0002] With the rapid development of computer vision computing, there is a demand in many fields such as e-commerce and film and television production for generating target images containing specific objects. For example, generating background images of products containing a certain item, or poster images containing a movie or television character.

[0003] Currently, the image generation model typically inputs the description information corresponding to the target image as text prompts, generating a target image containing the specific object through a text-to-image generation process. However, due to the potential for long text prompts and limitations of the image generation model itself, the final target image generated by the model is often of poor quality and fails to meet user expectations. Summary of the Invention

[0004] This invention provides an image generation method, device, storage medium, and product to improve image generation quality.

[0005] In a first aspect, embodiments of the present invention provide an image generation method, the method comprising:

[0006] Based on the editing operations on the first image element and the second image element on the editing canvas, an image to be processed containing the first image element and the second image element is generated;

[0007] Obtain the first prompt word and the second prompt word corresponding to the second image element;

[0008] Based on the position information of the second image element on the image to be processed, the image to be processed is segmented into regions to determine the image region corresponding to the second image element;

[0009] The second prompt word is used as the region prompt word for the image region corresponding to the second image element, and the first prompt word is used as the global prompt word for the image to be processed to generate a target prompt word.

[0010] The image to be processed and the target prompt are input into the image generation model to obtain the first target image generated by the image generation model.

[0011] In a second aspect, embodiments of the present invention provide an image generation apparatus, the apparatus comprising:

[0012] An editing module is used to generate an image to be processed, which includes the first image element and the second image element, based on editing operations on the first image element and the second image element on the editing canvas.

[0013] The acquisition module is used to acquire the first prompt word and the second prompt word corresponding to the second image element;

[0014] The segmentation module is used to perform region segmentation on the image to be processed based on the position information of the second image element on the image to be processed, so as to determine the image region corresponding to the second image element;

[0015] The generation module is used to generate a target prompt word by using the second prompt word as the region prompt word for the image region corresponding to the second image element and the first prompt word as the global prompt word for the image to be processed.

[0016] The processing module is used to input the image to be processed and the target prompt word into the image generation model to obtain the first target image generated by the image generation model.

[0017] Thirdly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a communication interface; wherein, the memory stores executable code, and when the executable code is executed by the processor, the processor performs the image generation method as described in the first aspect.

[0018] Fourthly, embodiments of the present invention provide a non-transitory machine-readable storage medium storing executable code, which, when executed by a processor of an electronic device, enables the processor to at least implement the image generation method as described in the first aspect.

[0019] Fifthly, embodiments of the present invention provide a computer program product, the computer program product including a computer program, which, when executed by a processor, can implement the image generation method as described in the first aspect.

[0020] During the process of generating a target image containing specific image elements, an editing canvas can be provided to the user so that the user can add specific image elements to the editing canvas and perform editing operations on the added image elements to generate the target image containing the specific image elements that the user expects.

[0021] Based on this, in the process of generating the target image, an image to be processed containing the first and second image elements can be generated first by editing the first and second image elements on the editing canvas. Then, a global first prompt word and a second prompt word corresponding to the second image element are obtained. Next, based on the position information of the second image element in the image to be processed, region segmentation is performed to determine the image region corresponding to the second image element. Furthermore, using the second prompt word as the region prompt word for the image region corresponding to the second image element and the first prompt word as the global prompt word for the image to be processed, a target prompt word is generated to indicate that a second image element is added to a specific region in the target image generated primarily from the first image element. Finally, the image to be processed and the target prompt word are input into the image generation model to obtain the first target image generated by the image generation model.

[0022] In the above scheme, by determining the first prompt word as the global prompt word corresponding to the image to be processed, and the second prompt word as the regional prompt word of the image region corresponding to the second image element, the image generation model is guided to control the second image element in the final generated target image to generate a high-quality target image at the expected position relative to the first image element, so that the generated target image presents the effect expected by the user. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart of an image generation method provided in an embodiment of the present invention;

[0025] Figure 2 A schematic diagram of an image generation interface provided in an embodiment of the present invention;

[0026] Figure 3 A schematic diagram of an edge image provided in an embodiment of the present invention;

[0027] Figure 4 A flowchart of another image generation method provided in an embodiment of the present invention;

[0028] Figure 5 A flowchart illustrating yet another image generation method provided in an embodiment of the present invention;

[0029] Figure 6This is a schematic diagram illustrating the application of an image generation method provided in an embodiment of the present invention;

[0030] Figure 7 A schematic diagram of another image generation interface provided in an embodiment of the present invention;

[0031] Figure 8 This is a schematic diagram of the structure of an image generation device provided in an embodiment of the present invention;

[0032] Figure 9 This is a schematic diagram of the structure of an electronic device provided in this embodiment. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the timing of the steps in the following method embodiments is only an example and not a strict limitation.

[0034] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0035] To address the problem of poor image quality in traditional image generation schemes, which fails to meet user expectations, this invention provides a novel image generation scheme. In this scheme, prompts corresponding to multiple image elements on an editing canvas are obtained, and each prompt is associated with its corresponding image region. This guides the image generation model to precisely control the image generation of each region based on the prompts, ensuring that the multiple image elements in the generated target image accurately reflect the effect on the editing canvas. This not only improves the quality of the generated target image but also ensures that the generated target image achieves the user's desired effect.

[0036] The following detailed description of some embodiments of the present invention is provided in conjunction with the accompanying drawings. Where there is no conflict between the embodiments, the following embodiments and features thereof can be combined with each other.

[0037] The image generation method provided in this embodiment of the invention can be executed by an electronic device, which can be a terminal device such as a PC, laptop, or smartphone, or a server. The server can be a physical server containing an independent host, a virtual server, or a cloud server or server cluster.

[0038] Figure 1 A flowchart of an image generation method provided in an embodiment of the present invention is shown below. Figure 1 As shown, the execution subject of this method can be an image generation device, and the method can include the following steps:

[0039] 101. Based on the editing operations on the first image element and the second image element on the editing canvas, generate an image to be processed containing the first image element and the second image element.

[0040] 102. Obtain the first prompt word and the second prompt word corresponding to the second image element.

[0041] 103. Based on the position information of the second image element on the image to be processed, perform region segmentation on the image to be processed to determine the image region corresponding to the second image element.

[0042] 104. Using the second prompt word as the region prompt word for the image region corresponding to the second image element, and using the first prompt word as the global prompt word for the image to be processed, generate the target prompt word.

[0043] 105. Input the image to be processed and the target prompt word into the image generation model to obtain the first target image generated by the image generation model.

[0044] When a user wants to add a second image element to a first image element to generate a target image containing both the first and second image elements, the user can create a new editing canvas in the image generation interface. The user can then add the first and second image elements to the editing canvas and perform editing operations on them, hoping that the added image elements will be accurately presented according to the editing effect on the editing canvas.

[0045] The first and second image elements can be any type of image element. For example, in an e-commerce scenario, the first image element could include a product image, and the second image element could include a background element added to the product image. In a film or television drama scenario, the first image element could include a scene image, and the second image element could include a character image added to the scene image. Alternatively, the first image element could include a landscape image, and the corresponding second image element could include a character image added to the landscape image.

[0046] Based on the above examples, it can be seen that, optionally, the first image element can be an image of an object / person that the user has taken or designed. The purpose of generating the target image is to add some image elements (called second image elements) to the first image element, and the second image elements can be selected from a pre-generated element library.

[0047] In practice, optionally, the first image element can be an image of a foreground object, and the second image element can be an image of a background object added to the foreground object; or, optionally, the first image element can be an image of a background object, and the second image element is an image of a foreground object added to the background image.

[0048] Additionally, when creating an editing canvas, you can also set its size. In practice, you can set the editing canvas size according to the desired size of the target image, and the size of the subsequently generated target image can be the same as the size of the editing canvas. For example, Figure 2 The image generation interface shown is an editing canvas created in this interface, which is a 9*16cm rectangular canvas.

[0049] After creating the editing canvas, you can add a first image element and a second image element to it. Users can load their own first image element into the editing canvas and edit its position and size. The image generation interface includes a material addition control. Users can directly trigger this control to select the corresponding second image element and add it to the editing canvas, or they can add a second image element by uploading a custom image and then edit its size and position within the editing canvas.

[0050] So, after the user completes the editing operation in the editing canvas, an image to be processed, containing the first image element and the second image element, can be generated first. In fact, the first image element and the second image element are located on different layers in the editing canvas; the two layers can be merged to generate the image to be processed.

[0051] It's important to note that the user's editing of the first and second image elements in the editing canvas only involves simple adjustments such as position and size; it does not alter their original visual features, such as color, brightness, and outline edges. Therefore, the generated image to be processed only presents the original outlines, sizes, and positional relationships of these two image elements. In contrast, the target image aims to present the desired visual features of these two image elements, especially the second image element. This means that in the target image, while retaining the aforementioned features, other visual features, such as lighting, color, and texture, can be further modified.

[0052] Therefore, in order for the image generation model to know the desired effect of the first and second image elements in the target image to be generated, the user also needs to provide a first cue word and a second cue word corresponding to the second image element. The second cue word describes the expected effect of the second image element in the target image. The first cue word actually describes the expected effect of the scene in the target image excluding the second image element.

[0053] For example, such as Figure 2 As shown, when a user wants to create a product background image with a specific background element (a pink flower) for a perfume bottle image, the first image element is the perfume bottle image, and the second image element is a flower. In practical applications, since a background element is being added to a user-provided product image, the product image, i.e., the first image element, can generally be a color image (such as an image taken by the user of the product, or a designed image). The user can then... Figure 2 The example upload control adds the first image element (a perfume bottle image) to the editing canvas. The second image element is often added by the user as needed. Figure 2 The illustrated material addition control is added from the element library, which can be used by many different users. Each user has different expectations for the effect of the same material in the corresponding target image. Therefore, optionally, each image element in the element library can be an edge image, that is, the second image element is an edge image, called the second edge image. Simply put, an edge image is a black and white image that only contains the edge outline of the corresponding element.

[0054] like Figure 2 As shown, optionally, the image generation interface can also provide an input box for the first prompt word and an input box for the second prompt word corresponding to the second image element, so that the user can enter the prompt word in the corresponding input box. For example, in the example above, the first prompt word is: In an outdoor desert garden, a translucent yellow perfume bottle is placed on a light-colored rock display stand, surrounded by lush cacti and succulents, and the second prompt word corresponding to the second image element is: A pink flower, green roots.

[0055] It is important to note that the first prompt word can include all image description information except for the second image element, while the second prompt word only includes the image description information corresponding to the second image element. In this way, the first prompt word can be used as a global prompt word or a basic prompt word, which determines the macroscopic appearance of the target image.

[0056] The methods for obtaining the first and second clue words are not based on... Figure 2 The illustrations provided are for illustrative purposes only; other methods will be introduced later and will not be elaborated upon here. After obtaining the first prompt word and the second prompt word corresponding to the second image element, the image to be processed can be segmented according to the coordinates of the second image element to determine the image region corresponding to the second image element.

[0057] In one optional embodiment, the image to be processed can be divided into grids based on the position information of the second image element on the image to be processed, so as to determine the image region corresponding to the second image element. For example, Figure 3 The image to be processed shown can be divided horizontally and vertically to determine the image region corresponding to the second image element. This image region can be represented by the coordinates of its vertices.

[0058] In practical applications, the first cue word is usually used to describe the overall expected visual effect in the target image, excluding the second image element, while the second cue word is usually only used to describe the expected visual effect of the second image element. This can lead to situations where the first cue word is very long and the second cue word is relatively short. If this happens, the second cue word will be overwhelmed by the first cue word, especially when the second cue word is appended to the first cue word and input into the image generation model. This overwhelming situation will be more pronounced, meaning that the image generation model may ignore parts of the second cue word when recognizing it. This can result in image quality issues in the target image generated based on the cue word recognition, such as inaccurate placement of the second image element, cross-contamination between the second image element and other visual content, and insufficient texture precision of the generated second image element.

[0059] Therefore, in this embodiment of the invention, in order to enable the image generation model to accurately identify the second prompt word corresponding to the second image element, before inputting the second prompt word into the image generation model, the image to be processed can be segmented into regions to determine the image region corresponding to the second image element. The second prompt word is bound to the image region corresponding to the second image element as a region prompt word, and the first prompt word corresponding to the first image element is used as a global prompt word for the image to be processed. This allows the image generation model to clearly understand the expected presentation effect corresponding to the image region corresponding to the second image element, thereby helping to improve the image generation quality of the first target image.

[0060] Specifically, the second prompt word can be used as the region prompt word for the image region corresponding to the second image element, and the first prompt word can be used as the global prompt word for the image to be processed. These two prompt words are then concatenated to generate the target prompt word. Finally, the image to be processed and the target prompt word are input into the image generation model to obtain the first target image generated by the image generation model.

[0061] In practical applications, a pre-set template for concatenating prompt words can be used to concatenate the first and second prompt words to generate the target prompt word. For example, the concatenation template could be in the following format: Global Prompt Word#Separator 1#Region Prompt Word 1#Separator 2#Region Prompt Word 2#Separator 2#Region Prompt Word 3. Here, Region Prompt Word 1, Region Prompt Word 2, and Region Prompt Word 3 are used to illustrate the scenario where there may be more than one second image element added in a practical application.

[0062] In practical applications, image generation models can be various large models capable of performing text-to-image and image-to-image tasks, such as the Stable Diffusion (SD) model and the Generative Adversarial Network (GAN) model.

[0063] This invention utilizes a large-scale text-to-image model based on a Generative Adversarial Network (GAN) architecture. This model consists of a generator and a discriminator. The generator is responsible for generating corresponding images based on the input text description, while the discriminator distinguishes the generated images from real images. The model is trained on a large-scale image-text pair dataset containing over one million images and their corresponding descriptive texts. During training, an optimization method based on adversarial loss is used, and additional perceptual loss is introduced to improve the quality of the generated images. The large-scale text-to-image model in this invention receives a processed target prompt word and the image to be processed as input, and can generate high-resolution (1024x1024 pixels), color-rich images as output.

[0064] This invention also utilizes a text-to-image model based on a diffusion process—the SD model. This model generates images through a reverse diffusion process. Specifically, noise is gradually added to the input data until it is completely randomized, and then the noise is gradually removed through a reverse process to restore the original image or generate a new image. In this embodiment, the SD model receives a processed target cue word and the image to be processed as input, for example, 'Global cue word: In an outdoor desert garden, a small transparent yellow perfume bottle is placed on a light-colored rock display stand, surrounded by lush cacti and succulents. #Separator1#Region cue word: A red rose, green roots', and can generate a high-quality image matching this description. During training, the model uses a large amount of image-text pair data and improves the quality of the generated images by optimizing a specific loss function.

[0065] In an optional embodiment, when generating the target prompt word, the weights corresponding to the first prompt word and the second prompt word can be set separately to increase the weight of the second prompt word. For example, the preset weight of the first prompt word is 0.5, and the preset weight of the second prompt word is 0.5.

[0066] The generated target prompt words can also carry the weights corresponding to the first prompt word and the second prompt word. This can further improve the image generation model's effective recognition of the second prompt word, thereby improving the quality of the generated target image.

[0067] In this embodiment of the invention, by determining the first prompt word as the global prompt word corresponding to the image to be processed, and the second prompt word as the regional prompt word of the image region corresponding to the second image element, the image generation model can be guided to control each image region to be presented according to the expected effect based on the prompt words corresponding to each image region. This can improve the generation quality of the target image and meet user expectations.

[0068] Furthermore, in an optional embodiment, to improve the quality of the target image generated by the image generation model, the user can also upload a style reference image during the actual generation of the first target image, so that the target image generated by the image generation model exhibits the style features of the style reference image. Specifically, a style adapter (IP-adapter) can be used with the image generation model to better incorporate the style features from the style reference image into the image generation process.

[0069] Specifically, in the process of generating the first target image, a style reference image can also be obtained. The style reference image, the image to be processed, and the target prompt words are input into the image generation model to obtain the first target image generated by the image generation model. The first target image presents the style features in the style reference image.

[0070] Figure 4 A flowchart of another image generation method provided in an embodiment of the present invention; as follows: Figure 4 As shown, the method may include the following steps:

[0071] 401. Based on the editing operations on the first image element and the second image element on the editing canvas, generate an image to be processed containing the first image element and the second image element.

[0072] 402. Obtain the first prompt word and the second prompt word corresponding to the second image element.

[0073] 403. Perform edge detection processing on the image to be processed to obtain the edge image corresponding to the image to be processed. The edge image contains the first edge image corresponding to the first image element and the second edge image corresponding to the second image element.

[0074] 404. Based on the position information of the second edge image on the image to be processed, perform region segmentation on the edge image corresponding to the image to be processed to determine the image region corresponding to the second image element.

[0075] 405. Using the second prompt word as the region prompt word for the image region corresponding to the second image element, and using the first prompt word as the global prompt word for the image to be processed, generate the target prompt word.

[0076] 406. Input the edge image and target prompt word corresponding to the image to be processed into the image generation model to obtain the first target image.

[0077] When generating the target image, an image to be processed containing the first and second image elements is first generated based on the editing operations performed on the first and second image elements on the editing canvas. Then, the first prompt word as a global prompt word and the second prompt word corresponding to the second image element are obtained.

[0078] Next, edge detection processing is performed on the image to be processed to obtain the edge image corresponding to the image to be processed. The edge image includes a first edge image corresponding to the first image element and a second edge image corresponding to the second image element.

[0079] In other words, both the first and second edge images are black and white line drawings, containing only the shape information corresponding to the first and second image elements. Compared to the richly colored original image, the edge images contain less information, making it easier for image generation models to learn and generate, especially under limited computing resources. Furthermore, using edge images helps image generation models better understand the position and shape of the first and second image elements. Since the image to be processed is the input image, the image generation model focuses more on the shape, i.e., the edge contour information, of the first and second image elements, and the generation effect comes from the corresponding prompt words. Therefore, using the aforementioned edge images not only reduces the computational load of the image generation model but also reduces excessive interference from visual features such as color in the original image to be processed. Edge detection methods such as Canny edge detection, convolutional neural networks, and deep learning models can be used to perform edge detection on the image to be processed to obtain the corresponding edge images.

[0080] After obtaining the edge image corresponding to the image to be processed, the edge image corresponding to the image to be processed is segmented into regions based on the position information of the second edge image on the image to be processed, so as to determine the image region corresponding to the second image element, that is, the image region corresponding to the second edge image.

[0081] Then, using the second prompt word as the region prompt word for the image region corresponding to the second image element, and using the first prompt word as the global prompt word for the image to be processed, a target prompt word is generated. The edge image corresponding to the image to be processed and the target prompt word are then input into the image generation model to obtain the first target image.

[0082] In other words, in this embodiment of the invention, the edge image input to the image generation model only includes the shape information corresponding to the first image element and the second image element, so as to guide the image generation model to generate the first target image by referring to the shape information corresponding to the first image element and the second image element in the edge image, as well as the target prompt word.

[0083] In this embodiment of the invention, edge detection processing is performed on the image to be processed to obtain the edge image corresponding to the image to be processed. Based on the position information of the second edge image on the image to be processed, the edge image corresponding to the image to be processed is segmented into regions to determine the image region corresponding to the second image element. A second prompt word is used as the region prompt word for the image region corresponding to the second image element, and a first prompt word is used as the global prompt word corresponding to the image to be processed to generate a target prompt word. The edge image corresponding to the image to be processed and the target prompt word are input into the image generation model to obtain the first target image. That is, the structural clues provided by the edge image and the specific prompt content corresponding to the target prompt word are used to guide the image generation process, making the generated target image result more controllable, thereby obtaining a high-quality first target image.

[0084] Since the edge image corresponding to the first image element in the edge image is a black and white line drawing, which only contains information such as the shape and position of the first image element, in order to make the first image element in the generated target image closer to the color information of the first image element (color image) added by the user on the editing canvas, that is, to more realistically restore the visual information of the first image element in focus, after obtaining the first target image, the cutout image corresponding to the first image element can be extracted from the image to be processed, and the cutout image can be superimposed on the first target image to obtain a higher quality second target image.

[0085] Figure 5 A flowchart of another image generation method provided in an embodiment of the present invention; as follows: Figure 5 As shown, based on the above embodiments, the method further includes the following steps:

[0086] 501. Extract the cutout image corresponding to the first image element from the image to be processed. The size of the cutout image is the same as the size of the image to be processed.

[0087] 502. Overlay the cut-out image onto the first target image to generate the second target image.

[0088] In the cutout image obtained by cutting out the image to be processed, the second image element in the image to be processed will actually be removed, and only the complete first image element will be included. This cutout image can be used as a layer and superimposed on the first target image of the same size, so that the first image element in the cutout image occludes the corresponding image area in the first target image, while other image areas in the first target image are unaffected.

[0089] The following embodiments illustrate the operation and interaction process on the image generation interface in this invention.

[0090] As mentioned above, when users add the first image element and the second image element to the editing canvas in the image generation interface, they can also enter the first prompt word and the second prompt word corresponding to the second image element in the image generation interface.

[0091] In an optional embodiment, two prompt word input boxes can be directly displayed in the image generation interface for the user to input the corresponding prompt words.

[0092] In another optional embodiment, to simplify the user's operation, the input method for the second prompt word can be as follows: In response to an editing operation on the second image element displayed on the editing canvas, a corresponding prompt word editing box is displayed following the second image element, and the prompt word editing box displays the initial prompt word corresponding to the second image element. The second prompt word is determined to be either the confirmed initial prompt word or a prompt word obtained after modifying the initial prompt word within the prompt word editing box. This allows different users to flexibly edit and modify the prompt words of the second image element in real time as needed.

[0093] like Figure 6 As shown, in the image generation interface, an input box for the first prompt word can be directly displayed, where the user enters the first prompt word corresponding to the first image element. In the editing canvas, after the user adds a second image element, either directly or after the user triggers a prompt word editing operation on the second image element, the corresponding prompt word editing box can be displayed following the second image element, for example... Figure 6 The prompt text editing box appears below the second image element and will always follow the second image element regardless of how the user moves it within the editing canvas.

[0094] Optionally, the second image element added to the element library may have a default prompt word pre-configured when added to the element library: an initial prompt word, thus enabling... Figure 6 In the above-mentioned prompt word editing box, the initial prompt word can be displayed. If the current user decides to use the initial prompt word, they can confirm it. If they want to modify it, they can directly modify it in the prompt word editing box. For example, the second image element is a flower illustrated with a black and white edge image as shown above. The initial prompt word is: red rose, green stem. The user can change it to: pink rose, green stem, with 4 green leaves on the stem.

[0095] Based on this, when adding a second image element to the element library, the following approach can be adopted: display the element library in the image generation interface; in response to the element addition operation for the element library, display the second image element uploaded by the user in the element addition interface, and display a prompt word editing box associated with the second image element, and save the initial prompt word corresponding to the second image element entered by the user in the prompt word editing box.

[0096] The image generation interface can include a display area for an element library, which may contain controls for adding image elements. Users can click these controls to trigger the element addition operation. Optionally, an element addition interface can be displayed on the image generation interface. This interface can overlap the editing canvas area, and is not limited to this; it can even be the same interface as the editing canvas. The element addition interface displays the second image element uploaded by the user. If the initially uploaded second image element is a color image, edge detection can be performed to generate a corresponding edge image, which is then stored in the element library. Based on this, a prompt word editing box is displayed in association with the second image element (which may be an edge image). The user enters the initial prompt word corresponding to the second image element in this prompt word editing box, and this is saved. Since the prompt word editing box and the second image element are displayed together, the user can remain in the element addition interface while editing the initial prompt word, making the operation convenient.

[0097] Similarly, regarding the method of obtaining the first prompt word, in one optional embodiment, the image generation interface includes an edit box for the first prompt word, where the user can enter the first prompt word. Alternatively, other methods can be used to obtain the first prompt word corresponding to the first image element, and this embodiment of the invention does not limit this method.

[0098] In one optional embodiment, in addition to displaying the prompt word editing box corresponding to the second image element in the editing canvas, multiple editing controls corresponding to the second image element, such as a delete control, can also be displayed in the editing canvas in accordance with the second image element.

[0099] The following is combined Figure 7 This illustrates a practical application process of an image generation method provided in an embodiment of the present invention. For example... Figure 7 As shown, the user wants to create a product background image with a specific background element (a pink flower) for a perfume bottle image. First, the user can create a new editing canvas in the provided image generation interface and set the editing canvas to a 9*16 rectangular canvas.

[0100] Then, the user can add a perfume bottle image to the editing canvas by triggering the upload control in the image generation interface, and enter a prompt word in the first prompt word input box. For example, the user's first prompt word might be: "In an outdoor desert garden, a small transparent yellow perfume bottle sits on a light-colored rock display stand, surrounded by lush cacti and succulents." Next, the user can add a black-and-white edge-image flower to the editing canvas by triggering the material addition control in the image generation interface, and a corresponding prompt word editing box will appear, displaying preset prompt words. For example, the displayed preset prompt word might be: "A red rose with green stems." The user defaults to the displayed preset prompt word and does not edit it; therefore, the preset prompt word corresponding to the black-and-white edge-image flower is set as the second prompt word.

[0101] Next, the user uploads a style reference image by triggering the reference image upload control in the image generation interface.

[0102] After the user finishes editing, the system can generate a new image containing both the perfume bottle and the flower with black and white edges, based on the editing operations on the perfume bottle image and the flower with black and white edges on the editing canvas. Then, it retrieves the first prompt word and the second prompt word corresponding to the flower with black and white edges.

[0103] Next, edge detection processing is performed on the image to be processed to obtain the edge image corresponding to the image to be processed. This edge image includes a first edge image corresponding to the perfume bottle image and a second edge image corresponding to a flower with black and white edges. Based on the position information of the second edge image corresponding to the flower with black and white edges on the image to be processed, region segmentation is performed on the edge image corresponding to the image to be processed to determine the image region corresponding to the flower with black and white edges. Using the second prompt word as the region prompt word for the image region corresponding to the flower with black and white edges, and using the first prompt word as the global prompt word for the image to be processed, a target prompt word is generated.

[0104] Then, the style reference image uploaded by the user is obtained, and the style reference image, the image to be processed, and the target prompt are input into the image generation model to obtain the first target image generated by the image generation model. The first target image presents the style features of the style reference image, and the image generation model includes a style adapter.

[0105] Finally, a cutout image corresponding to the perfume product image is extracted from the image to be processed. The size of the cutout image is the same as that of the image to be processed. The cutout image is then overlaid on the first target image to generate the second target image.

[0106] The specific implementation process involved in the embodiments of the present invention can be referred to the content of the above embodiments, and will not be repeated here.

[0107] The image generation apparatus of one or more embodiments of the present invention will be described in detail below. Those skilled in the art will understand that these apparatuses can all be configured using commercially available hardware components through the steps taught in this solution.

[0108] Figure 8 This is a schematic diagram of the structure of an image generation device provided in an embodiment of the present invention, such as... Figure 8 As shown, the device includes: an editing module 11, an acquisition module 12, a segmentation module 13, a generation module 14, and a processing module 15.

[0109] The editing module 11 is used to generate an image to be processed, which includes the first image element and the second image element, based on the editing operations on the first image element and the second image element on the editing canvas.

[0110] The acquisition module 12 is used to acquire the first prompt word and the second prompt word corresponding to the second image element.

[0111] The segmentation module 13 is used to perform region segmentation on the image to be processed based on the position information of the second image element on the image to be processed, so as to determine the image region corresponding to the second image element.

[0112] The generation module 14 is used to generate a target prompt word by using the second prompt word as the region prompt word for the image region corresponding to the second image element and the first prompt word as the global prompt word for the image to be processed.

[0113] The processing module 15 is used to input the image to be processed and the target prompt word into the image generation model to obtain the first target image generated by the image generation model.

[0114] Optionally, the segmentation module 13 is specifically used to: perform edge detection processing on the image to be processed to obtain an edge image corresponding to the image to be processed, wherein the edge image corresponding to the image to be processed includes a first edge image corresponding to the first image element and a second edge image corresponding to the second image element; and perform region segmentation on the edge image corresponding to the image to be processed according to the coordinates of the second edge image to determine the image region corresponding to the second image element.

[0115] Optionally, in the image to be processed, the first image element is a color image, and the second image element is the second edge image.

[0116] Optionally, the processing module 15 is specifically used to: input the edge image corresponding to the image to be processed and the target prompt word into the image generation model.

[0117] Optionally, the device may further include a matting module, which is specifically used to: matte out the matted image corresponding to the first image element from the image to be processed, wherein the size of the matted image is the same as the size of the image to be processed; and overlay the matted image onto the first target image to generate a second target image.

[0118] Optionally, the processing module 15 is specifically used to: acquire a style reference image; input the style reference image, the image to be processed, and the target prompt word into the image generation model to obtain a first target image generated by the image generation model, wherein the first target image presents the style features in the style reference image, and the image generation model includes a style adapter.

[0119] Optionally, the acquisition module 12 is specifically used to: in response to the editing operation of the second image element displayed on the editing canvas, display a corresponding prompt word editing box following the second image element, wherein the prompt word editing box displays the initial prompt word corresponding to the second image element; determine that the second prompt word is the confirmed initial prompt word or the prompt word obtained after modifying the initial prompt word in the prompt word editing box.

[0120] Optionally, the device may further include a material adding module, which is specifically used for: displaying an element library in the image generation interface; in response to an element adding operation for the element library, displaying the second image element uploaded by the user in the element adding interface, and displaying a prompt word editing box associated with the second image element; and saving the initial prompt word corresponding to the second image element entered by the user in the prompt word editing box.

[0121] Optionally, the first image element includes a product image, and the second image element includes a background element added to the product image.

[0122] Figure 8 The apparatus shown can perform the steps in the image generation method in the foregoing embodiments. For detailed execution process and technical effects, please refer to the description in the foregoing embodiments, which will not be repeated here.

[0123] This invention also provides an electronic device, such as... Figure 9 As shown, the electronic device may include: a processor 21, a memory 22, and a communication interface 23. The memory 22 stores executable code, which, when executed by the processor 21, enables the processor 21 to implement the image generation method as described in the foregoing embodiments.

[0124] In addition, embodiments of the present invention provide a non-transitory machine-readable storage medium storing executable code, which, when executed by a processor of an electronic device, enables the processor to at least implement the image generation method provided in the foregoing embodiments.

[0125] The invention also provides a computer program product comprising a computer program that, when executed by a processor, enables the processor to at least implement the image generation method provided in the foregoing embodiments.

[0126] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0127] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of a necessary general-purpose hardware platform, or by a combination of hardware and software. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a computer product. The present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image generation method, characterized in that, include: Based on the editing operations on the first image element and the second image element on the editing canvas, an image to be processed containing the first image element and the second image element is generated; Obtain the first prompt word and the second prompt word corresponding to the second image element; Based on the position information of the second image element on the image to be processed, the image to be processed is segmented into regions to determine the image region corresponding to the second image element; The second prompt word is used as the region prompt word for the image region corresponding to the second image element, and the first prompt word is used as the global prompt word for the image to be processed to generate a target prompt word. The image to be processed and the target prompt are input into the image generation model to obtain the first target image generated by the image generation model.

2. The method according to claim 1, characterized in that, The step of segmenting the image to be processed based on the coordinates of the second image element to determine the image region corresponding to the second image element includes: Edge detection processing is performed on the image to be processed to obtain an edge image corresponding to the image to be processed, wherein the edge image corresponding to the image to be processed includes a first edge image corresponding to the first image element and a second edge image corresponding to the second image element; Based on the coordinates of the second edge image, the edge image corresponding to the image to be processed is segmented to determine the image region corresponding to the second image element.

3. The method according to claim 2, characterized in that, In the image to be processed, the first image element is a color image, and the second image element is the second edge image.

4. The method according to claim 2, characterized in that, The step of inputting the image to be processed and the target prompt word into the image generation model includes: The edge image corresponding to the image to be processed and the target prompt word are input into the image generation model.

5. The method according to claim 4, characterized in that, The method further includes: Extract the cutout image corresponding to the first image element from the image to be processed, wherein the size of the cutout image is the same as the size of the image to be processed; The cut-out image is superimposed on the first target image to generate the second target image.

6. The method according to any one of claims 1-5, characterized in that, The step of inputting the image to be processed and the target prompt word into the image generation model to obtain the first target image generated by the image generation model includes: Obtain style reference images; The style reference image, the image to be processed, and the target prompt are input into the image generation model to obtain a first target image generated by the image generation model. The first target image presents the style features in the style reference image, and the image generation model includes a style adapter.

7. The method according to any one of claims 1-5, characterized in that, The acquisition of the second prompt word includes: In response to an editing operation on the second image element displayed on the editing canvas, a corresponding prompt word editing box is displayed following the second image element, and the prompt word editing box displays the initial prompt word corresponding to the second image element; The second prompt word is determined to be either the confirmed initial prompt word or a prompt word obtained after modifying the initial prompt word in the prompt word editing box.

8. The method according to claim 7, characterized in that, The editing canvas is located in the designated image generation interface, and the method further includes: The element library is displayed in the image generation interface; In response to the element addition operation for the element library, the second image element uploaded by the user is displayed in the element addition interface, and a prompt word editing box is displayed in association with the second image element; Save the initial prompt word corresponding to the second image element entered by the user in the prompt word edit box.

9. The method according to any one of claims 1-5, characterized in that, The first image element includes a product image, and the second image element includes a background element added to the product image.

10. An electronic device, characterized in that, include: The system includes a memory, a processor, and a communication interface; wherein the memory stores executable code, which, when executed by the processor, causes the processor to perform the image generation method as described in any one of claims 1 to 9.

11. A non-transitory machine-readable storage medium, characterized in that, The non-transitory machine-readable storage medium stores executable code that, when executed by a processor of an electronic device, causes the processor to perform the image generation method as described in any one of claims 1 to 9.

12. A computer program product, characterized in that, include: A computer program, when executed by a processor of an electronic device, causes the processor to perform the image generation method as described in any one of claims 1 to 9.

Images