Image generation method, apparatus and electronic device

Abstract

This invention provides an image generation method, apparatus, and electronic device, relating to the field of image technology. The method includes: in a free creation mode, acquiring a first creation instruction input by a user, the first creation instruction including a first creation prompt word; converting the first creation prompt word to obtain a parameter set, the parameter set including a creation description, sampling steps, and model weights; inputting the creation description, the sampling steps, and the model weights into a composition skeleton generation model to obtain a composition skeleton output by the composition skeleton generation model; and generating a first image based on the composition skeleton and the first creation prompt word. This invention generates a composition skeleton based on the parameter set obtained from the first creation prompt word, using the composition skeleton as an intermediate constraint to finally generate the image, which can avoid spatial distortion and thus improve the accuracy of image generation.

Description

Technical Field

[0001] This invention relates to the field of image technology, and more particularly to an image generation method, apparatus, and electronic device. Background Technology

[0002] Automatic image generation technology is one of the most groundbreaking research areas in the field of artificial intelligence in recent years. This technology, with the help of deep learning models, enables the automatic generation of images based on text descriptions.

[0003] In related technologies, text descriptions are typically input into an image generation model to obtain the image output by the model.

[0004] However, among the aforementioned technologies, generating images solely based on text descriptions can lead to spatial distortion in the image generation model, thereby reducing the accuracy of image generation. Summary of the Invention

[0005] This invention provides an image generation method, apparatus, and electronic device to address the shortcomings of existing technologies that reduce the accuracy of image generation.

[0006] This invention provides an image generation method, comprising the following steps.

[0007] In free creation mode, the user inputs a first creation instruction, which includes a first creation prompt word; The first creative prompt word is converted to obtain a parameter set, which includes creative description, sampling steps and model weights; The creation description, the number of sampling steps, and the model weights are input into the composition skeleton generation model to obtain the composition skeleton output by the composition skeleton generation model. Based on the composition skeleton and the first creation prompt, a first image is generated.

[0008] According to an image generation method provided by the present invention, the first creation instruction further includes a reference image; The step of inputting the creative description, the number of sampling steps, and the model weights into the composition skeleton generation model to obtain the composition skeleton output by the composition skeleton generation model includes: The creation description, the number of sampling steps, the model weights, and the reference image are input into the composition skeleton generation model to obtain the composition skeleton output by the composition skeleton generation model.

[0009] According to an image generation method provided by the present invention, generating a first image based on the compositional skeleton and the first creative prompt word includes: Generate at least two candidate sketches based on the composition skeleton; Receive the target candidate sketch selected by the user from the at least two candidate sketches; The target candidate sketch and the first creative prompt are input into the image generation model to obtain the first image output by the image generation model.

[0010] According to an image generation method provided by the present invention, the method further includes: In the free creation mode, the correspondence between the first information and the first task identifier in the entire image generation process is stored in the cloud server, and the first task identifier is added to the local history interface. The first information includes at least one of the following: the first creation prompt word, the parameter group, the reference image, the composition skeleton, the candidate sketch, and the first image.

[0011] According to an image generation method provided by the present invention, the method further includes: Upon receiving a user's selection operation on the first task identifier in the history interface, an information retrieval request is sent to the cloud server, the information retrieval request including the first task identifier. Receive the first information corresponding to the first task identifier sent by the cloud server; Load the first information into the editing interface; Receive the user's modification operation on the first information in the editing interface, and obtain the modified first information; A new image is generated based on the modified first information.

[0012] According to an image generation method provided by the present invention, the method further includes: In template replacement mode, a second creation instruction input by the user is obtained. The second creation instruction includes a second creation prompt, a background template, and a reference image including the main object. The background template is input into the semantic segmentation model to obtain the background preservation area mask and the subject replacement area mask output by the semantic segmentation model. Feature extraction is performed on the reference image including the main object to obtain the key features of the main object; The background preservation area mask, the subject replacement area mask, and the key features are input into the redrawing model to generate the subject latent image; A second image is generated based on the subject latent image and the second creative prompt.

[0013] According to an image generation method provided by the present invention, the semantic segmentation model further outputs template parameter information, which includes at least one of the following: illumination direction, depth information, and style label; The process of generating a second image based on the subject latent image and the second creative prompt includes: The latent image of the subject, the second creative prompt word, and the template parameter information are input into the image generation model to obtain the second image output by the image generation model.

[0014] According to an image generation method provided by the present invention, the method further includes: In the template replacement mode, the correspondence between the second information and the second task identifier in the entire image generation process is stored in the cloud server. The second information includes at least one of the following: the second creation prompt word, the background template, the reference image including the main object, the background retention area mask, the main replacement area mask, the main latent image, the template parameter information, and the second image.

[0015] The present invention also provides an image generation apparatus, comprising: The first acquisition unit is used to acquire a first creation instruction input by the user in the free creation mode, wherein the first creation instruction includes a first creation prompt word; The conversion unit is used to convert the first creation prompt word to obtain a parameter group, which includes creation description, sampling steps and model weights; The first generation unit is used to input the creation description, the number of sampling steps and the model weights into the composition skeleton generation model to obtain the composition skeleton output by the composition skeleton generation model; The second generation unit is used to generate the first image based on the composition skeleton and the first creation prompt.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image generation method as described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image generation method as described above.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the image generation method as described above.

[0019] The image generation method, apparatus, and electronic device provided by this invention, in a free creation mode, acquire a first creation instruction input by the user, including a first creation prompt word; transform the first creation prompt word to obtain a parameter set, which includes a creation description, sampling steps, and model weights; input the creation description, sampling steps, and model weights into a composition skeleton generation model to obtain a composition skeleton output by the composition skeleton generation model; and finally, based on the composition skeleton and the first creation prompt word, generate a first image. It can be seen that this invention generates a composition skeleton based on the parameter set obtained from the transformation of the first creation prompt word, and uses the composition skeleton as an intermediate constraint to finally generate the image, which can avoid spatial distortion and thus improve the accuracy of image generation. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is one of the flowcharts illustrating the image generation method provided in this embodiment of the invention.

[0022] Figure 2 This is the second flowchart illustrating the image generation method provided in this embodiment of the invention.

[0023] Figure 3 This is the third flowchart of the image generation method provided in the embodiments of the present invention.

[0024] Figure 4 This is the fourth flowchart of the image generation method provided in the embodiments of the present invention.

[0025] Figure 5 This is the fifth flowchart illustrating the image generation method provided in this embodiment of the invention.

[0026] Figure 6 This is a schematic diagram of the image generation device provided in an embodiment of the present invention.

[0027] Figure 7 This is a schematic diagram of the physical structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0029] The following is combined Figures 1 to 5 The present invention describes an image generation method. The entity executing this image generation method can be an electronic device such as a terminal, tablet computer, computer, or server, or it can be an image generation device installed in the electronic device. This image generation device can be implemented through software, hardware, or a combination of both.

[0030] Figure 1 This is one of the flowcharts illustrating the image generation method provided in this embodiment of the invention, such as... Figure 1 As shown, the image generation method includes the following steps: Step 101: In the free creation mode, obtain the first creation instruction input by the user, the first creation instruction including the first creation prompt word.

[0031] For example, this image generation method can be integrated into a mini-program or application. Users can select a creation mode in the mini-program or application. If a user wants to use the free creation mode, they can click the free creation mode button to put the mini-program or application into free creation mode. The user can then input a first creation instruction, including a first creation prompt word. When the electronic device receives the first creation instruction, it parses the first creation instruction to obtain the first creation prompt word. For example, the first creation prompt word is: Please generate a cartoon-style picture of a kitten playing on the grass, without drawing the sky.

[0032] Step 102: Convert the first creation prompt word to obtain a parameter group, which includes creation description, sampling steps and model weights.

[0033] For example, the first creative prompt is processed using natural language understanding and transformed into a set of parameters that can directly drive the composition skeleton generation model. The parameter set includes a creative description, sampling steps, and model weights. The sampling steps refer to the number of iterations required for the composition skeleton generation model to generate the skeleton, and the model weights refer to a weight file selected based on the style expressed in the first creative prompt. Taking the first creative prompt "Please generate a cartoon-style picture of a kitten playing on the grass, without drawing the sky" as an example, the creative description can include positive and negative descriptions. A positive description could be: cartoon style, a kitten playing on lush green grass, bright colors, and simple lines. A negative description could be: exclude the sky and clouds; the sampling steps are 25; and the model weights should be a weight file that is good at cartoon styles.

[0034] Step 103: Input the creation description, the number of sampling steps and the model weights into the composition skeleton generation model to obtain the composition skeleton output by the composition skeleton generation model.

[0035] For example, after inputting the creative description, sampling steps, and model weights into the composition skeleton generation model, the model drives the generation process according to the parameter set: First, the model weights, as the core style and knowledge base, are loaded, determining the basic visual language and structural rules of the generation (such as exaggerated proportions and dynamic lines in a cartoon style); then, positive descriptions in the creative description (such as a cartoon style, a kitten playing in a green grass) guide the composition skeleton generation model to construct the subject, scene, and style elements, while negative descriptions (such as excluding the sky and clouds) constrain the sampling space to avoid the appearance of irrelevant content; during the generation process, the sampling steps (such as 25 steps) control the number of iterations for optimization. In each step, the composition skeleton generation model gradually refines the transformation from random noise to a clear outline, continuously adjusting the relationship between lines, shapes, and layout, and finally outputs a composition skeleton that meets the description requirements and includes the positions, proportions, and dynamic relationships of the main elements, providing a foundation for subsequent detailed drawing or coloring stages. For example, the composition skeleton can be a Canny edge map or an OpenPose pose map.

[0036] It should be noted that the composition skeleton generation model can be trained based on creative description samples and corresponding composition skeleton labels. The specific training process is as follows: input the creative description samples into the initial composition skeleton generation model to obtain the predicted composition skeleton output by the initial composition skeleton generation model; construct a loss function based on the predicted composition skeleton and composition skeleton labels; iteratively update the model parameters of the initial composition skeleton generation model based on the loss function until the convergence condition is reached, and finally obtain the trained composition skeleton generation model.

[0037] Step 104: Generate the first image based on the composition skeleton and the first creation prompt.

[0038] For example, the composition skeleton and the first creative prompt can be input into a pre-trained image generation model. This allows the image generation model to use the composition skeleton as an intermediate constraint, based on the structured spatial layout and dynamic relationships provided by the composition skeleton, and to deeply integrate the style and subject details defined by the first creative prompt. This results in an efficient and accurate generation of a first image that conforms to the composition logic of the picture and fully meets the requirements of the text description.

[0039] The image generation method provided by this invention, in a free creation mode, obtains a first creation instruction input by the user, including a first creation prompt word. The first creation prompt word is then transformed to obtain a parameter set, which includes a creation description, sampling steps, and model weights. The creation description, sampling steps, and model weights are input into a composition skeleton generation model to obtain a composition skeleton output by the model. Finally, based on the composition skeleton and the first creation prompt word, a first image is generated. It can be seen that this invention generates a composition skeleton based on the parameter set obtained from the transformation of the first creation prompt word, and uses the composition skeleton as an intermediate constraint to finally generate the image, which can avoid spatial distortion and thus improve the accuracy of image generation.

[0040] In one embodiment, the first creation instruction further includes a reference diagram; step 103 above inputs the creation description, the number of sampling steps, and the model weights into the composition skeleton generation model to obtain the composition skeleton output by the composition skeleton generation model, which can be implemented in the following ways: The creation description, the number of sampling steps, the model weights, and the reference image are input into the composition skeleton generation model to obtain the composition skeleton output by the composition skeleton generation model.

[0041] For example, the reference image can be a user-uploaded image used for style or composition constraints, which includes a creation description, sampling steps, model weights, and references used for style or composition constraints. Figure 1After inputting the composition skeleton generation model, the model drives the generation process based on the parameter set: First, the model weights, as the core style and knowledge base, are loaded, determining the basic visual language and structural rules of the generation (such as exaggerated proportions and dynamic lines in a cartoon style); then, positive descriptions in the creative description (such as a cartoon style, a kitten playing in a green grass) guide the composition skeleton generation model to construct the subject, scene, and style elements, while negative descriptions (such as excluding the sky and clouds) constrain the sampling space to avoid the appearance of irrelevant content; at the same time, the input reference image provides the composition skeleton generation model with specific visual references in terms of style strokes, color tendencies, or overall composition, enabling it to more accurately align with the reference in style or composition. Throughout the generation process, the number of sampling steps (such as 25 steps) controls the number of iterations for optimization. In each step, the composition skeleton generation model gradually refines the transformation from random noise to clear outlines, continuously adjusting the relationship between lines, shapes, and layouts, and finally outputs a composition skeleton that meets the description requirements and includes the positions, proportions, and dynamic relationships of the main elements.

[0042] In this embodiment, the reference image uploaded by the user is also input into the composition skeleton generation model as a style or composition constraint, so that the final generated composition skeleton is more accurate.

[0043] In one embodiment, step 104 above generates a first image based on the composition skeleton and the first creation prompt, which can be implemented in the following way: At least two candidate sketches are generated based on the composition skeleton; a target candidate sketch selected by the user from the at least two candidate sketches is received; the target candidate sketch and the first creation prompt are input into the image generation model to obtain the first image output by the image generation model.

[0044] For example, multiple low-resolution candidate sketches can be generated based on the composition skeleton using image generation technology for users to preview and select. The candidate sketches that meet the requirements selected by the user are determined as target candidate sketches. The target candidate sketches are used as the composition locked by the user. The target candidate sketches and the first creative prompt are input into a pre-trained image generation model. The image generation model adjusts the target candidate sketches based on the first creative prompt and finally outputs the first image.

[0045] It should be noted that when the image generation model outputs the first image, it also outputs a random seed corresponding to the first image. The random seed covers the entire processing logic of the image generation model when generating the first image.

[0046] In this embodiment, multiple candidate sketches generated based on the composition skeleton are displayed to the user, allowing the user to select a target candidate sketch that meets the requirements. Then, the image generation model generates a first image based on the target candidate sketch and the first creative prompt, thereby making the generated first image more in line with the user's requirements and further improving the accuracy of image generation.

[0047] In one embodiment, Figure 2 This is a second schematic flowchart of the image generation method provided in this embodiment of the invention, as shown below. Figure 2 As shown, after step 104 above, the image generation method further includes the following steps: Step 105: In the free creation mode, the correspondence between the first information and the first task identifier in the entire image generation process is stored in the cloud server, and the first task identifier is added to the local history interface. The first information includes at least one of the following: the first creation prompt word, the parameter group, the reference image, the composition skeleton, the candidate sketch, and the first image.

[0048] For example, in the free creation mode, a unique first task identifier is generated for this image generation task, and a correspondence is established between the first task identifier and the first information generated throughout the entire process of this image generation task. This correspondence is stored in the cloud server. The first information covers some or all of the key data from the initial input to the final output, including at least one of the following: the first creative prompt word to start the image generation task, the parameter group that drives the generation, the reference image used for guidance, the composition skeleton produced in the middle, multiple candidate sketches for the user to choose from, and the final generated first image. At the same time, the first task identifier is also added to the user's local history interface, usually presented in the form of an interactive item, so that the user can conveniently trace and manage the task history locally, and achieve complete traceability of the creative process and persistent management of task data by relying on cloud storage.

[0049] It should be noted that the first information may also include a random seed, which facilitates secondary creation based on the random seed. This invention does not limit this.

[0050] In this embodiment, the correspondence between the first information and the first task identifier in the entire image generation process under the free creation mode is stored on the cloud server, realizing the complete traceability of the creation process and the persistent management of task data by relying on cloud storage, and solving the problem of data loss caused by local caching limitations.

[0051] In one embodiment, Figure 3 This is the third flowchart illustrating the image generation method provided in this embodiment of the invention, as shown below. Figure 3As shown, after step 105 above, the image generation method further includes the following steps: Step 301: Upon receiving a user's selection operation on the first task identifier in the history interface, send an information retrieval request to the cloud server, the information retrieval request including the first task identifier.

[0052] For example, when it is detected that a user has performed a selection operation (such as clicking or touching) on ​​the first task identifier corresponding to an image generation task in the history interface, a remote data retrieval process will be triggered based on this selection operation: a structured information retrieval request will be automatically sent to the cloud server, which includes the first task identifier selected by the user.

[0053] Step 302: Receive the first information corresponding to the first task identifier sent by the cloud server.

[0054] For example, after receiving an information retrieval request, the cloud server will quickly retrieve and verify the corresponding first information (which covers the entire chain of information for this image generation task, from the first creation prompt, parameter group, reference image, composition skeleton, candidate sketch to the final first image) in its stored association mapping based on the first task identifier carried in the information retrieval request, and then securely return this first information to the local electronic device.

[0055] Step 303: Load the first information into the editing interface.

[0056] For example, upon receiving the first information corresponding to the first task identifier, the first information is loaded into the editing interface for the user to edit.

[0057] Step 304: Receive the user's modification operation on the first information in the editing interface, and obtain the modified first information.

[0058] For example, users can modify the displayed first information in the editing interface based on their secondary creation needs to obtain the modified first information. For instance, users can replace the reference image included in the first information (such as replacing the first image in the first information with the reference image), while keeping other information unchanged.

[0059] Step 305: Generate a new image based on the modified first information.

[0060] For example, if the modified first information only includes the first creative prompt, parameter group, and modified reference image, then the modified reference image, the creative description included in the parameter group, the number of sampling steps, and the model weights are input into the composition skeleton generation model to obtain a new composition skeleton output by the composition skeleton generation model. Then, a new image for secondary creation is generated based on the new composition skeleton and the first creative prompt. The specific steps required for secondary creation are determined by the content included in the modified first information. The above-mentioned corresponding steps can be performed based on the content included in the modified first information, without performing extra steps. For example, if the modified first information includes the modified composition skeleton, then a new image is directly generated based on the modified composition skeleton and the first creative prompt, without performing the step of generating the composition skeleton.

[0061] In this embodiment, in the free creation mode, the first information corresponding to the first task identifier can be loaded into the editing interface, so that the user can change the content of the first information in the editing interface, and then generate a new image based on the changed first information to complete the secondary creation and realize the efficient reuse of creative assets.

[0062] In one embodiment, Figure 4 This is the fourth flowchart illustrating the image generation method provided in this embodiment of the invention, as shown below. Figure 4 As shown, the image generation method also includes the following steps: Step 401: In template replacement mode, obtain the second creation instruction input by the user. The second creation instruction includes a second creation prompt, a background template, and a reference image including the main object.

[0063] For example, if a user wants to use the template replacement mode, they can click the template replacement mode button to put the mini-program or application into template replacement mode. The user can then input a second creation instruction, including a second creation prompt, a background template, and a reference image including the main object. When the electronic device receives the second creation instruction, it parses the instruction to obtain the second creation prompt, the background template, and the reference image including the main object. For example, the second creation prompt might be: "Please replace the main object in the reference image in the background template." The main object can be a human figure or other objects, and this invention does not limit this.

[0064] Step 402: Input the background template into the semantic segmentation model to obtain the background preservation area mask and the subject replacement area mask output by the semantic segmentation model.

[0065] For example, the obtained background template is input into the semantic segmentation model. The semantic segmentation model performs pixel-level semantic parsing and region division on the background template and outputs two key structured masks: one is the background preservation area mask, which can accurately mark the continuous areas in the background template that need to be preserved and can be used as the background of the new composite image (such as the sky, distant mountains, static streets, etc.) in the form of a binary image; the other is the subject replacement area mask, which also clearly defines the area range in the background template that can be reserved for the placement of the subject object in the form of a binary image (such as a blank area in the center of the image, a clear object outline space, or a person's portrait placeholder area).

[0066] Step 403: Extract features from the reference image including the main object to obtain the key features of the main object.

[0067] For example, a reference image including the main object is input into a feature extraction model for feature extraction to obtain the key features of the main object. For example, if the main object is a portrait, the key features are the facial and body features of the portrait.

[0068] Step 404: Input the background preservation area mask, the subject replacement area mask, and the key features into the redrawing model to generate the subject latent image.

[0069] For example, the background-preserving region mask and the subject replacement region mask generated by the semantic segmentation model, along with the extracted key features of the subject object, are input into the redrawing model (which can be a conditional generation model based on diffusion or GAN architecture). The redrawing model uses the subject replacement region mask as a spatial guide, adapting and fusing the subject visual information represented by the key features into the target area represented by the subject replacement region mask. At the same time, it follows the constraints of the background-preserving region mask to ensure that the background content is not destroyed. Through multiple rounds of iteration, it generates a latent image of the subject that is consistent with the background in terms of lighting, perspective, and style, instead of directly pasting the pixels of the subject object. This latent image of the subject is an intermediate visual representation that is embedded in the template background and has correct geometric and semantic alignment, preparing for subsequent fine-grained fusion and rendering output.

[0070] Step 405: Generate a second image based on the subject latent image and the second creative prompt.

[0071] For example, the latent image of the subject and the second creative prompt are input into the image generation model. The image generation model uses the latent image of the subject as the basis for the predetermined spatial layout and content, deeply integrates the new visual features and style instructions specified by the second creative prompt, and performs pixel-level decoding, fine redrawing and global coordination optimization of the latent image of the subject through a complete generation process. The model focuses on optimizing the pixel blending and color transfer of the subject object's edges to ensure that the subject object and the background are consistent in terms of tone, granularity and lighting logic. Finally, it outputs a high-quality second image in which the subject object and the background are highly coordinated and fully meet the requirements of the second creative prompt.

[0072] In this embodiment, in template replacement mode, the user inputs a second creation instruction including a second creation prompt word, a background template, and a reference image including the main object. The background template is input into a semantic segmentation model to obtain a background retention area mask and a main object replacement area mask output by the semantic segmentation model. Feature extraction is performed on the reference image including the main object to obtain the key features of the main object. The background retention area mask, the main object replacement area mask, and the key features are input into a redrawing model to generate a latent image of the main object. Finally, based on the latent image of the main object and the second creation prompt word, a second image is generated. As can be seen, this invention uses background preservation area masks and subject replacement area masks as intermediate constraints to generate the final image, which can avoid spatial distortion and thus improve the accuracy of image generation. In addition, by introducing background preservation area masks, subject replacement area masks, and feature extraction of the subject object, it not only preserves the high quality and stylization of the background template, but also realizes the personalized implantation of the subject object, solving the problems of heavy processing traces caused by traditional simple compositing techniques and uncontrollable background caused by end-to-end generation. Furthermore, this invention presents only a minimalist "select template-upload image-generate image" interaction logic on the front end, taking into account both automation efficiency and human personalization needs, enabling users to complete creative image production at a near-professional retoucher level within a mini-program or application, thereby lowering the threshold for image creation and truly serving the creative scenarios of a wide range of social media users.

[0073] In one embodiment, the semantic segmentation model also outputs template parameter information, which includes at least one of the following: illumination direction, depth information, and style tag; step 405 above generates a second image based on the subject latent image and the second creative prompt word, which can be implemented in the following ways: The latent image of the subject, the second creative prompt word, and the template parameter information are input into the image generation model to obtain the second image output by the image generation model.

[0074] For example, the latent image of the subject and the second creative prompt are input into the image generation model. At the same time, the lighting direction, depth information and style tags extracted from the background template are used as additional input conditions. The image generation model uses the latent image of the subject as the basis for the spatial layout and content, and deeply integrates the new visual features and style instructions specified by the second creative prompt. During the generation process, it strictly follows the lighting direction extracted from the background template to unify the global lighting logic, simulates the blurring transition of the foreground, middle ground and background based on the depth information, and refers to the style tags to ensure the overall style consistency of the image. Through the complete generation process, the latent image of the subject is decoded at the pixel level, finely redrawn and globally coordinated and optimized. The focus is on optimizing the pixel blending and color transfer of the subject's edges to ensure that the subject and background are completely integrated in terms of tone, graininess, lighting logic and style expression. Finally, a high-quality second image is output, in which the subject and background are highly coordinated and fully meet the requirements of the second creative prompt.

[0075] In this embodiment, a second image is generated based on the latent image of the subject, the second creative prompt, and the template parameter information. This avoids problems such as harsh edge blending, inconsistent lighting direction with the background template, and disproportionate ratio in the generated image, thereby further improving the accuracy of the generated image.

[0076] In one embodiment, Figure 5 This is the fifth flowchart of the image generation method provided in this embodiment of the invention. After step 405 above, the image generation method further includes the following steps: Step 406: In the template replacement mode, the correspondence between the second information and the second task identifier in the entire image generation process is stored in the cloud server. The second information includes at least one of the following: the second creation prompt word, the background template, the reference image including the main object, the background retention area mask, the main replacement area mask, the main latent image, the template parameter information, and the second image.

[0077] For example, in template replacement mode, a unique second task identifier is generated for this image generation task, and a correspondence is established between the second task identifier and the second information generated throughout the entire process of this image generation task. This correspondence is stored in the cloud server. The second information covers some or all of the key data from the initial input to the final output, including at least one of the following: the second creative prompt word to start the image generation task, the background template and the reference image including the subject, the background retention area mask of the intermediate output, the subject replacement area mask, the subject latent image and template parameter information, and the final generated second image. At the same time, the second task identifier is also added to the user's local history interface, usually presented in the form of an interactive item, so that the user can conveniently trace and manage the task history locally, and achieve complete traceability of the creative process and persistent management of task data by relying on cloud storage.

[0078] In this embodiment, the correspondence between the second information and the second task identifier in the entire image generation process under the template replacement mode is stored in the cloud server, realizing the complete traceability of the creation process and the persistent management of task data by relying on cloud storage, and solving the problem of data loss caused by local caching limitations.

[0079] In one embodiment, in template replacement mode, the image generation method further includes the following steps: Upon receiving the user's selection of the second task identifier in the history interface, a target information retrieval request is sent to the cloud server. The target information retrieval request includes the second task identifier. The second information corresponding to the second task identifier sent by the cloud server is received, and the second information is loaded into the editing interface. The user's modification operation on the second information in the editing interface is received, and the modified second information is obtained. A secondary creation image is generated based on the modified second information.

[0080] For example, when a user is detected to have performed a selection operation (such as clicking or touching) on ​​the second task identifier corresponding to an image generation task in the history interface, a remote data retrieval process is triggered based on this selection operation: a structured target information retrieval request is automatically sent to the cloud server, which includes the second task identifier selected by the user. After receiving the target information retrieval request, the cloud server quickly retrieves and verifies the corresponding second information (which covers the entire chain of information for this image generation task, from the second creation prompt, background template, reference image including the subject, background retention area mask, subject replacement area mask, subject latent image, template parameter information to the final second image) based on the second task identifier carried in the target information retrieval request in its stored association mapping. Then, this second information is securely returned to the local electronic device. Upon receiving the second information corresponding to the second task identifier, the second information is loaded into the editing interface for user editing. Users can modify the displayed second information based on their creative needs to obtain the modified second information. For example, users can replace the background template included in the second information while keeping other information unchanged. Finally, the corresponding steps described above can be performed based on the content included in the modified second information, without needing to perform redundant steps. For example, if the modified second information includes a second creative prompt, a reference image of the main object, and a modified background template, then steps similar to steps 401 to 405 described above are re-executed based on the modified second information to finally obtain the secondary creative image. If the modified second information only includes a second creative prompt and a modified latent image of the main object, then the secondary creative image is directly generated based on the modified latent image of the main object and the second creative prompt, without needing to repeat the step of generating the latent image of the main object.

[0081] In this embodiment, in template replacement mode, the second information corresponding to the second task identifier can be loaded into the editing interface, so that the user can change the content of the second information in the editing interface, and then generate a new image based on the changed second information to complete the secondary creation and realize the efficient reuse of creative assets.

[0082] The image generation apparatus provided by the present invention will be described below. The image generation apparatus described below can be referred to in correspondence with the image generation method described above.

[0083] Figure 6 This is a schematic diagram of the structure of the image generation device provided in an embodiment of the present invention, as shown below. Figure 6 As shown, the image generation device 600 includes a first acquisition unit 601, a conversion unit 602, a first generation unit 603, and a second generation unit 604; wherein: The first acquisition unit 601 is used to acquire a first creation instruction input by the user in the free creation mode, wherein the first creation instruction includes a first creation prompt word. The conversion unit 602 is used to convert the first creation prompt word to obtain a parameter group, the parameter group including creation description, sampling steps and model weights; The first generation unit 603 is used to input the creation description, the number of sampling steps and the model weights into the composition skeleton generation model to obtain the composition skeleton output by the composition skeleton generation model. The second generation unit 604 is used to generate a first image based on the composition skeleton and the first creation prompt.

[0084] The image generation apparatus provided by this invention, in free creation mode, acquires a first creation instruction input by the user, including a first creation prompt word; transforms the first creation prompt word to obtain a parameter set, which includes a creation description, sampling steps, and model weights; inputs the creation description, sampling steps, and model weights into a composition skeleton generation model to obtain a composition skeleton output by the model; and finally, based on the composition skeleton and the first creation prompt word, generates a first image. It can be seen that this invention generates a composition skeleton based on the parameter set obtained from the transformation of the first creation prompt word, and uses the composition skeleton as an intermediate constraint to finally generate the image, which can avoid spatial distortion and thus improve the accuracy of image generation.

[0085] Based on any of the above embodiments, the first creation instruction further includes a reference figure; the first generation unit 603 is specifically used for: The creation description, the number of sampling steps, the model weights, and the reference image are input into the composition skeleton generation model to obtain the composition skeleton output by the composition skeleton generation model.

[0086] Based on any of the above embodiments, the second generation unit 604 is specifically used for: Generate at least two candidate sketches based on the composition skeleton; Receive the target candidate sketch selected by the user from the at least two candidate sketches; The target candidate sketch and the first creative prompt are input into the image generation model to obtain the first image output by the image generation model.

[0087] Based on any of the above embodiments, the image generation apparatus 600 further includes: The first storage unit is used to store the correspondence between the first information and the first task identifier in the entire image generation process to the cloud server in the free creation mode, and to add the first task identifier to the local history interface. The first information includes at least one of the following: the first creation prompt word, the parameter group, the reference image, the composition skeleton, the candidate sketch, and the first image.

[0088] Based on any of the above embodiments, the image generation apparatus 600 further includes: The sending unit is configured to send an information retrieval request to the cloud server when it receives a user's selection operation on the first task identifier in the history interface, the information retrieval request including the first task identifier; The receiving unit is configured to receive the first information corresponding to the first task identifier sent by the cloud server; The loading unit is used to load the first information into the editing interface; The modification unit is used to receive the user's modification operation on the first information in the editing interface and obtain the modified first information. The third generation unit is used to generate a new image based on the modified first information.

[0089] Based on any of the above embodiments, the image generation apparatus 600 further includes: The second acquisition unit is used to acquire a second creation instruction input by the user in the template replacement mode. The second creation instruction includes a second creation prompt, a background template, and a reference image including the main object. The segmentation unit is used to input the background template into the semantic segmentation model to obtain the background preservation area mask and the subject replacement area mask output by the semantic segmentation model. The extraction unit is used to extract features from the reference image including the main object to obtain the key features of the main object; The redrawing unit is used to input the background preservation area mask, the subject replacement area mask, and the key features into the redrawing model to generate the subject latent image; The fourth generation unit is used to generate a second image based on the subject latent image and the second creative prompt.

[0090] Based on any of the above embodiments, the semantic segmentation model also outputs template parameter information, which includes at least one of the following: illumination direction, depth information, and style tag; The fourth generation unit is specifically used for: The latent image of the subject, the second creative prompt word, and the template parameter information are input into the image generation model to obtain the second image output by the image generation model.

[0091] Based on any of the above embodiments, the image generation apparatus 600 further includes: In the template replacement mode, the correspondence between the second information and the second task identifier in the entire image generation process is stored in the cloud server. The second information includes at least one of the following: the second creation prompt word, the background template, the reference image including the main object, the background retention area mask, the main replacement area mask, the main latent image, the template parameter information, and the second image.

[0092] Figure 7 This is a schematic diagram of the physical structure of the electronic device provided in the embodiments of the present invention, such as... Figure 7 As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute an image generation method, the method including: in a free creation mode, acquiring a first creation instruction input by the user, the first creation instruction including a first creation prompt word; The first creative prompt word is converted to obtain a parameter set, which includes creative description, sampling steps and model weights; The creation description, the number of sampling steps, and the model weights are input into the composition skeleton generation model to obtain the composition skeleton output by the composition skeleton generation model. Based on the composition skeleton and the first creation prompt, a first image is generated.

[0093] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0094] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, the computer program being executed by a processor, the computer being able to execute the image generation method provided by the above methods, the method including: in a free creation mode, obtaining a first creation instruction input by a user, the first creation instruction including a first creation prompt word; The first creative prompt word is converted to obtain a parameter set, which includes creative description, sampling steps and model weights; The creation description, the number of sampling steps, and the model weights are input into the composition skeleton generation model to obtain the composition skeleton output by the composition skeleton generation model. Based on the composition skeleton and the first creation prompt, a first image is generated.

[0095] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the image generation method provided by the above methods, the method comprising: in a free creation mode, acquiring a first creation instruction input by a user, the first creation instruction including a first creation prompt word; The first creative prompt word is converted to obtain a parameter set, which includes creative description, sampling steps and model weights; The creation description, the number of sampling steps, and the model weights are input into the composition skeleton generation model to obtain the composition skeleton output by the composition skeleton generation model. Based on the composition skeleton and the first creation prompt, a first image is generated.

[0096] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0097] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image generation method, characterized in that, include: In free creation mode, the user inputs a first creation instruction, which includes a first creation prompt word; The first creative prompt word is converted to obtain a parameter set, which includes creative description, sampling steps and model weights; The creation description, the number of sampling steps, and the model weights are input into the composition skeleton generation model to obtain the composition skeleton output by the composition skeleton generation model. Based on the composition skeleton and the first creation prompt, a first image is generated.

2. The image generation method according to claim 1, characterized in that, The first creation instruction also includes a reference image; The step of inputting the creative description, the number of sampling steps, and the model weights into the composition skeleton generation model to obtain the composition skeleton output by the composition skeleton generation model includes: The creation description, the number of sampling steps, the model weights, and the reference image are input into the composition skeleton generation model to obtain the composition skeleton output by the composition skeleton generation model.

3. The image generation method according to claim 2, characterized in that, The process of generating a first image based on the composition skeleton and the first creative prompt includes: Generate at least two candidate sketches based on the composition skeleton; Receive the target candidate sketch selected by the user from the at least two candidate sketches; The target candidate sketch and the first creative prompt are input into the image generation model to obtain the first image output by the image generation model.

4. The image generation method according to claim 3, characterized in that, The method further includes: In the free creation mode, the correspondence between the first information and the first task identifier in the entire image generation process is stored in the cloud server, and the first task identifier is added to the local history interface. The first information includes at least one of the following: the first creation prompt word, the parameter group, the reference image, the composition skeleton, the candidate sketch, and the first image.

5. The image generation method according to claim 4, characterized in that, The method further includes: Upon receiving a user's selection operation on the first task identifier in the history interface, an information retrieval request is sent to the cloud server, the information retrieval request including the first task identifier. Receive the first information corresponding to the first task identifier sent by the cloud server; Load the first information into the editing interface; Receive the user's modification operation on the first information in the editing interface, and obtain the modified first information; A new image is generated based on the modified first information.

6. The image generation method according to claim 1, characterized in that, The method further includes: In template replacement mode, a second creation instruction input by the user is obtained. The second creation instruction includes a second creation prompt, a background template, and a reference image including the main object. The background template is input into the semantic segmentation model to obtain the background preservation area mask and the subject replacement area mask output by the semantic segmentation model. Feature extraction is performed on the reference image including the main object to obtain the key features of the main object; The background preservation area mask, the subject replacement area mask, and the key features are input into the redrawing model to generate the subject latent image; A second image is generated based on the subject latent image and the second creative prompt.

7. The image generation method according to claim 6, characterized in that, The semantic segmentation model also outputs template parameter information, which includes at least one of the following: illumination direction, depth information, and style tag; The process of generating a second image based on the subject latent image and the second creative prompt includes: The latent image of the subject, the second creative prompt word, and the template parameter information are input into the image generation model to obtain the second image output by the image generation model.

8. The image generation method according to claim 7, characterized in that, The method further includes: In the template replacement mode, the correspondence between the second information and the second task identifier in the entire image generation process is stored in the cloud server. The second information includes at least one of the following: the second creation prompt word, the background template, the reference image including the main object, the background retention area mask, the main replacement area mask, the main latent image, the template parameter information, and the second image.

9. An image generation apparatus, characterized in that, include: The first acquisition unit is used to acquire a first creation instruction input by the user in the free creation mode, wherein the first creation instruction includes a first creation prompt word; The conversion unit is used to convert the first creation prompt word to obtain a parameter group, which includes creation description, sampling steps and model weights; The first generation unit is used to input the creation description, the number of sampling steps and the model weights into the composition skeleton generation model to obtain the composition skeleton output by the composition skeleton generation model; The second generation unit is used to generate the first image based on the composition skeleton and the first creation prompt.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the image generation method as described in any one of claims 1 to 8.

Images