Image generation method, device, equipment and storage medium
Through a layered controllable layout model, the target picture is generated using the backbone and side branch generation models, the problem of uncontrollable image content layout in the existing technology is solved, and the layout controllability and quality improvement of image generation is achieved.
Patent Information
- Application Number
- CN202410153823.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-02
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-02-02
AI Technical Summary
The layout of the image content generated by the existing text-generating image technology based on diffusion model is uncontrollable and lacks controllability.
By obtaining the image generation description and location information of the target area box, using a hierarchical controllable layout model, including the backbone generation model and the side branch generation model, training the side branch generation model to generate layout generation conditions, and generating the target picture through the backbone generation model to ensure that Controllability of image layout.
The layout controllability of image generation is realized, the controllability and quality of image generation is improved, and the specific needs of users for image layout are met.
Smart Images

Figure CN117876524B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and storage medium for generating an image. Background Art
[0002] As the most popular application of generative AI in recent years, the text-to-image technology of Generative AI (AIGC) has a wide range of derivative application scenarios. The basic text-to-image technology based on the diffusion model can be summarized as two processes: diffusion process and denoising process. The diffusion process is the process of converting the original image into latent space noise and gradually increasing the noise, while the denoising process is the process of predicting the noise of each step of the noisy image through the model, gradually denoising and restoring the original image.
[0003] The advantage of text-to-image generation technology based on the diffusion model is that it can generate images with diversity and realism, but the generated images have a certain randomness and the layout of the image content is uncontrollable. Therefore, it is of great significance to achieve the ability to generate with controllable layout based on the diffusion model, which is a technical direction that urgently needs to be developed. Summary of the invention
[0004] The main purpose of the present invention is to provide a picture generation method, device, equipment and storage medium, aiming to solve the technical problem that the image content layout of the image generated by the prior art is uncontrollable.
[0005] To achieve the above object, the present invention provides a method for generating an image, the method comprising the following steps:
[0006] Obtaining a picture generation description and area location information corresponding to at least one target area frame;
[0007] A target image is generated according to the area location information and the image generation description.
[0008] Optionally, before the step of generating a target image according to the area location information and the image generation description, the step further includes:
[0009] Acquire an initial generation model, wherein the initial generation model includes a trunk generation model and a branch generation model;
[0010] The side branch generation model in the initial generation model is trained by using a model sample set to obtain a hierarchical controllable layout model;
[0011] The step of generating a target image according to the area location information and the image generation description comprises:
[0012] Generate a layout generation condition corresponding to the target area frame according to the area position information and the picture generation description by using a branch generation model in the hierarchical controllable layout model;
[0013] A target image is generated according to the layout generation conditions through a backbone generation model in the hierarchical controllable layout model.
[0014] Optionally, the step of generating the layout generation condition corresponding to the target area frame according to the area position information and the picture generation description by using the side branch generation model in the hierarchical controllable layout model includes:
[0015] If the language type of the image generation description is inconsistent with the input type of the side branch generation model in the hierarchical controllable layout model, performing semantic extraction on the image generation description to obtain image generation semantics;
[0016] Generate description input information according to the image generation semantics and the input type of the side branch generation model in the hierarchical controllable layout model;
[0017] The layout generation condition corresponding to the target area frame is generated according to the area position information and the description input information through the branch generation model in the hierarchical controllable layout model.
[0018] Optionally, the step of generating a target image according to the layout generation condition by using a backbone generation model in the hierarchical controllable layout model includes:
[0019] Merging the layout generation conditions to obtain a fused generation condition;
[0020] A target image is generated according to the fusion generation condition through the backbone generation model in the hierarchical controllable layout model.
[0021] Optionally, the step of generating the layout generation condition corresponding to the target area frame according to the area position information and the picture generation description by using the side branch generation model in the hierarchical controllable layout model includes:
[0022] Generate the layout generation condition and the area weight corresponding to the target area frame according to the area position information and the picture generation description by using the branch generation model in the hierarchical controllable layout model;
[0023] The step of fusing the layout generation conditions to obtain a fused generation condition comprises:
[0024] The layout generation conditions are fused based on the area weights to obtain fused generation conditions.
[0025] Optionally, the step of obtaining a picture generation description and area location information corresponding to at least one target area frame includes:
[0026] Upon receiving the generated description information input by the user, determining a generation mode corresponding to the generated description information;
[0027] If the generation mode is a custom mode, the image generation description and area position information corresponding to at least one target area frame are determined according to the generation description information.
[0028] Optionally, after the step of determining the generation mode corresponding to the generation description information when the generation description information input by the user is received, the step further includes:
[0029] If the generation mode is a recommendation mode, generating at least one target area frame;
[0030] The region position information corresponding to the at least one generated target region frame is acquired, and a picture generation description corresponding to the at least one target region frame is generated according to the generation description information.
[0031] Optionally, the step of generating at least one target area frame includes:
[0032] Performing semantic extraction on the generated description information to obtain description semantic information;
[0033] If there is a region-related description in the description semantic information, extracting the region-related description from the description semantic information;
[0034] At least one target area frame is generated according to the area-related description.
[0035] Optionally, after the step of performing semantic extraction on the generated description information to obtain the description semantic information, the step further includes:
[0036] If there is no region-related description in the description semantic information, obtaining a historical generation record corresponding to the user;
[0037] Selecting a target generation record from the historical generation records according to the generation description information;
[0038] At least one target area frame is generated according to the area division information corresponding to the target generation record.
[0039] Optionally, the step of selecting a target generation record from the historical generation records according to the generation description information includes:
[0040] Obtaining a historical generation description corresponding to the historical generation record;
[0041] Determine the description similarity corresponding to each historical generation record according to the generation description information and the historical generation description;
[0042] A target generation record is selected from the historical generation records based on the description similarity.
[0043] Optionally, after the step of generating a target image according to the area location information and the image generation description, the step further includes:
[0044] Displaying the target image;
[0045] Receiving generated evaluation information submitted by a user based on the displayed target image;
[0046] Generate a description generation model training sample according to the generated evaluation information, the target image and the image;
[0047] The model training samples are added to the model sample set.
[0048] In addition, to achieve the above object, the present invention also provides a picture generating device, which includes the following modules:
[0049] An acquisition module, used to acquire a picture generation description and area location information corresponding to at least one target area frame;
[0050] A generation module is used to generate a target image according to the area location information and the image generation description.
[0051] Optionally, the generation module is further used to obtain an initial generation model, the initial generation model including a trunk generation model and a side branch generation model; train the side branch generation model in the initial generation model through a model sample set to obtain a hierarchical controllable layout model;
[0052] The generation module is also used to generate layout generation conditions corresponding to the target area box according to the area position information and the image generation description through the branch generation model in the hierarchical controllable layout model; and generate a target image according to the layout generation conditions through the trunk generation model in the hierarchical controllable layout model.
[0053] Optionally, the generation module is also used to perform semantic extraction on the image generation description to obtain image generation semantics if the language type of the image generation description is inconsistent with the input type of the side branch generation model in the hierarchical controllable layout model; generate description input information according to the image generation semantics and the input type of the side branch generation model in the hierarchical controllable layout model; and generate layout generation conditions corresponding to the target area box according to the area position information and the description input information through the side branch generation model in the hierarchical controllable layout model.
[0054] Optionally, the generation module is further used to fuse the layout generation conditions to obtain fused generation conditions; and generate a target image according to the fused generation conditions through a backbone generation model in the hierarchical controllable layout model.
[0055] Optionally, the generation module is further used to generate a layout generation condition and an area weight corresponding to the target area frame according to the area position information and the image generation description through a branch generation model in the hierarchical controllable layout model;
[0056] The generation module is further used to fuse the layout generation conditions based on the area weights to obtain fused generation conditions.
[0057] Optionally, the acquisition module is also used to determine the generation mode corresponding to the generation description information when receiving the generation description information input by the user; if the generation mode is a custom mode, determine the image generation description and area location information corresponding to at least one target area box according to the generation description information.
[0058] Optionally, the acquisition module is also used to generate at least one target area box if the generation mode is a recommended mode; obtain area position information corresponding to the at least one generated target area box, and generate a picture generation description corresponding to the at least one target area box according to the generation description information.
[0059] In addition, to achieve the above-mentioned purpose, the present invention also proposes a picture generating device, which includes: a processor, a memory, and a picture generating program stored in the memory and executable on the processor, and the picture generating program implements the steps of the picture generating method described above when executed by the processor.
[0060] In addition, to achieve the above-mentioned purpose, the present invention also proposes a computer-readable storage medium, on which a picture generation program is stored, and when the picture generation program is executed, the steps of the picture generation method as described above are implemented.
[0061] The present invention obtains a picture generation description and regional position information corresponding to at least one target area frame; generates a target picture according to the regional position information and the picture generation description. Since the image content and the layout information of the image content corresponding to each target area frame are first generated according to the picture generation description and the regional position information corresponding to the target area frame, and then the image content and the layout information of the image content corresponding to each target area frame are summarized, and a complete target picture is generated according to the summarized overall layout, the generated picture layout is controllable. BRIEF DESCRIPTION OF THE DRAWINGS
[0062] Figure 1 It is a schematic diagram of the structure of an electronic device in a hardware operating environment involved in an embodiment of the present invention;
[0063] Figure 2 This is a schematic diagram of the process of the first embodiment of the image generation method of the present invention;
[0064] Figure 3 This is a schematic diagram of the flow chart of the second embodiment of the image generation method of the present invention;
[0065] Figure 4 A schematic diagram of a hierarchical controllable layout model structure according to an embodiment of the present invention
[0066] Figure 5 A schematic diagram of a picture generation process according to an embodiment of the present invention;
[0067] Figure 6 A schematic diagram of multiple concepts of a hierarchical controllable layout model according to an embodiment of the present invention;
[0068] Figure 7 This is a schematic diagram of the flow chart of the third embodiment of the image generation method of the present invention;
[0069] Figure 8 This is a structural block diagram of the first embodiment of the image generating device of the present invention.
[0070] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0071] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0072] Reference Figure 1 , Figure 1 The figure is a schematic diagram of the structure of a picture generation device of the hardware operating environment involved in the embodiment of the present invention.
[0073] like Figure 1As shown, the electronic device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (WIreless-FIdelity, WI-FI) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM), or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk storage. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.
[0074] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.
[0075] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a picture generation program.
[0076] exist Figure 1 In the electronic device shown, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the electronic device of the present invention can be set in a picture generating device, and the electronic device calls the picture generating program stored in the memory 1005 through the processor 1001, and executes the picture generating method provided by an embodiment of the present invention.
[0077] The embodiment of the present invention provides a method for generating an image. Figure 2 , Figure 2 The figure is a flow chart of a first embodiment of a method for generating an image according to the present invention.
[0078] In this embodiment, the image generation method includes the following steps:
[0079] Step S10: Obtaining a picture generation description and area location information corresponding to at least one target area frame.
[0080] It should be noted that the executor of this embodiment can be the image generating device, and the image generating device can be an electronic device such as a personal computer or a server, or other devices that can achieve the same or similar functions. This embodiment does not limit this. In this embodiment and the following embodiments, the image generating method of the present invention is described by taking the image generating device as an example.
[0081] It should be noted that the image generation description may be text description information, the text description information may be an open domain (OpenDomain) description text, and the region position information may be position information of the target region frame.
[0082] In the specific implementation, the controllable layout text graph technology based on the diffusion model learns the relationship between text and image layout by training a deep learning model. Specifically, the technology first encodes the text description into a latent space vector, and then gradually generates a structured image from random noise. During the generation process, the layout information or conditional constraints can be introduced to control the generated image layout and content.
[0083] The current generation technology can be divided into two types based on conditions: generation based on category (label) and region box (bounding box), and generation based on text description (caption) and region box (bounding box).
[0084] Among them, the generation based on label and boundingbox can only generate layout images of limited categories (CloseDomain), and the generated semantic granularity is coarse and the image quality is poor. The generation based on caption and boundingbox can generate layout images of open domain (Open Domain), and the generated semantics is more detailed and the image quality is better, but it is also more difficult. In order to ensure that the image quality of the hierarchical controllable layout generated pictures is good, the generation method of text description (caption) and area box (boundingbox) is adopted in this embodiment. Therefore, the picture generation description (i.e., caption) and area location information (used to characterize the target area box) corresponding to at least one target area box can be obtained first.
[0085] Step S20: Generate a target image according to the area location information and the image generation description.
[0086] It should be noted that generating a target image according to the region location information and the image generation description may be inputting the region location information and the image generation description into a pre-trained hierarchical controllable layout model, and generating a target image with layout information by the hierarchical controllable layout model.
[0087] Among them, the hierarchical controllable layout model will first determine the image content and layout information of the image content in each target area box according to the area position information corresponding to the target area box and the picture generation description, and then summarize the image content and layout information of the image content in each target area box to generate a complete target picture.
[0088] This embodiment obtains the image generation description and area location information corresponding to at least one target area frame; and generates the target image according to the area location information and the image generation description. Since the image content and the layout information of the image content corresponding to each target area frame are first generated according to the image generation description and the area location information corresponding to the target area frame, and then the image content and the layout information of the image content corresponding to each target area frame are summarized, and a complete target image is generated according to the summarized overall layout, the generated image layout is controllable.
[0089] refer to Figure 3 , Figure 3 FIG. 4 is a flow chart of a second embodiment of a method for generating an image according to the present invention.
[0090] Based on the first embodiment, the picture generation method of this embodiment further includes, before step S20:
[0091] Step S11: Obtain an initial generation model.
[0092] Step S12: training the side branch generation model in the initial generation model through the model sample set to obtain a hierarchical controllable layout model.
[0093] The step S20 may include:
[0094] Step S201: Generate layout generation conditions corresponding to the target area frame according to the area position information and the image generation description through the branch generation model in the hierarchical controllable layout model.
[0095] Step S202: Generate a target image according to the layout generation conditions through the backbone generation model in the hierarchical controllable layout model.
[0096] It should be noted that the layout generation condition corresponding to the target area box generated according to the area location information and the image generation description can be generated according to the area location information and the image generation description by a pre-trained layout generation model. Among them, the layout generation condition can be used to describe the image layout information corresponding to the target area box, for example: the layout generation condition may include the position of the target area box, the image content that should be set in the target area box, and the layout method of the image content in the target area box (such as the layout position of each part of the content in the target area box, etc.).
[0097] In actual use, generating a target image according to layout generation conditions can be to fuse the layout generation conditions corresponding to each target area box, and input the fused layout generation conditions into the image generation model, and the image generation model generates the target image according to the input fused layout generation conditions.
[0098] In practical applications, the model for generating layout generation conditions and the model for generating target images can both be pre-trained and set up. In order to ensure the correlation between the two, they can be set up in a trunk and branch manner. At this time, the initial generation model can include a trunk generation model and a branch generation model. The trunk generation model can be the above-mentioned image generation model, and the branch generation model can be the above-mentioned layout generation model.
[0099] The main generation model can be a diffusion model, such as stablediffusion; the branch generation model can be set based on the controlnet structure. Since multiple branches with shared weights are used to generate different layout information based on the main generation model (i.e., multiple target area boxes, each target box corresponds to a different layout generation condition, and the weight is consistent), it can fully adapt to the basic models of different styles in different communities. Therefore, the main generation mode of this embodiment can also be other generation models similar to stablediffusion, such as various image generation models provided by the civital website.
[0100] In actual use, since the trunk generation model adopts the commonly used diffusion image generation model, it is already in a trained state. When the initial generation model is trained, the trunk generation model does not need to participate in the training. Therefore, the side branch generation model in the initial generation model can be trained through the model sample set. After the training is completed, that is, when the side branch generation model is trained to convergence, the initial generation model is used as a hierarchical controllable layout model. At this time, the side branch generation model in the hierarchical controllable layout model can generate the layout generation conditions corresponding to the target area box according to the regional position information and the image generation description.
[0101] Among them, if there are multiple target area boxes, for each target area box, the corresponding layout generation condition can be generated by the side branch generation model in the hierarchical controllable layout model respectively without interfering with each other. In this process, each target area box can share the weight, that is, the weight corresponding to each target area box is consistent.
[0102] In a specific implementation, in order to ensure the validity of the generated layout generation conditions, step S201 in this embodiment may include:
[0103] If the language type of the image generation description is inconsistent with the input type of the side branch generation model in the hierarchical controllable layout model, performing semantic extraction on the image generation description to obtain image generation semantics;
[0104] Generate description input information according to the image generation semantics and the input type of the side branch generation model in the hierarchical controllable layout model;
[0105] The layout generation condition corresponding to the target area frame is generated according to the area position information and the description input information through the branch generation model in the hierarchical controllable layout model.
[0106] It should be noted that, at present, most models generally handle input data of a certain language type better during development. For example, some models handle English relatively well, but support other language types poorly, or even fail to receive input data of a specific language type. On this basis, in order to ensure the validity of the generated layout generation conditions, it is necessary to ensure that the language type of the data input to the branch generation model in the hierarchical controllable layout model is consistent with the input type supported by the branch generation model.
[0107] Therefore, if it is detected that the language type of the image generation description is inconsistent with the input type of the side branch generation model in the hierarchical controllable layout model (for example, the language type of the image generation description is Chinese, but the input type of the side branch generation model in the hierarchical controllable layout model is English), it is necessary to regenerate the description input information according to the image generation semantics of the image generation description and the input type of the side branch generation model in the hierarchical controllable layout model, and then generate the layout generation conditions corresponding to the target area box according to the area position information and the description input information through the side branch generation model in the hierarchical controllable layout model, so as to ensure that the side branch generation model in the hierarchical controllable layout model can be processed normally and the validity of the generated layout generation conditions is guaranteed.
[0108] In actual use, semantic extraction is performed on the image generation description to obtain the image generation semantics, which can be achieved by using an intelligent AI big language model to perform semantic extraction on the image generation description to obtain the image generation semantics. Similarly, the description input information generated according to the image generation semantics and the input type of the side branch generation model in the hierarchical controllable layout model can be generated according to the image generation semantics and the input type of the side branch generation model in the hierarchical controllable layout model using an intelligent AI big language model, the semantics of which are consistent with the image generation description, but the language type is consistent with the input type of the side branch generation model in the hierarchical controllable layout model, thereby obtaining the description input information. Among them, the intelligent AI big language model can be ChatGpt or a similar model, which is not limited in this embodiment.
[0109] In a specific implementation, in order to ensure the effect of the generated target image, step S30 in this embodiment may include:
[0110] Merging the layout generation conditions to obtain a fused generation condition;
[0111] A target image is generated according to the fusion generation condition through the backbone generation model in the hierarchical controllable layout model.
[0112] It should be noted that the backbone generation model is an image generation model. However, at present, if only the layout generation conditions corresponding to each target area box are obtained, the layout generation conditions are scattered layout information. In order to ensure the effect of image generation, the layout generation conditions corresponding to each target area box can be fused first, and then fused into an overall fused generation condition to describe the overall layout division. The fused generation condition is then input into the backbone generation model in the hierarchical controllable layout model, so that the backbone generation model in the hierarchical controllable layout model automatically generates images according to the fused generation conditions, thereby obtaining the target image.
[0113] Among them, since shared weights are used when generating the layout generation conditions corresponding to each target area box, the weights of the layout generation conditions corresponding to each target area box are equivalent at this time. When fusion is performed at this time, if there is area overlap among the target area boxes, the layout information therein can be averaged and fused for the overlapping areas.
[0114] In a specific implementation, in order to ensure that the actual image generation requirements of the user can be met, it is allowed to perform targeted weight adjustment for the layout information of different target area boxes when the layout generation conditions corresponding to each target area box are obtained by inference. That is, the weights of different target area boxes are different. At this time, when performing layout generation condition fusion, the weight needs to be considered. Therefore, step S201 in this embodiment may include:
[0115] Generate the layout generation condition and the area weight corresponding to the target area frame according to the area position information and the picture generation description by using the branch generation model in the hierarchical controllable layout model;
[0116] Accordingly, the step of fusing the layout generation conditions to obtain the fused generation conditions may include:
[0117] The layout generation conditions are fused based on the area weights to obtain fused generation conditions.
[0118] It should be noted that the regional weight may be the weight corresponding to the layout information in the target region box. Based on the regional weight, the layout generation conditions are merged to obtain the merged generation conditions, which may be to merge the layout generation conditions of the non-overlapping regions in each target region box, and then, based on the regional weight, to merge the layout generation conditions of the overlapping regions in each target region box by weighted summing according to the regional weight, and finally to generate the merged generation conditions.
[0119] In actual use, when fusing layout generation conditions, models can also be used for fusion. For example, the layout generation conditions corresponding to each target area box are fused based on the area weight through the Lora model to obtain the fused generation conditions.
[0120] In a specific implementation, in order to ensure the generation effect of the hierarchical controllable layout model, after step S30 in this embodiment, the following may also be included:
[0121] Displaying the target image;
[0122] Receiving generated evaluation information submitted by a user based on the displayed target image;
[0123] Generate a description generation model training sample according to the generated evaluation information, the target image and the image;
[0124] The model training samples are added to the model sample set.
[0125] It should be noted that the generated evaluation information may be the user's evaluation information on the generated target image, which may be divided into multiple levels, such as: accurate, calibrated, general, poor, extremely poor, etc.
[0126] It can be understood that after obtaining the target image, the user is asked to evaluate it, and then the model training sample can be constructed according to the obtained generated evaluation information, the target image and the image generation description. Then, the model training sample can be added to the model sample set, thereby expanding the scale of the model sample set and strengthening the training of the hierarchical controllable layout model, and finally gradually improving the generation effect of the hierarchical controllable layout model.
[0127] For ease of understanding, now combined Figure 4 , 5 4 is a schematic diagram of the hierarchical controllable layout model structure of this embodiment, Figure 5 This is a schematic diagram of the picture generation process of this embodiment. Figure 6 Schematic diagram of multiple concepts of the hierarchical controllable layout model of this embodiment.
[0128] like Figure 4 As shown, the hierarchical controllable layout model includes a trunk generation model and a branch generation model, wherein the trunk generation model is as follows Figure 4 The stable diffusion model shown in a (i.e. Figure 4 The model consists of multiple layers on the left side of the figure), and the side branch generation model is as follows Figure 4 The controlnet structure model shown in b (i.e. Figure 4 A model composed of multiple layers on the right side of the figure).
[0129] In the process of image generation, the weight sharing method is adopted to generate the model through the side branch (i.e. Figure 5 The control net in the figure generates the layout generation conditions corresponding to each target area box (i.e., the layout information at the position corresponding to each target area box). Finally, the layout information generated by all the side branches (i.e., the layout generation conditions corresponding to each target area box) is fused, and the fused information is input into the trunk generation model, which generates the target image.
[0130] The basic stable diffusion model only has the ability to embed a single concept, and multi-concept embedding will cause the image generated by the model to collapse due to the direct merging of different Lora model parameters. In this embodiment, the proposed hierarchical controllable layout model not only has the ability to embed a single concept, but also has the ability to embed multiple concepts.
[0131] The effect of the hierarchical controllable layout model combined with single concept embedding is shown in the figure below. Figure 6 a and Figure 6 As shown in b, Figure 6 a is the shrek concept trained on the stable diffusion 1.5 base model, Figure 6 b is the blind concept trained on the stable diffusion 1.5 base model. Figure 6 c is the effect generated by the blind concept in the specified area, Figure 6 d is the effect of generating both the concepts of blind and shrek at the same time.
[0132] This embodiment obtains an initial generation model, which includes a trunk generation model and a side branch generation model; trains the side branch generation model in the initial generation model through a model sample set to obtain a hierarchical controllable layout model; and generates layout generation conditions corresponding to the target area frame according to the area location information and the image generation description through the side branch generation model in the hierarchical controllable layout model. Since the model for generating layout generation conditions is associated with the model for generating the target image in a trunk branch manner, the association between the two is guaranteed, and the layout of the image generated by the hierarchical controllable layout model is guaranteed to be controllable.
[0133] refer to Figure 7 , Figure 7 FIG. 4 is a flow chart of a third embodiment of a method for generating an image according to the present invention.
[0134] Based on the first embodiment, step S10 of the picture generation method of this embodiment includes:
[0135] Step S101: upon receiving generation description information input by a user, determining a generation mode corresponding to the generation description information.
[0136] It should be noted that the picture generation description may be the description information input by the user when the picture needs to be automatically generated.
[0137] In actual use, in order to ensure user experience and meet the actual needs of users, the image generation device provides users with two generation modes: custom model and recommended mode.
[0138] Among them, the custom mode allows the user to customize at least one target area box according to actual needs, and set a corresponding picture generation description for each target area box; while the recommendation mode is that the user sets a global description information, and then the picture generation device automatically recommends and generates at least one target area box, and generates a corresponding picture generation description for each target area box.
[0139] In actual use, determining the generation mode corresponding to the generated description information may be to extract a mode specifying parameter from the generated description information, and determining the generation mode corresponding to the generated description information according to the value of the mode specifying parameter. For example, in the generated description information, a mode specifying parameter is set as "type", if type = 1, it means that the recommended mode is adopted; if type = 0, or type does not exist in the generated description information, it means that the custom mode is adopted.
[0140] Step S102: If the generation mode is a custom mode, determining a picture generation description and area position information corresponding to at least one target area frame according to the generation description information.
[0141] It can be understood that if the generation mode is a custom mode, it means that the user is allowed to customize at least one target area box according to actual needs, and set a corresponding image generation description for each target area box. At this time, the generation description information must include the image generation description and area location information corresponding to at least one target area box. Therefore, the generation description information can be parsed to read the image generation description and area location information corresponding to at least one target area box.
[0142] In a specific implementation, if the generation mode is the recommendation mode, it means that the target area frame is automatically set by the image generation device at this time. Then, after step S101 in this embodiment, the following may also be included:
[0143] If the generation mode is a recommendation mode, generating at least one target area frame;
[0144] The region position information corresponding to the at least one generated target region frame is acquired, and a picture generation description corresponding to the at least one target region frame is generated according to the generation description information.
[0145] It should be noted that generating at least one target area frame may be dividing the target area frame according to a preset template, wherein the preset template may be pre-set by a manager of the image generating device.
[0146] In actual use, obtaining the region position information corresponding to the at least one generated target region frame may be reading the position information corresponding to the at least one generated target region frame to obtain the region position information. Since the recommendation mode is adopted, the generated description information set by the user is global description information, that is, it is effective for all target region frames. Therefore, the generated description information can be directly used as the image generated description corresponding to the target region frame.
[0147] Furthermore, in order to ensure that the divided target area frame meets the actual needs of the user as much as possible, the step of generating at least one target area frame in this embodiment may include:
[0148] Performing semantic extraction on the generated description information to obtain description semantic information;
[0149] If there is a region-related description in the description semantic information, extracting the region-related description from the description semantic information;
[0150] At least one target area frame is generated according to the area-related description.
[0151] It should be noted that semantic extraction is performed on the description generated by the image to obtain the descriptive semantic information. The method can be to perform semantic extraction on the description generated by the image through a large intelligent AI model, and use the extracted semantics as the descriptive semantic information.
[0152] It is understandable that although the user sets the generated description information that applies to the global environment, the description may include descriptions related to the region, i.e., region-related descriptions. If such descriptions are included, you can try to divide the region based on such descriptions.
[0153] For example, assuming the semantic information is "someone is drying clothes on a clothesline under a clear sky", then there are region-related descriptions "clear sky", "clothesline", and "person", and three target region boxes can be divided according to the region-related descriptions.
[0154] Furthermore, in order to ensure that when there is no region-related description, the divided target region frame can also meet the actual needs of the user as much as possible, then after the step of performing semantic extraction on the generated description information and obtaining the description semantic information described in this embodiment, the following may also be included:
[0155] If there is no region-related description in the description semantic information, obtaining a historical generation record corresponding to the user;
[0156] Selecting a target generation record from the historical generation records according to the generation description information;
[0157] At least one target area frame is generated according to the area division information corresponding to the target generation record.
[0158] It should be noted that if there is no region-related description in the descriptive semantic information, it means that it is impossible to deduce the relevant method of region division based on the content in the descriptive semantic information. At this time, in order to ensure that the divided target area box can meet the user's actual needs as much as possible, the user's historical generation records can be used as a reference. Therefore, the user's corresponding historical generation records can be obtained.
[0159] In actual use, selecting a target generation record from historical generation records according to the image generation description may be selecting a historical generation record that has the highest correlation with the generation description information from the historical generation records as the target generation record.
[0160] It is understandable that the target generation record has the highest correlation with the generation description information, and the probability of the similarity being consistent when performing regional division is the highest. Therefore, at least one target region frame can be generated according to the regional division information corresponding to the target generation record. The regional division information corresponding to the target generation record can be the division rule when dividing the target region frame corresponding to the target generation record, and can include the number of target region frames and the regional location information corresponding to each target region frame.
[0161] In actual use, in order to accurately determine the target generation record, the step of selecting the target generation record from the historical generation record according to the generation description information in this embodiment may include:
[0162] Obtaining a historical generation description corresponding to the historical generation record;
[0163] Determine the description similarity corresponding to each historical generation record according to the generation description information and the historical generation description;
[0164] A target generation record is selected from the historical generation records based on the description similarity.
[0165] It should be noted that the historical generation description corresponding to the historical generation record may be generation description information input by the user when generating the image corresponding to the historical generation record.
[0166] In actual use, determining the description similarity corresponding to each historical generation record based on the generated description information and the historical generation description can be to determine the similarity between the generated description information and the historical generation description corresponding to each historical generation record respectively, and use the obtained similarity as the description similarity corresponding to the historical generation record.
[0167] In a specific application, selecting a target generation record from historical generation records based on description similarity may be taking a historical generation record with the highest corresponding description similarity as the target generation record.
[0168] When receiving the generation description information input by the user, this embodiment will determine the image generation description and area location information corresponding to at least one target area box in different ways according to the generation mode corresponding to the generation description information input by the user, so that the user can generate images in the way he needs according to actual needs, thereby improving the user's actual usage experience.
[0169] In addition, an embodiment of the present invention further proposes a storage medium, on which a picture generation program is stored. When the picture generation program is executed by a processor, the steps of the picture generation method described above are implemented.
[0170] Reference Figure 8 , Figure 8 This is a structural block diagram of the first embodiment of the image generating device of the present invention.
[0171] like Figure 8 As shown, the picture generating device proposed in the embodiment of the present invention includes:
[0172] An acquisition module 10 is used to acquire a picture generation description and area location information corresponding to at least one target area frame;
[0173] The generating module 20 is used to generate a target image according to the area location information and the image generation description.
[0174] This embodiment obtains the image generation description and area location information corresponding to at least one target area frame; and generates the target image according to the area location information and the image generation description. Since the image content and the layout information of the image content corresponding to each target area frame are first generated according to the image generation description and the area location information corresponding to the target area frame, and then the image content and the layout information of the image content corresponding to each target area frame are summarized, and a complete target image is generated according to the summarized overall layout, the generated image layout is controllable.
[0175] Optionally, the generation module 20 is further used to obtain an initial generation model, the initial generation model including a trunk generation model and a side branch generation model; train the side branch generation model in the initial generation model through a model sample set to obtain a hierarchical controllable layout model;
[0176] The generation module 20 is further used to generate layout generation conditions corresponding to the target area box according to the area location information and the image generation description through the branch generation model in the hierarchical controllable layout model; and generate a target image according to the layout generation conditions through the trunk generation model in the hierarchical controllable layout model.
[0177] Optionally, the generation module 20 is also used to perform semantic extraction on the image generation description to obtain image generation semantics if the language type of the image generation description is inconsistent with the input type of the side branch generation model in the hierarchical controllable layout model; generate description input information according to the image generation semantics and the input type of the side branch generation model in the hierarchical controllable layout model; and generate layout generation conditions corresponding to the target area box according to the area position information and the description input information through the side branch generation model in the hierarchical controllable layout model.
[0178] Optionally, the generation module 20 is further configured to fuse the layout generation conditions to obtain fused generation conditions; and generate a target image according to the fused generation conditions through a backbone generation model in the hierarchical controllable layout model.
[0179] Optionally, the generation module 20 is further configured to generate a layout generation condition and an area weight corresponding to the target area frame according to the area position information and the image generation description through a branch generation model in the hierarchical controllable layout model;
[0180] The generation module 20 is further configured to fuse the layout generation conditions based on the region weights to obtain fused generation conditions.
[0181] Optionally, the acquisition module 10 is also used to determine the generation mode corresponding to the generation description information when receiving the generation description information input by the user; if the generation mode is a custom mode, determine the image generation description and area location information corresponding to at least one target area box according to the generation description information.
[0182] Optionally, the acquisition module 10 is also used to generate at least one target area box if the generation mode is a recommended mode; obtain area position information corresponding to the at least one generated target area box, and generate a picture generation description corresponding to the at least one target area box according to the generation description information.
[0183] Optionally, the acquisition module 10 is further used to perform semantic extraction on the generated description information to obtain description semantic information; if there is region-related description in the description semantic information, the region-related description is extracted from the description semantic information; and at least one target region box is generated according to the region-related description.
[0184] Optionally, the acquisition module 10 is also used to obtain the historical generation record corresponding to the user if there is no area-related description in the descriptive semantic information; select the target generation record from the historical generation record according to the generation description information; and generate at least one target area box according to the area division information corresponding to the target generation record.
[0185] Optionally, the acquisition module 10 is further used to obtain the historical generation description corresponding to the historical generation record; determine the description similarity corresponding to each historical generation record according to the generation description information and the historical generation description; and select the target generation record from the historical generation records based on the description similarity.
[0186] Optionally, the generation module 20 is also used to display the target image; receive generation evaluation information submitted by the user based on the displayed target image; generate model training samples according to the generation evaluation information, the target image and the image generation description; and add the model training samples to the model sample set.
[0187] It should be understood that the above is only an example and does not constitute any limitation on the technical solution of the present invention. In specific applications, technicians in this field can make settings as needed, and the present invention does not limit this.
[0188] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, technicians in this field can select part or all of them according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.
[0189] In addition, for technical details that are not described in detail in this embodiment, reference can be made to the image generating method provided in any embodiment of the present invention, and will not be repeated here.
[0190] In addition, it should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.
[0191] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0192] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory (ROM) / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.
[0193] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
[0194] The present application discloses A1, a method for generating an image, the method comprising the following steps:
[0195] Obtaining a picture generation description and area location information corresponding to at least one target area frame;
[0196] A target image is generated according to the area location information and the image generation description.
[0197] A2. The image generation method as described in A1, before the step of generating a target image according to the area location information and the image generation description, further comprising:
[0198] Acquire an initial generation model, wherein the initial generation model includes a trunk generation model and a branch generation model;
[0199] The side branch generation model in the initial generation model is trained by using a model sample set to obtain a hierarchical controllable layout model;
[0200] The step of generating a target image according to the area location information and the image generation description comprises:
[0201] Generate a layout generation condition corresponding to the target area frame according to the area position information and the picture generation description by using a branch generation model in the hierarchical controllable layout model;
[0202] A target image is generated according to the layout generation conditions through a backbone generation model in the hierarchical controllable layout model.
[0203] A3. The image generation method as described in A2, wherein the step of generating the layout generation condition corresponding to the target area frame according to the area position information and the image generation description by using the branch generation model in the hierarchical controllable layout model comprises:
[0204] If the language type of the image generation description is inconsistent with the input type of the side branch generation model in the hierarchical controllable layout model, performing semantic extraction on the image generation description to obtain image generation semantics;
[0205] Generate description input information according to the image generation semantics and the input type of the side branch generation model in the hierarchical controllable layout model;
[0206] The layout generation condition corresponding to the target area frame is generated according to the area position information and the description input information through the branch generation model in the hierarchical controllable layout model.
[0207] A4. The image generation method as described in A2, wherein the step of generating a target image according to the layout generation condition by using the backbone generation model in the hierarchical controllable layout model comprises:
[0208] Merging the layout generation conditions to obtain a fused generation condition;
[0209] A target image is generated according to the fusion generation condition through the backbone generation model in the hierarchical controllable layout model.
[0210] A5. The image generation method as described in A4, wherein the step of generating the layout generation condition corresponding to the target area frame according to the area position information and the image generation description by using the branch generation model in the hierarchical controllable layout model comprises:
[0211] Generate the layout generation condition and the area weight corresponding to the target area frame according to the area position information and the picture generation description by using the branch generation model in the hierarchical controllable layout model;
[0212] The step of fusing the layout generation conditions to obtain a fused generation condition comprises:
[0213] The layout generation conditions are fused based on the area weights to obtain fused generation conditions.
[0214] A6. In the image generation method described in A1, the step of obtaining the image generation description and area location information corresponding to at least one target area frame comprises:
[0215] Upon receiving the generated description information input by the user, determining a generation mode corresponding to the generated description information;
[0216] If the generation mode is a custom mode, the image generation description and area position information corresponding to at least one target area frame are determined according to the generation description information.
[0217] A7. The image generation method as described in A6, after the step of determining the generation mode corresponding to the generation description information when receiving the generation description information input by the user, further comprising:
[0218] If the generation mode is a recommendation mode, generating at least one target area frame;
[0219] The region position information corresponding to the at least one generated target region frame is acquired, and a picture generation description corresponding to the at least one target region frame is generated according to the generation description information.
[0220] A8. In the image generation method as described in A7, the step of generating at least one target area frame comprises:
[0221] Performing semantic extraction on the generated description information to obtain description semantic information;
[0222] If there is a region-related description in the description semantic information, extracting the region-related description from the description semantic information;
[0223] At least one target area frame is generated according to the area-related description.
[0224] A9. The image generation method as described in A8, after the step of performing semantic extraction on the generated description information to obtain the description semantic information, further comprising:
[0225] If there is no region-related description in the description semantic information, obtaining a historical generation record corresponding to the user;
[0226] Selecting a target generation record from the historical generation records according to the generation description information;
[0227] At least one target area frame is generated according to the area division information corresponding to the target generation record.
[0228] A10. In the picture generation method described in A9, the step of selecting a target generation record from the historical generation records according to the generation description information comprises:
[0229] Obtaining a historical generation description corresponding to the historical generation record;
[0230] Determine the description similarity corresponding to each historical generation record according to the generation description information and the historical generation description;
[0231] A target generation record is selected from the historical generation records based on the description similarity.
[0232] A11. The image generation method as described in any one of A1 to A10, after the step of generating a target image according to the area location information and the image generation description, further comprising:
[0233] Displaying the target image;
[0234] Receiving generated evaluation information submitted by a user based on the displayed target image;
[0235] Generate a description generation model training sample according to the generated evaluation information, the target image and the image;
[0236] The model training samples are added to the model sample set.
[0237] The present application also discloses B12, a picture generating device, the picture generating device comprising the following modules:
[0238] An acquisition module, used to acquire a picture generation description and area location information corresponding to at least one target area frame;
[0239] A generation module is used to generate a target image according to the area location information and the image generation description.
[0240] B13. The image generation device as described in B12, wherein the generation module is further used to obtain an initial generation model, wherein the initial generation model includes a trunk generation model and a side branch generation model; the side branch generation model in the initial generation model is trained by a model sample set to obtain a hierarchical controllable layout model;
[0241] The generation module is also used to generate layout generation conditions corresponding to the target area box according to the area position information and the image generation description through the branch generation model in the hierarchical controllable layout model; and generate a target image according to the layout generation conditions through the trunk generation model in the hierarchical controllable layout model.
[0242] B14. In the image generation device as described in B13, the generation module is also used to perform semantic extraction on the image generation description to obtain image generation semantics if the language type of the image generation description is inconsistent with the input type of the side branch generation model in the hierarchical controllable layout model; generate description input information according to the image generation semantics and the input type of the side branch generation model in the hierarchical controllable layout model; and generate layout generation conditions corresponding to the target area box according to the area position information and the description input information through the side branch generation model in the hierarchical controllable layout model.
[0243] B15. In the image generation device as described in B13, the generation module is further used to fuse the layout generation conditions to obtain fused generation conditions; and generate the target image according to the fused generation conditions through the backbone generation model in the hierarchical controllable layout model.
[0244] B16, the image generation device as described in B15, the generation module is further used to generate the layout generation conditions and area weights corresponding to the target area frame according to the area position information and the image generation description through the branch generation model in the hierarchical controllable layout model;
[0245] The generation module is further used to fuse the layout generation conditions based on the area weights to obtain fused generation conditions.
[0246] B17. In the image generating device as described in B12, the acquisition module is further used to determine the generation mode corresponding to the generation description information when receiving the generation description information input by the user; if the generation mode is a custom mode, the image generation description and area position information corresponding to at least one target area box are determined according to the generation description information.
[0247] B18. In the image generation device as described in B17, the acquisition module is also used to generate at least one target area box if the generation mode is a recommended mode; obtain area position information corresponding to the at least one generated target area box, and generate an image generation description corresponding to the at least one target area box according to the generation description information.
[0248] The present application also discloses C19, a picture generating device, which includes: a processor, a memory, and a picture generating program stored in the memory and executable on the processor, wherein the picture generating program implements the steps of the picture generating method as described above when executed by the processor.
[0249] The present application also discloses D20, a computer-readable storage medium, on which a picture generation program is stored, and when the picture generation program is executed, the steps of the picture generation method as described above are implemented.
Claims
1. A method for generating an image, characterized in that: The image generation method comprises the following steps: Obtaining a picture generation description and area location information corresponding to at least one target area frame; Generate a target image according to the area location information and the image generation description; The step of generating a target image according to the area location information and the image generation description includes: Generate layout generation conditions corresponding to the target area frame according to the area position information and the image generation description by using a side branch generation model in a hierarchical controllable layout model, wherein the hierarchical controllable layout model is obtained by pre-training a side branch generation model in an initial generation model using a model sample set; A target image is generated according to the layout generation conditions through a backbone generation model in the hierarchical controllable layout model.
2. The image generation method according to claim 1, characterized in that: Before the step of generating a target image according to the area location information and the image generation description, the step further includes: Acquire an initial generation model, wherein the initial generation model includes a trunk generation model and a branch generation model; The side branch generation model in the initial generation model is trained through the model sample set to obtain a hierarchical controllable layout model.
3. The image generation method according to claim 2, characterized in that: The step of generating the layout generation condition corresponding to the target area frame according to the area position information and the picture generation description by using the branch generation model in the hierarchical controllable layout model comprises: If the language type of the image generation description is inconsistent with the input type of the side branch generation model in the hierarchical controllable layout model, performing semantic extraction on the image generation description to obtain image generation semantics; Generate description input information according to the image generation semantics and the input type of the side branch generation model in the hierarchical controllable layout model; The layout generation condition corresponding to the target area frame is generated according to the area position information and the description input information through the branch generation model in the hierarchical controllable layout model.
4. The image generation method according to claim 2, characterized in that: The step of generating a target image according to the layout generation condition by using the backbone generation model in the hierarchical controllable layout model includes: Merging the layout generation conditions to obtain a fused generation condition; A target image is generated according to the fusion generation condition through the backbone generation model in the hierarchical controllable layout model.
5. The image generation method according to claim 4, characterized in that: The step of generating the layout generation condition corresponding to the target area frame according to the area position information and the picture generation description by using the branch generation model in the hierarchical controllable layout model comprises: Generate the layout generation condition and the area weight corresponding to the target area frame according to the area position information and the picture generation description by using the branch generation model in the hierarchical controllable layout model; The step of fusing the layout generation conditions to obtain a fused generation condition comprises: The layout generation conditions are fused based on the area weights to obtain fused generation conditions.
6. The image generation method according to claim 1, characterized in that: The step of obtaining a picture generation description and area location information corresponding to at least one target area frame includes: Upon receiving the generated description information input by the user, determining a generation mode corresponding to the generated description information; If the generation mode is a custom mode, the image generation description and area position information corresponding to at least one target area frame are determined according to the generation description information.
7. The image generation method according to claim 6, characterized in that: After the step of determining the generation mode corresponding to the generated description information when receiving the generated description information input by the user, the method further includes: If the generation mode is a recommendation mode, generating at least one target area frame; The region position information corresponding to the at least one generated target region frame is acquired, and a picture generation description corresponding to the at least one target region frame is generated according to the generation description information.
8. A picture generating device, characterized in that: The image generation device includes the following modules: An acquisition module, used to acquire a picture generation description and area location information corresponding to at least one target area frame; A generating module, used for generating a target image according to the area location information and the image generation description; The generation module is also used to generate layout generation conditions corresponding to the target area box according to the area position information and the image generation description through a side branch generation model in a hierarchical controllable layout model, wherein the hierarchical controllable layout model is obtained by pre-training the side branch generation model in the initial generation model through a model sample set; and generate a target image according to the layout generation conditions through a trunk generation model in the hierarchical controllable layout model.
9. A picture generating device, characterized in that: The image generation device comprises: a processor, a memory, and an image generation program stored in the memory and executable on the processor, wherein the image generation program, when executed by the processor, implements the steps of the image generation method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a picture generation program, and when the picture generation program is executed, the steps of the picture generation method according to any one of claims 1 to 7 are implemented.
