Image generation method and apparatus

Abstract

An image generation method and apparatus. The method comprises: acquiring an image generation instruction, wherein the image generation instruction comprises input text and first position information, and the first position information is used for indicating at least one of a plurality of partitions obtained by dividing an image area in advance (201); parsing the image generation instruction to obtain prompt text and second position information (202); and inputting the prompt text and the second position information into an image generation model, to acquire an image generated by the image generation model by using the prompt text and the second position information (203). At least one partition among a plurality of partitions obtained by dividing an image area is indicated as position control information for image generation, improving the position control capability, simplifying user operations, and improving usability.

Description

Image generation method and device

[0001] This application claims priority to Chinese patent application No. 202410176223.7, filed on February 7, 2024, with invention name “Image Generation Method and Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the fields of artificial intelligence technology and image processing technology, and in particular to image generation methods and devices. Background Art

[0003] In the current field of artificial intelligence, the automatic generation of images from text is gaining increasing popularity. This technology provides content creators with a new tool, making it easier and more cost-effective to create content that previously required professional expertise, significantly improving work efficiency and creative freedom.

[0004] However, current image generation technologies have poor capabilities for controlling the position of generated images. Summary of the Invention

[0005] In view of this, the present application provides an image generation method and apparatus for improving the ability to control the position of a generated image.

[0006] This application provides the following solutions:

[0007] In a first aspect, a method for generating an image is provided, the method comprising:

[0008] Obtaining an image generation instruction, the image generation instruction including input text and first position information, the first position information being used to indicate at least one of a plurality of partitions obtained by pre-dividing the image area;

[0009] Parsing the image generation instruction to obtain a prompt text and second position information;

[0010] The prompt text and the second position information are input into an image generation model, and an image generated by the image generation model using the prompt text and the second position information is obtained.

[0011] According to an implementable manner in an embodiment of the present application, the image generation instruction includes a text sequence, and the text sequence includes input text and first position information of entity words marked in the input text.

[0012] According to an achievable method in an embodiment of the present application, the partitioning includes: partitioning obtained by dividing the image area into multiple grids.

[0013] According to an implementable method in an embodiment of the present application, the first location information includes a partition number.

[0014] According to an achievable method in an embodiment of the present application, parsing the image generation instruction to obtain the prompt text and the second location information includes:

[0015] Parsing the input text and first position information of entity words included in the input text from the image generation instruction;

[0016] Obtaining prompt text using the input text;

[0017] The first position information annotated for the entity word included in the input text is used to obtain the position box information corresponding to the entity word as the second position information.

[0018] According to an achievable method in an embodiment of the present application, obtaining the location box information corresponding to the entity word by using the first location information annotated for the entity word included in the input text includes:

[0019] Identifying first position information where the entity word is marked;

[0020] If the first position information marked with the entity word indicates a partition, taking the edge of the partition as the position box corresponding to the entity word;

[0021] If the first position information marked with the entity word indicates multiple partitions, the multiple partitions are fused, and the edges of the fused regions are used as the position boxes corresponding to the entity word.

[0022] According to an achievable embodiment of the present application, the method further includes:

[0023] According to the set disturbance factor, the position of the position box is randomly adjusted within the disturbance range corresponding to the disturbance factor.

[0024] According to an achievable method in an embodiment of the present application, the image generation model is pre-trained based on a diffusion model.

[0025] According to an achievable method in an embodiment of the present application, the image generation model generates an image using the prompt text and the second position information, including:

[0026] The image generation model encodes the prompt text to obtain a feature representation of the prompt text;

[0027] encoding the control condition including the second position information to obtain a feature representation of the control condition;

[0028] Using the feature representation of the prompt text and the feature representation of the control condition, noise prediction and denoising are performed on the noisy image for T time steps to obtain a generated image, where T is a preset positive integer.

[0029] In a second aspect, an image generation method is provided, which is applied to a cloud server, and the method includes:

[0030] Obtaining an image generation instruction from a user device, the image generation instruction including input text and first position information, where the first position information is used to indicate at least one of a plurality of partitions into which an image area is pre-divided;

[0031] Parsing the image generation instruction to obtain a prompt text and second position information;

[0032] Inputting the prompt text and the second position information into an image generation model, and obtaining an image generated by the image generation model using the prompt text and the second position information;

[0033] The generated image is returned to the user device.

[0034] According to an achievable manner in an embodiment of the present application, obtaining an image generation instruction from a user device includes: providing an interactive interface to the user device, and obtaining the image generation instruction input by the user in a first area of ​​the interactive interface;

[0035] The method further includes: displaying the prompt text and the second location information in a second area of ​​the interactive interface; and / or displaying the generated image in a third area of ​​the interactive interface.

[0036] According to a third aspect, an image generating device is provided, the device comprising:

[0037] an instruction acquisition unit, configured to acquire an image generation instruction, wherein the image generation instruction includes input text and first position information, wherein the first position information is used to indicate at least one of a plurality of partitions into which the image area is pre-divided;

[0038] An instruction parsing unit, which parses the image generation instruction to obtain a prompt text and second position information;

[0039] The image generation unit inputs the prompt text and the second position information into an image generation model, and obtains an image generated by the image generation model using the prompt text and the second position information.

[0040] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the steps of the method described in any one of the first and second aspects are implemented.

[0041] According to a fifth aspect, there is provided an electronic device, comprising:

[0042] one or more processors; and

[0043] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the first and second aspects above.

[0044] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0045] 1) This application proposes a new position indication method, which indicates at least one partition in a plurality of partitions into which the image area is pre-divided as the position control information for image generation, namely the first position information, and guides the image generation model to generate the image after converting it into the second position information suitable for the image generation model through analysis, thereby improving the position control capability of image generation.

[0046] 2) In this application, users are only required to input text and annotate partition information for entity words in the input text. Compared with the traditional method of manually drawing position boxes for each entity word, this is much simpler, significantly reduces the cost of position control, and improves ease of use. In addition, it can achieve more refined position control for each entity.

[0047] 3) In this application, the image area is divided into multiple partitions, and each partition is indicated using a partition number, which simplifies the position indication method and improves the accuracy of position indication.

[0048] 4) In this application, the first position information indicating the partition is parsed into position frame information, and then provided to the image generation model for position control. It can adapt to the existing position control image generation model based on position frame information without modifying the underlying model structure of the image generation model, and has good versatility.

[0049] 5) This application sets a disturbance factor when determining the position frame, and performs a small dither on the position frame information obtained by analysis, so as to make each position control more random and enhance the diversity of image generation.

[0050] 6) The present application interacts with the user by providing an interactive interface. The interactive interface has an area for the user to input image generation instructions, as well as a display area for prompt information, location information and generated images, which improves the interactivity between the device and the user during the image generation process and reduces the user's operating difficulty.

[0051] Of course, any product implementing the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0053] FIG1 is a diagram of a system architecture applicable to an embodiment of the present application;

[0054] FIG2 is a flow chart of an image generation method provided in an embodiment of the present application;

[0055] FIG3 is a schematic diagram of parsing location frame information provided by an embodiment of the present application;

[0056] FIG4 is a schematic diagram of an interactive interface provided in an embodiment of the present application;

[0057] FIG5 is a schematic diagram of an image generation model provided in an embodiment of the present application;

[0058] FIG6 is a schematic block diagram of an image generating device provided in an embodiment of the present application;

[0059] FIG7 is a schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0061] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "an", "the" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0062] It should be understood that the term "and / or" as used herein is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0063] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0064] Automatic image generation technology generates high-quality, creative images by analyzing input text. However, precisely controlling the position of objects within the generated images based on the text remains challenging. Current image generation techniques that rely solely on a single text modality to control the generated object's position are ineffective, while the introduction of additional hand-drawn position boxes complicates the process and hinders user experience and the widespread adoption of position-controllable models.

[0065] In view of this, the present application provides a new approach. To facilitate understanding of the present application, the system architecture on which the present application is based is first described. FIG1 illustrates an exemplary system architecture to which embodiments of the present application can be applied. As shown in FIG1 , the system architecture may include a user device and an image generation device located on the server side.

[0066] User devices may include, but are not limited to, smart mobile terminals, smart home devices, wearable devices, and personal computers (PCs). Smart mobile devices may include mobile phones, tablets, laptops, PDAs (Personal Digital Assistants), and internet-connected cars. Smart home devices may include smart TVs and smart refrigerators. Wearable devices may include smart watches, smart glasses, virtual reality devices, augmented reality devices, and mixed reality devices (i.e., devices that support both virtual reality and augmented reality).

[0067] The image generation device can use the method provided in the embodiments of the present application to generate an image according to an image generation instruction, wherein the image generation process involves the use of an image generation model.

[0068] The image generation device can be deployed as a standalone server, within a server cluster, or even on a cloud server. A cloud server, also known as a cloud computing server or cloud host, is a host product within the cloud computing service ecosystem. It addresses the management difficulties and scalability limitations of traditional physical hosts and virtual private servers (VPS). In addition to the architecture shown in Figure 1, the image generation device can also be deployed on a computer terminal with significant computing power.

[0069] In one possible implementation, a user can input an image generation instruction through a user device, which then sends the instruction to an image generation device on a server. The image generation device generates an image based on the instruction and returns the generated image to the user device via a network. The user device then displays the image to the user.

[0070] It should be understood that the user equipment, image generation apparatus and image generation model in Figure 1 are merely illustrative and any number of user equipment, image generation apparatus and image generation model may be provided as required.

[0071] FIG2 is a flow chart of an image generation method provided by an embodiment of the present application, which can be executed by the image generation device in the system shown in FIG1. ​​As shown in FIG2, the method may include the following steps:

[0072] Step 201: Acquire an image generation instruction, where the image generation instruction includes input text and first position information, where the first position information is used to indicate at least one of a plurality of partitions into which an image area is pre-divided.

[0073] Step 202: Parse the image generation instruction to obtain the prompt text and the second position information.

[0074] Step 203: Input the prompt text and the second position information into the image generation model, and obtain an image generated by the image generation model using the prompt text and the second position information.

[0075] It can be seen from the above process that the present application proposes a new position indication method, which improves the position control ability of image generation by indicating at least one partition among multiple partitions into which the image is pre-divided as the position control information for image generation, namely the first position information, and then guiding the image generation model to generate the image after converting it into the second position information suitable for the image generation model through analysis, thereby improving the position control ability of image generation.

[0076] The following describes in detail each step of the above process and the effects that can be further produced, in conjunction with the embodiments. It should be noted that the terms "first" and "second" in this disclosure do not have limitations on size, order, or quantity, but are merely used to distinguish between them in terms of name. For example, "first location information" and "second location information" are used to distinguish between two pieces of location information.

[0077] First, the above step 201, namely “obtaining an image generation instruction”, is described in detail with reference to the embodiment.

[0078] The image generation instruction is used to indicate the user's demand for generating an image, and the image generation instruction includes input text and first position information.

[0079] The input text is a user-entered description of the image to be generated. This text description primarily describes the content of the image, such as the objects contained in the image, the relationships between objects, and their attributes. For example, a text description might be something like "A rabbit stood by a bed by the river, and there was a bright moon in the sky." This indicates that the generated image must include objects such as the rabbit, river, bed, sky, and bright moon. These objects are typically represented as entity words in the text description.

[0080] Therefore, in an embodiment of the present application, the above-mentioned image generation instruction may include a text sequence, that is, the image generation instruction may be input in the form of a text sequence, and the text sequence includes the input text and the first position information of the entity word annotation included in the input text.

[0081] The first position information is used to indicate at least one of a plurality of partitions into which the image area is pre-divided.

[0082] The image region is the area of ​​the screen where the image is generated. This region corresponds to the final image. The image region can be divided in a variety of ways, such as horizontally or vertically, diagonally, or using multiple grids. The number of partitions can also be flexibly determined based on actual needs.

[0083] As a preferred implementation solution, the image area can be divided into nine partitions in a nine-square grid manner, as shown in the left part of FIG3 .

[0084] Partitions can be represented by coordinates, partition numbers, partition names, etc. As one possible implementation, each partition can be numbered from 1 to N (N is the total number of partitions), and each partition is referred to by its corresponding partition number. In this case, the first location information includes the partition number. The use of the partition number method requires less user input and is less expensive. In the embodiments of this application, the partition number is used as an example for description.

[0085] Continuing with the above example, the text sequence included in the image generation instruction can be: "There is a <rabbit:4> standing beside a <bed:5,6> by the <river:7,8,9>, and there is a <bright moon:3> in the sky." Among them, "<river:7,8,9>; <rabbit:4>; <bed:5,6>; <bright moon:3>" are the annotations of the first position information for the entity words "river", "rabbit", "bed", and "bright moon" in the input text. The annotation form uses the <:> form, and the numbers therein are the partition numbers where the entity words are annotated. For example, the partition numbers annotated for the entity word "river" are 7, 8, and 9, indicating that the position of the "river" in the image area (i.e., in the finally generated image) is the positions corresponding to the numbers 7, 8, and 9 in the nine-square grid shown on the left side of Figure 3. Another example is that the partition number annotated for the entity word "rabbit" is 4, indicating that the position of the "rabbit" in the image area is the position corresponding to the number 4 in the nine-square grid shown on the left side of Figure 3. In addition to this annotation form, other annotation forms can also be used.

[0086] It should be noted that the first position information can be annotated for all entity words in the input text, or only for some of the entity words. For example, in the above example, the partition number is not annotated for the entity word "sky".

[0087] As one of the realizable ways, an interactive interface can be provided for the user. The user can input the image generation instruction through the interactive interface. During the input process of the first position information, the interactive interface can guide the user, such as showing the partition positions and the corresponding partition numbers in the form of a schematic diagram.

[0088] The user inputs the image generation instruction according to their own needs through the interactive interface, and the user device sends the image generation instruction to the server side. As one of the realizable ways, the server side can push the interactive interface to the user device, and some components (such as input boxes, dropdown boxes, options, etc.) are provided on this interface for the user to input a structured image generation instruction.

[0089] Figure 4 shows a user interactive interface provided by an embodiment of the present application. A first area is set for the user device to input the image generation instruction. As shown in Figure 4, the left half area of the interactive interface is the first area. The user inputs the image generation instruction in this area and clicks the generation button to submit the image generation instruction to the server. Among them, the first area shown in Figure 4 can include an input component for the text sequence. For example, the input box at the top of the first area in Figure 4. The user inputs "There is a <rabbit:4> standing beside a <bed:5,6> by the <river:7,8,9>, and there is a <bright moon:3> in the sky." in this input box.

[0090] In addition, the image generation instruction may further include other information such as image style information, model selection information, image size, image clarity, random perturbation factor (which will be discussed in subsequent embodiments). For example, in FIG4 , the first area may further include a component for selecting a model type, a component for performing style control, etc. For example, a user can select a specific image generation model for image generation by selecting a model type component. FIG4 shows two models, Model 1 and Model 2.

[0091] The above step 202, namely "parsing the image generation instruction to obtain the prompt text and the second position information", is described in detail below with reference to an embodiment.

[0092] Since the image generation instruction input by the user may not be in a form that can be understood and used by the image generation model, the image generation device can parse the image generation instruction input by the user to obtain prompt text and second position information. The prompt text and second position information are in an instruction format suitable for inputting the image generation model.

[0093] In an embodiment of the present application, the input text and the first position information of each entity word marked, such as the partition number, can be first parsed from the image generation instruction, and then the input text and the first position information of each entity word marked, such as the partition number, can be further parsed to obtain the prompt text and the second position information.

[0094] Among them, when parsing the input text and the first position information of each entity word, such as the partition number, a regular matching method can be used. Furthermore, the first area on the interactive interface shown in Figure 4 can further display a component of the parsing result. The system uses regular expressions to parse the entity words and the marked partition number information from the user input, and then displays the entity words and the marked partition number information on the interactive interface shown in Figure 4.

[0095] In addition to the display method of the parsing results shown in FIG4 , other methods may also be used, such as highlighting entity words and partition numbers marked with entity words in a text sequence.

[0096] The above parsing results can help users check whether the parsing results meet their needs. If the parsing result is incorrect, it means that the text sequence entered by the user is incorrect, such as entering the wrong partition number or the incorrect annotation format. The user can readjust the text sequence entered until the correct parsing result is obtained.

[0097] The prompt text is used to indicate the content information of the generated image, and is mainly obtained using the input text in the image generation instruction. As one of the feasible methods, the parsed input text can be used as the prompt text. For example, "There is a rabbit standing by the bed by the river, and there is a bright moon in the sky" can be used as the prompt text. In addition to this method, other methods can also be used to obtain the prompt text, such as combining the parsed input text with a preset template to obtain the prompt text, etc.

[0098] The second position information is used to indicate the specific position of the object corresponding to the entity word in the generated image, and the second position information is obtained based on the first position information. The second position information can be described in a variety of ways. As one of the feasible ways, the second position information can use the position box (BoundingBox) information, such as the coordinates of the position box. In addition to the position box information, the second position information can also use other forms, such as "in the upper half of the image", "occupying one-third of the image" and other direction and proportion information.

[0099] Taking the position box information as an example, first, the input text and the first position information of the entity words included in the input text are parsed from the image generation instruction; secondly, the prompt text is obtained using the input text; finally, the position box information corresponding to the entity word is obtained as the second position information using the first position information of the entity word included in the input text.

[0100] The position box information can be obtained specifically in the following manner: identifying the first position information of the entity word; if the first position information of the entity word indicates a partition, the edge of the partition is the position box corresponding to the entity word; if the first position information of the entity word indicates multiple partitions, the multiple partitions are fused, and the edge of the fused area is the position box corresponding to the entity word.

[0101] The location box information may be the location box coordinates corresponding to the location box. It is understood that for partitions in a grid format such as a nine-square grid, each partition may be represented using diagonal coordinates. For example, the location box coordinates corresponding to a partition may be uniquely identified by the coordinates of the upper left corner and the lower right corner. Another example is the location box coordinates corresponding to a partition may be uniquely identified by the coordinates of the upper right corner and the lower left corner.

[0102] As one feasible approach, the coordinates of the upper left corner and lower right corner of each partition may be calculated in advance according to the size of the image to be generated, and stored in advance in the form of a file, so that the file can be directly called when parsing the second position information.

[0103] As another feasible method, the position frame coordinates can be calculated in real time when parsing the second position information. Taking the nine-square grid segmentation as an example, the position frame coordinates can be calculated as follows: according to the size of the image to be generated, the height and width of each partition are calculated; according to the height and width of each partition, the vertex coordinates of each partition are calculated; the partition number string in the first position information is converted into an array; the partition number in each array is used as an index to query the vertex coordinates of each partition. More specifically, if there is only one partition number in the array, the two coordinates corresponding to the partition number as the index are the position frame information; if there are multiple partition numbers in the array, when the area after the partition fusion is a rectangle, the upper left corner coordinates and the lower right corner coordinates of the fused area are used as the position frame information; when the area after the partition fusion is an irregular polygon, the coordinates of each vertex are used as the position frame information.

[0104] Furthermore, when parsing the position box information, the present application can perform dynamic processing to cause the position box determined during each image generation process to fluctuate within a small range, making each position control more random, thereby improving the diversity of image generation. Specifically, based on a set perturbation factor, the position of the position box can be randomly adjusted within the perturbation range corresponding to the perturbation factor.

[0105] Among them, the disturbance factor can adopt a preset value or be open to the user and set by the user through the interactive interface. The disturbance factor is used to limit the degree of disturbance. In the embodiment of the present application, the maximum range of disturbance can be pre-set. For example, R represents the maximum range threshold of the disturbance. R can be determined according to a certain ratio of the image size. For example, for an image of 512*512 size, R can be set to 20. The disturbance factor can be a value between 0 and 1. If a represents the disturbance factor, the corresponding disturbance range is [-a*R, a*R]. During the image generation process, the position coordinate value of each position box will have a random disturbance amount from the range of [-a*R, a*R] added to its original value.

[0106] Figure 3 is a schematic diagram of the location frame information obtained by analysis. As shown in Figure 3, the image generation instruction input by the user is "There is a <rabbit:4> standing next to the <bed:5,6> beside the <river:7,8,9>, and there is a <bright moon:3> in the sky." The image generation instruction defines the partitions corresponding to the entity words "rabbit", "bed", "river", and "moon" respectively. The regional fusion result is shown in the right figure. The image generation device fuses multiple partitions corresponding to the same entity word according to the method described above, and generates location frames corresponding to each entity word.

[0107] As one possible implementation, the recognition results of the location frame information can also be displayed on the interactive interface. For example, the location frame information shown on the right side of Figure 3 can be displayed on the interactive interface, and the corresponding entity word can be marked on each location frame to facilitate the user to confirm whether it meets their needs.

[0108] Furthermore, on the interactive interface, the hint text and the second position information obtained by the analysis can be further displayed in the second area, as shown in Figure 4. The second position information is represented in the form of the position box coordinates corresponding to each entity word.

[0109] In addition, it should be noted that the above embodiment uses a text sequence as an example to illustrate the input method of inputting text and first position information. In addition to this method, other methods can also be used. For example, after the user enters a text description as input text, the image generation device automatically identifies entity words in the input text and displays the entity words to the user in the interactive interface; the user then uses the components provided on the interactive interface to label the partition numbers for the entity words, etc.

[0110] The above step 203, namely "inputting the prompt text and the second position information into the image generation model, and obtaining the image generated by the image generation model using the prompt text and the second position information" will be described in detail below with reference to an embodiment.

[0111] After obtaining the prompt text and the second position information that the image generation model can understand, the image generation model can automatically generate an image, and the position of the object contained in the generated image is generated according to the partition position indicated by the first position information.

[0112] As an implementable approach for the present application, the image generation model can be implemented based on a diffusion model, that is, pre-trained based on a diffusion model, such as StableDiffustion (a latent diffusion model-based image model) and LDM (Latent Diffusion Model).

[0113] Existing image generation models based on diffusion models support inputting prompt text to generate images that match the prompt text description. Some image generation models already provide excellent support for the "text + location box" input format. In the embodiments of the present application, if the second position information is the location box coordinates, existing position-controlled image generation models based on location box information can be directly adapted without modifying the underlying model structure of the image generation model, thus demonstrating good versatility.

[0114] As one possible implementation, as shown in Figure 5, the image generation model can encode the prompt text to obtain a feature representation of the prompt text, and encode the second position information, such as the position box information, as one of the control conditions to obtain a feature representation of the control condition. The feature representation of the prompt text and the feature representation of the control condition are input into the diffusion model. The diffusion model uses the feature representation of the prompt text and the feature representation of the control condition to perform noise prediction and denoising on the noisy image for T time steps, obtaining the final generated image.

[0115] The denoising process described above can be understood as predicting normally distributed noise and performing denoising at each time step to restore the true image content. This process simulates the reverse process of a Markov chain of length T. T is the total time step of the diffusion model, a preset positive integer. A longer T improves denoising, but also increases the impact on computational performance. Therefore, a trade-off needs to be made between the two. This value can be empirically or experimentally determined.

[0116] During denoising, the denoising network in the image generation model performs attention-based processing on the feature representation of the prompt text and the latent vector corresponding to the noisy image to obtain the image latent vector. Denoising is then performed on this image latent vector for T time steps. At each time step, the image latent vector obtained at the previous time step is used to predict the noise at the current time step. The predicted noise is then used to denoise the image latent vector to obtain the image latent vector for the current time step. This process is repeated in this manner to obtain the final denoised image latent vector.

[0117] Finally, the denoised image latent vector obtained in the last step is input into the decoding network in the image generation model, and the decoding network uses the denoised image latent vector to perform decoding processing to obtain the generated image.

[0118] During the denoising process described above, the feature representation of the control condition may be input at all or some time steps. If the feature representation of the control condition is input at the current time step, the feature representation of the control condition is input into the denoising network, which fuses the image latent vector with the feature representation of the control condition at that time step. For example, each hidden layer in the denoising network fuses the latent vector predicted by that hidden layer with the feature representation of the control condition input to that hidden layer, and then inputs the fused latent vector into the next hidden layer.

[0119] Furthermore, in an embodiment of the present application, the final result of the generated image can be displayed to the user through the interactive interface, for example, the generated image can be displayed in the third area of ​​the interactive interface. If the user is not satisfied with the generated image, the input text or partition number can be adjusted and the image generation request can be resubmitted until the user is satisfied. The user can also download the image to facilitate its application in different occasions. Continuing with the above example, the corresponding generated image can be displayed in the third area of ​​the upper right part of the interactive interface shown in Figure 4. The "river", "rabbit", "bed" and "bright moon" in the image are all generated according to the partition position indicated by the user.

[0120] Furthermore, a reset function is also provided in the above interactive interface. If the user wants to re-enter the image generation instruction, or is not satisfied with the generated image and wants to adjust the image generation instruction, the input box corresponding to the image generation instruction can be cleared through the "Reset" component in Figure 4.

[0121] It can be seen from the above embodiments that users only need to mark the entity words in the input text with partition numbers when inputting text. This slightly expanded method can be used to control the position of objects in the generated image. There is no need to manually draw position boxes for each entity word, nor is there any need to manually enter the coordinates of the position boxes corresponding to each entity word. This makes the implementation simpler and more efficient, with low user input costs and a better experience.

[0122] The above method provided in the embodiment of the present application can be applied to a variety of scenarios. In the field of advertising and promotional design, the method of the present application can be used to generate advertising images with specific element positions. In the field of game development, game designers can specify the positions of characters, props and environments in the game through text descriptions, and then use models to generate corresponding virtual scenes. In the field of architectural design, designers can specify the positions of furniture, decorations and structural elements through text descriptions, and then use models to generate corresponding design drawings. At the same time, the above method provided in the embodiment of the present application can be applied to artistic creation, and artists and designers can use this model to realize their creativity and ideas. They can specify the position and attributes of graphic elements through text descriptions, and then use models to generate corresponding works of art or visualization effects. This text-based graph model that specifies the position of graphic elements can provide greater flexibility and position controllability, allowing users to accurately specify the desired graphic element positions to meet specific scenarios and needs.

[0123] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0124] According to another embodiment, an image generation device is provided. FIG6 shows a schematic block diagram of the image generation device according to one embodiment. The device can be provided on the server side of the architecture shown in FIG1 . As shown in FIG6 , the device 600 may include: an instruction acquisition unit 601, an instruction parsing unit 602, and an image generation unit 603. The main functions of each component unit are as follows:

[0125] The instruction acquisition unit 601 acquires an image generation instruction, where the image generation instruction includes input text and first position information, where the first position information is used to indicate at least one of a plurality of partitions into which the image area is pre-divided.

[0126] The instruction parsing unit 602 parses the image generation instruction to obtain the prompt text and the second position information.

[0127] The image generation unit 603 inputs the prompt text and the second position information into the image generation model, and obtains an image generated by the image generation model using the prompt text and the second position information.

[0128] As one possible implementation method, the image generation instruction may include a text sequence, where the text sequence includes input text and first position information annotated for entity words included in the input text.

[0129] As one possible implementation, the partitioning may include partitioning the image area into multiple grids, such as the 9-grid pattern shown in the left portion of FIG3 .

[0130] As one possible implementation method, the first location information may include a partition number.

[0131] As one of the possible implementation methods, the instruction parsing unit 602 can be specifically configured to: parse the input text and the first position information of the entity words annotated in the input text from the image generation instruction; use the input text to obtain the prompt text; use the first position information of the entity words annotated in the input text to obtain the position box information corresponding to the entity word as the second position information.

[0132] As one of the possible implementation methods, when the instruction parsing unit 602 obtains the position box information corresponding to the entity word by using the first position information annotated for the entity word included in the input text, it can be specifically configured as follows: identifying the first position information annotated for the entity word; if the first position information annotated for the entity word indicates a partition, then the edge of a partition is the position box corresponding to the entity word; if the first position information annotated for the entity word indicates multiple partitions, then the multiple partitions are fused, and the edge of the fused area is the position box corresponding to the entity word.

[0133] As one achievable manner, the instruction parsing unit 602 may be specifically configured to: randomly adjust the position of the position box within a disturbance range corresponding to the disturbance factor according to the set disturbance factor.

[0134] As one of the possible approaches, the image generation model is pre-trained based on the diffusion model.

[0135] As one of the feasible methods, the image generation model encodes the prompt text to obtain a feature representation of the prompt text; encodes the control condition containing the second position information to obtain a feature representation of the control condition; and uses the feature representation of the prompt text and the feature representation of the control condition to perform noise prediction and denoising on the noisy image for T time steps to obtain a generated image, where T is a preset positive integer.

[0136] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For the relevant parts, refer to the partial description of the method embodiment. The device embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0137] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0138] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0139] And an electronic device comprising:

[0140] one or more processors; and

[0141] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.

[0142] The present application also provides a computer program product, comprising a computer program, which implements the steps of any one of the methods described in the aforementioned method embodiments when executed by a processor.

[0143] 7 exemplarily shows the architecture of an electronic device, which may include a processor 710, a video display adapter 711, a disk drive 712, an input / output interface 713, a network interface 714, and a memory 720. The processor 710, the video display adapter 711, the disk drive 712, the input / output interface 713, the network interface 714, and the memory 720 may be communicatively connected via a communication bus 730.

[0144] Among them, the processor 710 can be implemented by a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solutions provided in this application.

[0145] The memory 720 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 720 can store an operating system 721 for controlling the operation of the electronic device 700, and a basic input and output system (BIOS) 722 for controlling the low-level operations of the electronic device 700. In addition, a web browser 723, a data storage management system 724, and an image generating device 725, etc. can also be stored. The above-mentioned image generating device 725 can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided by the present application is implemented by software or firmware, the relevant program code is stored in the memory 720 and is called and executed by the processor 710.

[0146] The input / output interface 713 is used to connect input / output modules to implement information input and output. The input / output modules can be configured as components in the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0147] The network interface 714 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0148] The bus 730 comprises a pathway for transmitting information between the various components of the device (eg, the processor 710 , the video display adapter 711 , the disk drive 712 , the input / output interface 713 , the network interface 714 , and the memory 720 ).

[0149] It should be noted that although the above device only shows the processor 710, video display adapter 711, disk drive 712, input / output interface 713, network interface 714, memory 720, bus 730, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may also include only the components necessary to implement the solution of the present application, and does not necessarily include all the components shown in the figure.

[0150] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer program product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present application or certain parts of the embodiments.

[0151] The above is a detailed introduction to the technical solutions provided by this application. Specific examples are used herein to illustrate the principles and implementation methods of this application. The description of the above embodiments is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the contents of this specification should not be understood as limiting this application.

Claims

1. An image generation method, characterized in that: The method comprises: Obtaining an image generation instruction, the image generation instruction including input text and first position information, the first position information being used to indicate at least one of a plurality of partitions into which the image area is pre-divided; Parsing the image generation instruction to obtain a prompt text and second position information; The prompt text and the second position information are input into an image generation model, and an image generated by the image generation model using the prompt text and the second position information is obtained.

2. The method according to claim 1, characterized in that The image generation instruction includes a text sequence, where the text sequence includes input text and first position information annotated for entity words included in the input text.

3. The method according to claim 1, characterized in that The partitioning includes partitioning obtained by dividing the image area into multiple grids.

4. The method according to claim 1, wherein The first location information includes a partition number.

5. The method according to claim 1, wherein The image generation instruction is parsed to obtain the prompt text and the second location information, including: Parsing the input text and first position information of entity words included in the input text from the image generation instruction; Obtaining prompt text using the input text; The first position information annotated for the entity word included in the input text is used to obtain the position box information corresponding to the entity word as the second position information.

6. The method according to claim 5, characterized in that The obtaining of the position box information corresponding to the entity word by using the first position information annotated for the entity word included in the input text includes: Identifying first position information where the entity word is marked; If the first position information marked with the entity word indicates a partition, taking the edge of the partition as the position box corresponding to the entity word; If the first position information marked with the entity word indicates multiple partitions, the multiple partitions are fused, and the edges of the fused regions are used as the position boxes corresponding to the entity word.

7. The method according to claim 5, characterized in that The method further comprises: According to the set disturbance factor, the position of the position box is randomly adjusted within the disturbance range corresponding to the disturbance factor.

8. The method according to any one of claims 1 to 7, characterized in that The image generation model is pre-trained based on a diffusion model.

9. The method according to claim 8, characterized in that The image generation model generates an image using the prompt text and the second position information, including: The image generation model encodes the prompt text to obtain a feature representation of the prompt text; encoding the control condition including the second position information to obtain a feature representation of the control condition; Using the feature representation of the prompt text and the feature representation of the control condition, noise prediction and denoising are performed on the noisy image for T time steps to obtain a generated image, where T is a preset positive integer.

10. An image generation method, characterized in that: Applied to a cloud server, the method includes: Obtaining an image generation instruction from a user device, the image generation instruction including input text and first position information, where the first position information is used to indicate at least one of a plurality of partitions into which an image area is pre-divided; Parsing the image generation instruction to obtain a prompt text and second position information; Inputting the prompt text and the second position information into an image generation model, and obtaining an image generated by the image generation model using the prompt text and the second position information; The generated image is returned to the user device.

11. The method according to claim 10, characterized in that Acquiring the image generation instruction from the user device includes: providing an interactive interface to the user device, and acquiring the image generation instruction input by the user in a first area of the interactive interface; The method further includes: displaying the prompt text and the second location information in a second area of the interactive interface; and / or displaying the generated image in a third area of the interactive interface.

12. An image generating device, characterized in that: The device comprises: an instruction acquisition unit, configured to acquire an image generation instruction, wherein the image generation instruction includes input text and first position information, wherein the first position information is used to indicate at least one of a plurality of partitions into which the image area is pre-divided; An instruction parsing unit, which parses the image generation instruction to obtain a prompt text and second position information; The image generation unit inputs the prompt text and the second position information into an image generation model, and obtains an image generated by the image generation model using the prompt text and the second position information.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

14. An electronic device, characterized in that: include: one or more processors; as well as A memory associated with the one or more processors, the memory being configured to store program instructions, wherein when the program instructions are read and executed by the one or more processors, the program instructions execute the steps of the method according to any one of claims 1 to 11.

Images