Image generation method and related apparatus, device, and storage medium

Abstract

The application discloses an image generation method and related devices, equipment and storage media, wherein the image generation method comprises: obtaining an image generation request; obtaining the expected coordinates of an instance object based on the image generation request; normalizing based on the expected coordinates of the instance object, and determining the index token of the instance object in the index token corresponding to each of a plurality of preset normalized coordinates; constructing a prompt text instruction based on the index token of the instance object; and obtaining a target generated image for responding to the image generation request based on the model generated image output by the generative model in response to the prompt text instruction. The above scheme can reduce the calculation and video memory overhead of image generation, and improve the accuracy of the spatial layout of the instance object in the generated image, especially in a multi-instance object scene.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to an image generation method and related apparatus, devices and storage media. Background Technology

[0002] With the development of diffusion models and large-scale language model training techniques, image generation systems (such as DALL-E, Stable Diffusion, etc.) and unified multimodal models have integrated perception, reasoning and generation capabilities.

[0003] However, existing image generation techniques struggle to achieve accurate spatial positioning of instance objects, especially in scenarios with multiple instance objects (e.g., four or more instance objects), where performance degrades significantly and computational overhead is substantial. Therefore, reducing the computational and memory overhead of image generation while improving the accuracy of the spatial layout of instance objects in generated images, particularly in scenarios with multiple instance objects, has become a pressing issue. Summary of the Invention

[0004] This application provides at least one image generation method, related apparatus, device, and storage medium.

[0005] A first aspect of this application provides an image generation method, comprising: acquiring an image generation request; wherein the image generation request describes an instance object in an image to be generated and the layout position of the instance object; obtaining the expected coordinates of the instance object based on the image generation request; normalizing the expected coordinates of the instance object and determining the index words of the instance object in a plurality of preset index words corresponding to the normalized coordinates; constructing a prompt text instruction based on the index words of the instance object; wherein the prompt text instruction describes the image to be generated in natural language and embeds the index words of the instance object in the prompt text instruction at the words of the instance object; and generating a model image based on the model output by the generative model in response to the prompt text instruction to obtain a target generated image for responding to the image generation request.

[0006] A second aspect of this application provides an image generation apparatus, comprising: a content acquisition module, a coordinate determination module, an index determination module, a prompt construction module, and a model generation module. The content acquisition module is used to acquire an image generation request, wherein the image generation request describes an instance object in the desired generated image and the layout position of the instance object. The coordinate determination module is used to obtain the desired coordinates of the instance object based on the image generation request. The index determination module is used to normalize the desired coordinates of the instance object and determine the index terminology of the instance object from the index terminology corresponding to each of a preset plurality of normalized coordinates. The prompt construction module is used to construct a prompt text instruction based on the index terminology of the instance object, wherein the prompt text instruction describes the desired generated image in natural language, and embeds the index terminology of the instance object at the words corresponding to the instance object in the natural language statement. The model generation module is used to generate a target generated image in response to the image generation request based on the model generated image output by the generative model in response to the prompt text instruction.

[0007] A third aspect of this application provides an electronic device, including at least a memory and a processor coupled to each other, wherein the memory stores at least program instructions, and the processor executes the program instructions to implement the image generation method of the first aspect described above.

[0008] A fourth aspect of this application provides a computer-readable storage medium storing program instructions executable by a processor, the program instructions being used to implement the image generation method of the first aspect described above.

[0009] The above scheme obtains an image generation request, which describes the instance objects and their layout positions in the desired generated image. Based on the image generation request, the desired coordinates of the instance objects are obtained, and then normalized. Index terms for the instance objects are determined from the index terms corresponding to several preset normalized coordinates. Based on these index terms, a prompt text instruction is constructed. This prompt text instruction describes the desired generated image in natural language, and the index terms for the instance objects are embedded in the words related to them. Finally, the model generates an image based on the output of the generative model in response to the prompt text instruction, resulting in the target generated image used to respond to the image generation request. This process involves embedding the index terms for the instance objects in the prompt text instruction. By embedding index terms representing the layout coordinates of instance objects into the language flow, layout constraints can be directly encoded as text terms embedded in the language stream. This allows for a deep fusion of spatial and semantic constraints, enabling generative models to generate images. This improves the accuracy of the spatial layout of instance objects in the generated images, especially in scenarios with multiple instance objects. Furthermore, since each normalized coordinate has its own pre-set index terminology, and normalization is performed during image generation in conjunction with the expected coordinates of the instance objects to determine these index terms, and the instance object's index terms are directly embedded at the words in the prompt text instead of the expected coordinates, the input terminology overhead during image generation is reduced. This helps save sequence length in complex layout scenarios, thus reducing computational and memory overhead. Therefore, it reduces the computational and memory overhead of image generation and improves the accuracy of the spatial layout of instance objects in the generated images, especially in scenarios with multiple instance objects. Attached Figure Description

[0010] Figure 1 This is a schematic flowchart of an embodiment of the image generation method of this application;

[0011] Figure 2a This is a schematic diagram of an embodiment of the image generation method of this application; Figure 2b This is a schematic diagram of an embodiment of text-guided image generation in this application; Figure 2c This is a schematic diagram of an embodiment of image generation guided by text and graphics in this application; Figure 3 This is a schematic diagram of the framework of an embodiment of the image generation apparatus of this application; Figure 4 This is a schematic diagram of the framework of an embodiment of the electronic device of this application; Figure 5 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0012] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0013] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0014] In this paper, the terms "system" and "network" are often used interchangeably. The term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the slash " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this paper indicates two or more objects.

[0015] Please see Figure 1 , Figure 1 This is a schematic flowchart of an embodiment of the image generation method of this application. It should be noted that the process operations in this embodiment can be performed by an electronic device with computing capabilities or related equipment containing an electronic device; therefore, the specific structure and type of the electronic device and related equipment are not limited herein. Specifically, this embodiment may include the following steps: Step S11: Obtain the image generation request.

[0016] In this embodiment of the disclosure, the image generation request can describe the instance objects in the desired generated image and the layout positions of these instance objects. It should be noted that the desired generated image represents the image data that the target object triggering the image generation request expects to generate, and it can at least include which instance objects are included and their layout positions in the desired generated image. The number of instance objects can be one or more, and the type of each instance object can be a person, animal, building, plant, or object, etc. For ease of understanding, taking an office scene poster as an example, the desired generated image can include three instance objects: a coffee cup, a notebook, and a pen. The layout positions of these instance objects can be: the coffee cup in the upper right area, the notebook in the middle, and the pen in the lower left area. In this case, the image generation request can include, but is not limited to, the following: "Generate an office scene poster containing a coffee cup, a notebook, and a pen, with the following layout: coffee cup in the upper right area, notebook in the middle, and pen in the lower left area." Of course, the above example is merely one possible example in practical application, and the application scenarios and specific content of the image generation request are not limited here, nor will they be listed in detail.

[0017] In one implementation scenario, the layout position of the instance object described in the image generation request can be either a positional coordinate or a relative orientation. For example, when the layout position of the instance object described in the image generation request is a relative orientation, the relative orientation can be described using terms such as, but not limited to, top left, top right, bottom left, bottom right, center, etc., without limitation. For ease of understanding, taking the aforementioned office scene poster as an example, the layout position in the image generation request in the above example is a relative orientation. Alternatively, for another example, when the layout position of the instance object described in the image generation request is a positional coordinate, the image generation request can also describe the desired image size (e.g., 64). 64, 128 128, 1080 (e.g., 1024). To facilitate understanding, let's take the aforementioned office scene poster as an example again. The image generation request may include, but is not limited to, the following: "Generate an image containing a coffee cup, a notebook, and a pen (64 images)." The following is a suggested layout for an office scene poster (model 64): a coffee cup in the boundary region between (40,40) and (64,64), a notebook in the boundary region between (20,20) and (40,40), and a pen in the boundary region between (0,0) and (20,20) (where the two sets of coordinates for each instance object are the coordinates of the two diagonal vertices of its bounding box). Of course, the above examples are merely a few possible descriptions of the layout of instance objects in image generation requests in practical applications. Other possible descriptions are not limited here, nor will they be listed individually.

[0018] In one implementation scenario, to facilitate the subsequent determination of index terms for instance objects, a set of index terms corresponding to several preset normalized coordinates can be obtained in advance. Index terms are tokens that the generative model can understand. When processing input data, the generative model needs to map the input data to index terms before subsequent processing and output. In this embodiment, there is a pre-defined mapping relationship between the preset normalized coordinates and the index terms. Optionally, the index terms corresponding to the preset normalized coordinates are obtained in the following way: a preset number of index terms are registered as special tokens in the generative model's vocabulary, and the preset normalized coordinates are mapped to their corresponding index terms. Large models typically need to map input text, images, audio, etc., into tokens that can be recognized and uniformly encoded by the model. In this embodiment, the vocabulary serves as a token mapping tool for the generative model, used to characterize the correspondence between the data input to the generative model and the tokens that the generative model can recognize. In this way, by pre-registering a certain number of index terms as special tokens in the vocabulary, compared to the situation in generative models where BPE (Byte Pair Encoding) is directly applied to normalized coordinates without registering special tokens, resulting in each dimension of the coordinate being divided into multiple tokens, the token overhead required for the expected coordinates of instance objects can be further reduced. This shortens the sequence length processed by the model in complex layout scenarios, reduces computation and memory overhead, and improves training and inference efficiency.

[0019] Optionally, the maximum number of decimal places for the preset normalized coordinates is the target number of decimal places (e.g., 2 decimal places when retaining two decimal places, 3 decimal places when retaining three decimal places, 4 decimal places when retaining four decimal places, etc.). The total number of index terms is 10 raised to the power of the target number of decimal places, so that the number of index terms corresponds to the granularity of the normalized coordinates, thus enabling a one-to-one correspondence between the normalized coordinates and the index terms. Furthermore, the higher the precision of the layout position expression, the larger the target number of decimal places (in the aforementioned example, the more decimal places retained, the larger the target number of decimal places); conversely, the less precision the layout position expression, the smaller the target number of decimal places (in the aforementioned example, the fewer decimal places retained, the smaller the target number of decimal places).

[0020] In a specific implementation scenario, the index terms corresponding to several preset normalized coordinates can be obtained as follows: First, an integer interval can be obtained based on the target integer corresponding to the target number of digits. It should be noted that the total number of integers in the interval can be 10 raised to the power of the target number of digits (e.g., when the target number of digits is 3, the total number of integers in the interval can be 1000, meaning the preset number of registered index terms is 1000). The upper limit of the interval can be the target integer, which can be the largest integer with the target number of digits (e.g., when the target number of digits is 3, the target integer can be the largest three-digit integer, i.e., 999) or 10 raised to the power of the target number of digits (e.g., when the target number of digits is 3, the target integer can be 10 to the power of 3, i.e., 1000). Based on this, index terms corresponding to the normalized coordinates with the same order as the selected integers can be constructed based on the integers selected sequentially within the interval. For example, for the first normalized coordinate in a sequence of normalized coordinates sorted from smallest to largest, the first integer in the integer range can be selected to construct its corresponding index term. To facilitate the subsequent distinction between index terms and ordinary terms, the index term can be constructed using the method of "index identifier" + "selected integer". Taking the first integer as 0 and the index identifier as coord_q as an example, the index term corresponding to the first normalized coordinate can be constructed as coord_q0. To further facilitate the distinction between ordinary terms and index terms, the construction method of "index identifier" + "selected integer" can be further enclosed with tags such as "<>", as mentioned above, the index term corresponding to the first normalized coordinate can be constructed as...<coord_q0> Of course, the above examples are merely a few possible examples of constructing index terms in practical applications. Other possible construction methods are not limited here, nor will they be listed one by one. In the above method, the decimal places of several normalized coordinates are at most the target number of digits. An integer interval is obtained based on the target integer corresponding to the target number of digits, and the total number of integers in the integer interval is 10 raised to the power of the target number of digits. The upper limit of the integer interval is the target integer, and the target integer is the largest integer of the target number of digits or 10 raised to the power of the target number of digits. Then, based on each integer selected in sequence in the integer interval, an index term corresponding to the normalized coordinates with the same order as the selected integers is constructed. Since the normalized coordinates are mapped to the index terms constructed with their corresponding integers, it can reduce the overhead compared to directly embedding the normalized coordinates themselves in the prompt text instructions.

[0021] In a specific implementation scenario, for ease of understanding, let's take a target length of 3 as an example. As mentioned earlier, we can pre-construct 1000 index terms, each representing a different normalized coordinate when the target length is 3. These 1000 index terms are denoted as follows:<coord_000> ,<coord_001> ...<coord_q> ...<coord_999> Not generally, for any normalized coordinate... As for (normalized coordinates are not limited to a maximum of the target number of decimal places, meaning they can be more or less than the target number of decimal places, but the total number of indexed terms is constrained to be 10 to the power of the target number of decimal places), the indexed term corresponding to it in a total number of indexed terms raised to the power of the target number of decimal places can be calculated using the following formula: q=clip(round(u 1000), 0, 999) In the above formula, the `round` function represents rounding, and the `clip` function represents a numerical constraint function, such as when `round(u...`... When 1000 is between 0 and 999 (i.e., the largest integer of the target number of bits), it can be constrained to itself. When 1000 is less than 0 (where 1000 represents the target number of bits raised to the power of 10), it can be constrained to 0. When 1000 is higher than 999, it can be constrained to 999. That is, [0, 999] represents the aforementioned integer range, and 999 represents the aforementioned target integer. For ease of understanding, let's take the normalized coordinate 0.1342, located between 0 and 1, as an example. Multiplying it by 1000 and rounding it gives 134. Since it falls between 0 and 999, its corresponding index term is...<coord_134> In this way, each normalized coordinate occupies only one lexical unit. Of course, the above example is just one possible example in practical application. When the target number of digits is set to other values, the same principle can be applied (e.g., when the target number of digits is set to four, the above formula can be adapted to q=clip(round(u 10000), 0, 9999), and the range of indexed terms is<coord_000> to<coord_9999> Other possible scenarios will not be listed here.

[0022] Step S12: Based on the image generation request, obtain the expected coordinates of the instance object.

[0023] In one implementation scenario, as a possible approach, as mentioned earlier, the layout position of the instance object described in the image generation request can be either positional coordinates or relative orientation. To obtain the desired coordinates of the instance object for subsequent normalization mapping to corresponding index terms, the description method of the layout position in the image generation request can be detected. This description method includes either positional coordinates or relative orientation. If the layout position is described using positional coordinates, coordinate extraction can be performed based on the image generation request to obtain the desired coordinates of the instance object. If the layout position is described using relative orientation, information interaction can be performed between the image generation request and the target object to obtain the desired coordinates of the instance object. It should be noted that, as mentioned earlier, the image generation request can be triggered by the target object. The above method detects whether the layout position in the image generation request is described using positional coordinates or relative orientation. If the layout position is described using positional coordinates, coordinate extraction is performed based on the image generation request to obtain the expected coordinates of the instance object. If the layout position is described using relative orientation, information interaction is performed between the image generation request and the target object to obtain the expected coordinates of the instance object. Furthermore, the image generation request is triggered by the target object. In practical applications, this method can support both direct description of the instance object's layout position using positional coordinates and description of the instance object's layout position using relative orientation.

[0024] In a specific implementation scenario, where the layout is described using location coordinates, for ease of understanding, let's continue with the aforementioned office scene poster as an example. We can directly request the image generation function to "generate an image containing a coffee cup, notebook, and pen (64...)". The following coordinate extraction is performed on an office scene poster with the following layout: a coffee cup in the boundary area between (40,40) and (64,64), a notebook in the boundary area between (20,20) and (40,40), and a pen in the boundary area between (0,0) and (20,20). This yields the expected coordinates of the instance objects "coffee cup" (40,40, (64,64), "notebook" (20,20, (40,40)," and "pen" (0,0, (20,20)). This example is merely one possible illustration of coordinate extraction for obtaining expected coordinates in a practical application using an office scene poster as an example. Other possible scenarios are not limited here, nor will they be listed individually.

[0025] In a specific implementation scenario, when the layout position is described in relative orientation, a drawing canvas of the desired generated image can be displayed on the interaction interface with the target object, prompting the target object to specify the absolute position of the instance object on the drawing canvas. For example, a prompt message such as "Please specify the instance object at the desired position on the drawing canvas" can be output; the specific content of the prompt message is not limited here. Furthermore, the canvas size can be the same as the image size of the desired generated image. If the target object does not specify the image size of the desired generated image, the canvas size can be set to a default value; the specific value of the default value is not limited here. Based on this, the desired coordinates of the instance object can be obtained in response to the absolute position specified by the target object on the drawing canvas. For example, the target object can specify the absolute position of the instance object by drawing a bounding box on the drawing canvas. In this case, the desired coordinates of the instance object can be obtained from the vertex coordinates of the bounding box (e.g., the actual coordinates of the bottom left and top right vertices of the bounding box can be taken, or the application can take the actual coordinates of the top left and bottom right vertices of the bounding box). The above method allows the target object to specify the absolute position of the instance object on the drawing canvas, thus adapting to more flexible and diverse image generation needs.

[0026] In another implementation scenario, as a possible alternative, distinct from the aforementioned implementation where the layout position is described in relative orientation, in response to the relative orientation description of the layout position, the layout positions of each instance object described in relative orientation can be transformed based on a coordinate transformation model to obtain the desired coordinates of each instance object. It should be noted that the coordinate transformation model can include, but is not limited to, ordinary neural network models such as convolutional neural networks, or large-scale network models such as large language models; the network architecture of the coordinate transformation model is not limited here. For ease of understanding, taking a large language model as an example, a large model instruction can be constructed based on the image size of the desired generated image and the layout positions of each instance object described in relative orientation. This large model instruction instructs the large language model to convert the layout positions of the instance objects described in relative orientation into image coordinates. The output coordinates of the large language model in response to the large model instruction can then be obtained as the desired coordinates of the instance objects. Taking the aforementioned office scene poster as an example, as mentioned before, the image generation request can include "generating an office scene poster containing a coffee cup, a notebook, and a pen, with the following layout: coffee cup in the upper right area, notebook in the middle, and pen in the lower left area." Therefore, the desired image size is 64. In the case of 64, the large model instruction may include, but is not limited to, the following: "The expected image size of the generated image is 64". Example 64 contains three instance objects: a coffee cup, a notebook, and a pen. The coffee cup is located in the upper right area, the notebook in the middle, and the pen in the lower left area. Based on this, the image coordinates of the three instance objects are determined. The above large model instruction is input into the large language model for processing, and the output coordinates of the large language model are obtained: "Coffee cup: (40,40), (64,64); Notebook: (20,20), (40,40); Pen: (0,0), (20,20)". Of course, the above example is only one possible example of using the large language model as an example to perform coordinate transformation on the layout positions described by relative orientation to obtain the desired coordinates. Other possible situations are not limited here, nor will they be listed one by one.

[0027] Step S13: Normalize the expected coordinates of the instance object, and determine the index term of the instance object from the index terms corresponding to the preset normalized coordinates.

[0028] Specifically, as mentioned earlier, the number of decimal places of several normalized coordinates can be up to the target number of decimal places (e.g., the target number of decimal places is 3 when retaining three decimal places, and the target number of decimal places is 4 when retaining four decimal places, etc.). In order to determine the index term of the instance object, the expected coordinates of the instance object can be normalized to the target number of decimal places to obtain the first coordinate of the instance object. Then, the first coordinate of the instance object is constrained based on the numerical range of several normalized coordinates (i.e., the numerical range from 0 to 1) to obtain the second coordinate of the instance object. It should be noted that for the x-axis, it can be divided by the width of the desired generated image, retaining the target number of decimal places. Finally, it can be constrained using the aforementioned `clip` function (the difference being that the constraint range of the `clip` function here is the numerical range between 0 and 1) to obtain the normalized x-axis. Similarly, for the y-axis, it can be divided by the height of the desired generated image, retaining the target number of decimal places. Finally, it can be constrained using the aforementioned `clip` function (the difference being that the constraint range of the `clip` function here is the numerical range between 0 and 1) to obtain the normalized y-axis. Furthermore, the normalized x-axis and y-axis are the second coordinates of the instance object. Based on this, index terms corresponding to the second coordinates of the instance object can be selected as the index terms of the instance object. As one possible example, since the index terms corresponding to several preset normalized coordinates have already been obtained through the aforementioned methods, after obtaining the second coordinate of the example object, the closest normalized coordinate can be found among the aforementioned preset normalized coordinates, and the index term corresponding to the closest normalized coordinate can be used as the index term for the instance. As another possible example, the aforementioned formula for calculating index term q can also be used. The second coordinate can be substituted (with the x-coordinate and y-coordinate substituted separately) into the formula for calculating index term q to obtain the index terms for the instance object (i.e., the index term corresponding to the x-coordinate and the index term corresponding to the y-coordinate). The above method first normalizes the expected coordinates of the instance object to the target number of bits to obtain the first coordinates of the instance object. Then, the first coordinates of the instance object are constrained based on the numerical range of several normalized coordinates to obtain the second coordinates of the instance object. The index term corresponding to the second coordinate of the instance object can then be selected as the index term for the instance object. This method allows the expected coordinates to be mapped to the corresponding index terms through coordinate normalization and coordinate numerical constraints, which helps improve the convenience of index mapping.

[0029] For ease of understanding, let's continue with the aforementioned office scene poster as an example. For the instance object "coffee cup," its expected coordinates include two sets of coordinates: (40, 40) and (64, 64). For the first set of coordinates, we can divide its horizontal coordinate by the width of the desired generated image, retaining the desired number of decimal places (e.g., three decimal places), and finally constrain it using the aforementioned `clip` function (where the constraint range of the `clip` function is 0 to 1). This yields a normalized horizontal coordinate of 0.625. Referring to the aforementioned formula for calculating the index term `q`, we can obtain its corresponding index term as...<coord_625> Similarly, its ordinate can be divided by the desired height of the generated image, and the number of decimal places can be retained according to the target number of decimal places (e.g., three decimal places). Finally, the ordinate can be constrained by the aforementioned clip function (where the constraint range of the clip function is 0 to 1), resulting in a normalized ordinate of 0.625. Referring to the aforementioned formula for calculating the index term q, its corresponding index term can be obtained as follows:<coord_625> Furthermore, for the second set of coordinates, its x-coordinate can be divided by the width of the desired generated image, and the number of decimal places can be retained according to the target number of digits (e.g., three decimal places). Finally, the x-coordinate is constrained by the aforementioned clip function (the constraint range of the clip function is 0 to 1), resulting in a normalized x-coordinate of 1. Referring to the aforementioned formula for calculating the index term q, its corresponding index term can be obtained as follows:<coord_999> Similarly, its ordinate can be divided by the desired height of the generated image, and the number of decimal places can be retained according to the target number of decimal places (e.g., three decimal places). Finally, the ordinate can be constrained by the aforementioned clip function (where the constraint range of the clip function is 0 to 1) to obtain the normalized ordinate 1. Referring to the aforementioned formula for calculating the index term q, its corresponding index term can be obtained as follows.<coord_999> In other words, the index term for the instance object "coffee cup" can be represented as:<coord_625><coord_625><coord_999><coord_999> To facilitate explicit distinction between the index terminology and the subsequent prompt text instructions, a starting term (e.g., ...) can be placed around the index terminology of the instance object. <bbox> ) and ending words (e.g., < / bbox> The index term of the instance object "coffee cup" in the subsequently constructed prompt text instruction can be represented as follows: <bbox><coord_625><coord_625><coord_999><coord_999>< / bbox> Similarly, for the instance object "notebook", its expected coordinates (20,20) and (40,40) can also be converted into corresponding index terms in the same way as described above: <bbox><coord_313><coord_313><coord_625><coord_625>< / bbox> Similarly, for the example object "pen", its expected coordinates (0,0) and (20,20) can also be converted into the corresponding index terms in the aforementioned manner: <bbox><coord_0><coord_0><coord_313><coord_313>< / bbox>Of course, the above example is just one possible way to convert expected coordinates into index terms using the aforementioned office scene poster as an example. Other possible scenarios are not limited here, nor will they be listed one by one.

[0030] Step S14: Construct prompt text instructions based on the index terms of the instance object.

[0031] In this embodiment of the disclosure, the prompt text instruction can describe the desired image to be generated in natural language, and embed the index terminology of the instance object at the words related to the instance object in the prompt text instruction. For example, the index terminology of the instance object is directly bound to the semantic phrase of the corresponding instance object in the description of the desired image.

[0032] In one implementation scenario, after obtaining the index terms of the instance objects, a prompt text instruction can be constructed based on the aforementioned image generation request and the instance object's index terms. As a possible example, to improve the efficiency of constructing the prompt text instruction, the aforementioned image generation request and instance object's index terms can be processed based on a large language model, allowing the large language model to construct the prompt text instruction. For instance, a large model instruction can be constructed based on the image generation request and instance object's index terms, instructing the large language model to generate a descriptive statement of the desired generated image in conjunction with the image generation request, embedding the instance object's index terms after the semantic phrase of the instance object in the descriptive statement. Of course, the large model instruction can also contain other content, such as requirements for generating the descriptive statement (e.g., processing requirements when multiple instance objects of the same type are present, modification requirements for the image generation request, model output format requirements), and relevant examples to facilitate the large language model's understanding of the task. Further information that the large model instruction can include is not limited here, nor will it be listed in detail. Based on this, the output text of the large language model responding to the large model instruction can be obtained as the prompt text instruction.

[0033] In one implementation scenario, for ease of understanding, let's take the aforementioned office scene poster as an example again. Based on the index terms determined for each instance object, we can construct prompt text instructions including but not limited to: "a coffee cup". <bbox><coord_625><coord_625><coord_999><coord_999>< / bbox> On the table, a notebook <bbox><coord_313><coord_313><coord_625><coord_625>< / bbox> There is a blank page and a fountain pen. <bbox><coord_0><coord_0><coord_313><coord_313>< / bbox> "Leaning against the laptop, with soft lighting in the office background." Of course, the above example is just one possible instance of prompt text instructions in practical applications. Other possible scenarios are not limited here, nor will they be listed one by one.

[0034] Step S15: Based on the model-generated image output by the generative model in response to the prompt text command, obtain the target generated image used to respond to the image generation request.

[0035] In one implementation scenario, word segmentation can be performed based on prompt text instructions to obtain the first word sequence, and then based on the removal of index words (i.e., removing the aforementioned words)...<coord_q> The index terminology also includes embedded start terminology. <bbox> and ending word< / bbox> In this case, it is necessary to further remove the starting word. <bbox> and ending word< / bbox> The prompt text following the first word sequence is segmented to obtain a second word sequence. This second word sequence is then diffused to obtain a first velocity field, and the same diffusion process is repeated to obtain a second velocity field. These two velocity fields can then be fused to obtain a fused velocity field, which is then used for decoding to obtain the target generated image. It's important to note that the velocity field is a key physical quantity defined from the perspective of the stochastic dynamics of the diffusion process. Its core physical meaning is the instantaneous evolution rate of the state vector (pixel value or feature vector) at each location in the image pixel space at any given moment during the diffusion process. Essentially, it's a quantitative description of the "denoising / generation direction" of the diffusion process. Of course, the above description is merely an exemplary description of the velocity field obtained through the diffusion operation; for details, please refer to the technical details of the diffusion mechanism, which will not be elaborated upon here. The above method performs word segmentation based on the prompt text instruction to obtain a first word sequence, and then performs word segmentation based on the prompt text instruction after removing the index words to obtain a second word sequence. The first word sequence is then diffused to obtain a first velocity field, and the second word sequence is diffused to obtain a second velocity field. The first and second velocity fields are then fused to obtain a fused velocity field, and finally, the target generated image is obtained by decoding based on the fused velocity field. Since the velocity field is fused in two branches during the image generation process, with both processing branches containing word indexes and processing branches without word indexes, the double fidelity of layout constraints and semantic constraints can be enhanced.

[0036] In a specific implementation scenario, the detailed processes of diffusion based on the first word sequence and diffusion based on the second word sequence can be found in the relevant descriptions of diffusion models, and will not be repeated here. Please refer to the relevant documentation. Figure 2a , Figure 2a This is a schematic diagram illustrating an embodiment of the image generation method of this application. Figure 2a As shown, a generative model can include a tokenizer, a multimodal unified model (MMRM), and a decoder (e.g., a VAE decoder). It should be noted that the tokenizer can be used to perform word segmentation on the aforementioned word sequence, the MMRM can be used to perform the aforementioned diffusion operation, and the decoder can be used to perform the aforementioned decoding operation. For example, the generative model can be based on a Mixture of Transformers (MoT) architecture, which can integrate dual Transformer expert modules (responsible for multimodal understanding and generation, respectively). As a possible example, the number of Transformer layers can be set to 28, the number of attention heads can be set to 28, and the embedding dimension d can be set to 3584. Of course, the above example is only one possible setup in practical applications; the parameter configuration of the Mixture of Transformers is not limited here, nor will it be listed in detail. Furthermore, for the specific mechanism of the generative model, please refer to the technical details of the Mixture of Transformers architecture, which will not be elaborated here.

[0037] In a specific implementation scenario, after obtaining the first and second velocity fields, they can be fused to obtain a fused velocity field. Specifically, the difference velocity field between the first and second velocity fields can be obtained, along with its scaling factor. This difference velocity field is then scaled based on the scaling factor to obtain a scaled velocity field. Finally, the fused velocity field can be obtained by summing the scaled velocity field with the second velocity field. For ease of description, the first velocity field at time t during the diffusion process can be denoted as... Let the second velocity field at time t be denoted as Let the scaling factor be denoted as Then the fusion velocity field at time t It can be represented as:

[0038] In the above formula, the scaling factor can be set according to the actual application needs. For example, if more emphasis is placed on the first velocity field, the scaling factor can be set larger; conversely, if more emphasis is placed on the second velocity field, the scaling factor can be set smaller. This balances layout consistency and generation quality based on the scaling factor. The specific value of the scaling factor is not limited here. The above method obtains the difference velocity field between the first and second velocity fields, obtains the scaling factor of the difference velocity field, and then scales the difference velocity field based on the scaling factor to obtain the scaled velocity field. Finally, the scaled velocity field is summed with the second velocity field to obtain the fused velocity field. This allows the scaling factor to guide the full fusion of the first and second velocity fields.

[0039] In a specific implementation scenario, after obtaining the fused velocity field, decoding can be performed directly based on it. Alternatively, after obtaining the fused velocity field, decoding can be delayed. Instead, a normalization factor can be obtained based on the ratio of the magnitudes of the second velocity field and the target velocity field. The target velocity field is the sum of the fused velocity field and a preset minimum value. The fused velocity field is then normalized based on the normalization factor to obtain a new fused velocity field. Decoding can then be performed based on this new fused velocity field to obtain the target generated image. For ease of description, the preset minimum value can be denoted as ε. The new fused velocity field... It can be represented as:

[0040] In the above formula, α represents the normalization factor. This represents the standard text-guided model velocity prediction (which in this example could be the second velocity field), ||.|| N The above method, before decoding based on the fused velocity field, first obtains a normalization factor based on the ratio of the magnitude of the second velocity field to the target velocity field. The target velocity field is the sum of the fused velocity field and a preset minimum value. The fused velocity field is then normalized based on the normalization factor to obtain a new fused velocity field. Decoding is then performed based on the new fused velocity field to obtain the target generated image, which can stabilize the intensity through velocity normalization.

[0041] In one implementation scenario, the image generation request can include not only the instance objects and their layout positions in the desired generated image, but also reference images of the instance objects. For ease of understanding, using the aforementioned office scene poster as an example, the image generation request can also include reference images of the instance objects "coffee cup," "notebook," and "pen," guiding the generative model to generate the desired generated image containing each instance object according to its reference images. Of course, in practical applications, the image generation request may not include reference images of the instance objects, may include reference images of individual instance objects, or may include reference images of all instance objects; this is not limited here. When the image generation request also includes reference images of the instance objects, a new prompt text instruction can be constructed based on the instance object's index terms and reference images. Furthermore, the source identifier of the instance object is embedded at the word position in the new prompt text instruction. This source identifier identifies the reference image from which the instance object originates, with each reference image corresponding to a different source identifier. Based on this, the target generated image used to respond to the image generation request can be obtained by using the model-generated image output by the generative model in response to the new prompt text instruction and reference images. The above method, when the image generation request also includes a reference image of the instance object, constructs a new prompt text instruction based on the index terms of the instance object and the reference image. In the new prompt text instruction, the source identifier of the instance object is also embedded at the word of the instance object. The source identifier is used to identify the reference image from which the instance object originates. Each reference image corresponds to a different source identifier. Thus, the model-generated image output by the generative model in response to the new prompt text instruction and the reference image is used to obtain the target generated image for responding to the image generation request. In this way, when the image generation request also includes a reference image of the instance object, the correspondence between the generation source and spatial layout of the instance object can be further clarified in the new prompt text instruction. This helps to achieve both accurate layout and lossless identity of the instance object as much as possible when there is a reference image to guide image generation.

[0042] In a specific implementation scenario, the source identifier can be represented as "original image 1", "original image 2", ... "image1", "image2", ... "reference image 1", "reference image 2", etc. The specific expression of the source identifier is not limited here. In practical applications, it is advisable to make the source identifier accurately locate the reference image.

[0043] In a specific implementation scenario, for ease of description, let's take the aforementioned office scene poster as an example. The source identifier for the reference image of the instance object "coffee cup" is "Reference Image 1", the source identifier for the reference image of the instance object "notebook" is "Reference Image 2", and the source identifier for the reference image of the instance object "pen" is "Reference Image 3". Then, the new prompt text instruction can be represented as: "A coffee cup..." <bbox><coord_625><coord_625><coord_999><coord_999>< / bbox> From 'Reference Image 1' on the table, a notebook <bbox><coord_313><coord_313><coord_625><coord_625>< / bbox> From 'Reference Image 2' there is a blank page and a pen. <bbox><coord_0><coord_0><coord_313><coord_313>< / bbox> "From 'Reference Image 3', leaning against a laptop, with a soft office background." Of course, the above example is only one possible example of a prompt text instruction in a practical application center. Other possible situations are not limited here, nor will they be listed one by one.

[0044] In a specific implementation scenario, please refer to the following: Figure 2a After constructing the new prompt text instructions, the generative model can process them to obtain the target generated image. Specifically, the reference image can be encoded using a ViT encoder to obtain a first encoded sequence, and then encoded using a VAE encoder to obtain a second encoded sequence. Furthermore, the prompt text instructions are segmented to obtain a first word sequence, and then segmented again using the prompt text instructions after removing index words to obtain a second word sequence. In other words, each reference image needs to be encoded using both a ViT encoder and a VAE encoder. Additionally, the generative model can also include a ViT encoder and a VAE encoder. For example, the ViT encoder can be initialized using SigLIP for semantic awareness, and the VAE encoder can be initialized using FLUX for image generation. The input resolution of the ViT encoder can be set to 384. 384, the patch size can be set to 14. 14. The output feature dimension can be 1152, and the ViT encoder can integrate NaViT technology to support the ViT encoder in processing input images with arbitrary aspect ratios; the input resolution of the VAE encoder can be set to 512. 512, the encoder network layer number can be set to 8, the number of latent variable channels can be set to 4, and the latent variable resolution can be set to 64. 64. Furthermore, the generative model may also include a first mapping network (e.g., a multilayer perceptron) between the ViT encoder and the multimodal unified model, and a second mapping network between the VAE encoder and the multimodal unified model. The first mapping network can be used to adjust (e.g., expand) the feature dimensions of the ViT encoder output features to align with the embedding dimensions of the prompt text instructions; the second mapping network can be used to 2... After 2-patch word merging and projection, the latent variable sequence can be flattened into 1024 words. Furthermore, the generative model can also include a third mapping network between the multimodal unified model and the decoder to restore the predicted velocity field back to the feature dimensions of the image encoder for decoding. It should be noted that... Figure 2a The example shown is merely one possible instance of a generative model employing a hybrid Transformer architecture; the specific architecture of the generative model is not limited here. Of course, after adopting the above model architecture, the generative model can support the projection of all modalities (e.g., text, ViT semantic features, VAE latent variables) into the same embedding space, facilitating the subsequent achievement of consistent representations between perception and generation through a self-attention mechanism. This overcomes the technical deficiency of lacking a unified image generation framework in other related embodiments outside of this disclosure. Furthermore, as mentioned earlier, the MoT architecture contains dual Transformer modules (an understanding expert and a generation expert). The understanding expert can process text tokens and ViT feature tokens in the input sequence, while the generation expert can process VAE latent variable tokens and noise tokens in the input sequence. The understanding expert and the generation expert can also share feedforward networks, positional encoding layers, and self-attention layers. For details, please refer to the technical details of the MoT architecture; they will not be elaborated upon here.

[0045] In a specific implementation scenario, after obtaining the first encoded sequence, the second encoded sequence, the first word sequence, and the second word sequence, diffusion can be performed based on these sequences to obtain a predicted velocity field. Then, decoding is performed based on this predicted velocity field to obtain the generated target image. For example, the above sequence combination can be input into the diffusion mechanism of a generative model (such as...). Figure 2a(Multimodal Unified Model). Similar to the aforementioned diffusion operation, the diffusion mechanism that inputs the above sequence combination into the generative model can also obtain a first velocity field and a second velocity field. These two can be further fused to obtain a fused velocity field (which can be regarded as a predicted velocity field), and decoded accordingly to obtain the target generated image. For details, please refer to the relevant descriptions above, which will not be repeated here. In addition, the scaling factor in the fusion stage can also be adaptively adjusted based on whether there is a reference image. For example, in the case of no reference image (i.e., pure text-guided image generation), the scaling factor can be set to 1.6, while in the case of a reference image (i.e., image-text-guided image generation), the scaling factor can be set between 0.4 and 0.8. Of course, the above example is only one possible example of adaptive adjustment of the scaling factor, and the specific value of the scaling factor is not limited here. The above method encodes the reference image using a ViT encoder to obtain a first encoded sequence, and then encodes the reference image using a VAE encoder to obtain a second encoded sequence. It also segments the prompt text instruction to obtain a first word sequence, and segments the prompt text instruction after removing index words to obtain a second word sequence. Diffusion is then performed based on the first encoded sequence, the second encoded sequence, the first word sequence, and the second word sequence to obtain a predicted velocity field. Finally, decoding is performed based on the predicted velocity field to obtain the target generated image. This method combines the ViT encoder, the VAE encoder, and the word segmentation processing of the prompt text instruction during image generation, thus maximizing compatibility with multimodal data input.

[0046] In one implementation scenario, a generative model responding to prompt text commands can output multiple model-generated images. For each model-generated image, the mean Intersection over Union (mIoU) can be calculated based on the expected coordinates of the instance object and its actual coordinates within the model-generated image. The target generated image can then be determined from the mIoU of all the model-generated images. For example, the model-generated image with the highest mIoU can be selected as the target generated image. This approach, by outputting multiple model-generated images in response to prompt text commands, and by calculating the mIoU for each model-generated image based on the expected coordinates of the instance object and its actual coordinates within the model-generated image, and by determining the target generated image from the mIoU of all the model-generated images, can generate a target generated image that closely matches the desired layout.

[0047] In one implementation scenario, the generative model can be trained based on a first sample set. The first sample set can contain several sets of first sample data. The first sample data can contain first sample prompts (as input data for the generation task during training, and the processing method during training can be referred to the prompt text instructions in the aforementioned reasoning process) and first sample target images (as output targets when training the generation task). That is, the first sample set is a text-guided training sample set. To construct the first sample data, multiple instance images from the first dataset can be selected as the first sample target images. Each instance image contains multiple first sample instance objects, and each instance image is labeled with a first bounding box for each first sample instance object. Based on these first bounding boxes, the first normalized coordinates of the first sample instance objects are obtained. Then, based on these first normalized coordinates and the corresponding index terms for each normalized coordinate, a first major model instruction can be constructed. This first major model instruction instructs the first major model to map the first normalized coordinates of the first sample instance objects to their corresponding index terms, and based on these index terms, a first descriptive statement for the multiple instance images is generated. The first descriptive statement embeds the index terms of the first sample instance objects at the corresponding words. This allows the first output statement of the first major model in response to the first major model instruction to be obtained, serving as the first sample prompt instruction. This method allows for the generation of the first sample prompt instruction based on easily accessible multiple instance images combined with the first major model, thus constructing text-guided sample data and minimizing the construction cost of the first sample set.

[0048] In a specific implementation scenario, the first dataset may include, but is not limited to, datasets such as LayoutSAM, where each multi-instance image may contain 2 to 6 first sample instance objects, and each first sample instance object is accompanied by a first bounding box.

[0049] In a specific implementation scenario, before selecting multiple instance images from the first dataset as the first sample target images, the multiple instance images in the first dataset can be filtered first, such as retaining those with a resolution of not less than 512. The remaining multi-instance images, after selecting 512 instances, retaining those with at least 95% completeness of instance object annotations, and removing those with blurriness or large-area occlusion (e.g., more than 50%), can each be selected as the first sample target image. It should be noted that the first sample target image is the image data that the generative model is expected to generate under the guidance of the first sample prompts during training based on the first sample set.

[0050] In a specific implementation scenario, after selecting and obtaining the first sample target image, the first normalized coordinates of the first sample instance object can be obtained based on the first bounding box of the first sample instance object. Specifically, the first bounding box of the first sample instance object in the first sample target image can be obtained, and the vertex coordinates of the first bounding box (e.g., the coordinates of the lower left vertex and the upper right vertex, or the coordinates of the upper left vertex and the lower right vertex) can be selected and normalized (see the aforementioned relevant description for details) to obtain the first normalized coordinates of the first sample instance object.

[0051] In a specific implementation scenario, after obtaining the first normalized coordinates of the first sample instance object, the first major model instruction can be constructed by combining the index terms corresponding to several normalized coordinates. Specifically, the first major model instruction can be used to instruct the first major model to reference the index terms corresponding to several normalized coordinates, map the first normalized coordinates of the first sample instance object to the index terms of the first sample instance object, and generate a first descriptive statement for the multi-instance image based on the index terms of each first sample instance object. Furthermore, the first descriptive statement embeds the index terms of the first sample instance object at the corresponding words. For example, to facilitate the first major model's understanding of this generation task, relevant generation instances can be embedded in the first major model instruction. That is, by using an in-context-learning mechanism, the index terms corresponding to the first normalized coordinates can be appended to the words of the first sample instance object. This leverages the understanding and generation capabilities of the first major model to directly generate the first sample prompt instruction, minimizing or even eliminating the need for manual intervention.

[0052] In one implementation scenario, the generative model can be trained based on a second sample set. The second sample set can contain several sets of second sample data. The second sample data can include second sample prompts (as input data for the generation task during training, and the processing method during training can refer to the prompt text instructions in the aforementioned reasoning process), second sample target images (as output targets for the generation task during training), and several sample reference images (as input data for the generation task during training, and the processing method during training can refer to the reference images in the aforementioned reasoning process). That is, the second sample set is a training sample set guided by images and text. To construct the second sample data, multiple single-instance images from the second dataset can be selected as sample reference images. Based on these single-instance images, a second large model instruction is constructed. This instruction instructs the second large model to refer to the single-instance images and generate a second descriptive statement containing multiple second sample instance objects. Then, a multi-instance image conforming to the second descriptive statement is generated, with each single-instance image containing different second sample instance objects. This allows the acquisition of the model-generated image output by the second large model in response to the second large model instruction, serving as the second sample target image. Object detection is then performed on the second sample target image to obtain the second bounding boxes of each second sample instance object. Based on these second bounding boxes, a second sample prompt instruction can be generated. This prompt instruction describes the second sample target image in natural language, embedding index terms of the second sample instance objects at the corresponding words. This method allows for the generation of second sample prompt instructions based on easily accessible single-instance images combined with the second large model, thus constructing image-guided sample data and minimizing the construction cost of the second sample set.

[0053] In a specific implementation scenario, the second dataset may include, but is not limited to, Subjects200K (containing 20K single-topic cropped images) and UNO (containing 50K single-topic cropped images). It should be noted that Subjects200K and UNO both belong to the 40 core categories under the Object365 category system (e.g., animals, furniture, food, etc.).

[0054] In a specific implementation scenario, before selecting multiple single-instance images from the second dataset as sample reference images, topic filtering can be performed first. For example, 5K high-quality samples (e.g., CLIP feature similarity not less than 0.8, no background interference, etc.) can be retained for each category to obtain 200K candidate topic images. Based on this, multiple (e.g., 2-3) topic images (i.e., single-instance images) from different categories can be randomly combined and used as sample reference images.

[0055] In a specific implementation scenario, after obtaining multiple single-instance images, a second large-scale model instruction can be constructed based on these images. This instruction instructs the second large-scale model to reference the multiple single-instance images, generate a second descriptive statement containing multiple second sample instance objects, and then generate a multi-instance image that conforms to the second descriptive statement. Each of the multiple single-instance images contains different second sample instance objects. Based on this, the model-generated image output by the second large-scale model (e.g., LLaVA, Qwen-VL, etc., multimodal large models) in response to the second large-scale model instruction can be obtained as the second sample target image. For example, taking the selection of single-instance images "dolphin" and "boat" as an example, a second major model instruction can be constructed: "Generate an image description statement, which describes a new image set containing the input images [i.e., multiple single-instance images]". This second major model instruction is then input to the second major model (or, the second major model instruction and multiple single-instance images are input to the second major model). The second major model outputs a model-generated image containing the aforementioned multiple single-instance images (i.e., the single-instance images "dolphin" and "boat") (e.g., a boat floating on a vast ocean, with a dolphin leaping out of the water), which serves as the second sample target image. Of course, the above example is merely one possible illustration in practical applications. The specific content of the second major model instruction and the model output image in response to the second major model instruction is not limited here, nor will it be listed in detail. Furthermore, the second sample target image can be considered as the model-generated image expected to be output by the generative model during the training process, guided by the sample reference image and the second sample prompt instruction.

[0056] In a specific implementation scenario, after obtaining the second sample target image, in order to further construct a second sample prompt instruction based on it, target detection can be performed on the second sample target image to obtain the second bounding boxes of each second sample instance object in the second sample target image. Then, based on the second bounding boxes of the second sample instance objects, the second normalized coordinates of the second sample instance objects can be obtained. Thus, based on the second normalized coordinates of the second sample instance objects and the index words corresponding to each of the normalized coordinates, a third major model instruction can be constructed. The third major model instruction is used to instruct the third major model to refer to the index words corresponding to each of the normalized coordinates to map the second normalized coordinates of the second sample instance objects to the index words of the second sample instance objects. Based on the index words of each second sample instance object, a third descriptive statement of the second sample target image is generated. The index words of the second sample instance objects are embedded in the words of the second sample instance objects in the third descriptive statement. Then, the second output statement of the third major model in response to the third major model instruction can be obtained as the second sample prompt instruction. Specifically, please refer to the aforementioned description of the process of constructing the second sample data: "Based on the first bounding box of the first sample instance object, obtain the first normalized coordinates of the first sample instance object; based on the first normalized coordinates of the first sample instance object and the index terms corresponding to each of the pre-set normalized coordinates, construct the first major model instruction, and obtain the first output statement of the first major model in response to the first major model instruction, as the first sample prompt instruction." This will not be repeated here. The above method, by combining the second normalized coordinates of the second sample instance object and the index terms corresponding to each pre-set normalized coordinate, can construct a third major model instruction to instruct the second major model to generate the second sample prompt instruction, which helps reduce the construction cost of the second sample prompt instruction.

[0057] In a specific implementation scenario, after the second sample data is constructed, it can be validated to determine whether it should be used in the training phase. Specifically, the CLIP similarity and DINOv2 similarity between the image data of the second sample instance object within the second sample target image and the sample reference image can be validated. If the former similarity meets a first screening condition regarding a first similarity threshold (e.g., higher than, or not lower than, such as, a first similarity threshold of 0.7) and the latter similarity meets a second screening condition regarding a second similarity threshold (e.g., higher than, such as, a second similarity threshold of 0.6) or not lower than, such as, a second similarity threshold of 0.6, the second sample data can be retained; otherwise, it can be considered that the identities are inconsistent and the second sample data can be filtered out.

[0058] In one implementation scenario, after obtaining at least one of the aforementioned first and second sample sets, the generative model can be trained accordingly. Specifically, the generative model can employ a two-stage training approach. In the first stage (which can be called the alignment stage), the training data can include the aforementioned first and second sample sets used for the generation task. In addition, as a possible example, other datasets used for the understanding task (e.g., plain text understanding, mixed text / image understanding, etc.) can be included, in which case they can be mixed in a 1:1:0.5 ratio of the three types of data. The optimizer can be AdamW, with β1 set to 0.9, β2 set to 0.95, ε set to 1e-15, a learning rate set to 2.5e-5, no warmup, weight decay set to 0.01, and 16,000 training steps. In the second stage (which can be called the hybrid supervised fine-tuning stage), the training data can include the aforementioned first and second sample sets used for the generation task. In addition, as a possible example, other datasets for understanding tasks (e.g., plain text understanding, mixed image understanding, etc.) could be included. In this case, 2000 data points from each of these three types of data could be selected for training. The optimizer could be AdamW, with a learning rate adjusted to 1.25e-5. In rounds 3-5, only datasets with resolutions higher than 1024 could be used. For sample data of 1024 (e.g., representing 30% of the total data), the input resolution of the VAE encoder can be adjusted to 1024. 1024. Based on this, in any of the above training stages, a loss function that combines the generation and understanding tasks can be used to adjust the parameters of the generative model.

[0059] In a specific implementation scenario, during the processing of the generative model, for flow matching velocity prediction, the sampling diffusion time step t ~ U(0,1) generates mixed latent variables: x t =(1-t)x0+tx1 In the above formula, x0 represents the initial distribution of the data sample, such as random noise, and x1 represents the target distribution of the data sample, such as the target image (e.g., the first target image, the second target image). t The intermediate state data at time t is represented. The velocity field prediction is based on the fused token sequence (e.g., image encoded sequence, text word sequence, etc.) to obtain the predicted velocity field. The generation loss L is obtained based on the difference between the predicted velocity field and the sample velocity field (which can be obtained by measuring the data sample x0 of the initial distribution and the data sample x1 of the target distribution). FM :

[0060] In the above formula, x0-x1 represents the expected sample velocity field of the generation task, v θ (x t |c) represents the predicted velocity field; the difference between the two represents the generation loss. In addition, the understanding loss can be calculated, specifically based on the next-token-prediction mechanism to obtain the understanding loss L. LM By combining the aforementioned generation loss with parameter adjustments, the basic dialogue understanding and image-based storytelling abilities of the generative model can be maintained, while also enhancing its understanding of layout coordinates in both the language and generation parts. For example, the understanding loss L... LM It can be represented as:

[0061] In the above formula, z i Let z represent the i-th token. t P represents the tokens preceding the i-th token. θ This means predicting the probability of the i-th token based on the tokens preceding the i-th token. Based on this, the generation loss L... FM And understanding loss L LM We then perform a weighted calculation to obtain the training loss L of the generative model: L=λ LM L LM +λ FM L FM In the above formula, λ LM λ represents the weighting factor of the understanding loss. FM This represents the weighting factor of the generation loss. By combining the generation loss and the understanding loss, the generation and understanding performance of the generative model can be jointly optimized.

[0062] In a specific implementation scenario, please refer to the relevant documents. Figure 2a During parameter adjustment, only the network parameters of the first mapping network, the second mapping network, the multimodal unified model, and the third mapping network in the generative model can be adjusted. As for the network parameters of the ViT encoder, the VAE encoder, the word segmenter, and the decoder (e.g., the VAE decoder), they can all be frozen.

[0063] In one implementation scenario, experiments verified the layout control accuracy and general multimodal capability of the technical solution of this disclosure embodiment. Regarding layout control accuracy, it was found that on COCO-Position, the average instance success rate was 92.6%, the image success rate was 76.1%, mIoU was 85.3%, and AP was 70.9%, which is 13.7% higher than GLIGEN. Regarding general multimodal capability, it was found that the score was 81.4% in MMBEC, 42.3 in MMMU, and 0.88 in GenEval, which is on par with the Bagel base model.

[0064] In one implementation scenario, to facilitate a comprehensive understanding of the embodiments of this disclosure, several specific examples are provided below for illustration. The first example is the aforementioned office scene poster, which can be referred to in the aforementioned related descriptions and will not be repeated here. The second example is e-commerce product image optimization. For example, if the original product image contains three instance objects: "mobile phone," "earphone," and "charger," and the image generation request is to rearrange the above three instance objects, requiring the mobile phone to be centered, the earphone to be in the upper left, and the charger to be in the lower right, while keeping the product appearance unchanged, then the original product image can be used as a reference image. Alternatively, image data of the three instance objects can be extracted from the original product image and used as reference images for the three instance objects respectively. After determining the expected coordinates according to the image generation request, normalization mapping is performed to obtain the index words of the three instance objects. Combined with the reference images, prompt text instructions are constructed, such as including but not limited to "mobile phone." <bbox><coord_350><coord_300><coord_650><coord_750>< / bbox> From 'Original Image', Headphones <bbox><coord_100><coord_100><coord_280><coord_350>< / bbox> From 'Original Image', Charger <bbox><coord_720><coord_600><coord_900><coord_850>< / bbox> "From 'Original Image', clean white background." Additionally, a scaling factor can be set. coord With a value of 1.4, a sampling step count of 50, a temperature coefficient of 0.9, and a desired image resolution of 768, the desired output image resolution is [missing value]. 768. The generative model can respond to the above prompt text instructions to obtain the target generated image. Each instance object in the target generated image retains the appearance of the original product image, but the layout is rearranged according to the image generation request. The third example is also e-commerce product image optimization. For instance, if the original product image contains two instance objects, "dining table" and "tableware," and the image generation request is to rearrange these two instance objects, maintaining the original appearance of the product and placing them according to a specified layout (e.g., tableware is located in the lower left position of the dining table), then, similar to the previous operation, the original product image can be used as a reference image, or image data of the dining table and tableware can be extracted from the original product image separately as reference images for the dining table and tableware, respectively. Prompt text instructions can then be constructed based on these, such as including but not limited to "dining table." <bbox><coord_200><coord_300><coord_800><coord_900>< / bbox> From original image 1, ceramic dinner plate <bbox><coord_100><coord_100><coord_300><coord_300>< / bbox>"From original image 2", the generative model will then respond to this prompt text instruction and generate a target generated image that matches the image generation request and preserves the details of the product's appearance. In addition, a scaling factor s can be set. coord The value was 0.6. After testing and verification, it was found that the product appearance was accurate (DINO similarity reached 0.93), the layout was accurate (IoU>0.85), and the batch processing efficiency was improved by 18 times. The fourth example is the restoration of film and television prop scenes. For example, if it is desired to restore a close-up image of a battlefield containing the instance objects "Bronze Sword" and "Animal Pattern Shield," it is required to retain the details of the reference image and follow the storyboard layout. If conventional 3D modeling is used, it usually takes several days (e.g., a typical designer would need three days). Based on the technical solution of this disclosure embodiment, prompt text instructions can be directly constructed, such as including but not limited to "Bronze Sword." <bbox><coord_250><coord_400><coord_450><coord_850>< / bbox> From reference Figure 1 Animal-patterned shield <bbox><coord_550><coord_400><coord_750><coord_850>< / bbox> "From reference Figure 2", the generative model can then respond to this prompt text instruction and generate a target generated image that matches the image generation request and retains the details of the reference image. After testing and verification, it was found that the prop details are faithfully reproduced (DINO similarity of 0.68), the layout is accurate (IoU=0.89), and a naturally blended target generated image can be generated in just 30 minutes, significantly shortening the production cycle.

[0065] In one implementation scenario, to more intuitively understand the relevant implementation methods in the embodiments of this disclosure, please refer to the following: Figure 2b and Figure 2c , Figure 2b This is a schematic diagram of an embodiment of text-guided image generation in this application. Figure 2c This is a schematic diagram illustrating an embodiment of image generation guided by text and graphics in this application. For example... Figure 2b As shown, the image generation request is "generate an image of a puppy swimming in clear water". Following the aforementioned process, we can first determine the expected coordinates based on the implicit layout position "image global" of the instance object "clear water", and then normalize the mapping to obtain the index terms.<coord_000><coord_000><coord_999><coord_999> The expected coordinates of the instance object "puppy" are determined by its layout position "middle", and then the index words are obtained by normalization mapping based on these coordinates.<coord_201><coord_202><coord_801><coord_902> Based on this, it is possible to construct... Figure 2b The prompt text shown in the middle left image reads: "In clear water..." <bbox><coord_000><coord_000><coord_999><coord_999>< / bbox> In the middle, a puppy <bbox><coord_201><coord_202><coord_801><coord_902>< / bbox> "Swimming," from which a generative model can generate results such as... Figure 2b The generated image of the target is shown in the middle right figure. It should be noted that... Figure 2b In the middle right image, the green and red boxes in the generated image are used to indicate the layout positions of the instance objects "puppy" and "clear water" in the generated image, respectively, and do not represent that the generated image contains these two boxes. Figure 2c As shown, the image generation request could be "adjust the mug in the reference image to the right center position, adjust the buns on the plate in the reference image to the left front side of the mug, and adjust the red crab to the right front side of the mug." Following the aforementioned process, the desired coordinates can be determined first based on the layout position "right center position" of the instance object "mug" (e.g., by first detecting...). Figure 2c The coordinates of the "mug" (highlighted by a green dashed box) in the reference image shown in the middle left image, combined with... Figure 2c The image size of the reference image shown in the middle left figure and the layout position of the "mug" ("right center") are used to determine its expected coordinates without changing the size of the "mug". Then, the index terms are obtained by normalization mapping based on these coordinates.<coord_382><coord_388><coord_651><coord_666> Similarly, we can obtain the index terms of the instance object "the buns on the plate".<coord_128><coord_605><coord_356><coord_823> and the index term of the instance object "red crab".<coord_421><coord_611><coord_817><coord_802> Based on this, it is possible to construct... Figure 2c The prompt text shown in the middle left image is: "A mug". <bbox><coord_382><coord_388><coord_651><coord_666>< / bbox> From the 'reference image', a steamed bun is prominently displayed in the center right of a rustic wooden table, bathed in the warm morning sunlight. The bun is on a freshly steamed plate. <bbox><coord_128><coord_605><coord_356><coord_823>< / bbox> From 'reference image'. Meanwhile, a red crab. <bbox><coord_421><coord_611><coord_817><coord_802>< / bbox> "The 'reference image' is slowly crawling across the desktop," from which a generative model can generate something like... Figure 2c The generated image of the target is shown in the middle right figure. It should be noted that... Figure 2c In the middle right image, the green, orange, and blue boxes in the generated image are used to indicate the layout positions of the instance objects "mug," "bun on a plate," and "red crab" in the generated image, respectively, and do not represent that the generated image contains these three boxes. It should be noted that... Figure 2b and Figure 2c The examples shown are just a few possible examples of text-guided image generation and image-text-guided image generation in practical applications. Other possible situations are not limited here, nor will they be listed one by one.

[0066] The above scheme obtains an image generation request, which describes the instance objects and their layout positions in the desired generated image. Based on the image generation request, the desired coordinates of the instance objects are obtained, and then normalized. Index terms for the instance objects are determined from the index terms corresponding to several preset normalized coordinates. Based on these index terms, a prompt text instruction is constructed. This prompt text instruction describes the desired generated image in natural language, and the index terms for the instance objects are embedded in the words related to them. Finally, the model generates an image based on the output of the generative model in response to the prompt text instruction, resulting in the target generated image used to respond to the image generation request. This process involves embedding the index terms for the instance objects in the prompt text instruction. By embedding index terms representing the layout coordinates of instance objects into the language flow, layout constraints can be directly encoded as text terms embedded in the language stream. This allows for a deep fusion of spatial and semantic constraints, enabling generative models to generate images. This improves the accuracy of the spatial layout of instance objects in the generated images, especially in scenarios with multiple instance objects. Furthermore, since each normalized coordinate has its own pre-set index terminology, and normalization is performed during image generation in conjunction with the expected coordinates of the instance objects to determine these index terms, and the instance object's index terms are directly embedded at the words in the prompt text instead of the expected coordinates, the input terminology overhead during image generation is reduced. This helps save sequence length in complex layout scenarios, thus reducing computational and memory overhead. Therefore, it reduces the computational and memory overhead of image generation and improves the accuracy of the spatial layout of instance objects in the generated images, especially in scenarios with multiple instance objects.

[0067] Please see Figure 3 , Figure 3 This is a schematic diagram of the framework of an embodiment of the image generation apparatus of this application. The image generation apparatus 30 includes: a content acquisition module 31, a coordinate determination module 32, an index determination module 33, a prompt construction module 34, and a model generation module 35. The content acquisition module 31 is used to acquire an image generation request; wherein, the image generation request describes the instance object and the layout position of the instance object in the image to be generated; the coordinate determination module 32 is used to obtain the expected coordinates of the instance object based on the image generation request; the index determination module 33 is used to normalize the expected coordinates of the instance object and determine the index words of the instance object in a number of preset normalized coordinates; the prompt construction module 34 is used to construct a prompt text instruction based on the index words of the instance object; wherein, the prompt text instruction describes the image to be generated in natural language and embeds the index words of the instance object in the natural language statement; the model generation module 35 is used to generate a target generated image for responding to the image generation request based on the model generated image output by the generative model in response to the prompt text instruction.

[0068] In the above scheme, the image generation device 30 acquires an image generation request, which describes the instance objects and their layout positions in the desired generated image. Based on the image generation request, it obtains the desired coordinates of the instance objects, normalizes these coordinates, and determines the index terms of the instance objects from the index terms corresponding to several preset normalized coordinates. Based on these index terms, it constructs a prompt text instruction, which describes the desired generated image in natural language. The prompt text instruction embeds the index terms of the instance objects at the corresponding words. Then, based on the generative model's response to the prompt text instruction, it generates an image, resulting in the target generated image used to respond to the image generation request. On the one hand, the prompt text instruction contains the instance objects... By embedding index terms representing the layout coordinates of instance objects at the word positions of objects, layout constraints can be directly encoded as text terms embedded in the language stream. This allows for a deep fusion of spatial and semantic constraints, enabling generative models to generate images. This helps improve the accuracy of the spatial layout of instance objects in the generated images, especially in scenarios with multiple instance objects. Furthermore, since each normalized coordinate has its own pre-set index terms, and these are normalized during image generation in conjunction with the expected coordinates of the instance objects to determine the index terms, and the instance object's index terms are directly embedded at the word positions of the instance objects in the prompt text instructions instead of the expected coordinates, the input terminology overhead during image generation can be reduced. This helps save sequence length in complex layout scenarios, thereby reducing computational and memory overhead. Therefore, it can reduce the computational and memory overhead of image generation and improve the accuracy of the spatial layout of instance objects in the generated images, especially in scenarios with multiple instance objects.

[0069] In some disclosed embodiments, the coordinate determination module 32 includes a request detection submodule for detecting the description method of the layout position in the image generation request, wherein the description method includes description by position coordinates or description by relative orientation; the coordinate determination module 32 includes a first response submodule for extracting coordinates based on the image generation request in response to the layout position being described by position coordinates, thereby obtaining the expected coordinates of the instance object; the coordinate determination module 32 includes a second response submodule for interacting with the target object based on the image generation request in response to the layout position being described by relative orientation, thereby obtaining the expected coordinates of the instance object; wherein the image generation request is triggered by the target object.

[0070] In some disclosed embodiments, the second response submodule includes a human-computer interaction unit for displaying a drawing canvas on the interaction interface with the target object to generate the image, and prompting the target object to specify the absolute position of the instance object on the drawing canvas; the second response submodule includes a coordinate acquisition unit for obtaining the expected coordinates of the instance object in response to the absolute position specified by the target object on the drawing canvas.

[0071] In some disclosed embodiments, the number of decimal places of several normalized coordinates is at most the target number of decimal places. The index determination module 33 includes a normalization submodule, which is used to normalize the expected coordinates of the instance object to the target number of decimal places to obtain the first coordinates of the instance object. The index determination module 33 includes a coordinate constraint submodule, which is used to constrain the first coordinates of the instance object based on the numerical range of several normalized coordinates to obtain the second coordinates of the instance object. The index determination module 33 includes a word mapping submodule, which is used to select the index word corresponding to the second coordinates of the instance object as the index word of the instance object.

[0072] In some disclosed embodiments, the image generation apparatus 30 includes an index acquisition module, which is used to acquire index terms corresponding to a number of preset normalized coordinates. Specifically, the index acquisition module is used to: register a preset number of index terms as special tokens in the vocabulary of the generative model, and map the preset number of normalized coordinates to the corresponding index terms. The vocabulary is used to characterize the correspondence between the data input to the generative model and the tokens that the generative model can recognize.

[0073] In some disclosed embodiments, the model generation module 35 includes a text segmentation submodule, used to segment words based on prompt text instructions to obtain a first word sequence, and to segment words based on prompt text instructions after removing index words to obtain a second word sequence; the model generation module 35 includes a diffusion prediction submodule, used to diffuse based on the first word sequence to obtain a first velocity field, and to diffuse based on the second word sequence to obtain a second velocity field; the model generation module 35 includes a representation fusion submodule, used to fuse based on the first velocity field and the second velocity field to obtain a fused velocity field; the model generation module 35 includes an image decoding submodule, used to decode based on the fused velocity field to obtain a target generated image.

[0074] In some disclosed embodiments, the model generation module 35 includes a factor calculation submodule for obtaining a normalization factor based on the ratio of the magnitude of the second velocity field to that of the target velocity field; wherein the target velocity field is the sum of the fused velocity field and a preset minimum value; the model generation module 35 includes a representation update submodule for normalizing the fused velocity field based on the normalization factor to obtain a new fused velocity field; the image decoding submodule is specifically used for decoding based on the new fused velocity field to obtain the target generated image.

[0075] In some disclosed embodiments, when the image generation request also includes a reference image of the instance object, the prompt construction module 34 is specifically used to construct a new prompt text instruction based on the index terms of the instance object and the reference image; wherein, at the word of the instance object in the new prompt text instruction, a source identifier of the instance object is also embedded, the source identifier is used to identify the reference image from which the instance object originates, and each reference image corresponds to a different source identifier; the model generation module 35 is specifically used to obtain a target generated image for responding to the image generation request based on the model generated image output by the generative model in response to the new prompt text instruction and the reference image.

[0076] In some disclosed embodiments, the model generation module 35 includes an image encoding submodule, used to encode a reference image based on a ViT encoder to obtain a first encoding sequence, and to encode the reference image based on a VAE encoder to obtain a second encoding sequence; the model generation module 35 includes a text segmentation submodule, used to segment words based on prompt text instructions to obtain a first word sequence, and to segment words based on prompt text instructions after removing index words to obtain a second word sequence; the model generation module 35 includes a diffusion prediction submodule, used to diffuse based on the first encoding sequence, the second encoding sequence, the first word sequence, and the second word sequence to obtain a predicted velocity field; and the model generation module 35 includes an image decoding submodule, used to decode based on the predicted velocity field to obtain a target generated image.

[0077] In some disclosed embodiments, the generative model is trained based on a first sample set, which includes several sets of first sample data. The first sample data includes first sample prompts and first sample target images. The image generation device 30 includes a first selection module for selecting multiple instance images from the first dataset as first sample target images. Each multiple instance image contains multiple first sample instance objects, and each instance image is labeled with a first bounding box for each first sample instance object. The image generation device 30 includes a coordinate normalization module for obtaining first normalized coordinates of the first sample instance objects based on their first bounding boxes. The image generation device 30 also includes a first construction module for... The image generation device 30 includes a first generation module for constructing a first large model instruction based on the first normalized coordinates of a first sample instance object and the index terms corresponding to several normalized coordinates. The first large model instruction is used to instruct the first large model to map the first normalized coordinates of the first sample instance object to the index terms of the first sample instance object by referring to the index terms corresponding to several normalized coordinates, and to generate a first description statement of the multi-instance image based on the index terms of each first sample instance object. The first description statement contains the index terms of the first sample instance object embedded in the words of the first sample instance object. The image generation device 30 includes a first generation module for obtaining the first output statement of the first large model in response to the first large model instruction, which serves as the first sample prompt instruction.

[0078] In some disclosed embodiments, the generative model is trained based on a second sample set, which includes several sets of second sample data. The second sample data includes second sample prompt instructions, second sample target images, and several sample reference images. The image generation device 30 includes a second selection module for selecting multiple single-instance images from the second dataset as sample reference images. The image generation device 30 also includes a second construction module for constructing a second major model instruction based on the multiple single-instance images. The second major model instruction instructs the second major model to refer to the multiple single-instance images, generate a second description statement containing multiple second sample instance objects, and then generate multiple instance images conforming to the second description statement. Each of the multiple single-instance images contains different second sample objects. This example object; the image generation device 30 includes a sample acquisition module, used to acquire the model-generated image output by the second large model in response to the second large model instruction, as the second sample target image; the image generation device 30 includes a sample detection module, used to perform target detection based on the second sample target image, to obtain the second bounding box of each second sample instance object in the second sample target image; the image generation device 30 includes a second generation module, used to generate a second sample prompt instruction based on the second bounding box of each second sample instance object in the second sample target image; wherein, the second sample prompt instruction describes the second sample target image in natural language, and the index terminology of the second sample instance object is embedded at the words of the second sample instance object in the second sample prompt instruction.

[0079] In some disclosed embodiments, the second generation module includes a sample normalization submodule, used to obtain the second normalized coordinates of the second sample instance object based on the second bounding box of the second sample instance object; the second generation module includes an instruction construction submodule, used to construct a third major model instruction based on the second normalized coordinates of the second sample instance object and the index terms corresponding to several normalized coordinates; wherein, the third major model instruction is used to instruct the third major model to map the second normalized coordinates of the second sample instance object to the index terms of the second sample instance object by referring to the index terms corresponding to several normalized coordinates, and to generate a third descriptive statement of the second sample target image based on the index terms of each second sample instance object, and the index terms of the second sample instance object are embedded in the words of the second sample instance object in the third descriptive statement; the second generation module includes an instruction generation submodule, used to obtain the second output statement of the third major model in response to the third major model instruction, as the second sample prompt instruction.

[0080] In some disclosed embodiments, the generative model responds to prompt text instructions and outputs multiple model-generated images. For each model-generated image, the average intersection-union ratio is calculated based on the expected coordinates of the instance object and the actual coordinates of the instance object in the model-generated image. The target generated image is determined by the average intersection-union ratio of the various model-generated images.

[0081] In some disclosed embodiments, the index term of any instance object in the prompt text instruction is surrounded by a start term and an end term.

[0082] In some disclosed embodiments, the index term includes an index identifier, which is used to distinguish between ordinary terms and index terms in the prompt text instruction.

[0083] Please see Figure 4 , Figure 4 This is a schematic diagram of a framework of an embodiment of the electronic device of this application. The electronic device 40 includes at least a memory 41 and a processor 42 coupled to each other. The memory 41 stores at least program instructions, and the processor 42 is used to execute the program instructions to implement the steps in any of the above-described image generation method embodiments. For details, please refer to the foregoing disclosed embodiments, which will not be repeated here. As a possible example, the electronic device 40 may include, but is not limited to, servers, tablet computers, mobile phones, and other devices, and the specific type of the electronic device 40 is not limited here.

[0084] Specifically, processor 42 controls itself and memory 41 to implement the steps in any of the above-described image generation method embodiments. Processor 42 can also be referred to as a CPU (Central Processing Unit). Processor 42 may be an integrated circuit chip with signal processing capabilities. Processor 42 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 42 can be implemented using integrated circuit chips.

[0085] In the above scheme, the electronic device 40 acquires an image generation request, which describes the instance objects and their layout positions in the desired generated image. Based on the image generation request, it obtains the desired coordinates of the instance objects, normalizes these coordinates, and determines the index terms of the instance objects from the index terms corresponding to several preset normalized coordinates. Based on these index terms, it constructs a prompt text instruction, which describes the desired generated image in natural language. The prompt text instruction embeds the index terms of the instance objects at the corresponding words. Then, based on the generative model's response to the prompt text instruction, it generates an image, resulting in the target generated image used to respond to the image generation request. On the one hand, the prompt text instruction contains the instance objects... By embedding index terms representing the layout coordinates of instance objects at the word positions, layout constraints can be directly encoded as text terms embedded in the language stream. This allows for a deep fusion of spatial and semantic constraints, enabling generative models to generate images. This improves the accuracy of the spatial layout of instance objects in the generated images, especially in scenarios with multiple instance objects. Furthermore, since each normalized coordinate has its own pre-set index terminology, and normalization is performed during image generation in conjunction with the expected coordinates of the instance objects to determine these index terms, and the instance object's index terms are directly embedded at the word positions in the prompt text instructions instead of the expected coordinates, the input terminology overhead during image generation is reduced. This helps save sequence length in complex layout scenarios, thus reducing computational and memory overhead. Therefore, it reduces the computational and memory overhead of image generation and improves the accuracy of the spatial layout of instance objects in the generated images, especially in scenarios with multiple instance objects.

[0086] Please see Figure 5 , Figure 5 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. The computer-readable storage medium 50 stores program instructions 51 that can be executed by a processor. The program instructions 51 are used to implement the steps in any of the above-described image generation method embodiments.

[0087] In the above scheme, the computer-readable storage medium 50 acquires an image generation request, which describes the instance objects and their layout positions in the desired generated image. Based on the image generation request, the desired coordinates of the instance objects are obtained, and then normalized. The index terms of the instance objects are determined from the index terms corresponding to several preset normalized coordinates. Based on these index terms, a prompt text instruction is constructed. This prompt text instruction describes the desired generated image in natural language, and embeds the index terms of the instance objects at the words related to them. Finally, based on the generative model's response to the prompt text instruction, a model-generated image is output, resulting in the target generated image used to respond to the image generation request. On the one hand, the prompt text instruction... By embedding index terms representing the layout coordinates of instance objects at the word positions of instance objects, layout constraints can be directly encoded as text terms embedded in the language stream. This allows for a deep fusion of spatial and semantic constraints, enabling generative models to generate images. This helps improve the accuracy of the spatial layout of instance objects in the generated images, especially in scenarios with multiple instance objects. Furthermore, since each normalized coordinate has its own pre-set index terms, and these are normalized during image generation in conjunction with the expected coordinates of the instance objects to determine the index terms, and the instance object's index terms are directly embedded at the word positions of the instance objects in the prompt text instructions instead of the expected coordinates, the input terminology overhead during image generation can be reduced. This helps save sequence length in complex layout scenarios, thereby reducing computational and memory overhead. Therefore, it can reduce the computational and memory overhead of image generation and improve the accuracy of the spatial layout of instance objects in the generated images, especially in scenarios with multiple instance objects.

[0088] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0089] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0090] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0091] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0092] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0093] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0094] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

Claims

1. An image generation method, characterized in that, include: Obtain an image generation request; wherein, the image generation request describes the instance objects in the image to be generated and the layout positions of the instance objects; Based on the image generation request, the expected coordinates of the instance object are obtained; The expected coordinates of the instance object are normalized, and the index terms of the instance object are determined from the index terms corresponding to the preset normalized coordinates. Based on the index terms of the instance object, a prompt text instruction is constructed; wherein, the prompt text instruction describes the desired generated image in natural language, and the index terms of the instance object are embedded in the words of the instance object in the prompt text instruction; The model-generated image output by the generative model in response to the prompt text instruction is used to obtain the target generated image for responding to the image generation request.

2. The method according to claim 1, characterized in that, The step of obtaining the expected coordinates of the instance object based on the image generation request includes: The description method of the layout position in the image generation request is detected; wherein, the description method includes describing it with position coordinates or describing it with relative orientation; In response to the layout position being described in position coordinates, coordinate extraction is performed based on the image generation request to obtain the expected coordinates of the instance object; In response to the layout position being described in relative orientation, information is exchanged with the target object based on the image generation request to obtain the desired coordinates of the instance object; wherein, the image generation request is triggered by the target object.

3. The method according to claim 2, characterized in that, The step of interacting with the target object based on the image generation request to obtain the expected coordinates of the instance object includes: The drawing canvas for the image to be generated is displayed in the interactive interface with the target object, and the target object is prompted to specify the absolute position of the instance object on the drawing canvas; In response to the absolute position of the target object on the drawing canvas, the desired coordinates of the instance object are obtained.

4. The method according to any one of claims 1 to 3, characterized in that, The number of decimal places of the normalized coordinates is at most the target number of decimal places, and the total number of index terms is 10 raised to the power of the target number of decimal places. The process of normalizing the expected coordinates of the instance object and determining the index terms of the instance object from the index terms corresponding to each of the preset normalized coordinates includes: The first coordinates of the instance object are obtained by normalizing the expected coordinates of the instance object to the target number of bits; The first coordinate of the instance object is constrained based on the numerical range of the several normalized coordinates to obtain the second coordinate of the instance object; The index term corresponding to the second coordinate of the instance object is selected as the index term of the instance object.

5. The method according to any one of claims 1 to 4, characterized in that, The index terms corresponding to the preset normalized coordinates are obtained in the following way: A preset number of index terms are registered as special tokens in the vocabulary of the generative model, and several preset normalized coordinates are mapped to the corresponding index terms. The vocabulary is used to represent the correspondence between the data input to the generative model and the tokens that the generative model can recognize.

6. The method according to any one of claims 1 to 5, characterized in that, The model-generated image output by the generative model in response to the prompt text instruction, to obtain the target generated image for responding to the image generation request, includes: Based on the prompt text instruction, word segmentation is performed to obtain a first word element sequence, and based on the prompt text instruction after removing the index word element, word segmentation is performed to obtain a second word element sequence; A first velocity field is obtained by diffusion based on the first word sequence, and a second velocity field is obtained by diffusion based on the second word sequence. The first velocity field and the second velocity field are fused to obtain the fused velocity field; The target generated image is obtained by decoding based on the fused velocity field.

7. The method according to claim 6, characterized in that, After fusing the first velocity field and the second velocity field to obtain a fused velocity field, and before decoding the fused velocity field to obtain the target generated image, the method further includes: A normalization factor is obtained based on the ratio of the magnitudes of the second velocity field and the target velocity field; wherein the target velocity field is the sum of the fused velocity field and a preset minimum value; The fusion velocity field is normalized based on the normalization factor to obtain a new fusion velocity field. The decoding based on the fused velocity field to obtain the target generated image includes: The target generated image is obtained by decoding based on the new fused velocity field.

8. The method according to any one of claims 1 to 7, characterized in that, If the image generation request also includes a reference image of the instance object, the step of constructing the prompt text instruction based on the index terms of the instance object includes: Based on the index terms and reference images of the instance object, a new prompt text instruction is constructed; wherein, at the words of the instance object in the new prompt text instruction, the source identifier of the instance object is also embedded, the source identifier is used to identify the reference image from which the instance object originates, and each reference image corresponds to a different source identifier; The model-generated image output by the generative model in response to the prompt text instruction, to obtain the target generated image for responding to the image generation request, includes: Based on the generative model's response to the new prompt text instruction and the reference image, a model-generated image is output, resulting in a target generated image for responding to the image generation request.

9. The method according to claim 8, characterized in that, The model-generated image output based on the generative model in response to the new prompt text instruction and the reference image, to obtain the target generated image for responding to the image generation request, includes: The reference image is encoded using a ViT encoder to obtain a first encoding sequence, and the reference image is encoded using a VAE encoder to obtain a second encoding sequence. The prompt text instruction is then segmented to obtain a first word sequence, and the prompt text instruction after removing the index word is then segmented to obtain a second word sequence. Based on the first encoded sequence, the second encoded sequence, the first word sequence, and the second word sequence, a diffusion is performed to obtain the predicted velocity field; The target image is obtained by decoding based on the predicted velocity field.

10. The method according to any one of claims 1 to 9, characterized in that, The generative model is trained based on a first sample set, which contains several sets of first sample data. The first sample data includes first sample prompt instructions and first sample target images. The steps for obtaining the first sample data include: Select a multi-instance image from the first dataset as the first sample target image; wherein the multi-instance image contains a plurality of first sample instance objects, and the multi-instance image is labeled with a first bounding box for each of the first sample instance objects; Based on the first bounding box of the first sample instance object, the first normalized coordinates of the first sample instance object are obtained; Based on the first normalized coordinates of the first sample instance object and the index terms corresponding to each of the plurality of normalized coordinates, a first large model instruction is constructed; wherein, the first large model instruction is used to instruct the first large model to map the first normalized coordinates of the first sample instance object to the index terms of the first sample instance object by referring to the index terms corresponding to each of the plurality of normalized coordinates, and to generate a first description statement of the multi-instance image based on the index terms of each of the first sample instance objects, and the index terms of the first sample instance object are embedded in the words of the first sample instance object in the first description statement. Obtain the first output statement of the first large model in response to the first large model instruction, and use it as the first sample prompt instruction.

11. The method according to any one of claims 1 to 10, characterized in that, The generative model is trained based on a second sample set, which contains several sets of second sample data. The second sample data includes second sample prompts, second sample target images, and several sample reference images. The steps for obtaining the second sample data include: Multiple single-instance images from the second dataset are selected as the sample reference images; Based on the multiple single-instance images, a second large model instruction is constructed; wherein, the second large model instruction is used to instruct the second large model to refer to the multiple single-instance images, generate a second description statement that contains multiple second sample instance objects, and then generate a multi-instance image that conforms to the second description statement, and the multiple single-instance images respectively contain different second sample instance objects. Obtain the model-generated image output by the second large model in response to the second large model instruction, and use it as the second sample target image; Target detection is performed based on the second sample target image to obtain the second bounding box of each second sample instance object in the second sample target image; Based on the second bounding boxes of each second sample instance object in the second sample target image, a second sample prompt instruction is generated; wherein, the second sample prompt instruction describes the second sample target image in natural language, and the index terminology of the second sample instance object is embedded at the words of the second sample instance object in the second sample prompt instruction.

12. The method according to claim 11, characterized in that, The step of generating the second sample prompt instruction based on the second bounding box of each second sample instance object in the second sample target image includes: Based on the second bounding box of the second sample instance object, the second normalized coordinates of the second sample instance object are obtained; Based on the second normalized coordinates of the second sample instance object and the index terms corresponding to each of the plurality of normalized coordinates, a third major model instruction is constructed; wherein, the third major model instruction is used to instruct the third major model to map the second normalized coordinates of the second sample instance object to the index terms of the second sample instance object by referring to the index terms corresponding to each of the plurality of normalized coordinates, and to generate a third descriptive statement of the second sample target image based on the index terms of each of the second sample instance objects, and the index terms of the second sample instance object are embedded in the words of the second sample instance object in the third descriptive statement; Obtain the second output statement of the third major model in response to the third major model instruction, and use it as the second sample prompt instruction.

13. The method according to any one of claims 1 to 12, characterized in that, The generative model responds to the prompt text instruction and outputs multiple model-generated images. For each model-generated image, the average intersection-union ratio is calculated based on the expected coordinates of the instance object and the actual coordinates of the instance object in the model-generated image. The target generated image is determined by the average intersection-union ratio of each model-generated image. And / or, the index term of any instance object in the prompt text instruction is surrounded by a start term and an end term; And / or, the index term includes an index identifier, which is used to distinguish between ordinary terms in the prompt text instruction and the index term.

14. An image generation apparatus, characterized in that, include: The content acquisition module is used to acquire an image generation request; wherein, the image generation request describes the instance objects in the image to be generated and the layout position of the instance objects; The coordinate determination module is used to obtain the expected coordinates of the instance object based on the image generation request; The index determination module is used to normalize the expected coordinates of the instance object and determine the index term of the instance object from the index terms corresponding to several preset normalized coordinates. The prompt construction module is used to construct prompt text instructions based on the index terms of the instance object; wherein, the prompt text instructions describe the desired generated image in natural language statements, and embed the index terms of the instance object at the words of the instance object in the natural language statements; The model generation module is used to generate a model image based on the prompt text instruction output by the generative model, and obtain a target generated image for responding to the image generation request.

15. An electronic device, characterized in that, The method includes at least a memory and a processor coupled to each other, wherein the memory stores at least program instructions, and the processor executes the program instructions to implement the image generation method according to any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that, The device stores program instructions that can be executed by a processor, the program instructions being used to implement the image generation method according to any one of claims 1 to 13.

Images