Image generation method, device, equipment and storage medium

Abstract

The present invention discloses an image generation method, device, equipment and storage medium, the method comprising: in response to detecting target image layout data, inputting the target image layout data into a pre-trained target image labeling model; outputting an object feature vector based on the target image layout data through a feature conversion module in the target image labeling model; obtaining image block coordinate data corresponding to a plurality of image blocks in the image to be generated through an image labeling module in the target image labeling model, constructing a connection matrix based on each image block coordinate data and each boundary coordinate data, and outputting target image block identifiers corresponding to a plurality of image blocks based on the connection matrix and the object feature vector; inputting each target image block identifier into a pre-trained image generation model to obtain an output image to be generated. This embodiment solves the problem of unreasonable connection matrix construction in the prior art and improves the quality of the generated image.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an image generation method, device, equipment and storage medium. Background Art

[0002] Image generators have always been a focus of machine vision and play an important role in poster design, virtual scene design, etc.

[0003] There are mainly two types of existing image generators. One is an image generation method based on a convolutional neural network model, and the other is an image generation method based on a transformer model.

[0004] In the process of implementing the present invention, it is found that there are at least the following technical problems in the prior art:

[0005] Image generation methods based on convolutional neural network models cannot generate the correct object structure, such as generating an image of a person with three hands. In the image generation methods based on transformer models, the connection matrix is ​​constructed in an unreasonable way and cannot accurately understand the relationship between multiple objects in the image. For example, an image of a person riding a horse is generated instead of an image of a person standing on a horse. Therefore, the image quality generated by existing image generators is poor. Summary of the invention

[0006] The present invention provides an image generation method, device, equipment and storage medium to solve the problem of unreasonable construction method of connection matrix in the prior art and improve the quality of images generated by an image generator.

[0007] According to one aspect of the present invention, there is provided an image generation method, the method comprising:

[0008] In response to detecting the target image layout data, the target image layout data is input into a pre-trained target image labeling model; wherein the target image local data includes boundary coordinate data corresponding to at least one object contained in the image to be generated;

[0009] Outputting an object feature vector based on the target image layout data through a feature conversion module in the target image labeling model;

[0010] Obtain image block coordinate data corresponding to a plurality of image blocks in the to-be-generated image through an image labeling module in the target image labeling model, construct a connection matrix based on each of the image block coordinate data and each of the boundary coordinate data, and output target image block identifiers corresponding to the plurality of image blocks based on the connection matrix and the object feature vector; wherein the connection matrix represents interaction information between the object and the image block and / or interaction information between the image blocks in the to-be-generated image;

[0011] Each target image block identifier is input into a pre-trained image generation model to obtain an output image to be generated.

[0012] According to another aspect of the present invention, there is provided an image generating device, the device comprising:

[0013] A target image layout data input module, configured to input the target image layout data into a pre-trained target image labeling model in response to detecting the target image layout data; wherein the target image local data includes boundary coordinate data corresponding to at least one object contained in the image to be generated;

[0014] An object feature vector output module, configured to output an object feature vector based on the target image layout data through a feature conversion module in the target image labeling model;

[0015] a target image block identification determination module, configured to obtain image block coordinate data corresponding to a plurality of image blocks in the to-be-generated image through an image marking module in the target image marking model, construct a connection matrix based on each of the image block coordinate data and each of the boundary coordinate data, and output target image block identifications corresponding to a plurality of image blocks based on the connection matrix and the object feature vector; wherein the connection matrix represents interaction information between the object and the image block and / or interaction information between the image blocks in the to-be-generated image;

[0016] The image to be generated output module is used to input each of the target image block identifiers into a pre-trained image generation model to obtain an output image to be generated.

[0017] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0018] at least one processor; and

[0019] a memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the image generating method described in any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the image generation method described in any embodiment of the present invention when executed.

[0022] The technical solution of the embodiment of the present invention obtains image block coordinate data corresponding to multiple image blocks in the image to be generated through an image marking module in a pre-trained target image marking module, and constructs a connection matrix based on each image block coordinate data and each boundary coordinate data. The connection matrix represents the interaction information between the object and the image block and / or the interaction information between the image blocks in the image to be generated, and outputs the target image block identifiers corresponding to the multiple image blocks based on the connection matrix and the object feature vector output based on the target image layout data, thereby solving the problem that the interaction information represented by the connection matrix constructed in the prior art is entangled at the object level and the image block level, improving the rationality of the constructed connection matrix, thereby ensuring the accuracy of the object structure in the generated image and the rationality of the association relationship between objects, and improving the image quality of the generated image.

[0023] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 is a flow chart of an image generating method provided according to an embodiment of the present invention;

[0026] Figure 2A is a schematic diagram of an image block provided according to an embodiment of the present invention;

[0027] Figure 2B is a schematic diagram of a connection matrix provided according to an embodiment of the present invention;

[0028] Figure 2CIt is a schematic diagram of a connection matrix constructed by the prior art;

[0029] Figure 2D is a schematic diagram of a connection matrix provided according to an embodiment of the present invention;

[0030] Figure 2E is a schematic diagram of another connection matrix provided according to an embodiment of the present invention;

[0031] Figure 3 is a flowchart of another image generating method provided according to an embodiment of the present invention;

[0032] Figure 4 is a schematic diagram of another connection matrix provided according to an embodiment of the present invention;

[0033] Figure 5 is a flowchart of another image generating method provided according to an embodiment of the present invention;

[0034] Figure 6 is a schematic diagram of another connection matrix provided according to an embodiment of the present invention;

[0035] Figure 7 is a flowchart of another image generating method provided according to an embodiment of the present invention;

[0036] Figure 8 is a flowchart of a method for training a target image labeling model provided according to an embodiment of the present invention;

[0037] Fig. 9 is a structural schematic diagram of an image generating device provided according to an embodiment of the present invention;

[0038] Fig.10 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0040] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0041] Figure 1 is a flowchart of an image generation method provided according to an embodiment of the present invention. This embodiment is applicable to the case where an image is automatically generated based on an image layout requirement input by a user. The method can be executed by an image generation device, which can be implemented in the form of hardware and / or software, and can be configured in a terminal device. Figure 1 As shown, the method includes:

[0042] S110 . In response to detecting the target image layout data, inputting the target image layout data into a pre-trained target image labeling model.

[0043] In this embodiment, the target image local data includes boundary coordinate data corresponding to at least one object contained in the image to be generated. Specifically, the boundary coordinate data can be used to describe the object area to which the object belongs in the image to be generated. Exemplarily, when the shape of the object area is a square, the boundary coordinate data includes the upper left corner coordinate data (x 1 ,y 1 ) and the lower right corner coordinate data (x 2 ,y 2 ).

[0044] In one embodiment, optionally, the target image layout data further includes an object index corresponding to at least one object contained in the image to be generated. The object index is used to represent the object category to which the object belongs. Exemplarily, the target image layout data can be expressed as:

[0045]

[0046] Where c represents the target image layout data, l i represents the category index corresponding to the i-th object in the image to be generated, b i =[x 1i ,y 1i ,x2i ,y 2i ] represents the boundary coordinate data corresponding to the i-th object in the image to be generated, where (x 1i ,y 1i ) represents the coordinate data of the upper left corner of the i-th object, (x 2i ,y 2i ) represents the upper right corner coordinate data of the ith object, and N represents the total number of objects contained in the image to be generated.

[0047] In one embodiment, optionally, before inputting the target image layout data into the pre-trained target image labeling model, the method further includes: performing a scaling operation on the boundary coordinate data according to the image size of the image to be generated to obtain the corrected boundary coordinate data. Wherein, for example, if the image size of the image to be generated is 256*256, the boundary coordinate data of the i-th object is (x 1i ,y 1i ,x 2i ,y 2i ), if the image size of the image to be generated is 128*128, the boundary coordinate data of the i-th object is The advantage of such a setting is that it ensures the adaptability of the boundary coordinate data to the image size, and avoids the situation where the object area defined by the boundary coordinate data is too large or too small, resulting in an unreasonable image to be generated.

[0048] In this embodiment, the model type of the target image labeling model is a transformer model.

[0049] S120. Output an object feature vector based on the target image layout data through a feature conversion module in the target image labeling model.

[0050] Specifically, the feature conversion module maps the target image layout data to the feature space to obtain an object feature vector. Exemplarily, the object feature vector F can be expressed as:

[0051] F=[Emb c (c)]

[0052] Among them, Emb c Represents a feature mapping network.

[0053] S130. Obtain image block coordinate data corresponding to a plurality of image blocks in the to-be-generated image through an image labeling module in the target image labeling model, construct a connection matrix based on the image block coordinate data and the boundary coordinate data, and output target image block identifiers corresponding to the plurality of image blocks based on the connection matrix and the object feature vector.

[0054] Specifically, the image block is a part of the image area corresponding to the image to be generated. Exemplarily, based on the image size of the image to be generated, the image to be generated is divided into multiple image blocks, and the image block coordinate data corresponding to the multiple image blocks are determined. Figure 2A is a schematic diagram of an image block provided according to an embodiment of the present invention. Specifically, the image size of the image to be generated is 8*8, and the size of the image block is 2*2, then the number of image blocks obtained by splitting is 4*4=16. Exemplarily, the image block coordinate data may be the upper left corner coordinate data, the lower right corner coordinate data, or the center point coordinate data of the image block, etc. Figure 2A For example, the image block coordinate data corresponding to the upper left corner of the image block can be (0,0), (2,2) or (1,1). The specific value position of the image block coordinate data is not limited here, and the user can customize it according to actual needs.

[0055] Figure 2B is a schematic diagram of a connection matrix provided according to an embodiment of the present invention. Specifically, Figure 2B The upper figure in the figure represents the connection matrix. The three borders in the lower left figure represent the three objects contained in the image to be generated, which are P1, P2 and P3 respectively. The lower left figure shows the layout of the three objects in the image to be generated. The lower right figure represents the image to be generated and the 25 image blocks corresponding to the image to be generated. The two cylinders and one quadrilateral in the lower right figure correspond to P1, P2 and P3 in the lower left figure respectively. Specifically, the horizontal axis of the connection matrix is ​​the multiple objects and multiple image blocks contained in the image to be generated, which together constitute the horizontal axis object. The vertical axis of the connection matrix is ​​also the multiple objects and multiple image blocks contained in the image to be generated, which together constitute the vertical axis object. The horizontal axis object is the same as the vertical axis object. Figure 2B The horizontal and vertical axes of the connection matrix in do not include all the image blocks in the lower right figure, and only image block 1, image block 9, image block 11, image block 12 and image block 14 are exemplified.

[0056] Figure 2B The area formed by the vertical boxes in the connection matrix represents the matrix area corresponding to objects, the area formed by the gray boxes represents the matrix area corresponding to objects and image blocks, the area formed by the horizontal boxes represents the matrix area corresponding to image blocks, and the area formed by the white boxes belongs to the undefined area, and the matrix value in this area defaults to -∞.

[0057] by Figure 2B For example, Figure 2C is a schematic diagram of a connection matrix constructed in the prior art. Specifically, the connection matrix in the prior art satisfies the formula:

[0058]

[0059] Among them, i represents the vertical axis coordinate value corresponding to the vertical axis object in the connection matrix, and j represents the horizontal axis coordinate value corresponding to the horizontal axis object in the connection matrix. Specifically, Figure 2C The medium gray matrix square corresponds to "1" and the white matrix square corresponds to "-∞".

[0060] from Figure 2C It can be seen from the connection matrix shown that the connection matrix defined in the prior art is entangled at the object level and the image block level, ignoring the semantic information between objects, resulting in an unreasonable connection matrix.

[0061] In this embodiment, the connection matrix represents the interaction information between the object and the image block in the image to be generated and / or the interaction information between the image blocks.

[0062] In one embodiment, optionally, a connection matrix is ​​constructed based on each image block coordinate data and each boundary coordinate data, including: for each boundary coordinate data, taking each image block coordinate data as target image block coordinate data in turn; based on the boundary coordinate data and the target image block coordinate data, judging whether the target image block corresponding to the target image block coordinate data belongs to the object corresponding to the boundary coordinate data; if so, setting the matrix value corresponding to the target image block and the object in the connection matrix to a first value, so that the connection matrix represents the interaction information between the target image block and the object in the image to be generated.

[0063] Specifically, the matrix value corresponding to the target image block and the object in the connection matrix is ​​the matrix area corresponding to the object and the image block in the connection matrix ( Figure 2B The matrix value corresponding to the object and the target image block in the area formed by the gray box in the middle does not include the undefined area ( Figure 2B The matrix values ​​within the area formed by the white box in .

[0064] In which, exemplarily, the first value is 1.

[0065] In one embodiment, optionally, based on the boundary coordinate data and the target image block coordinate data, determining whether the target image block corresponding to the target image block coordinate data belongs to the object corresponding to the boundary coordinate data includes: determining, based on the boundary coordinate data, an object region of the object corresponding to the boundary coordinate data in the image to be generated; determining, based on the target image block coordinate data and a preset image block size, a target image block region of the target image block corresponding to the target image block coordinate data in the image to be generated; if the overlapping area between the target image block region and the object region, and the area ratio of the target image block region exceed the preset area ratio, determining that the target image block belongs to the object corresponding to the boundary coordinate data.

[0066] Exemplarily, the preset area ratio may be 80%.

[0067] Based on the above embodiment, optionally, the method further includes, if the target image block corresponding to the target image block coordinate data does not belong to the object corresponding to the boundary coordinate data, setting the matrix value corresponding to the target image block and the object in the connection matrix to a second value. Exemplarily, the second value is -∞.

[0068] The connection matrix in this embodiment satisfies the formula:

[0069]

[0070] Where, when i represents the vertical axis coordinate value corresponding to the object on the vertical axis in the connection matrix, j represents the horizontal axis coordinate value corresponding to the object on the horizontal axis in the connection matrix, when i represents the vertical axis coordinate value corresponding to the image block on the vertical axis in the connection matrix, j represents the horizontal axis coordinate value corresponding to the image block on the horizontal axis in the connection matrix, and s x represents the xth image block corresponding to the vertical axis in the connection matrix, P y Represents the y-th object corresponding to the horizontal axis in the connection matrix.

[0071] by Figure 2B For example, Figure 2D is a schematic diagram of a connection matrix provided according to an embodiment of the present invention. Specifically, Figure 2D In the connection matrix shown, image block 9 and image block 14 belong to object P2, respectively, and image block 11 and image block 12 belong to object P1, respectively.

[0072] The advantage of this setting is that the connection matrix clearly represents the interactive information between the object and the image block, ensures the consistency between the object and the image block, and improves the rationality of the connection matrix.

[0073] Based on the above embodiment, optionally, the method also includes: if the target image block corresponding to the target image block coordinate data does not belong to any object corresponding to any boundary coordinate data, then the matrix values ​​corresponding to the target image block and each object in the connection matrix are all set to the first value, so that the connection matrix represents the interaction information between the target image block and each object in the image to be generated.

[0074] Specifically, it is determined whether the target image block corresponding to the target image block coordinate data belongs to the object corresponding to the current boundary coordinate data. If not, it is continued to be determined whether the target image block corresponding to the target image block coordinate data belongs to the object corresponding to the next boundary coordinate data, until it is determined whether the target image block corresponding to the target image block coordinate data belongs to the object corresponding to the last boundary coordinate data. If so, the matrix values ​​corresponding to the target image block and the object corresponding to the last boundary coordinate data in the connection matrix are all set to the first value. If not, the matrix values ​​corresponding to the target image block and each object in the connection matrix are all set to the first value.

[0075] by Figure 2B For example, Figure 2E is a schematic diagram of another connection matrix provided according to an embodiment of the present invention. Specifically, Figure 2E In the connection matrix shown, image blocks 9 and 14 belong to object P2, image blocks 11 and 12 belong to object P1. Since image block 1 does not belong to object P1, object P2, and object P3, the matrix values ​​corresponding to image block 1 and object P1, object P2, and object P3, respectively, are also set to the first value.

[0076] The advantage of this setting is that the target image block does not belong to any object in the image to be generated, which means that the target image block belongs to the background area. The matrix values ​​of the background area and each object are set to the first value, so that the connection matrix can represent the interaction information between the image block in the background area and all objects, thereby improving the image quality of the generated image.

[0077] Specifically, the image labeling module in this embodiment uses a self-attention mechanism, and the calculation process of the self-attention mechanism can be expressed as:

[0078]

[0079] Among them, T represents transpose, D K represents the vector dimension of the object feature vector, M represents the connection matrix, and Softmax represents the activation function. Q = W Q F,K=W K F,V=W V F, where W Q , W K and W V represents the learnable model parameters in the image labeling module, and F represents the feature vector.

[0080] Specifically, an activation parameter value is determined based on a connection matrix and an object feature vector through an image marking module, and target image block identifiers corresponding to the plurality of image blocks are determined based on the activation parameter value.

[0081] Specifically, the target image block identifier is used to represent the image information contained in the image block. For example, the target image block identifier is 1, and the corresponding image information is "head", the target image identifier is 2, and the corresponding image information is "palm", etc. The method for setting the target image block identifier and the image information it represents are not limited here.

[0082] S140: Input each target image block identifier into a pre-trained image generation model to obtain an output image to be generated.

[0083] Among them, exemplarily, the image generation model can be a decoder of VQ-VAE (image vectorization network).

[0084] The technical solution of this embodiment is to obtain image block coordinate data corresponding to multiple image blocks in the image to be generated through the image marking module in the pre-trained target image marking module, take each image block coordinate data as the target image block coordinate data in turn, judge whether the target image block corresponding to the target image block coordinate data belongs to an object corresponding to a certain boundary coordinate data, and if so, set the matrix value corresponding to the target image block and the object in the connection matrix to the first value, so that the connection matrix represents the interaction information between the object and the image block in the image to be generated, and output the target image block identifiers corresponding to the multiple image blocks based on the connection matrix and the object feature vector output based on the target image layout data, so as to solve the problem that the interaction information represented by the connection matrix constructed in the prior art is entangled at the object level and the image block level, improve the rationality of the constructed connection matrix, thereby ensuring the accuracy of the object structure in the generated image and the rationality of the association relationship between objects, and improve the image quality of the generated image.

[0085] Figure 3 is a flow chart of another image generation method provided according to an embodiment of the present invention. This embodiment further refines the technical feature of "building a connection matrix based on the coordinate data of each image block and each boundary coordinate data" in the above embodiment. Figure 3 As shown, the method includes:

[0086] S210 . In response to detecting the target image layout data, input the target image layout data into a pre-trained target image labeling model.

[0087] S220. Output an object feature vector based on the target image layout data through a feature conversion module in the target image labeling model.

[0088] S230 , obtaining image block coordinate data corresponding to a plurality of image blocks in the image to be generated, respectively, through an image marking module in the target image marking model.

[0089] S240 . For each image block coordinate data, determine the object to which the image block corresponding to the image block coordinate data belongs based on the image block coordinate data and each boundary coordinate data.

[0090] Specifically, for each boundary coordinate data, based on the boundary coordinate data, the object area of ​​the object corresponding to the boundary coordinate data in the image to be generated is determined; based on the image block coordinate data and the preset image block size, the image block area of ​​the image block corresponding to the image block coordinate data in the image to be generated is determined; if the overlapping area between the target image block area and the object area, and the area ratio between the target image block area exceed the preset area ratio, it is determined that the image block belongs to the object corresponding to the boundary coordinate data; if the area ratio is less than the preset area ratio, it is determined that the image block does not belong to the object corresponding to the boundary coordinate.

[0091] Among them, illustratively, the preset area ratio is 80%.

[0092] S250, determining whether the objects to which any two image blocks belong are the same, if yes, executing S260, if no, executing S270.

[0093] S260, setting the matrix value corresponding to the two image blocks in the connection matrix to a first value, so that the connection matrix represents the interaction information between the two image blocks in the image to be generated, and executing S270.

[0094] The connection matrix in this embodiment satisfies the formula:

[0095]

[0096] Among them, i represents the vertical axis coordinate value corresponding to the vertical axis object in the connection matrix, j represents the horizontal axis coordinate value corresponding to the horizontal axis object in the connection matrix, and s x represents the xth image block corresponding to the vertical axis in the connection matrix, s y represents the yth image block corresponding to the vertical axis in the connection matrix, x represents the vertical axis coordinate value corresponding to the image block on the vertical axis in the connection matrix, y represents the horizontal axis coordinate value corresponding to the image block on the horizontal axis in the connection matrix, P m Represents the mth object in the image to be generated.

[0097] Specifically, the connection matrix represents the interaction information between two image blocks of the same object in the image to be generated.

[0098] by Figure 2B For example, Figure 4 is a schematic diagram of another connection matrix provided according to an embodiment of the present invention. Specifically, Figure 4In the connection matrix shown, image block 9 and image block 14 belong to the same object P2, and image block 11 and image block 12 belong to the same object P1.

[0099] S270: Set the matrix value corresponding to two image blocks in the connection matrix as a second value.

[0100] Based on the above embodiment, optionally, the method further includes: if the two image blocks respectively corresponding to any two image block coordinate data do not belong to the object corresponding to the same boundary coordinate data, then determining whether the two image blocks respectively corresponding to any two image block coordinate data do not belong to the object corresponding to any boundary coordinate data; if so, setting the matrix value corresponding to the two image blocks in the connection matrix to the first value, so that the connection matrix represents the interaction information between the two image blocks in the image to be generated.

[0101] Specifically, whether the two image blocks corresponding to any two image block coordinate data do not belong to the objects corresponding to any boundary coordinate data, it means that the two image blocks belong to the background area, and the matrix values ​​corresponding to the two image blocks in the background area are set to the first value, so that the connection matrix represents the interaction information between the two image blocks in the background area in the image to be generated. Figure 2B The example in does not contain all image patches. Figure 2B The five types of image blocks do not include two image blocks belonging to the background area, so the schematic diagram of the connection matrix provided by this embodiment is not shown.

[0102] S280 , outputting target image block identifiers corresponding to the plurality of image blocks respectively based on the connection matrix and the object feature vector.

[0103] S290, inputting each target image block identifier into a pre-trained image generation model to obtain an output image to be generated.

[0104] The technical solution of this embodiment is to determine, for any two image block coordinate data, whether the two image blocks respectively corresponding to any two image block coordinate data belong to the object corresponding to the same boundary coordinate data based on the any two image block coordinate data and each boundary coordinate data; if so, the matrix value commonly corresponding to the two image blocks in the connection matrix is ​​set to a first value, so that the connection matrix represents the interaction information between the two image blocks in the image to be generated, and output the target image block identifiers respectively corresponding to the multiple image blocks based on the connection matrix and the object feature vector output based on the target image layout data, thereby solving the problem that the interaction information represented by the connection matrix constructed in the prior art is entangled at the object level and the image block level, improving the rationality of the constructed connection matrix, thereby ensuring the accuracy of the object structure in the generated image and the rationality of the association relationship between objects, and improving the image quality of the generated image.

[0105] Figure 5 FIG. 1 is a flow chart of another image generation method provided according to an embodiment of the present invention. This embodiment further refines the “connection matrix” in the above embodiment. Figure 5 , the method comprising:

[0106] S310 . In response to detecting the target image layout data, input the target image layout data into a pre-trained target image labeling model.

[0107] S320. Output an object feature vector based on the target image layout data through a feature conversion module in the target image labeling model.

[0108] S330 , obtaining image block coordinate data corresponding to a plurality of image blocks in the image to be generated, respectively, through an image labeling module in the target image labeling model.

[0109] S340: construct a connection matrix based on the coordinate data of each image block and the coordinate data of each boundary.

[0110] In this embodiment, a connection matrix is ​​constructed based on each image block coordinate data and each boundary coordinate data, including: for each boundary coordinate data, taking each image block coordinate data as target image block coordinate data in turn; based on the boundary coordinate data and the target image block coordinate data, judging whether the target image block corresponding to the target image block coordinate data belongs to the object corresponding to the boundary coordinate data; if so, setting the matrix value corresponding to the target image block and the object in the connection matrix to a first value, so that the connection matrix represents the interaction information between the target image block and the object in the image to be generated.

[0111] The connection matrix defined in this section satisfies the formula:

[0112]

[0113] Among them, s i represents the i-th image block corresponding to the vertical axis in the connection matrix, P j Represents the jth object in the image to be generated.

[0114] In this embodiment, a connection matrix is ​​constructed based on each image block coordinate data and each boundary coordinate data, and the method further includes: for any two image block coordinate data, based on the any two image block coordinate data and each boundary coordinate data, determining whether two image blocks respectively corresponding to any two image block coordinate data belong to an object corresponding to the same boundary coordinate data; if so, setting the matrix value corresponding to the two image blocks in the connection matrix to the first value, so that the connection matrix represents the interaction information between the two image blocks in the image to be generated.

[0115] The connection matrix defined in this section satisfies the formula:

[0116]

[0117] Among them, s i represents the i-th image block in the image to be generated, s j represents the jth image block in the image to be generated, i represents the vertical axis coordinate value corresponding to the image block on the vertical axis in the connection matrix, j represents the horizontal axis coordinate value corresponding to the image block on the horizontal axis in the connection matrix, P m Represents the mth object in the image to be generated.

[0118] S350: Set the matrix value corresponding to any two objects in the connection matrix to a first value, so that the connection matrix also represents the interaction information between any two objects in the image to be generated.

[0119] Among them, the connection matrix defined in S350 satisfies the formula:

[0120] M oo [s i ,s j ]=1

[0121] Among them, s i represents the i-th object in the image to be generated, s j Represents the jth object in the image to be generated.

[0122] The connection matrix in this embodiment satisfies the formula:

[0123]

[0124] Figure 6 is a schematic diagram of another connection matrix provided according to an embodiment of the present invention. Specifically, Figure 6 In the connection matrix shown, the 9 matrix values ​​corresponding to objects P1, P2 and P3 are first values ​​respectively, the matrix value corresponding to image block 9 and image block 14 is the first value, the matrix value corresponding to image block 11 and image block 12 is the first value, the 3 matrix values ​​corresponding to image block 1 and three object decibels are the first values, the matrix values ​​corresponding to image block 9 and image block 14 and object P2 are the first values ​​respectively, and the matrix values ​​corresponding to image block 11 and image block 12 and object P1 are the first values ​​respectively.

[0125] S360: Based on the connection matrix and the object feature vector, output target image block identifiers corresponding to the multiple image blocks.

[0126] Specifically, the image marking module includes multiple image marking units, and the number of image marking units is greater than or equal to the total number of image blocks corresponding to the image to be generated. Specifically, through the first image marking unit, based on each boundary coordinate data, the first connection matrix is ​​constructed, and based on the first connection matrix and the object feature vector, the target image block identifier corresponding to the first image block is output. Through the i-th image marking unit, in response to receiving the target image identifier corresponding to the i-1th image block output by the i-1th image marking unit, based on each boundary coordinate data and the image block coordinate data corresponding to the first i image blocks, the i-th connection matrix is ​​constructed, and based on the i-th connection matrix, the object feature vector and the target image block identifier corresponding to the first i-1 image blocks, the target image identifier corresponding to the i-th image block is output; wherein 2<i≤N, N is the total number of image blocks.

[0127] S370: Input each target image block identifier into a pre-trained image generation model to obtain an output image to be generated.

[0128] The technical solution of this embodiment constructs a connection matrix based on the coordinate data of each image block and the coordinate data of each boundary, and further sets the matrix value corresponding to any two objects in the connection matrix to be a first value, so that the connection matrix represents the interaction information between objects, between objects and image blocks, and between image blocks in the image to be generated, and outputs target image block identifiers corresponding to multiple image blocks based on the connection matrix and the object feature vector output based on the target image layout data, thereby solving the problem that the interaction information represented by the connection matrix constructed in the prior art is entangled at the object level and the image block level, and improving the rationality of the constructed connection matrix from the dimensions of three types of interaction information, further ensuring the accuracy of the object structure in the generated image and the rationality of the association relationship between objects, thereby further improving the image quality of the generated image.

[0129] Figure 7 FIG. 1 is a flowchart of another image generation method provided according to an embodiment of the present invention. This embodiment further defines the training method of the “target image labeling model” in the above embodiment. Figure 7 As shown, the method includes:

[0130] S410: Obtain sample image layout data corresponding to the sample image and real image block identifiers corresponding to a plurality of image blocks in the sample image.

[0131] In this embodiment, the sample image layout data includes boundary coordinate data corresponding to at least one object included in the sample image.

[0132] Specifically, the sample image is input into a pre-trained encoder to obtain real image block identifiers corresponding to multiple image blocks in the output sample image. Exemplarily, the encoder may be a VQ-VAE (image vectorization network) encoder. The VQ-VAE (image vectorization network) encoder encodes the sample image into a set of real image block identifiers indexed by a code number, and the real image block identifier set s can be expressed as:

[0133] s={s 1 ,s 2 ,...,s i ,...,s H}

[0134] Among them, s i represents the real image block identifier corresponding to the i-th image block in the sample image, and H represents the total number of image blocks corresponding to the sample image.

[0135] S420: Input the sample image layout data and the identifiers of each real image block into an untrained initial image labeling model to obtain a set of prediction probabilities corresponding to each image block.

[0136] Specifically, the sample image layout data is converted into an object feature vector, and each real image block identifier is converted into an image block feature vector through a feature conversion module in the initial image labeling model. Specifically, the feature conversion module maps the sample image layout data and each real image block identifier to a feature space to obtain an image feature vector. Exemplarily, the image feature vector F can be expressed as:

[0137] F=[Emb c (c);Emb s (s)]

[0138] Among them, Emb c and Emb s Both represent feature mapping networks; represent concatenation operations.

[0139] Specifically, the image labeling module in the initial image labeling model includes multiple image labeling units, and the number of image labeling units is greater than or equal to the total number of image blocks corresponding to the sample image. Specifically, the first image labeling unit constructs the first connection matrix based on each boundary coordinate data, and outputs the prediction probability set corresponding to the first image block based on the first connection matrix and the object feature vector. Through the i-th image labeling unit, based on each boundary coordinate data and the image block coordinate data corresponding to the first i image blocks, the i-th connection matrix is ​​constructed, and based on the i-th connection matrix, the object feature vector and the image block feature vectors corresponding to the first i-1 image blocks, the prediction probability set corresponding to the i-th image block is output; wherein 2<i≤N, N is the total number of image blocks.

[0140] In this embodiment, the prediction probability set includes prediction probability values ​​corresponding to the corresponding image block and at least one preset image block identifier, respectively. The sum of at least one prediction probability value is 1. For example, assuming that the preset image block identifiers include 1, 2, and 3, the prediction probability set includes P(x=1)=0.9, P(x=2)=0.1, and P(x=3)=0.

[0141] S430: construct a loss function based on the real image block identifiers and prediction probability sets corresponding to each image block.

[0142] In one embodiment, optionally, a loss function is constructed based on the real image block identifiers and prediction probability sets corresponding to each image block, including: for each image block, taking the prediction probability value corresponding to the real image block identifier of the image block in the prediction probability set corresponding to the image block as the target prediction probability value; and constructing a loss function based on each target prediction probability value.

[0143] Among them, the loss function satisfies the formula:

[0144]

[0145] Among them, H represents the total number of image blocks corresponding to the sample image, P(x i =real image identification) is the target prediction probability value selected from the prediction probability set based on the real image identification.

[0146] S440: Adjust the model parameters in the initial image labeling model based on the loss function to obtain a trained target image labeling model.

[0147] Specifically, when the loss function converges, a trained target image labeling model is obtained.

[0148] Figure 8It is a flowchart of a training method of a target image labeling model provided according to an embodiment of the present invention. Specifically, a sample image is input into an image vectorization network encoder to obtain a real image block label set of multiple image blocks corresponding to the sample image, and the sample image layout data and the real image block identification set are input into a transformer model based on a focus attention mechanism to obtain a prediction probability set corresponding to each of the multiple image blocks output, the prediction probability set includes a prediction probability value corresponding to at least one preset image block identification, and a loss function is constructed based on each prediction probability set and the real image block identification set, and the model parameters in the transformer are adjusted based on the loss function. When the loss function converges, a trained target image labeling model is obtained.

[0149] S450: In response to detecting the target image layout data, input the target image layout data into a pre-trained target image labeling model.

[0150] S460. Output an object feature vector based on the target image layout data through a feature conversion module in the target image labeling model.

[0151] S470. Obtain image block coordinate data corresponding to a plurality of image blocks in the to-be-generated image through an image labeling module in the target image labeling model, construct a connection matrix based on the image block coordinate data and the boundary coordinate data, and output target image block identifiers corresponding to the plurality of image blocks based on the connection matrix and the object feature vector.

[0152] In another embodiment, optionally, based on the connection matrix and the object feature vector, target image block identifiers corresponding to multiple image blocks are output, including: for each image marking unit, the image marking unit outputs a prediction probability set corresponding to the corresponding image block based on the corresponding connection matrix and the object feature vector; wherein the prediction probability set includes prediction probability values ​​corresponding to the corresponding image block and at least one preset image block identifier; and the preset image block identifier corresponding to the prediction probability value that meets the preset selection rule in the prediction probability set is used as the output target image block identifier. Exemplarily, the preset selection rule may be the largest prediction probability value or the second largest prediction probability value in the prediction probability set. The preset selection rules corresponding to different image marking units may be the same or different.

[0153] S480: Input each target image block identifier into a pre-trained image generation model to obtain an output image to be generated.

[0154] By setting the preset selection rules corresponding to different image marking units in a diversified manner, different combinations of multiple target image block identifiers can be obtained, and then multiple images to be generated can be obtained.

[0155] The technical solution of this embodiment is to input the sample image layout data and the real image block identifiers of each image block corresponding to the sample image into an untrained initial image labeling model to obtain the predicted probability sets corresponding to each output image block, construct a loss function based on the real image block identifiers and the predicted probability sets corresponding to each image block, adjust the model parameters in the initial image labeling model based on the loss function, and obtain a trained target image labeling model, thereby solving the training problem of the target image labeling model, improving the integrity of the image generation method, ensuring the training effect of the target image labeling model, and further ensuring the quality of subsequently generated images.

[0156] Fig. 9 is a schematic diagram of the structure of an image generating device provided according to an embodiment of the present invention. Fig. 9 As shown, the device includes: a target image layout data input module 510, an object feature vector output module 520, a target image block identification determination module 530 and a to-be-generated image output module 540.

[0157] The target image layout data input module 510 is used to input the target image layout data into a pre-trained target image labeling model in response to detecting the target image layout data; wherein the target image local data includes boundary coordinate data corresponding to at least one object contained in the image to be generated;

[0158] An object feature vector output module 520, configured to output an object feature vector based on the target image layout data through a feature conversion module in the target image labeling model;

[0159] The target image block identification determination module 530 is used to obtain the image block coordinate data corresponding to the multiple image blocks in the image to be generated through the image marking module in the target image marking model, construct a connection matrix based on the image block coordinate data and the boundary coordinate data, and output the target image block identifications corresponding to the multiple image blocks based on the connection matrix and the object feature vector; wherein the connection matrix represents the interaction information between the object and the image block and / or the interaction information between the image blocks in the image to be generated;

[0160] The to-be-generated image output module 540 is used to input each target image block identifier into a pre-trained image generation model to obtain an output image to be generated.

[0161] The technical solution of this embodiment obtains image block coordinate data corresponding to multiple image blocks in the image to be generated through the image marking module in the pre-trained target image marking module, and constructs a connection matrix based on the image block coordinate data and the boundary coordinate data. The connection matrix represents the interaction information between the object and the image block and / or the interaction information between the image blocks in the image to be generated. Based on the connection matrix and the object feature vector output based on the target image layout data, the target image block identifiers corresponding to the multiple image blocks are output, which solves the problem of entanglement between the interaction information represented by the connection matrix constructed in the prior art at the object level and the image block level, improves the rationality of the constructed connection matrix, and thus ensures the accuracy of the object structure in the generated image and the rationality of the association relationship between objects, thereby improving the image quality of the generated image.

[0162] Based on the above embodiment, optionally, the target image block identification determination module 530 includes:

[0163] A first connection matrix construction unit is used to use the image block coordinate data as the target image block coordinate data in sequence for each boundary coordinate data;

[0164] Based on the boundary coordinate data and the target image block coordinate data, determining whether the target image block corresponding to the target image block coordinate data belongs to the object corresponding to the boundary coordinate data;

[0165] If yes, the matrix value corresponding to the target image block and the object in the connection matrix is ​​set to the first value, so that the connection matrix represents the interaction information between the target image block and the object in the image to be generated.

[0166] Based on the above embodiment, optionally, the first connection matrix construction unit is specifically used to:

[0167] Based on the boundary coordinate data, determining an object region of the object corresponding to the boundary coordinate data in the image to be generated;

[0168] Based on the target image block coordinate data and the preset image block size, determining a target image block area of ​​the target image block corresponding to the target image block coordinate data in the image to be generated;

[0169] If the area ratio of the overlapping area between the target image block area and the object area to the target image block area exceeds a preset area ratio, it is determined that the target image block belongs to the object corresponding to the boundary coordinate data.

[0170] Based on the above embodiment, optionally, the first connection matrix construction unit is further used to:

[0171] If the target image block corresponding to the target image block coordinate data does not belong to any object corresponding to any boundary coordinate data, the matrix values ​​corresponding to the target image block and each object in the connection matrix are all set to the first value, so that the connection matrix represents the interaction information between the target image block and each object in the image to be generated.

[0172] Based on the above embodiment, optionally, the target image block identification determination module 530 includes:

[0173] A second connection matrix construction unit is used to determine, for any two image block coordinate data, based on the any two image block coordinate data and each boundary coordinate data, whether two image blocks respectively corresponding to any two image block coordinate data belong to an object corresponding to the same boundary coordinate data;

[0174] If yes, the matrix value corresponding to the two image blocks in the connection matrix is ​​set to the first value, so that the connection matrix represents the interaction information between the two image blocks in the image to be generated.

[0175] Based on the above embodiment, optionally, the second connection matrix construction unit is further used to:

[0176] If the two image blocks corresponding to any two image block coordinate data do not belong to the object corresponding to the same boundary coordinate data, then determining whether the two image blocks corresponding to any two image block coordinate data do not belong to the object corresponding to any boundary coordinate data;

[0177] If yes, the matrix value corresponding to the two image blocks in the connection matrix is ​​set to the first value, so that the connection matrix represents the interaction information between the two image blocks in the image to be generated.

[0178] Based on the above embodiment, optionally, the device further includes:

[0179] The second connection matrix construction module is used to set the matrix value corresponding to any two objects in the connection matrix to the first value before outputting the target image block identifiers corresponding to multiple image blocks respectively based on the connection matrix and the object feature vector, so that the connection matrix also represents the interaction information between any two objects in the image to be generated.

[0180] Based on the above embodiment, optionally, the device further includes:

[0181] The target image labeling model training module is used to obtain sample image layout data corresponding to the sample image and real image block identifiers corresponding to multiple image blocks in the sample image; wherein the sample image layout data includes boundary coordinate data corresponding to at least one object contained in the sample image;

[0182] The sample image layout data and each real image block identifier are input into an untrained initial image labeling model to obtain a set of prediction probabilities corresponding to each image block; wherein the set of prediction probabilities includes prediction probability values ​​corresponding to the corresponding image block and at least one preset image block identifier;

[0183] Construct a loss function based on the real image block identifiers and prediction probability sets corresponding to each image block;

[0184] The model parameters in the initial image labeling model are adjusted based on the loss function to obtain a trained target image labeling model.

[0185] Based on the above embodiment, optionally, the target image labeling model training module is specifically used for:

[0186] For each image block, the prediction probability value corresponding to the real image block identifier of the image block in the prediction probability set corresponding to the image block is used as the target prediction probability value;

[0187] Based on the predicted probability value of each target, a loss function is constructed.

[0188] The image generating device provided in the embodiment of the present invention can execute the image generating method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0189] Fig.10 1 is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0190] like Fig.10As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0191] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0192] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as an image generation method.

[0193] In some embodiments, the image generation method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the image generation method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the image generation method in any other appropriate manner (e.g., by means of firmware).

[0194] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0195] The computer programs for implementing the image generation method of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0196] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute an image generation method, the method comprising:

[0197] In response to detecting the target image layout data, the target image layout data is input into a pre-trained target image labeling model; wherein the target image local data includes boundary coordinate data corresponding to at least one object contained in the image to be generated;

[0198] Outputting an object feature vector based on the target image layout data through a feature conversion module in the target image labeling model;

[0199] Obtain image block coordinate data corresponding to a plurality of image blocks in the image to be generated through an image labeling module in the target image labeling model, construct a connection matrix based on the image block coordinate data and the boundary coordinate data, and output target image block identifiers corresponding to the plurality of image blocks based on the connection matrix and the object feature vector; wherein the connection matrix represents interaction information between the object and the image block and / or interaction information between the image blocks in the image to be generated;

[0200] The identifier of each target image block is input into the pre-trained image generation model to obtain the output image to be generated.

[0201] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0202] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0203] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0204] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0205] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0206] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An image generation method, characterized in that: include: In response to detecting the target image layout data, the target image layout data is input into a pre-trained target image labeling model; wherein the target image local data includes boundary coordinate data corresponding to at least one object contained in the image to be generated, and the model type of the target image labeling model is a transformer model; Outputting an object feature vector based on the target image layout data through a feature conversion module in the target image labeling model; Obtain image block coordinate data corresponding to a plurality of image blocks in the to-be-generated image through an image labeling module in the target image labeling model, construct a connection matrix based on each of the image block coordinate data and each of the boundary coordinate data, and output target image block identifiers corresponding to the plurality of image blocks based on the connection matrix and the object feature vector; wherein the connection matrix represents interaction information between the object and the image block and / or interaction information between the image blocks in the to-be-generated image; Each target image block identifier is input into a pre-trained image generation model to obtain an output image to be generated.

2. The method according to claim 1, characterized in that The step of constructing a connection matrix based on each of the image block coordinate data and each of the boundary coordinate data comprises: For each boundary coordinate data, the coordinate data of each image block is sequentially used as the coordinate data of the target image block; Based on the boundary coordinate data and the target image block coordinate data, determining whether the target image block corresponding to the target image block coordinate data belongs to the object corresponding to the boundary coordinate data; If yes, the matrix value corresponding to the target image block and the object in the connection matrix is ​​set to a first value, so that the connection matrix represents the interaction information between the target image block and the object in the image to be generated.

3. The method according to claim 2, characterized in that The determining, based on the boundary coordinate data and the target image block coordinate data, whether the target image block corresponding to the target image block coordinate data belongs to the object corresponding to the boundary coordinate data comprises: Based on the boundary coordinate data, determining an object region of the object corresponding to the boundary coordinate data in the image to be generated; Based on the target image block coordinate data and a preset image block size, determining a target image block area of ​​the target image block corresponding to the target image block coordinate data in the image to be generated; If the area ratio of the overlapping area between the target image block area and the object area to the target image block area exceeds a preset area ratio, it is determined that the target image block belongs to the object corresponding to the boundary coordinate data.

4. The method according to claim 2, characterized in that: The method further comprises: If the target image block corresponding to the target image block coordinate data does not belong to any object corresponding to any boundary coordinate data, the matrix values ​​corresponding to the target image block and each object in the connection matrix are all set to the first value, so that the connection matrix represents the interaction information between the target image block and each object in the image to be generated.

5. The method according to claim 1, characterized in that The step of constructing a connection matrix based on each of the image block coordinate data and each of the boundary coordinate data comprises: For any two image block coordinate data, based on the any two image block coordinate data and each of the boundary coordinate data, determining whether two image blocks respectively corresponding to the any two image block coordinate data belong to an object corresponding to the same boundary coordinate data; If yes, the matrix value commonly corresponding to the two image blocks in the connection matrix is ​​set to a first value, so that the connection matrix represents the interaction information between the two image blocks in the image to be generated.

6. The method according to claim 5, characterized in that The method further comprises: If the two image blocks respectively corresponding to the arbitrary two image block coordinate data do not belong to the object corresponding to the same boundary coordinate data, determining whether the two image blocks respectively corresponding to the arbitrary two image block coordinate data do not belong to the object corresponding to any boundary coordinate data; If yes, the matrix value commonly corresponding to the two image blocks in the connection matrix is ​​set to a first value, so that the connection matrix represents the interaction information between the two image blocks in the image to be generated.

7. The method according to any one of claims 1 to 6, characterized in that: Before outputting target image block identifiers corresponding to a plurality of image blocks respectively based on the connection matrix and the object feature vector, the method further includes: The matrix value commonly corresponding to any two objects in the connection matrix is ​​set to the first value, so that the connection matrix also represents the interaction information between the any two objects in the image to be generated.

8. The method according to claim 1, characterized in that The training method of the target image labeling model includes: Acquire sample image layout data corresponding to the sample image and real image block identifiers corresponding to a plurality of image blocks in the sample image; wherein the sample image layout data includes boundary coordinate data corresponding to at least one object contained in the sample image; Input the sample image layout data and each of the real image block identifiers into an untrained initial image labeling model to obtain a set of prediction probabilities corresponding to each of the output image blocks; wherein the set of prediction probabilities includes prediction probability values ​​corresponding to the corresponding image block and at least one preset image block identifier; Construct a loss function based on the real image block identifiers and prediction probability sets corresponding to each image block; The model parameters in the initial image labeling model are adjusted based on the loss function to obtain a trained target image labeling model.

9. The method according to claim 8, characterized in that The loss function is constructed based on the real image block identifiers and prediction probability sets corresponding to each image block, including: For each image block, taking a prediction probability value corresponding to a real image block identifier of the image block in a prediction probability set corresponding to the image block as a target prediction probability value; Based on the predicted probability values ​​of each target, a loss function is constructed.

10. An image generating device, characterized in that: include: A target image layout data input module, configured to input the target image layout data into a pre-trained target image labeling model in response to detecting the target image layout data; wherein the target image local data includes boundary coordinate data corresponding to at least one object contained in the image to be generated, and the model type of the target image labeling model is a transformer model; An object feature vector output module, configured to output an object feature vector based on the target image layout data through a feature conversion module in the target image labeling model; a target image block identification determination module, configured to obtain image block coordinate data corresponding to a plurality of image blocks in the to-be-generated image through an image marking module in the target image marking model, construct a connection matrix based on each of the image block coordinate data and each of the boundary coordinate data, and output target image block identifications corresponding to a plurality of image blocks based on the connection matrix and the object feature vector; wherein the connection matrix represents interaction information between the object and the image block and / or interaction information between the image blocks in the to-be-generated image; The image to be generated output module is used to input each of the target image block identifiers into a pre-trained image generation model to obtain an output image to be generated.

11. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the image generating method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the image generation method according to any one of claims 1 to 9 when executed.

Images