Image generation method and electronic device

Abstract

Embodiments of the present application provide an image generation method and an electronic device. The method comprises: first, acquiring a plurality of information groups, wherein each information group comprises attribute information and position information of one instance; then, on the basis of the attribute information of the instance comprised in each information group among the plurality of information groups, generating an image feature of the instance; and then, on the basis of the position information of the plurality of instances comprised in the plurality of information groups, integrating the image features of the plurality of instances to generate a target image. In this way, the problems of attribute confusion and instance loss in generated target images can be solved.

Description

Image generation method and electronic device Technical Field

[0001] The embodiments of the present application relate to the field of image processing, and in particular to an image generation method and electronic device. Background Art

[0002] In recent years, technologies for generating high-quality images from text have made significant progress, with a key technology being layout-to-image generation (L2I). The L2I task requires users to provide not only a textual description of the generated image but also the location and layout of each object (or object / instance) in the image. The L2I algorithm then generates an image that matches the object layout based on the user-provided information.

[0003] However, when the number, size, type, color, material, style, and other attributes of instances are relatively large, existing L2I-based image generation solutions may generate images with attribute confusion and instance loss. For example, if the user inputs the text "a blue cat and a green dog," existing technologies may generate an image of "two blue cats," "a blue cat and a blue dog," or "a blue cat and a black and white dog," and so on. Summary of the Invention

[0004] In view of this, the present application provides an image generation method and electronic device. The image generation method can solve the problems of attribute confusion and instance loss in the generated target image.

[0005] In a first aspect, the present application provides an image generation method, which includes: first, obtaining multiple information groups; wherein each information group includes attribute information and position information of an instance; then, generating image features of an instance based on the attribute information of an instance contained in each information group in the multiple information groups; then, integrating the image features of the multiple instances based on the position information of the multiple instances contained in the multiple information groups to generate a target image.

[0006] In other words, the present application adopts a "divide-rule-combine" approach to achieve image generation. Among them, "divide" can refer to "disassembling and dividing single-instance tasks", that is, disassembling the description information of each instance (including the attribute information and location information of the instance), that is, disassembling multiple information groups, and a single information group can correspond to a single-instance task. "Control" can refer to "generating and processing each single-instance task one by one", that is, executing the single-instance task multiple times, and each execution of the single-instance task can generate the image features of an instance. It should be noted that the present application can execute multiple single-instance tasks in parallel. "Combine" can refer to "integrating the generation results of each instance", that is, integrating the image features of multiple instances to generate a target image. In this way, complex multiple instances can be decomposed into multiple simpler single-instance generation tasks; and in the process of executing the single-instance generation task, since each information group only includes the attribute information of the single instance itself, there is no problem of attribute leakage; and thus the present application can ensure that each instance can be generated according to the attributes specified by the user; thereby, in the process of generating the target image, precise control of the attributes of each instance in the target image is achieved.

[0007] In addition, the position information of multiple instances is utilized in the global integration process. Therefore, the present application can also achieve precise control of the position of each instance in the target image.

[0008] It should be noted that the multiple information groups can be input by the user or obtained by parsing the information input by the user; this embodiment of the application does not limit this. When the multiple information groups are input by the user, "division" means that the user can disassemble the multiple information groups.

[0009] Illustratively, an instance can be any object; for example, a person, an animal, a plant, a hat, a stool, a head, limbs, a tail, an eye, etc.; this application does not impose any limitation on this.

[0010] Exemplarily, the attribute information of an instance may include information for describing the attributes of the instance, wherein the instance attributes may include but are not limited to: size, type (such as name), color, material, style, etc., which is not limited in this application.

[0011] Exemplarily, the number of information groups can be represented by n (n is a positive integer); that is, the present application can generate image features of n instances: image features of instance 1, image features of instance 2, ..., image features of instance n.

[0012] Exemplarily, a neural network may be used to process the attribute information of an instance contained in an information group and output an image feature of the instance.

[0013] For example, "integration" can refer to connecting scattered elements to form a whole. In this application, "integration" can be understood as connecting the image features of multiple instances to obtain a whole feature comprising n instances. Subsequently, a target image can be generated based on the whole feature comprising n instances. For example, a neural network can be used to process the whole feature comprising n instances to obtain the target image.

[0014] Exemplarily, the target image generated by the present application includes these n instances, the positions of the n instances in the target image correspond one-to-one to the position information in the n information groups, and the attributes of the n instances in the target image correspond one-to-one to the attribute information in the n information groups.

[0015] It should be noted that when there is only one information group obtained, the image generation method of the present application can also be used to generate the target image.

[0016] It should be noted that “multiple information groups” may refer to two information groups, or more than two information groups.

[0017] According to the first aspect, multiple mask images are generated based on the position information of multiple instances; wherein the multiple mask images correspond one-to-one to multiple pixels in the target image; the size of each mask image is the same as the size of the target image, the mask image corresponding to the first pixel is the mask image corresponding to the instance to which the first pixel belongs, and the first pixel is any pixel in the target image; based on the multiple mask images, global integration information is generated; based on the position information of multiple instances contained in the multiple information groups, the image features of the multiple instances are integrated to generate the target image, including: based on the position information of the multiple instances and the global integration information, the image features of the multiple instances are integrated to generate the target image.

[0018] That is, in the global integration process, global integration information is newly introduced; thus, the information used to guide the integration of image features of multiple instances is richer, thereby enabling more accurate position control of multiple instances in the target image.

[0019] In addition, in the process of generating global integrated information based on multiple mask images, the probability of each pixel being any instance or background can be fine-tuned, that is, the position of each instance in the target image can be adjusted; thereby, the position of each instance in the target image can be more closely matched with the position information of each instance, further improving the accuracy of the positions of multiple instances in the target image.

[0020] According to the first aspect, or any implementation of the first aspect above, global description information of the target image is obtained; global image features are generated based on the global description information; and image features of multiple instances are integrated based on the position information of multiple instances contained in multiple information groups to generate the target image, including: integrating image features of multiple instances based on the position information of multiple instances and global image features to generate the target image.

[0021] The global description information of the target image may include at least one of foreground description information and background description information. The foreground description information may include at least one of attribute information of all instances contained in the target image, positional relationships between the instances, and motion relationships between the instances. In this way, a target image matching the user-specified global description information can be generated.

[0022] For example, the source of the global description information of the target image may include, but is not limited to, the user's current input, the user's historical input, or the electronic device's pre-settings, etc., and this application does not limit this. This application uses the user's current input as an example for explanation.

[0023] According to the first aspect, or any implementation method of the first aspect above, multiple mask images are generated based on the position information of multiple instances; wherein the multiple mask images correspond one-to-one to multiple pixels in the target image; the size of each mask image is the same as the size of the target image, the mask image corresponding to the first pixel is the mask image corresponding to the instance to which the first pixel belongs, and the first pixel is any pixel in the target image; based on the multiple mask images, global integration information is generated; based on the position information of the multiple instances contained in the multiple information groups and the global image features, the image features of the multiple instances are integrated to generate the target image, including: based on the position information, global integration information and global image features of the multiple instances, the image features of the multiple instances are integrated to generate the target image.

[0024] According to the first aspect, or any implementation of the first aspect above, obtaining multiple information groups includes receiving multiple information groups input by a user. In this way, the user can input the information groups without requiring the terminal device to perform single instance decomposition, thereby improving the efficiency of the terminal device in generating the target image.

[0025] According to the first aspect, or any implementation method of the first aspect above, multiple information groups are obtained, including: performing text parsing based on the global description information of the target image to obtain attribute information of multiple instances; performing instance position prediction based on the global description information of the target image to obtain position information of multiple instances.

[0026] In other words, the input interface is diverse, and users can choose to input multiple information groups or global description information according to their needs; this can better meet user needs.

[0027] According to the first aspect, or any implementation of the first aspect above, based on the attribute information of an instance contained in each information group of multiple information groups, the image features of an instance are generated, including: dividing the attribute information of multiple instances into multiple inputs to an instance processing network, and the instance processing network generates features based on the attribute information of an instance to obtain the image features of the instance; wherein the attribute information of an instance is one input of the instance processing network.

[0028] Exemplarily, the instance processing network may be implemented by a neural network, and may also be referred to as an instance processing module.

[0029] According to the first aspect, or any implementation of the first aspect above, based on the position information of multiple instances contained in multiple information groups, the image features of multiple instances are integrated to generate a target image, including: inputting the position information of multiple instances and the image features of multiple instances into a global integration network to obtain the target image.

[0030] Exemplarily, the global integration network can be implemented by a neural network, and can also be called a global integration module.

[0031] According to the first aspect, or any implementation of the first aspect above, based on the attribute information of an instance contained in each information group of multiple information groups, an image feature of an instance is generated, including: during each generation process of the diffusion network performing T1 generation processes, the first intermediate image generated last time is input as one path and the attribute information of multiple instances is input as multiple paths into the diffusion network, and the instance processing network in the diffusion network performs multiple feature generation to obtain image features of the multiple instances; wherein the first intermediate image generated last time and the attribute information of one instance are used for one feature generation, and the attribute information of one instance is input as one path; based on the position information of multiple instances contained in the multiple information groups, the image features of the multiple instances are integrated to generate a target image, including: during each generation process of the diffusion network performing T1 generation processes, the position information of the multiple instances and the image features of the multiple instances are input into a global integration network in the diffusion network to obtain a first intermediate feature; the first intermediate feature is input into other networks in the diffusion network to obtain a first intermediate image; wherein the global integration network is located after the instance processing network, and the other networks are located after the global integration network; and the target image is determined based on the first intermediate image obtained by the diffusion network performing the T1 generation process.

[0032] According to the first aspect, or any implementation of the first aspect above, a target image is determined based on a first intermediate image obtained by performing a T1-th generation process on a diffusion network, including: in each generation process of the diffusion network performing T2-th generation process, the second intermediate image generated last is input as one path and the attribute information of multiple instances is input as one path into the diffusion network, and the instance processing network performs feature generation based on the second intermediate image generated last and the attribute information of the multiple instances to obtain a second intermediate feature; the second intermediate feature is input into other networks to obtain a second intermediate image; and the second intermediate image obtained by performing the T-th generation process on the diffusion network is used as the target image; wherein T is the sum of T1 and T2.

[0033] Since the subsequent stages of the chain generation process of the diffusion network can generate the details, attributes of each object in the target image and the lighting, style, and tone of the entire image, the interaction between the objects in the final target image can be natural, without any sense of splicing or fragmentation, and with harmonious tones and lighting, while ensuring that the image content is not changed or barely changed.

[0034] According to the first aspect, or any implementation of the first aspect above, determining the target image based on the first intermediate image obtained by the diffusion network performing the T1th generation process includes: using the first intermediate image obtained by the diffusion network performing the T1th generation process as the target image.

[0035] According to the first aspect, or any implementation of the first aspect above, the attribute information includes at least one of the following: an image or text; and the location information includes at least one of the following: an image or text.

[0036] For example, the attribute information is the text "a blue cat".

[0037] For example, the attribute information is an image of a blue cat.

[0038] For example, the attribute information is a cat posture image.

[0039] For example, the global description information may be text or an image, and this application does not impose any limitation on this.

[0040] For example, the position information is the coordinates of four vertices of the minimum bounding box of the instance.

[0041] For example, the location information is a mask image generated based on the instance's outline information. For example, the pixel values ​​of pixels within the instance's outline can be set to 0, and the pixel values ​​of pixels outside the instance's outline can be set to 1, to obtain a mask image as the location information. Alternatively, the pixel values ​​of pixels within the instance's outline can be set to 1, and the pixel values ​​of pixels outside the instance's outline can be set to 0, to obtain a mask image as the location information.

[0042] According to the first aspect, or any implementation of the first aspect above, the location information includes at least one of the following: outline information of the instance and information of a bounding box of the instance.

[0043] It should be noted that the image generation method of the present application can generate multiple instances with action relationships; the image generation method of the present application can also generate instances of different styles; the image generation method of the present application can also generate instances of different textures; the image generation method of the present application can also generate instances of different shapes; the image generation method of the present application can also generate instances of different numbers; the image generation method of the present application can also generate instances of different colors.

[0044] Furthermore, testing has shown that the present invention has a higher success rate in generating images containing a specified number of targets than existing technologies. Furthermore, compared to some existing technologies, the present invention takes less time to generate target images and has a higher success rate in generating images containing a specified number of targets. Furthermore, the locations of instances in the target images generated by the present invention more closely match those specified by the user. Furthermore, the quantitative metrics of the target images generated by the present invention are superior to those of existing technologies.

[0045] In a second aspect, the present application provides an image generating device, the device comprising:

[0046] An acquisition module, configured to acquire a plurality of information groups, wherein each information group includes attribute information and location information of an instance;

[0047] A first generating module is configured to generate an image feature of an instance based on attribute information of an instance contained in each information group of the plurality of information groups;

[0048] The integration module is used to integrate the image features of the multiple instances according to the position information of the multiple instances contained in the multiple information groups to generate a target image.

[0049] According to the second aspect, the image generating apparatus further includes:

[0050] a second generation module, configured to generate a plurality of mask images based on the position information of the plurality of instances; wherein the plurality of mask images correspond one-to-one to a plurality of pixels in the target image; the size of each mask image is the same as the size of the target image; the mask image corresponding to the first pixel is the mask image corresponding to the instance to which the first pixel belongs, and the first pixel is any pixel in the target image; and generate global integration information based on the plurality of mask images;

[0051] The integration module is specifically used to integrate the image features of multiple instances according to the location information of the multiple instances and the global integration information to generate a target image.

[0052] According to the second aspect, or any implementation of the second aspect above, the acquisition module is further configured to acquire global description information of the target image;

[0053] The image generating device further includes:

[0054] The third generation module is used to generate global image features based on the global description information;

[0055] The integration module is specifically used to integrate the image features of multiple instances according to the location information of the multiple instances and the global image features to generate a target image.

[0056] According to the second aspect, or any implementation of the second aspect above, the image generating apparatus further includes:

[0057] a second generation module, configured to generate a plurality of mask images based on the position information of the plurality of instances; wherein the plurality of mask images correspond one-to-one to a plurality of pixels in the target image; the size of each mask image is the same as the size of the target image; the mask image corresponding to the first pixel is the mask image corresponding to the instance to which the first pixel belongs, and the first pixel is any pixel in the target image; and generate global integration information based on the plurality of mask images;

[0058] The integration module is specifically used to integrate the image features of multiple instances according to the position information of the multiple instances, the global integration information and the global image features to generate a target image.

[0059] According to the second aspect, or any implementation of the second aspect above, the acquisition module is specifically configured to receive multiple information groups input by a user.

[0060] According to the second aspect, or any implementation method of the above second aspect, the acquisition module is specifically used to perform text parsing based on the global description information of the target image to obtain attribute information of multiple instances; and perform instance position prediction based on the global description information of the target image to obtain position information of multiple instances.

[0061] According to the second aspect, or any implementation of the second aspect above, the first generation module is specifically used to divide the attribute information of multiple instances into multiple inputs to the instance processing network, and the instance processing network generates features based on the attribute information of one instance to obtain the image features of one instance; wherein the attribute information of one instance is one input of the instance processing network.

[0062] According to the second aspect, or any implementation of the second aspect above, the integration module is specifically configured to input location information of multiple instances and image features of multiple instances into a global integration network to obtain a target image.

[0063] According to the second aspect, or any implementation of the second aspect above,

[0064] The first generation module is specifically configured to, during each generation process of the diffusion network performing T1 generation processes, input the previously generated first intermediate image as one path and attribute information of multiple instances as multiple paths into the diffusion network, so that the instance processing network in the diffusion network performs multiple feature generation to obtain image features of the multiple instances; wherein the previously generated first intermediate image and attribute information of one instance are used for one feature generation, and the attribute information of one instance is input as one path;

[0065] The integration module is specifically configured to input the position information and image features of multiple instances into a global integration network in the diffusion network during each generation process of the diffusion network performing T1 generation processes to obtain a first intermediate feature; input the first intermediate feature into other networks in the diffusion network to obtain a first intermediate image; wherein the global integration network is located after the instance processing network, and the other networks are located after the global integration network; and determine the target image based on the first intermediate image obtained by the diffusion network performing the T1 generation process.

[0066] According to the second aspect, or any implementation of the second aspect above, the integration module is specifically configured to, during each generation process of the diffusion network performing T2 generation processes thereafter, input the last generated second intermediate image as one path and the attribute information of multiple instances as one path into the diffusion network, and have the instance processing network perform feature generation based on the last generated second intermediate image and the attribute information of the multiple instances to obtain second intermediate features; input the second intermediate features into other networks to obtain a second intermediate image; and use the second intermediate image obtained by the diffusion network performing the T-th generation process as the target image; wherein T is the sum of T1 and T2.

[0067] According to the second aspect, or any implementation of the second aspect above, the integration module is specifically configured to use the first intermediate image obtained by the diffusion network performing the T1th generation process as the target image.

[0068] According to the second aspect, or any implementation of the second aspect above, the attribute information includes at least one of the following: an image or text; and the location information includes at least one of the following: an image or text.

[0069] According to the second aspect, or any implementation of the second aspect above, the location information includes at least one of the following: outline information of the instance and information of a bounding box of the instance.

[0070] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0071] In a third aspect, the present application provides an electronic device comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, enables the electronic device to execute the method in the first aspect or any possible implementation of the first aspect.

[0072] Exemplarily, the electronic device may be a terminal device or a server, and the embodiments of the present application do not limit this.

[0073] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0074] In a fourth aspect, the present application provides a chip comprising one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the method in the first aspect or any possible implementation of the first aspect are executed.

[0075] The fourth aspect and any implementation of the fourth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0076] In a fifth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, it enables the computer or the processor to execute the method in the first aspect or any possible implementation of the first aspect.

[0077] The fifth aspect and any implementation of the fifth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0078] In a sixth aspect, the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer or the processor executes the method in the first aspect or any possible implementation of the first aspect.

[0079] The sixth aspect and any implementation of the sixth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the sixth aspect and any implementation of the sixth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] FIG1 is a schematic diagram showing an exemplary framework of an image generation system 100;

[0081] FIG2A is a schematic diagram illustrating an exemplary image generation process;

[0082] FIG2B is a schematic diagram illustrating an exemplary image generation process;

[0083] FIG3A is a schematic diagram illustrating an exemplary mobile phone interface;

[0084] FIG3B is a schematic diagram illustrating location information of an example;

[0085] FIG3C is a schematic diagram illustrating an exemplary mobile phone interface;

[0086] FIG3D is a schematic diagram of an exemplary mobile phone interface;

[0087] FIG4A is a schematic structural diagram of an exemplary diffusion network;

[0088] FIG4B is a schematic diagram of an exemplary training process of a diffusion network;

[0089] FIG5A is a schematic diagram illustrating an exemplary image generation process 500;

[0090] FIG5B is a schematic diagram illustrating the processing process of the instance processing module and the global integration module;

[0091] FIG5C is a schematic diagram illustrating the processing process of the instance processing module and the global integration module;

[0092] FIG6A is a schematic diagram illustrating an exemplary image generation process 600;

[0093] FIG6B is a schematic diagram illustrating the processing process of the instance processing module and the global integration module;

[0094] FIG6C is a schematic diagram illustrating the processing process of the instance processing module and the global integration module;

[0095] FIG7 is a schematic diagram illustrating an exemplary image generation process 700;

[0096] FIG8A is a schematic diagram illustrating an exemplary image generation process 800;

[0097] FIG8B is a schematic diagram illustrating an exemplary mask image generation process;

[0098] FIG8C is a schematic diagram illustrating an exemplary process of generating global integrated information;

[0099] FIG8D is a schematic diagram illustrating an exemplary image generation process;

[0100] FIG9A is a schematic diagram illustrating an exemplary target image;

[0101] FIG9B is a schematic diagram illustrating an exemplary target image;

[0102] FIG9C is a schematic diagram illustrating an exemplary target image;

[0103] FIG9D is a schematic diagram illustrating an exemplary target image;

[0104] FIG10A is a schematic diagram illustrating an exemplary target image;

[0105] FIG10B is a schematic diagram illustrating an exemplary target image;

[0106] FIG10C is a schematic diagram illustrating an exemplary target image;

[0107] FIG10D is a schematic diagram illustrating an exemplary target image;

[0108] FIG10E is a schematic diagram illustrating an exemplary target image;

[0109] FIG10F is a schematic diagram illustrating an exemplary target image;

[0110] FIG11 is a schematic diagram illustrating an exemplary image generating apparatus 1100;

[0111] FIG12 is a schematic structural diagram of an exemplary device. DETAILED DESCRIPTION

[0112] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0113] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0114] In the description and claims of the embodiments of this application, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order of objects. For example, the terms "first target object" and "second target object" are used to distinguish different objects, rather than to describe a specific order of objects.

[0115] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0116] In the description of the embodiments of this application, unless otherwise specified, "multiple" means two or more. For example, "multiple processing units" means two or more processing units; "multiple systems" means two or more systems.

[0117] FIG1 is a schematic diagram of an exemplary framework of an image generation system 100. The image generation system 100 shown in FIG1 includes a terminal device and a server 110.

[0118] Exemplarily, the terminal device includes but is not limited to: a mobile phone 121, a laptop (personal computer) 122, a tablet computer 123, and a wearable device, etc., which is not limited in this application.

[0119] Exemplarily, the specific implementation form of the server 110 can be a cloud server, a physical (independent) server, a station cluster server, etc., and this application does not impose any restrictions on this.

[0120] Exemplarily, a user can use a terminal device to generate an image (hereinafter referred to as a target image) required by the user in an image generation application (Application, APP) / tool / webpage. Exemplarily, a user can enter description information of the instance and / or global description information of the target image in the image generation APP / tool / webpage; thereafter, the image generation operation can be performed. The terminal device can respond to the user's operation behavior and send the description information of the instance and / or the global description information of the target image to the server 110 via the network. Subsequently, the server 110 can generate a target image based on the description information of the instance and / or the global description information of the target image; then, the target image is sent to the terminal device via the network. The terminal device can display the target image in the interface of the image generation APP / tool / webpage.

[0121] Exemplarily, the description information of an instance may include attribute information and location information of the instance. The global description information of the target image may include at least one of foreground description information and background description information. The foreground description information may include at least one of attribute information of all instances contained in the target image, location relationships between instances, and action relationships between instances.

[0122] 1 , exemplarily, a user uses the image generation APP in the mobile phone 121 to generate a target image. Exemplarily, the user can input the attribute information of instance 1 (i.e., an image of a cat) and the location information of instance 1 (i.e., location 1); as well as the attribute information of instance 2 (i.e., the text "a pumpkin carriage") and the location information of instance 2 (i.e., location 2). Afterwards, the user can perform an image generation operation; the mobile phone 121 can respond to the user's operation and send to the server 110 via the network: an image of a cat and location 1, as well as the text "a pumpkin carriage" and location 2. Subsequently, the server 110 can generate a target image 11 containing a cat and a pumpkin carriage based on the information sent by the mobile phone 121; and then, send the target image 11 to the mobile phone 121 via the network. The image generation APP in the mobile phone 121 can display the target image 11.

[0123] Referring to Figure 1 , a user generates a target image using an image generation webpage on personal computer 122. For example, the user can enter a global description of the target image (i.e., the text "4 boats and a bench"). The user can then perform an image generation operation. In response to the user's operation, personal computer 122 can send the text "4 boats and a bench" to server 110 via the network. Subsequently, server 110 can generate target image 12 containing four boats and a bench based on the information sent by personal computer 122. The target image 12 can then be displayed on the image generation webpage on personal computer 122.

[0124] Referring to FIG. 1 , illustratively, a user generates a target image using an image generation webpage on tablet computer 123. For example, the user can enter global descriptive information for the target image (i.e., the text "a red rabbit wearing a white wig"), attribute information for instance 1 (i.e., the text "a red rabbit") and location information for instance 1 (i.e., location 3), and attribute information for instance 2 (i.e., the text "a white wig") and location information for instance 2 (i.e., location 4). The user can then perform an image generation operation; tablet computer 123 can respond to the user's operation by sending the text "a red rabbit wearing a white wig," the text "a red rabbit" and location 3, and the text "a white wig" and location 4 to server 110 via the network. Subsequently, server 110 can generate target image 13 containing a red rabbit wearing a white wig based on the information sent by tablet computer 123; and then send target image 13 to tablet computer 123 via the network. The image generation webpage on tablet computer 123 can display target image 13.

[0125] It should be understood that the user can also use other terminal devices to generate the target image by interacting with the server, and this application does not limit this.

[0126] It should be noted that FIG1 shows that the image generation method involved in this application is executed by the server 110; it should be understood that the image generation method involved in this application can also be performed locally by the terminal device, that is, the image generation method involved in this application can be executed by the terminal device. In addition, the image generation method involved in this application can also be executed collaboratively by the terminal device and the server. For example, the terminal device pre-processes the description information of the instance input by the user and / or the global description information of the target image before sending it to the server; thereafter, the server generates the target image based on the pre-processed information and returns the generated target image to the terminal device for display. In other words, this application does not limit the electronic device that executes the image generation method involved in this application.

[0127] The image generation process of this application is described below.

[0128] FIG. 2A is a schematic diagram illustrating an exemplary image generation process.

[0129] S201, obtaining multiple information groups; wherein each information group includes attribute information and location information of an instance.

[0130] In one possible approach, users can independently input description information for each instance in the target image in the image generation app. In this approach, the instance description information can include the instance's attribute information and the instance's location information. For example, the attribute information and location information of an instance can form an information group; that is, the user can independently input multiple information groups, or the user can input the description information of any instance separately from the description information of other instances.

[0131] Illustratively, an instance can be any object; for example, a person, an animal, a plant, a hat, a stool, a head, limbs, a tail, an eye, etc.; this application does not impose any limitation on this.

[0132] For example, the instance attribute information may include information used to describe the instance attributes, where the instance attributes may include, but are not limited to, size, type (such as name), color, material, style, etc., which is not limited in this application. For example, the instance attribute information may be text or an image, etc., which is not limited in this application.

[0133] Illustratively, the location information of the instance may include, but is not limited to, the instance's bounding box information, the instance's outline information, etc., which is not limited in this application.

[0134] Illustratively, the location information of the instance may be text (eg, coordinates) or an image, and this application does not impose any limitation on this.

[0135] FIG3A is a schematic diagram illustrating an exemplary mobile phone interface.

[0136] 3A , exemplarily, the image generation main interface 301 may include but is not limited to: property editing box 11 , position editing box 12 , property editing box 21 , position editing box 22 , image option 13 , image option 14 , image option 23 , image option 24 and confirmation option 302 , etc., and this application does not impose any restrictions on this.

[0137] Method (1):

[0138] 3A , illustratively, the user can enter the attribute information of instance 1 in the attribute edit box 11 and the location information of instance 1 in the location edit box 12. For example, the user can enter “a blue cat” in the attribute edit box 11 and the location information “(x11, y11), (x12, y12), (x13, y13), (x14, y14)” in the location edit box 12. Among them, (x11, y11), (x12, y12), (x13, y13), (x14, y14) are the coordinates of the four vertices (A1, A2, A3, and A4) of the minimum bounding box of the cat, as shown in FIG3B (1).

[0139] 3A , illustratively, the user can enter attribute information of instance 2 in attribute edit box 21 and position information of instance 2 in position edit box 22. For example, the user can enter "a green dog" in attribute edit box 21 and position information "(x21, y21), (x22, y22), (x23, y23), (x24, y24)" in position edit box 22.

[0140] Method (2):

[0141] Continuing with FIG3A , for example, a user can click on the image option 13 , and the phone can respond to the user's operation by displaying the album main interface; the user can select an image of instance 1 (e.g., an image of a cat) and can enter the location information of instance 1 in the location edit box 12 , such as "(x11, y11), (x12, y12), (x13, y13), (x14, y14)". In this case, the user can enter the attribute information of instance 1 in the attribute edit box 11 , or not, and this application does not limit this.

[0142] Continuing with FIG3A , for example, a user can click on the image option 23 . The phone can respond to the user's operation by displaying the album main interface. The user can then select an image of instance 2 (e.g., an image of a dog). Furthermore, the user can enter the location information of instance 2, "(x21, y21), (x22, y22), (x23, y23), (x24, y24)", in the location edit box 22 . In this case, the user can enter the attribute information of instance 2 in the attribute edit box 21 or not. This application does not limit this.

[0143] Method (3):

[0144] Continuing to refer to Figure 3A, illustratively, the user can enter the attribute information of instance 1 (such as "a blue cat") in the attribute editing box 11. And the image option 14 can be clicked, and the mobile phone can display the album main interface in response to the user operation; the user can select a mask image of instance 1 (which can be used to describe the contour information of instance 1); refer to Figure 3B (2). Figure 3B (2) shows a mask image of a cat; in the mask image of the cat, the pixel value of the pixel point within the cat's contour box is 0, and the pixel value of the pixel point outside the cat's contour box is 1. It should be understood that the pixel value of the pixel point within the contour box of the instance in the mask image can be 1, and the pixel value of the pixel point outside the contour box of the instance in the mask image can be 0; the embodiment of the present application does not limit this. In this case, the user can enter the location information of instance 1 in the location editing box 12, or not enter the location information of instance 1, and the present application does not limit this.

[0145] Continuing with FIG3A , for example, a user can enter attribute information for Instance 2 (e.g., "a green dog") in Attribute Edit Box 21. The user can then click Image Option 24. In response to the user's operation, the phone can display the main photo album interface. The user can then select a mask image for Instance 2 (which can be used to describe the location information of the outline of Instance 2), for example, a mask image of a dog. In this case, the user can enter the location information for Instance 2 in Location Edit Box 22, or not. This application does not limit this.

[0146] Method (4):

[0147] Continuing with FIG3A , for example, a user can click on image option 13, and the phone can respond to the user's operation by displaying the album main interface; the user can select an image from Example 1 (e.g., an image of a cat). Furthermore, by clicking on image option 14, the phone can respond to the user's operation by displaying the album main interface; the user can select a mask image from Example 1 (e.g., a mask image of a cat).

[0148] Continuing with FIG3A , for example, a user can click on image option 23, and the phone can respond to the user's operation by displaying the album main interface; the user can select an image from Example 2 (e.g., an image of a dog). Furthermore, by clicking on image option 24, the phone can respond to the user's operation by displaying the album main interface; the user can select a mask image from Example 2 (e.g., a mask image of a dog).

[0149] It should be noted that after the user clicks on the image option 13 or the image option 23, the mobile phone can respond to the user operation and display the album main interface, and the user can select a gesture image of an example.

[0150] It should be noted that when the user needs to generate a target image containing a green cat and a blue cat, the user also needs to input the attribute information and location information of two instances. Among them, the attribute information of instance 1 is "a green cat", and the location information of instance 1 is "(x31, y31), (x32, y32), (x33, y33), (x34, y34)". The attribute information of instance 2 is "a blue cat", and the location information of instance 2 is "(x41, y41), (x42, y42), (x43, y43), (x44, y44).

[0151] It should also be noted that when the user needs to generate a target image containing a cat with one blue eye and one green eye, the user needs to input the attribute information and position information of three instances. Among them, the attribute information of instance 1 is "a blue eye", and the position information of instance 1 is "(x51, y51), (x52, y52), (x53, y53), (x54, y54)". The attribute information of instance 2 is "a green eye", and the position information of instance 2 is "(x61, y41), (x62, y62), (x63, y63), (x64, y64). The attribute information of instance 3 is "a cat", and the position information of instance 3 is "(x71, y71), (x72, y72), (x73, y73), (x74, y74).

[0152] It should also be noted that when the user needs to generate a target image containing a red rabbit wearing a white hat, the user needs to input the attribute information and location information of two instances. Among them, the attribute information of instance 1 is "a white hat", and the location information of instance 1 is "(x81, y81), (x82, y82), (x83, y83), (x84, y84)". The attribute information of instance 2 is "a red rabbit", and the location information of instance 2 is "(x91, y41), (x92, y92), (x93, y93), (x94, y94).

[0153] After the user completes inputting the attribute information and location information of all instances, the user may click on the confirmation option 302 ; the mobile phone may respond to the user's operation information, obtain multiple information groups, and then execute S202 to S203 .

[0154] In one possible approach, the user can enter the global description information of the target image in the image generation APP. For example, the global description information of the target image can be text or an image, and this application does not limit this.

[0155] Method (1):

[0156] Exemplarily, the global description information input by the user is text, the global description information includes foreground description information, and the foreground description information includes attribute information of multiple instances.

[0157] 3C (1), the image generation main interface 301 may include, but is not limited to, an edit box 303, image options 304, and a confirmation option 302, etc., which are not limited in this application. For example, the user may enter "a blue dog and a green dog" in the edit box 303.

[0158] Method (2): The global description information input by the user is text, and the global description information includes foreground description information, and the foreground description information includes attribute information of multiple instances and the positional relationship of each instance.

[0159] 3C (2), the main image generation interface 301 may include, but is not limited to, an edit box 303, image options 304, and a confirmation option 302, etc., which are not limited in this application. For example, the user may enter "a blue dog to the left of a green dog" in the edit box 303.

[0160] Method (3): The global description information input by the user is text, and the global description information includes foreground description information, and the foreground description information includes attribute information of multiple instances and the action relationship between each instance.

[0161] 3C (3), the image generation main interface 301 may include, but is not limited to, an edit box 303, image options 304, and a confirmation option 302, etc., which are not limited in this application. For example, the user may enter "a player playing football" in the edit box 303.

[0162] Method (4): The global description information input by the user is text, and the global description information includes foreground description information and background description information. The foreground description information includes attribute information of multiple instances and the positional relationship of each instance.

[0163] 3C (4), the main image generation interface 301 may include, for example, but is not limited to, an edit box 303, image options 304, and a confirmation option 302, etc., which are not limited in this application. For example, the user may enter "a blue dog on the street to the left of a green dog" in the edit box 303.

[0164] It should be understood that other global description information may also be input, and this embodiment of the present application does not limit this.

[0165] After the user completes inputting the global description information for the target image, they can click on the confirmation option 302. In response to the user's operation information, the mobile phone can execute the image generation method involved in this application, namely, execute S201 to S203. One way to implement S201 can be to parse the global description information to extract the attribute information of each instance; and predict the instance location based on the global description information to obtain the location information of each instance; then, the attribute information and location information of an instance can be used to form an information group; in this way, the terminal device can obtain multiple information groups.

[0166] Method (5): The global description information input by the user is an image.

[0167] 3C(1) to 3C(4), exemplarily, the user can click on the image option 304, and the mobile phone can display the album main interface in response to the user operation; the user can select an image (the user expects the background of the target image to be similar to the background of the image, and the user expects the foreground of the target image to be similar to the foreground of the image). Afterwards, the user can click on the confirmation option 302. The mobile phone can execute the image generation method involved in this application in response to the user's operation information, that is, execute S201 to S203. One way to implement S201 can be to detect the image and extract background description information, attribute information and location information of each instance, and description text such as the location relationship and action relationship between each instance. Afterwards, the attribute information and location information of an instance can be used to form an information group; in this way, the terminal device can obtain multiple information groups.

[0168] In one possible approach, a user may input global description information of a target image and independently input description information of each instance in the target image in an image generation APP.

[0169] 3D , exemplarily, the main image generation interface 301 may include but is not limited to: property editing box 11, position editing box 12, property editing box 21, position editing box 22, image option 13, image option 14, image option 23, image option 24, global description information editing box 31, image option 32 and confirmation option 302, etc., and this application does not impose any restrictions on this.

[0170] For example, enter "a blue cat" in the attribute edit box 11, and enter the location information "(x11, y11), (x12, y12), (x13, y13), (x14, y14)" in the location edit box 12. Enter "a green dog" in the attribute edit box 21, and enter the location information "(x21, y21), (x22, y22), (x23, y23), (x24, y24)" in the location edit box 22. Enter "a blue cat on the street to the left of a green dog" in the global description information edit box 31.

[0171] It should be understood that the form of the attribute information and location information of each instance input by the user in FIG3D , as well as the form of the global description information, can refer to the above description of FIG3A and FIG3C , and will not be repeated here.

[0172] It should be understood that the present application does not limit the form of the image generation main interface, nor does it limit the form of user input information.

[0173] For example, the number of information groups can be represented by n (n is an integer greater than 1), that is, information group 1, information group 2, ..., information group n can be obtained. Information group 1 includes the attribute information and location information of instance 1, information group 2 includes the attribute information and location information of instance 2, ..., and information group n includes the attribute information and location information of instance n.

[0174] S202: Generate an image feature of an instance according to the attribute information of an instance contained in each information group of the multiple information groups.

[0175] For example, the present application can use the attribute information of an instance contained in an information group to generate an image feature of the instance. In this way, based on the description information of multiple information groups, image features of multiple instances can be generated. In other words, S202 can generate image features of n instances: image features of instance 1, image features of instance 2, ..., image features of instance n.

[0176] Exemplarily, a neural network may be used to process the attribute information of an instance contained in an information group and output an image feature of the instance.

[0177] S203 , integrating image features of the multiple instances according to the position information of the multiple instances included in the multiple information groups to generate a target image.

[0178] Illustratively, after obtaining the image features of each of the multiple instances, the image features of the multiple instances may be integrated to obtain the image features of the target image; thereafter, the image features of the target image may be processed to generate the target image.

[0179] For example, the image features of the multiple instances may be integrated according to their position information, so that each instance in the generated target image is at a position corresponding to its position information.

[0180] Exemplarily, a neural network may be used to process the position information of n instances and the image features of n instances to obtain a target image.

[0181] Exemplarily, the target image generated in S203 includes the n instances, and the positions of the n instances correspond one-to-one to the position information in the n information groups, and the attributes of the n instances in the target image correspond one-to-one to the attribute information in the n information groups.

[0182] FIG2B is a schematic diagram illustrating an exemplary image generation process.

[0183] Referring to Figure 2B , for example, a user inputs the global description of a target image, "a blue cat to the left of a green dog." Based on this global description, the positional layout and attribute information of instance 1 (cat) and instance 2 (dog) can be determined. Instance segmentation can then be performed to obtain the cat's attribute information, "a blue cat," and the dog's attribute information, "a green dog."

[0184] 2B , the cat's attribute information "a blue cat" may be processed to output a cat's image feature (i.e., Blue Cat's Feature). Also, the dog's attribute information "a green dog" may be processed to output a dog's image feature (i.e., Green Dog's Feature).

[0185] 2B , for example, the image features of the cat and the dog can be integrated based on their positional layout to obtain the image features of the target image. The image features of the target image can then be processed to obtain the target image. For example, the target image can be Image 1 or Image 2 in FIG. 2B .

[0186] In summary, the present application adopts a "divide-rule-combine" scheme to achieve image generation. Among them, "divide" can correspond to S201, and can refer to "disassembling and dividing single instance tasks", that is, disassembling the description information of each instance, that is, disassembling multiple information groups, and a single information group can correspond to a single instance task. It should be noted that when multiple information groups are input by the user, the disassembly of multiple information groups is implemented by the user. "Control" can correspond to S202, and can refer to "generating and processing each single instance task one by one", that is, executing the single instance task multiple times, and each execution of the single instance task can generate the image features of an instance. It should be noted that S202 can execute multiple single instance tasks in parallel. "Combine" can correspond to S203, and can refer to "integrating the generation results of each instance", that is, integrating the image features of multiple instances to generate the target image. In this way, complex multiple instances can be decomposed into multiple simpler single-instance generation tasks; and in the process of executing single-instance generation tasks, since each information group only includes the attribute information of the single instance itself, there is no problem of attribute leakage; and thus the present application can ensure that each instance can be generated according to the attributes specified by the user; thereby, in the process of generating the target image, precise control of the attributes of each instance in the target image can be achieved.

[0187] In addition, the position information of multiple instances is utilized in the global integration process. Therefore, the present application can also achieve precise control of the position of each instance in the target image.

[0188] It should be noted that when there is only one information group obtained, the image generation method of the present application can also be used to generate the target image.

[0189] For example, the present application can use a diffusion network to implement S202 and S203. Then, the structure of the diffusion network and the training process of the diffusion network can be described first. The following description is based on the example of the diffusion network implemented by U-NET.

[0190] FIG4A is a schematic structural diagram of an exemplary diffusion network.

[0191] Referring to FIG4A(1), for example, a U-NET (U-shaped structure) may include multiple blocks. The left and right halves of the U-NET may each include g (g is a positive integer) blocks. Each block may include two parts: a first part and a second part (as shown by the two white rectangles in each block in FIG4A(1)). The first and second parts may each include one or more network layers (such as convolutional layers, activation layers, etc.).

[0192] For example, the input Y of U-NET is a noisy image and a time node, and the output X can be the noise in the predicted noisy image.

[0193] In one possible approach, an instance processing module and a global integration module can be inserted into each block in the U-NET, as shown by the black rectangles in Block_1 to Block_g in Figure 4A(1). In this case, the diffusion network can include the U-NET, the instance processing module, and the global integration module.

[0194] Exemplarily, the example processing module may include one or more cross-attention modules. Optionally, the example processing module may further include an enhanced attention module. The example processing module may be implemented using a neural network, and thus the example processing module may also be referred to as an example processing network.

[0195] Exemplarily, the global integration module can adopt a neural network instance, so the global integration module can also be called a global integration network. For example, the global integration module can include two self-attention modules and a softmax function.

[0196] Referring to FIG4A(2), illustratively, the instance processing module may include m+1 inputs, where m is an integer greater than or equal to n. Input R0 may be the output of the first part of the block where the instance processing module is located (also referred to as the output of the previous network layer of the instance processing module). Any of inputs R1 to Rm may be used to input attribute information of an instance included in an information group.

[0197] Exemplarily, the instance processing module may include m outputs, any of which, from output U1 to output Um, can be used to output the image features of an instance. Output U1 is the output resulting from the instance processing module processing inputs R0 and R1; output U2 is the output resulting from the instance processing module processing inputs R0 and R2; and output Um is the output resulting from the instance processing module processing inputs R0 and Rm. It should be noted that the instance processing module can perform m independent processing steps simultaneously.

[0198] Exemplarily, the global integration module may include m+1 inputs. Among them, the m outputs of the instance processing module are the m inputs of the global integration module: the output U1 of the instance processing module serves as the input U1 of the global integration module; the output U2 of the instance processing module serves as the input U2 of the global integration module; ...; the output Um of the instance processing module serves as the input Um of the global integration module. In addition, the location information of all instances can be used as the input E0 of the global integration module. The global integration module may include one output, which can serve as the input of the second part of the Block where the global integration module is located (it can also be called the input of the next network layer of the global integration module). It should be understood that the processing performed by the global integration module is to integrate multiple inputs into one output.

[0199] In one possible approach, an instance processing module, a global integration information generation module, and a global integration module can be inserted into each block in the U-NET, as shown by the black rectangles in Block_1 to Block_g in Figure 4A(1). In this case, the diffusion network can include the U-NET, the instance processing module, the global integration information generation module, and the global integration module.

[0200] Exemplarily, the global integrated information generation module may include one or more self-attention modules. The global integrated information generation module may be implemented using a neural network, and thus the global integrated information generation module may be referred to as a global integrated information generation network.

[0201] Exemplarily, the instance processing module in FIG4A(3) is similar to the instance processing module in FIG4A(2), and will not be described again here.

[0202] Referring to Figure 4A (3), the global integration information generation module may include two inputs: input R0 and input H0. Among them, input R0 can be the output of the first part of the Block where the global integration information generation module is located (it can also be called the output of the previous network layer of the global integration information generation module). Input H0 can be a mask image corresponding to each pixel in the target image; wherein, the mask image corresponding to each pixel in the target image is generated based on the position information of all instances in the target image, which will be explained in detail later. The global integration information generation module includes one output, namely output D0 (D0 is the integration information).

[0203] For example, the global integration module can include m+2 inputs. The m outputs of the instance processing module serve as the m inputs of the global integration module; the single output D0 of the global integration information generation module serves as the single input E0 of the global integration module; and the location information of all instances serves as the input E0 of the global integration module. The global integration module can include one output, which serves as the input of the second part of the block in which the global integration module resides.

[0204] FIG4B is a schematic diagram illustrating an exemplary training process of a diffusion network.

[0205] 4B , illustratively, the training process of the diffusion network may include a forward process (or referred to as a diffusion process, as shown in FIG4B ( 1 )) and a reverse process (or referred to as a generation process, as shown in FIG4B ( 2 )).

[0206] For example, the training process of the diffusion network may be shown as follows S41 to S46:

[0207] S41, obtaining a training image and global description information of the training image.

[0208] Exemplarily, a plurality of image-text data pairs may be acquired, and each image-text data pair may include a training image and global description information of the training image.

[0209] S42: Parse multiple information groups from the global description information of the training image.

[0210] For example, each training image may include one or more instances. Attribute information for each instance contained in the training image can be extracted from the global description of the training image. Furthermore, location information (which can be called a bounding box) for each instance contained in the training image can be detected from the training image. The attribute information and location information for each instance contained in the training image can form an information group. Thus, multiple information groups can be determined for each training image.

[0211] S43: Sample a noise adding step number t from time node 0 to time node T, gradually add noise to the training image, and obtain a noisy image / noisy latent space feature.

[0212] 4B(1), exemplarily, Z0 is a training image, the corresponding time node is 0, and Z0 can be referred to as the state when the time node is 0.

[0213] For example, random Gaussian noise can be sampled between time nodes 0 and 1 to obtain a preset noise ε~N(0,1) (N(0,1) is a Gaussian distribution, and ε~N(0,1) indicates that the preset noise conforms to the Gaussian distribution). Then, based on the preset noise, Z0 is noised to obtain the state Z1 when time node is 1. The specific noise addition method can be referred to the description in the prior art and will not be repeated here.

[0214] Exemplarily, random Gaussian noise can be sampled between time node 1 and time node 2 to obtain a preset noise ε~N(0,1); then, based on the preset noise, Z1 is noised to obtain the state Z2 when the time node is 2; wherein, the specific noise adding method can refer to the description in the prior art and will not be repeated here.

[0215] Similarly, add noise t times and get the state Z at time node t t Among them, Z t It can be a noisy image / noisy latent space feature.

[0216] In step S44, the noisy image / noisy latent space features, the global description information of the training image, multiple information groups, and the number of noise addition steps are input into the diffusion network. The diffusion network performs t generation processes to obtain the predicted noise ε' of the noisy image / latent space features at time node 0.

[0217] For example, between time node t and time node t-1, the noisy image / noisy latent space features at time node t, the global description information of the training image, multiple information groups and t can be input into the diffusion network to obtain the state X at time node t-1. t-1 , and X t-1 The prediction noise in .

[0218] The attribute information in each of the n information groups serves as one input to the instance processing module. That is, the attribute information in the n information groups corresponds to inputs R1 to Rn of the instance processing module. The global description information of the training image serves as input R(n+1) of the instance processing module.

[0219] For example, between time node t-1 and time node t-2, the state X at time node t-1 can be t-1, the global description information of the training image, multiple information groups and t-1 are input to the diffusion network to obtain the state X at time node t-2 t-2 , and X t-2 The prediction noise in .

[0220] Similarly, after the diffusion network performs t generation processes, the predicted noise ε' of the noisy image / latent space feature at time node 0 can be obtained.

[0221] S45, calculating a loss value based on the predicted noise and the actual noise.

[0222] For example, the L1 distance and / or L2 distance between the predicted noise ε' of the noisy image / latent space feature at time node 0 and the actual noise sampled between time node 1 and time node 0 are calculated. The loss value is determined based on the L1 distance and / or L2 distance.

[0223] S46, performs back propagation according to the loss value and adjusts the network parameters of the diffusion network.

[0224] Similarly, steps S42 to S46 are executed in a loop until a preset condition is satisfied. The preset condition can be set as required, for example, the number of training times reaches a threshold; or the loss value is less than or equal to a loss threshold; etc. This application does not impose any restrictions on this.

[0225] Afterwards, the trained diffusion network can be used to achieve image generation; please refer to the following description for details.

[0226] FIG5A is a schematic diagram illustrating an exemplary image generation process 500. In the embodiment of FIG5A , the diffusion network employed includes a U-NET, an instance processing module, and a global integration module. The image generation main interface corresponding to the image generation process 500 may be as shown in FIG3A .

[0227] S501: Receive multiple information groups input by a user.

[0228] Exemplarily, the number of information groups can be represented by n.

[0229] For example, n=2, information group 1 includes: "a blue cat" and "(x11, y11), (x12, y12), (x13, y13), (x14, y14)"; information group 2 includes: "a green dog" and "(x21, y21), (x22, y22), (x23, y23), (x24, y24)".

[0230] S502: Input multiple information groups into the diffusion network, and the diffusion network performs T1 generation processing to obtain a first intermediate image.

[0231] Exemplarily, the diffusion network in S502 includes a U-NET, an instance processing module, and a global integration module.

[0232] For example, assume that the total number of times the diffusion network performs the generation process is T (T is a positive integer); that is, from time node T to time node 0, the diffusion network performs the generation process T times. Between time node T and time node T-1, the diffusion network performs the first generation process; between time node T-1 and time node T-2, the diffusion network performs the second generation process; and so on.

[0233] Exemplarily, T1 is a positive integer less than or equal to T. When T1=T, S503 does not need to be executed; when T1 is less than T, S503 can be executed. This application takes T1 less than T as an example for description.

[0234] The following uses the example of the instance processing module and global integration module embedded in Block_1 of U-NET during the i-th generation process of the diffusion network as an example to illustrate the process. Where i is a positive integer less than or equal to T1.

[0235] S5021: Input the last generated first intermediate image as one path and the attribute information of multiple instances as multiple paths into the diffusion network. The instance processing module in the diffusion network performs multiple feature generation to obtain image features of the multiple instances. The last generated first intermediate image and the attribute information of one instance are used for feature generation once, and the attribute information of one instance is input as one path.

[0236] For example, the first intermediate image (hereinafter referred to as the i-1th first intermediate image) and time node generated by the diffusion network during the i-1th generation process can be used as input Y and fed into the diffusion network. In other words, the i-1th first intermediate image and time node are fed into the first part of Block_1. After processing in the first part of Block_1, feature 1 is output. It should be noted that when i is 1, the first intermediate image generated by the i-1th generation process can refer to a preset noisy image.

[0237] Feature 1 is then used as input R0 of the instance processing module in Block_1. Furthermore, the attribute information in the n information groups can be used as n inputs to the instance processing module: the attribute information of the first information group is used as input R1 of the instance processing module, the attribute information of the second information group is used as input R2 of the instance processing module, and so on, and the attribute information of the nth information group is used as input Rn of the instance processing module.

[0238] That is, in the image generation process 500 , the instance processing module includes n+1 inputs.

[0239] FIG5B is a schematic diagram illustrating the processing process of the instance processing module and the global integration module.

[0240] 5B , illustratively, the instance processing module may perform feature generation based on feature 1 output by the first part of Block_1 and the attribute information of instance 1 to obtain image features of instance 1; perform feature generation based on feature 1 output by the first part of Block_1 and the attribute information of instance 2 to obtain image features of instance 2; ...; perform feature generation based on feature 1 output by the first part of Block_1 and the attribute information of instance n to obtain image features of instance n.

[0241] It can also be understood that the instance processing module performs n-way processing in parallel. The first way of processing is to generate features based on feature 1 output by the first part of Block_1 and the attribute information of instance 1. The second way of processing is to generate features based on feature 1 output by the first part of Block_1 and the attribute information of instance 2. ..., the nth way of processing is to generate features based on feature 1 output by the first part of Block_1 and the attribute information of instance n.

[0242] It should be noted that FIG5B shows that the n instance processing modules are actually the same instance processing module. The schematic diagram of FIG5B is to illustrate that the instance processing module processes the attribute information of each instance independently.

[0243] FIG5C is a schematic diagram illustrating the processing procedures of the instance processing module and the global integration module.

[0244] In Figure 5C , n = 2, the attribute information of the instance in information group 1 is "a blue cat," and the attribute information of the instance in information group 2 is "a green dog." The instance processing module can generate features based on feature 1 and "a blue cat" to output image features of the cat; and can also generate features based on feature 1 and "a green dog" to output image features of the dog.

[0245] S5022: Input the location information of the multiple instances and the image features of the multiple instances into a global integration module in the diffusion network to obtain a first intermediate feature.

[0246] For example, the image features of n instances can be used as n inputs of the global integration module, that is, the image features of instance 1 can be used as the input U1 of the global integration module, the image features of instance 2 can be used as the input U2 of the global integration module, ..., the image features of instance n can be used as the input Un of the global integration module. And the position information of the n instance can be used as one input of the global integration module (i.e., input E0). Then, the global integration module can process these n+1 inputs to obtain the first intermediate feature. Specifically, the global integration module can integrate the n inputs of input U1 to input Un based on input E0 to obtain the first intermediate feature.

[0247] 5C , for example, assuming that the position information in information group 1 is "(x11, y11), (x12, y12), (x13, y13), (x14, y14)", and the position information in information group 2 is "(x21, y21), (x22, y22), (x23, y23), (x24, y24)", then (x11, y11), (x12, y12), (x13, y13), (x14, y14), (x21, y21), (x22, y22), (x23, y23), (x24, y24)" can be used as input E0 and input into the global integration module.

[0248] S5023: Input the first intermediate feature into other modules in the diffusion network to obtain a third intermediate feature.

[0249] Exemplarily, the other modules may include the second part of Block in U-NET.

[0250] For example, the global integration module may input the first intermediate feature into the second part of Block_1, process the first intermediate feature by the second part of Block_1, and output the third intermediate feature to the second Block.

[0251] Afterwards, blocks 2 to 2g can be processed sequentially according to the method of Block_1, completing the i-th generation process of the diffusion network and obtaining the i-th first intermediate image. In other words, the i-th first intermediate image can refer to the output of the second part of the last block of the U-NET during the i-th generation process of the diffusion network (or in other words, the output of the diffusion network performing the i-th generation process).

[0252] By analogy, after the diffusion network performs T1 generation processing according to the above process, the T1th first intermediate image can be obtained.

[0253] It should be noted that when T1 is equal to T, S503 does not need to be executed. In this case, the T1-th first intermediate image can be used as the target image. When T1 is less than T, S503 can be executed.

[0254] S503: Input the first intermediate image into the diffusion network, and the diffusion network performs T2 generation processing to obtain the target image.

[0255] It should be noted that the diffusion network in S503 includes U-NET and instance processing module.

[0256] Exemplarily, S503 may include S5031 to S5033:

[0257] The following example illustrates the processing of the instance processing module embedded in Block_1 of U-NET during the T1+j generation process of the diffusion network. Here, j is a positive integer less than or equal to T2, T2 is a positive integer, and T1+T2=T.

[0258] S5031: The second intermediate image generated last time and the attribute information of multiple instances are input into the diffusion network as one path. The instance processing module generates features based on the second intermediate image generated last time and the attribute information of multiple instances to obtain second intermediate features.

[0259] For example, the second intermediate image generated by the diffusion network after the T1+j-1th generation process (hereinafter referred to as the T1+j-1th second intermediate image) and the time node can be used as input Y and input to the diffusion network; that is, the T1+j-1th second intermediate image and time node are input to the first part of Block_1. After processing in the first part of Block_1, feature 2 is output. It should be noted that when j is 1, the second intermediate image generated by the T1+j-1th generation process is the Tth first intermediate image generated in S502.

[0260] Feature 2 is then used as input R0 of the instance processing module in Block_1. Furthermore, the attribute information from the n information groups can be used as input R1 of the instance processing module. The instance processing module can then perform a feature generation operation based on feature 2 and the attribute information from the n information groups to obtain a second intermediate feature.

[0261] S5032: Input the second intermediate feature to other modules to obtain a second intermediate image.

[0262] Exemplarily, the other modules may include the second part of Block in U-NET.

[0263] Exemplarily, the instance processing module may input the second intermediate feature into the second part of Block_1, process the feature by the second part of Block_1, and output the fourth intermediate feature to the second Block.

[0264] Afterwards, blocks 2 through 2g can be processed sequentially according to the method of Block_1, completing the T1+j generation process of the diffusion network and obtaining the T1+j second intermediate image. In other words, the T1+j second intermediate image can refer to the output of the second part of the last block of the U-NET during the T1+j generation process of the diffusion network (or, in other words, the output of the T1+j generation process of the diffusion network).

[0265] By analogy, after the diffusion network performs T2 generation processing according to the above process, the T2th second intermediate image can be obtained.

[0266] S5033: The second intermediate image obtained by performing the T-th generation process of the diffusion network is used as the target image.

[0267] Exemplarily, the T2-th second intermediate image is used as the target image.

[0268] It should be understood that S503 is an optional step.

[0269] Since the subsequent stages of the chain generation process of the diffusion network can generate the details, attributes of each object in the target image and the lighting, style, and tone of the entire image, the interaction between the objects in the final target image can be natural, without any sense of splicing or fragmentation, and with harmonious tones and lighting, while ensuring that the image content is not changed or barely changed.

[0270] FIG6A is a schematic diagram illustrating an exemplary image generation process 600. In the embodiment of FIG6A , the diffusion network employed includes a U-NET / instance processing module and a global integration module. The image generation main interface corresponding to the image generation process 600 may be as shown in FIG3C .

[0271] S601: Receive global description information of a target image input by a user.

[0272] For example, the user may input global description information of the target image in the image generation main interface 301 in FIG. 3C .

[0273] S602: Determine multiple information groups based on the global description information of the target image input by the user.

[0274] Exemplarily, a text parser may be called to perform text parsing based on the global description information to obtain attribute information of each instance; and a position predictor may be called to perform instance position prediction based on the global description information to obtain position information of each instance.

[0275] S603: Input the multiple information groups into the diffusion network, and the diffusion network performs T1 generation processing to obtain a first intermediate image.

[0276] Exemplarily, T1 is a positive integer less than or equal to T. When T1=T, S604 does not need to be executed; when T1 is less than T, S604 can be executed. This application takes T1 less than T as an example for description.

[0277] The following uses the example of the instance processing module and global integration module embedded in Block_1 of U-NET during the i-th generation process of the diffusion network as an example to illustrate the process. Where i is a positive integer less than or equal to T1.

[0278] For example, the implementation of S603 may refer to the above S5021 to S5023, which will not be described in detail here.

[0279] In one possible approach, S603 may include: inputting multiple information groups and global description information into a diffusion network, and having the diffusion network perform T1 generation processing to obtain an intermediate image; this may be referred to as follows: S6031 to S6034:

[0280] S6031: Input the last generated first intermediate image as one path and the attribute information of multiple instances as multiple paths into the diffusion network. The instance processing module in the diffusion network performs multiple feature generation to obtain image features of the multiple instances. The last generated first intermediate image and the attribute information of one instance are used for feature generation once, and the attribute information of one instance is input as one path.

[0281] For example, S6031 may refer to the description of S5021 above and will not be repeated here.

[0282] S6032: Input the global description information into the instance processing module to generate features and obtain global image features.

[0283] Exemplarily, the global description information of the target image may be input into the instance processing module as the input R(n+1) of the instance processing module.

[0284] That is, in the image generation process 600 , the example processing module includes n+2 inputs.

[0285] FIG6B is a schematic diagram illustrating the processing process of the instance processing module and the global integration module.

[0286] 6B , illustratively, the instance processing module may perform feature generation based on feature 1 output by the first part of Block_1 and attribute information of instance 1 to obtain image features of instance 1; perform feature generation based on feature 1 output by the first part of Block_1 and attribute information of instance 2 to obtain image features of instance 2; ...; perform feature generation based on feature 1 output by the first part of Block_1 and attribute information of instance n to obtain image features of instance n; perform feature generation based on feature 1 output by the first part of Block_1 and global description information of the target image to obtain global image features.

[0287] It can also be understood that the instance processing module performs n+1 processing in parallel. The first processing path is to generate features based on feature 1 output by the first part of Block_1 and the attribute information of instance 1. The second processing path is to generate features based on feature 1 output by the first part of Block_1 and the attribute information of instance 2. ..., the nth processing path is to generate features based on feature 1 output by the first part of Block_1 and the attribute information of instance n; the n+1th processing path is to generate features based on feature 1 output by the first part of Block_1 and the global description information of the target image.

[0288] It should be noted that FIG6B shows that the n+1 instance processing modules are essentially the same instance processing module. The schematic diagram of FIG6B is to illustrate that the instance processing module independently processes the attribute information of each instance and the global description information of the target image.

[0289] FIG6C is a schematic diagram illustrating the processing procedures of the instance processing module and the global integration module.

[0290] In Figure 6C , the global description of the target image is "a blue cat to the left of a green dog." n = 2, the attribute information of the instance in information group 1 is "a blue cat," and the attribute information of the instance in information group 2 is "a green dog." The instance processing module can generate features based on feature 1 and "a blue cat" to output cat image features; it can also generate features based on feature 1 and "a green dog" to output dog image features; and it can generate features based on feature 1 and "a blue cat to the left of a green dog" to output global image features.

[0291] S6033: Input the location information of the multiple instances, the image features of the multiple instances, and the global image features into a global integration module in the diffusion network to obtain a first intermediate feature.

[0292] For example, compared to S5022, the global integration module in S6033 includes an additional input, namely the global image feature. This global image feature can be used as input U(n+1) of the global integration module. The global integration module can then integrate the n inputs, U1 through Un, based on the location information and global image features of multiple instances to obtain the first intermediate feature. For details, please refer to the description of S5022 and will not be repeated here.

[0293] S6034: Input the first intermediate feature into other modules in the diffusion network to obtain a third intermediate feature.

[0294] For example, S6034 may refer to the description of S5023 above, which will not be repeated here.

[0295] S604: Input the first intermediate image into the diffusion network, and the diffusion network performs T2 generation processing to obtain the target image.

[0296] For example, S604 may refer to the description of S503 above, which will not be repeated here.

[0297] It should be noted that, when the global description information of the target image does not include background description information, background prediction can be performed based on the global description information to obtain background description information of the target object.

[0298] In this way, a target image can be generated that matches the global description information specified by the user.

[0299] FIG7 is a schematic diagram illustrating an exemplary image generation process 700. In the embodiment of FIG7 , the diffusion network used includes an instance processing module and a global integration module. The image generation main interface corresponding to the image generation process 700 may be shown in FIG3D.

[0300] S701: Receive multiple information groups input by a user.

[0301] Exemplarily, the number of information groups can be represented by n.

[0302] For example, n=2, information group 1 includes: "a blue cat" and "(x11, y11), (x12, y12), (x13, y13), (x14, y14)"; information group 2 includes: "a green dog" and "(x21, y21), (x22, y22), (x23, y23), (x24, y24)".

[0303] S702: Receive global description information of a target image input by a user.

[0304] For example, the global description information of the target image is "a blue cat is to the left of a green dog".

[0305] S703: Input the multiple information groups and the global description information into the diffusion network, and the diffusion network performs T1 generation processing to obtain a first intermediate image.

[0306] Exemplarily, T1 is a positive integer less than or equal to T. When T1=T, S704 does not need to be executed; when T1 is less than T, S704 can be executed. This application takes T1 less than T as an example for description.

[0307] The following uses the example of the instance processing module and global integration module embedded in Block_1 of U-NET during the i-th generation process of the diffusion network as an example to illustrate the process. Where i is a positive integer less than or equal to T1.

[0308] Illustratively, S703 may include S7031 to S7034:

[0309] S7031: Input the last generated first intermediate image as one path and the attribute information of multiple instances as multiple paths into the diffusion network. The instance processing module in the diffusion network performs multiple feature generation to obtain image features of the multiple instances. The last generated first intermediate image and the attribute information of one instance are used for one feature generation, and the attribute information of one instance is input as one path.

[0310] S7032: Input the global description information into the instance processing module to generate features and obtain global image features.

[0311] S7033: Input the location information of the multiple instances, the image features of the multiple instances, and the global image features into a global integration module in the diffusion network to obtain a first intermediate feature.

[0312] S7034: Input the first intermediate feature into other modules in the diffusion network to obtain a third intermediate feature.

[0313] For example, S7033 to S7034 may refer to the description of S6033 to S6034 above, which will not be repeated here.

[0314] S704: Input the first intermediate image into the diffusion network, and the diffusion network performs T2 generation processing to obtain the target image.

[0315] For example, S704 may refer to the description of S503 above, which will not be repeated here.

[0316] Exemplarily, global integration information of the target image (also called a global integration template, which is an image feature) can also be generated based on the position information of multiple instances; the global integration information of the target image can be used to guide how to generate the target image using the image features of multiple instances.

[0317] Figure 8A is a schematic diagram illustrating an exemplary image generation process 800. Figure 8A is based on Figure 7. In the embodiment of Figure 8A, the diffusion network employed includes a U-NET, an instance processing module, a global integration module, and a global integration information generation module. The main image generation interface corresponding to image generation process 800 may be shown in Figure 3D.

[0318] S801: Receive multiple information groups input by a user.

[0319] Exemplarily, the number of information groups can be represented by n.

[0320] For example, n=2, information group 1 includes: "a blue cat" and "(x11, y11), (x12, y12), (x13, y13), (x14, y14)"; information group 2 includes: "a green dog" and "(x21, y21), (x22, y22), (x23, y23), (x24, y24)".

[0321] S802: Receive global description information of a target image input by a user.

[0322] For example, the global description information of the target image is "a blue cat is to the left of a green dog".

[0323] S803: Input the multiple information groups and the global description information into the diffusion network, and the diffusion network performs T1 generation processing to obtain a first intermediate image.

[0324] Exemplarily, the diffusion network in S803 includes a U-NET, an instance processing module, a global integration information generation module, and a global integration module.

[0325] Exemplarily, T1 is a positive integer less than or equal to T. When T1=T, S804 does not need to be executed; when T1 is less than T, S804 can be executed. This application takes T1 less than T as an example for description.

[0326] The following describes the processing of the instance processing module, global integration information generation module, and global integration module embedded in Block_1 of U-NET during the i-th generation process of the diffusion network as an example. Here, i is a positive integer less than or equal to T1.

[0327] Illustratively, S803 may include S8031 to S8036:

[0328] S8031: Input the last generated first intermediate image as one path and the attribute information of multiple instances as multiple paths into the diffusion network. The instance processing module in the diffusion network performs multiple feature generation to obtain image features of the multiple instances. The last generated first intermediate image and the attribute information of one instance are used for feature generation once, and the attribute information of one instance is input as one path.

[0329] S8032: Input the global description information into the instance processing module to generate features and obtain global image features.

[0330] For example, S8031 to S8032 may refer to the description of S6031 to S6032 above, which will not be repeated here.

[0331] S8033, generating multiple mask images based on the position information of multiple instances; wherein the multiple mask images correspond one-to-one to multiple pixels in the target image; the size of each mask image is the same as the size of the target image, the mask image corresponding to the first pixel is the mask image corresponding to the instance to which the first pixel belongs, and the first pixel is any pixel in the target image.

[0332] The following takes n=2 as an example to illustrate the process of generating a mask image corresponding to each pixel in the target image.

[0333] FIG8B is a schematic diagram illustrating an exemplary mask image generation process.

[0334] For example, the size of the layout diagram in FIG8B is the same as the size of the target image, and the pixels in the layout diagram correspond one-to-one with the pixels in the target image. The layout diagram in FIG8B includes the position layouts of two instances. Each square in the layout diagram can represent a pixel, with white pixels representing pixels in the background of the target image, gray pixels representing pixels within the bounding box of instance 1, and black pixels representing pixels within the bounding box of instance 2.

[0335] For example, for each pixel in the layout, a mask corresponding to the instance (or background) to which the pixel belongs (that is, the mask corresponding to the pixel) can be generated and stored. Assuming the layout has a height of H and a width of W, then for H*W pixels, H*W mask images can be generated, with one mask image corresponding to each pixel.

[0336] It should be noted that when the location information of the instance is the bounding box information of the instance, the mask image corresponding to the instance to which the pixel point P1 belongs can be generated based on the bounding box of the instance to which the pixel point P1 belongs. Specifically, the pixel values ​​of the pixels in the layout diagram that are within the bounding box of the instance to which the pixel point P1 belongs can be set to 0; the pixel values ​​of the pixels in the layout diagram that are outside the bounding box of the instance to which the pixel point P1 belongs can be set to 1, and the mask image corresponding to the instance to which the pixel point P1 belongs can be obtained. Alternatively, the pixel values ​​of the pixels in the layout diagram that are within the bounding box of the instance to which the pixel point P1 belongs can be set to 1; the pixel values ​​of the pixels in the layout diagram that are outside the bounding box of the instance to which the pixel point P1 belongs can be set to 0, and the mask image corresponding to the instance to which the pixel point P1 belongs can be obtained.

[0337] It should be noted that when the location information of the instance is the contour information of the instance, the mask image corresponding to the instance to which the pixel point P2 belongs can be generated based on the contour information of the instance to which the pixel point P2 belongs. Specifically, the pixel values ​​of the pixels in the layout diagram that are within the contour box of the instance to which the pixel point P2 belongs can be set to 0; the pixel values ​​of the pixels in the layout diagram that are outside the contour box of the instance to which the pixel point P2 belongs can be set to 1, and the mask image corresponding to the instance to which the pixel point P belongs can be obtained. Alternatively, the pixel values ​​of the pixels in the layout diagram that are within the contour box of the instance to which the pixel point P2 belongs can be set to 1; the pixel values ​​of the pixels in the layout diagram that are outside the contour box of the instance to which the pixel point P2 belongs can be set to 0, and the mask image corresponding to the instance to which the pixel point P belongs can be obtained.

[0338] For example, for the pixel point P3 belonging to the background, the pixel value of the pixel point located within the bounding box (or outline) of all instances in the layout diagram can be set to 0, and the pixel value of the pixel point located outside the bounding box (or outline) of all instances in the layout diagram can be set to 1 to obtain a mask image corresponding to the background.

[0339] It should be understood that the mask images corresponding to multiple pixels belonging to the same instance are the same, or the mask images corresponding to multiple pixels belonging to the background are the same, wherein the size of the mask image is the same as the size of the layout image.

[0340] Referring to Figure 8B , for a white pixel, a background mask can be generated, as shown in mask 1. For a black pixel, a mask corresponding to instance 2 can be generated, as shown in mask 2. For a gray pixel, a mask corresponding to instance 1 can be generated, as shown in mask 4. The mask images corresponding to the two black pixels within the bounding box of instance 2 are mask 2 and mask 3, respectively, and mask 2 and mask 3 are identical.

[0341] S8034: Generate global integration information based on multiple mask images.

[0342] For example, feature 1 output from the first part of the block and the mask corresponding to each pixel in the target image can be input into the global integrated information generation module. Feature 1 can serve as input R0 of the global integrated information generation module, and the mask corresponding to each pixel in the target image can serve as input H0 of the global integrated information generation module.

[0343] Next, the global integration information generation module can process the mask image corresponding to feature 1 and each pixel in the target image, and output the global integration information.

[0344] Figure 8C is a schematic diagram illustrating an exemplary process of generating global integrated information. Figure 8C shows a possible process of generating global integrated information by the global integrated information generation module. In Figure 8C, the global integrated information generation module may include a feature transformation module and a self-attention module.

[0345] 8C , illustratively, feature 1 can be first input into a feature transformation module, which performs three feature transformations (e.g., linear transformations) on feature 1 to obtain feature maps Q, K, and V. Feature maps Q, K, and V each have a height of H and a width of W. Next, feature maps Q and K are input into a self-attention module, which processes them to generate H*W attention maps, each with a height of H and a width of W. Subsequently, H*W mask maps are dot-multiplied with H*W attention maps to obtain H*W feature maps R (with a height of H and a width of W). The H*W feature maps R are then cross-multiplied with feature map V (i.e., matrix multiplication) to obtain global integrated information (shading template), which is a feature map with a height of H and a width of W.

[0346] S8035: Input the location information of the multiple instances, the image features of the multiple instances, the global integration information, and the global image features into a global integration module in the diffusion network to obtain a first intermediate feature.

[0347] For example, compared to S6033, the global integration module in S8035 has an additional input, namely global integration information. This global integration information can be used as input U(n+2) of the global integration module. The global integration module can then integrate the n inputs, U1 through Un, based on the location information of multiple instances, global image features, and the global integration information to obtain the first intermediate feature. For details, please refer to the description of S6033 and will not be repeated here.

[0348] S8036: Input the first intermediate feature into other modules in the diffusion network to obtain a third intermediate feature.

[0349] For example, S8036 can refer to the description of S5023 above, which will not be repeated here.

[0350] S804: Input the first intermediate image into the diffusion network, and the diffusion network performs T2 generation processing to obtain the target image.

[0351] For example, S804 may refer to the description of S503 above, which will not be repeated here.

[0352] FIG8D is a schematic diagram illustrating an exemplary image generation process.

[0353] Referring to Figure 8D , for example, a user inputs the global description of a target image, "a blue cat to the left of a green dog." Based on this global description, the positional layout and attribute information of instance 1 (cat) and instance 2 (dog) can be determined. Instance segmentation can then be performed to obtain the cat's attribute information, "a blue cat," and the dog's attribute information, "a green dog."

[0354] 8D , the instance processing module may include a cross attention module and an enhanced attention module. The attribute information of the cat "a blue cat" may be input into the cross attention module (Cross Attention) and the enhanced attention module (Enhancement Attention) respectively, and the feature 1 (i.e., the output of the first part of the Block) may be input into the cross attention module and the enhanced attention module respectively. Afterwards, the cross attention module may process the feature 1 and the attribute information of the cat "a blue cat" and output the first feature of the cat; and the enhanced attention module may process the feature 1 and the attribute information of the cat "a blue cat" and output the second feature of the cat. Afterwards, the first feature of the cat and the second feature of the cat are added to obtain the image feature of the cat.

[0355] Similarly, the dog's attribute information "a green dog" can be input into the Cross Attention module and the Enhancement Attention module, respectively. Feature 1 (i.e., the output of the first part of the block) can also be input into the Cross Attention module and the Enhancement Attention module. The Cross Attention module can then process Feature 1 and the dog's attribute information "a green dog" to output the first feature of the dog. The Enhancement Attention module can then process Feature 1 and the dog's attribute information "a green dog" to output the second feature of the dog. The first and second features of the dog are then added together to obtain the dog's image features.

[0356] 8D , feature 1 and global description information may be input into the cross-attention module of the instance processing module to obtain image features (also referred to as background features) of the target image.

[0357] 8D , the mask image corresponding to feature 1 and each pixel in the target image may be input into a global integration information generation module (which may include a layout attention module) to obtain global integration information.

[0358] Referring to FIG8D , the cat's and dog's position information, the global integration information, the target image's image features, the cat's and dog's image features, can be input into the global integration module to obtain global image features. Subsequently, the global image features are processed to obtain a target image, such as Target Image 1 or Target Image 2.

[0359] Compared with the above-mentioned image generation process 200, image generation process 400, image generation process 500, image generation process 600 and image generation process 700, image generation process 800 newly introduces global integration information in the global integration process; that is, the image generation process 800 has richer information for guiding the integration of image features of multiple instances. Therefore, the image generation process 800 can achieve more accurate position control of multiple instances in the target image.

[0360] In addition, the attention map represents the probability that each pixel in the target image is any instance or background. Since H*W mask maps are generated based on the position information of each instance, the dot product of H*W mask maps and H*W attention maps can fine-tune the probability that each pixel in the H*W attention maps is any instance or background, that is, adjust the position of each instance in the target image. This can make the position of each instance in the target image more consistent with the position information of each instance, further improving the accuracy of the positions of multiple instances in the target image.

[0361] FIG. 9A is a schematic diagram illustrating an exemplary target image.

[0362] For example, the global description information of the target image input by the user is "There is a fantasy world in the crystal ball on the table". Text parsing is performed based on the global description information to obtain the attribute information of instance 1 "a crystal ball", the attribute information of instance 2 "a white moon", the attribute information of instance 3 "a Christmas tree", the attribute information of instance 4 "a red house", the attribute information of instance 5 "snow" and the attribute information of instance 6 "a table". And instance position prediction is performed based on the global description information to obtain the position information of the above 6 instances; wherein, based on the position information of these 6 instances, the position layout of the external frames of these 6 instances (also referred to as the position layout of these 6 instances) is determined as shown in Figure 9A (11). The target image generated according to the global description information using the image generation method of the present application is shown in Figure 9A (12).

[0363] For example, the global description information of the target image input by the user is "a cute kitten and a pumpkin carriage". Text parsing is performed based on the global description information to obtain the attribute information of instance 1 "a cute kitten", the attribute information of instance 2 "a pumpkin carriage", the attribute information of instance 3 "a green donut wheel", the attribute information of instance 4 "a blue donut wheel" and the attribute information of instance 5 "hole". And instance position prediction is performed based on the global description information to obtain the position information of the above five instances; wherein, based on the position information of these five instances, the position layout of the external frames of these five instances (also referred to as the position layout of these five instances) is determined as shown in Figure 9A (21). The target image generated according to the global description information using the image generation method of the present application is shown in Figure 9A (22).

[0364] For example, the global description information of the target image input by the user is "a squirrel artist". Text parsing is performed based on the global description information to obtain the attribute information of instance 1 "a beret", the attribute information of instance 2 "a squirrel head" and the attribute information of instance 3 "a coat". And instance position prediction is performed based on the global description information to obtain the position information of the above three instances; wherein, based on the position information of the three instances, the position layout of the external frames of the three instances (also referred to as the position layout of the three instances) is shown in Figure 9A (31). The target image generated by the image generation method of the present application based on the global description information is shown in Figure 9A (32).

[0365] FIG9B is a schematic diagram showing an exemplary target image.

[0366] For example, the global description information of the target image input by the user is "a corgi wearing a Christmas hat sitting on a cake with a teddy bear standing next to it". Text parsing is performed based on the global description information to obtain the attribute information of instance 1 "a Christmas hat", the attribute information of instance 2 "a fuzzy Christmas tree", the attribute information of instance 3 "a cute corgi", the attribute information of instance 4 "a cake" and the attribute information of instance 5 "a fuzzy blue teddy bear". And instance position prediction is performed based on the global description information to obtain the position information of the above five instances; wherein, based on the position information of these five instances, the position layout of the external frames of the four instances (also referred to as the position layout of the five instances) is determined as shown in Figure 9B (11). The target image generated by the image generation method of the present application based on the global description information is shown in Figure 9B (12).

[0367] For example, the global description information of the target image input by the user is "On a green table, a blue plant grows in a yellow flowerpot, next to which is a red landline phone and a white cup". Text parsing is performed based on the global description information to obtain the attribute information of instance 1 "a blue plant", the attribute information of instance 2 "a yellow flowerpot", the attribute information of instance 3 "a red landline phone", the attribute information of instance 4 "a white cup" and the attribute information of instance 5 "a green table". And instance position prediction is performed based on the global description information to obtain the position information of the above five instances; wherein, based on the position information of these five instances, the position layout of the external frames of these five instances (also referred to as the position layout of these five instances) is determined as shown in Figure 9B (21). The target image generated according to the global description information using the image generation method of the present application is shown in Figure 9B (22).

[0368] For example, the global description information of the target image input by the user is "four boats". Text parsing is performed based on the global description information to obtain the attribute information of instance 1 "a blue boat", the attribute information of instance 2 "a red boat", the attribute information of instance 3 "a green boat", the attribute information of instance 4 "a black boat", the attribute information of instance 5 "a yellow bench" and the attribute information of instance 6 "a beach". And instance position prediction is performed based on the global description information to obtain the position information of the above 6 instances; wherein, based on the position information of these 6 instances, the position layout of the external frames of these 6 instances (also referred to as the position layout of these 6 instances) is determined as shown in Figure 9B (31). The target image generated by the image generation method of the present application based on the global description information is shown in Figure 9B (32).

[0369] FIG9C is a schematic diagram illustrating an exemplary target image.

[0370] For example, the global description information of the target image input by the user is "two teddy bears holding hands". Text parsing is performed based on the global description information to obtain the attribute information of instance 1 "a yellow teddy bear", the attribute information of instance 2 "a pink skirt", the attribute information of instance 3 "a red teddy bear", the attribute information of instance 4 "a blue tie" and the attribute information of instance 5 "a black dress". And instance position prediction is performed based on the global description information to obtain the position information of the above five instances; wherein, based on the position information of these five instances, the position layout of the external frames of these five instances (also referred to as the position layout of these five instances) is determined as shown in Figure 9C (11). The target image generated by the image generation method of the present application based on the global description information is shown in Figure 9C (12).

[0371] For example, the global description information of the target image input by the user is "a blue rabbit wearing a yellow wig tapping a green keyboard". Text parsing is performed based on the global description information to obtain the attribute information of instance 1 "a yellow wig", the attribute information of instance 2 "a blue rabbit", the attribute information of instance 3 "a green keyboard" and the attribute information of instance 4 "a table". And instance position prediction is performed based on the global description information to obtain the position information of the above four instances; wherein, based on the position information of the four instances, the position layout of the external frames of the four instances (also referred to as the position layout of the four instances) is determined as shown in Figure 9C (21). The target image generated by the image generation method of the present application based on the global description information is shown in Figure 9C (22).

[0372] For example, the global description information of the target image input by the user is "a squirrel in a space suit holding an apple". Text parsing is performed based on the global description information to obtain the attribute information of instance 1 "a squirrel", the attribute information of instance 2 "a space suit", the attribute information of instance 3 "a glass helmet", the attribute information of instance 4 "an apple" and the attribute information of instance 5 "a pair of gloves". And instance position prediction is performed based on the global description information to obtain the position information of the above five instances; wherein, based on the position information of these five instances, the position layout of the external frames of these five instances (also referred to as the position layout of these five instances) is determined as shown in Figure 9C (31). The target image generated according to the global description information using the image generation method of the present application is shown in Figure 9C (32).

[0373] FIG9D is a schematic diagram illustrating an exemplary target image.

[0374] For example, the global description information of the target image input by the user is "a corgi looking out the sunny window". Text parsing is performed based on the global description information to obtain the attribute information of instance 1 "a window with a sunny view", the attribute information of instance 2 "a corgi" and the attribute information of instance 3 "a dog leash". In addition, the background description information is determined to be "a dark background". And instance position prediction is performed based on the global description information to obtain the position information of the above three instances; wherein, based on the position information of these three instances, the position layout of the external frames of the three instances (also referred to as the position layout of the three instances) is determined as shown in Figure 9D (11). In addition, Figure 9D (11) also includes the position layout of the background. The target image generated by the image generation method of the present application based on the global description information is shown in Figure 9D (12).

[0375] For example, the global description information of the target image input by the user is "a blue lion is reading a red book on the beach". Text parsing is performed based on the global description information to obtain the attribute information of instance 1 "a coconut tree", the attribute information of instance 2 "a blue lion", the attribute information of instance 3 "a beach", the attribute information of instance 4 "a clear sky" and the attribute information of instance 5 "a red book". And instance position prediction is performed based on the global description information to obtain the position information of the above five instances; wherein, based on the position information of these five instances, the position layout of the external frames of these five instances (also referred to as the position layout of these five instances) is determined as shown in Figure 9D (21). The target image generated according to the global description information using the image generation method of the present application is shown in Figure 9D (22).

[0376] For example, the global description information of the target image input by the user is "a corgi sitting on the beach looking at the sunset sky". Text parsing is performed based on the global description information to obtain the attribute information of instance 1 "a lake", the attribute information of instance 2 "a sunset sky", the attribute information of instance 3 "a corgi" and the attribute information of instance 4 "a beach". And instance position prediction is performed based on the global description information to obtain the position information of the above four instances; wherein, based on the position information of these four instances, the position layout of the external frames of these four instances (also referred to as the position layout of these four instances) is determined as shown in Figure 9D (31). The target image generated according to the global description information using the image generation method of the present application is shown in 9D (32).

[0377] FIG10A is a schematic diagram of an exemplary target image. The target image of FIG10A(1) includes an orange cat, a blue cat, a green cat, a yellow sofa, and a red cake. The target image of FIG10A(2) includes a green cake, a yellow cake, a blue cake, a red cake, and an orange cake. It can be seen that the image generation method of the present application can generate instances of different colors.

[0378] FIG10B is a schematic diagram of an exemplary target image. FIG10B(1) shows a target image of three dogs and two cats. FIG10B(2) shows a target image of five birds. It can be seen that the image generation method of the present application can generate different numbers of instances.

[0379] FIG10C is a schematic diagram of exemplary target images. The target image of FIG10C(1) includes a triangular pyramid, a cuboid, a sphere, and a cylinder. The target image of FIG10C(2) includes a triangular cake, a circular cake, and a square cake. It can be seen that the image generation method of the present application can generate instances of different shapes.

[0380] FIG10D is a schematic diagram of an exemplary target image. The target image in FIG10D(1) includes vases made of five different materials: a wooden vase, a glass vase, a woven vase, a stone vase, and a metal vase; it can be seen that the image generation method of the present application can generate instances of different materials. The target image in FIG10D(2) includes five towels: a pure blue towel, a striped towel, a spotted towel, a checkered towel, and a printed towel; it can be seen that the image generation method of the present application can generate instances of different textures.

[0381] FIG10E is a schematic diagram of exemplary target images. The target image of FIG10E(1) includes a cyberpunk-style city, a realistic corgi, and a cartoon car. The target image of FIG10E(2) includes a watercolor elephant, a Van Gogh starry sky, a cartoon cat, and a realistic lake. It can be seen that the image generation method of the present application can generate examples of different styles.

[0382] Figure 10F is a schematic diagram of an exemplary target image. The target image in Figure 10F includes a player wearing two shoes of different colors, a football, and a blue chest; it can be seen that the image generation method of the present application can generate multiple instances with action relationships.

[0383] For example, Tables 1 to 3 show the comparison results of the effects of the present application and the prior art:

[0384] Table 1

[0385] In Table 1, when the number of instances is two, the instance generation success rate of prior art 1 is 6.87, the instance generation success rate of prior art 2 is 20.47, the instance generation success rate of prior art 3 is 24.61, the instance generation success rate of prior art 4 is 24.88, the instance generation success rate of prior art 5 is 42.30, and the instance generation success rate of the present application is 67.70. When the number of instances is 3, 4, 5, and 6, respectively, the instance generation success rates of prior art 1 to prior art 5, as well as the instance generation success rate of the present application can be referred to as shown in Table 1, and will not be repeated here. By comparison, it can be seen that the instance generation success rate of the present application is higher than that of the prior art.

[0386] Exemplarily, one way to determine whether an instance is generated successfully may be: for an instance in the target image (hereinafter referred to as instance 1), detect the position of instance 1 in the target image (hereinafter referred to as position 1); then, calculate the intersection-over-union (IoU) of position 1 and the position of instance 1 specified by the user (hereinafter referred to as position 0); when the IoU of position 1 and position 0 is greater than 0.5, calculate the IoU of the region in the target image where the attributes of instance 1 are the same as the attributes of instance 1 specified by the user (hereinafter referred to as region 1) and the region corresponding to instance 1 in the target image (hereinafter referred to as region 2, that is, the region in the target image that only contains instance 1). When the IoU of region 1 and region 2 is greater than 0.2, it is determined that instance 1 in the target image is generated successfully; otherwise, it is determined that instance 1 in the target image is generated unsuccessfully. Similarly, determine whether other instances in the target image are generated successfully.

[0387] In Table 1, the time taken by prior art 1 to generate the target image is 9.18s, the time taken by prior art 2 to generate the target image is 19.92s, the time taken by prior art 3 to generate the target image is 44.17s, the time taken by prior art 4 to generate the target image is 25.15s, the time taken by prior art 5 to generate the target image is 22.00s, and the time taken by prior art 4 to generate the target image is 15.16s. Compared with prior art 2 to prior art 5, the present application takes less time to generate the target image and has a high success rate in example generation.

[0388] Table 2

[0389] The intersection-over-union ratio in Table 2 refers to the intersection-over-union ratio of the aforementioned region 1 and region 2.

[0390] In Table 2, when the number of instances is two, the intersection-union ratio of prior art 1 is 18.92, the intersection-union ratio of prior art 2 is 29.34, the intersection-union ratio of prior art 3 is 32.64, the intersection-union ratio of prior art 4 is 29.41, the intersection-union ratio of prior art 5 is 37.58, and the intersection-union ratio of the present application is 59.39. When the number of instances is 3, 4, 5, and 6, respectively, the intersection-union ratios of prior art 1 to prior art 5, as well as the intersection-union ratio of the present application, can be referred to as shown in Table 2 and will not be repeated here. It can be seen that the intersection-union ratio of the present application is higher than that of the prior art; that is, the position of the instance in the target image generated by the present application is more consistent with the position of the instance specified by the user.

[0391] Table 3

[0392] Exemplarily, the success rates in Table 3 are success rates for generating target images; wherein, for each target image, if all instances in the target image are successfully generated, it can be determined that the target image is successfully generated.

[0393] Exemplarily, the intersection-over-union ratio in Table 3 refers to the intersection-over-union ratio of the positions of all instances in the target image to the positions of all instances specified by the user.

[0394] For example, AP (average precision) 50 refers to the proportion of target images in which the intersection-over-union ratio of the positions of all instances in multiple target images to the positions of all instances specified by the user is greater than 0.5.

[0395] For example, AP75 refers to the proportion of target images in which the intersection-over-union ratio of the positions of all instances in multiple target images to the positions of all instances specified by the user is greater than 0.75.

[0396] Illustratively, AP refers to the average of AP50, AP55, AP60, AP65, AP70, AP75, AP80, AP85, AP90, and AP95.

[0397] For example, CLIP (Contrastive Language-Image Pretraining) is used to determine the similarity between a target image and global description information.

[0398] For example, assuming that the target image includes two instances (instance 1 and instance 2), the region of instance 1 can be extracted from the target image, and CLIP can be used to determine the similarity between the region of instance 1 and the user-specified attributes of instance 1, obtaining similarity 1. Furthermore, the region of instance 2 can be extracted from the target image, and CLIP can be used to determine the similarity between the region of instance 2 and the user-specified attributes of instance 2, obtaining similarity 2. The average of similarity 1 and similarity 2 is the Local CLIP.

[0399] For example, the FID metric stands for Fréchet Inception Distance. It's a metric used to evaluate the performance of image generation models (such as GANs, generative adversarial networks). It measures the quality and diversity of generated images by comparing the distribution of generated images with real images within a specific space. 6K indicates that 6,000 target images were generated.

[0400] Exemplarily, the real image in Table 3 may refer to an image in a data set, and the position (that is, the position of the instance specified by the user) and attributes (that is, the attributes of the instance specified by the user) of each instance can be obtained by manually annotating the images in the data set. Afterwards, a detection tool may be used to detect the position and attributes of each instance in the real image; based on the position and attributes of each instance obtained by detecting the real image using the detection tool, and the position and attributes of the instance specified by the user, determine whether each instance is generated successfully (refer to the above description, which will not be repeated here); then, based on whether the instance is generated successfully, determine whether the real image is generated successfully. Afterwards, the intersection-over-union ratio, AP, AP50, and AP75 may also be calculated based on the position of the instance specified by the user and the position of each instance obtained by detecting the real image using the detection tool.

[0401] It should be understood that the success rate, intersection-over-union ratio, AP, AP50 and AP75 of the target image generated by prior arts 1 to prior arts 6, as well as the target image generated by the present application, can be determined in the same manner as the success rate, intersection-over-union ratio, AP, AP50 and AP75 of the real image; no further details will be given here.

[0402] Referring to Table 3, it can be seen that the quantitative indicators of the target images generated by this application: success rate, intersection-over-union ratio, AP, AP50 and AP75; as well as CLIP, Local CLIP and FID-6K, are all better than the existing technology.

[0403] 11 is a schematic diagram of an exemplary image generating apparatus 1100. The image generating apparatus 1100 can be used to execute the method of the aforementioned embodiment. Therefore, the beneficial effects achievable by the image generating apparatus 1100 can refer to the beneficial effects of the corresponding method provided above, and will not be described in detail here.

[0404] The acquisition module 1101 is used to acquire multiple information groups, wherein each information group includes attribute information and location information of an instance;

[0405] A first generating module 1102 is configured to generate an image feature of an instance based on the attribute information of an instance contained in each information group of the plurality of information groups;

[0406] The integration module 1103 is configured to integrate the image features of the multiple instances according to the position information of the multiple instances included in the multiple information groups to generate a target image.

[0407] Exemplarily, the image generating apparatus 1100 further includes:

[0408] a second generation module, configured to generate a plurality of mask images based on the position information of the plurality of instances; wherein the plurality of mask images correspond one-to-one to a plurality of pixels in the target image; the size of each mask image is the same as the size of the target image; the mask image corresponding to the first pixel is the mask image corresponding to the instance to which the first pixel belongs, and the first pixel is any pixel in the target image; and generate global integration information based on the plurality of mask images;

[0409] The integration module 1103 is specifically configured to integrate the image features of the multiple instances according to the position information of the multiple instances and the global integration information to generate a target image.

[0410] Exemplarily, the acquisition module 1101 is further configured to acquire global description information of the target image;

[0411] The image generating apparatus 1100 further includes:

[0412] The third generation module is used to generate global image features based on the global description information;

[0413] The integration module 1103 is specifically configured to integrate the image features of the multiple instances according to the location information of the multiple instances and the global image features to generate a target image.

[0414] Exemplarily, the image generating apparatus 1100 further includes:

[0415] a second generation module, configured to generate a plurality of mask images based on the position information of the plurality of instances; wherein the plurality of mask images correspond one-to-one to a plurality of pixels in the target image; the size of each mask image is the same as the size of the target image; the mask image corresponding to the first pixel is the mask image corresponding to the instance to which the first pixel belongs, and the first pixel is any pixel in the target image; and generate global integration information based on the plurality of mask images;

[0416] The integration module 1103 is specifically configured to integrate the image features of the multiple instances according to the position information of the multiple instances, the global integration information and the global image features, so as to generate a target image.

[0417] Exemplarily, the acquisition module 1101 is specifically configured to receive multiple information groups input by a user.

[0418] Exemplarily, the acquisition module 1101 is specifically configured to perform text parsing based on the global description information of the target image to obtain attribute information of multiple instances; and perform instance position prediction based on the global description information of the target image to obtain position information of multiple instances.

[0419] Exemplarily, the first generation module 1102 is specifically configured to divide the attribute information of multiple instances into multiple paths and input them into the instance processing network, and the instance processing network generates features based on the attribute information of one instance to obtain image features of the instance;

[0420] Among them, the attribute information of an instance is an input to the instance processing network.

[0421] Exemplarily, the integration module 1103 is specifically configured to input the location information of multiple instances and the image features of multiple instances into the global integration network to obtain a target image.

[0422] Exemplarily, the first generation module 1102 is specifically configured to, during each generation process of the diffusion network performing T1 generation processes, input the previously generated first intermediate image as one path and attribute information of multiple instances as multiple paths into the diffusion network, so that the instance processing network in the diffusion network performs multiple feature generation operations to obtain image features of the multiple instances; wherein the previously generated first intermediate image and attribute information of one instance are used for one feature generation operation, and the attribute information of one instance is input as one path.

[0423] The integration module 1103 is specifically configured to input the position information and image features of multiple instances into a global integration network in the diffusion network during each generation process of the diffusion network performing T1 generation processes to obtain a first intermediate feature; input the first intermediate feature into other networks in the diffusion network to obtain a first intermediate image; wherein the global integration network is located after the instance processing network, and the other networks are located after the global integration network; and determine the target image based on the first intermediate image obtained by the diffusion network performing the T1 generation process.

[0424] Exemplarily, the integration module 1103 is specifically configured to, during each generation process of the diffusion network performing T2 generation processes thereafter, input the second intermediate image generated last as one path and the attribute information of multiple instances as one path into the diffusion network, and have the instance processing network perform feature generation based on the second intermediate image generated last and the attribute information of multiple instances to obtain a second intermediate feature; input the second intermediate feature into other networks to obtain a second intermediate image; and use the second intermediate image obtained by the diffusion network performing the T-th generation process as the target image; wherein T is the sum of T1 and T2.

[0425] Exemplarily, the integration module 1103 is specifically configured to use the first intermediate image obtained by performing the T1th generation process of the diffusion network as the target image.

[0426] Exemplarily, the attribute information includes at least one of the following: an image or text;

[0427] The location information includes at least one of the following: an image or text.

[0428] Exemplarily, the location information includes at least one of the following: outline information of the instance and information of a bounding box of the instance.

[0429] In an example, FIG12 shows a schematic block diagram of a device 1200 according to an embodiment of the present application. The device 1200 may include: a processor 1201 and a transceiver / transceiver pin 1202 , and optionally, a memory 1203 .

[0430] The various components of the device 1200 are coupled together via a bus 1204, wherein the bus 1204 includes, in addition to a data bus, a power bus, a control bus, and a status signal bus. However, for the sake of clarity, all buses are referred to as bus 1204 in the figure.

[0431] Optionally, the memory 1203 may be used to store instructions in the aforementioned method embodiment. The processor 1201 may be used to execute the instructions in the memory 1203 and control the receiving pin to receive a signal and control the transmitting pin to send a signal.

[0432] The apparatus 1200 may be the electronic device or a chip of the electronic device in the above method embodiment.

[0433] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0434] The present application also provides a chip including one or more interface circuits and one or more processors. The one or more processors receive or send data via the one or more interface circuits. When the one or more processors execute computer instructions, the steps of the above-mentioned related methods are implemented. The interface circuit is a transceiver / transceiver pin 1202.

[0435] This embodiment further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the method in the above-mentioned embodiment.

[0436] This embodiment further provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer executes the above-mentioned related steps to implement the method in the above-mentioned embodiment.

[0437] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the methods in the above-mentioned method embodiments.

[0438] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0439] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0440] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0441] Units described as separate components may or may not be physically separate, and components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0442] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0443] Any content of each embodiment of this application, as well as any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of this application.

[0444] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0445] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in a hardware manner, or can be implemented by a processor executing a software instruction. The software instruction can be composed of corresponding software modules, and the software module can be stored in a random access memory (Random Access Memory, RAM), a flash memory, a read-only memory (Read Only Memory, ROM), an erasable programmable read-only memory (Erasable Programmable ROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disc (CD-ROM) or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0446] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer-readable storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0447] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. An image generation method, characterized in that: The method comprises: Acquire multiple information groups; wherein each information group includes attribute information and location information of an instance; generating an image feature of an instance according to the attribute information of an instance contained in each information group of the plurality of information groups; According to the position information of the multiple instances included in the multiple information groups, the image features of the multiple instances are integrated to generate a target image.

2. The method according to claim 1, characterized in that The method further comprises: Generate multiple mask images based on the position information of the multiple instances; wherein the multiple mask images correspond one-to-one to multiple pixels in the target image; the size of each mask image is the same as the size of the target image, the mask image corresponding to the first pixel is the mask image corresponding to the instance to which the first pixel belongs, and the first pixel is any pixel in the target image; Generate global integration information based on the multiple mask images; The step of integrating the image features of the plurality of instances according to the position information of the plurality of instances included in the plurality of information groups to generate a target image includes: The image features of the multiple instances are integrated according to the position information of the multiple instances and the global integration information to generate the target image.

3. The method according to claim 1, characterized in that The method further comprises: Obtaining global description information of the target image; generating the global image features according to the global description information; The step of integrating the image features of the plurality of instances according to the position information of the plurality of instances included in the plurality of information groups to generate a target image includes: The target image is generated by integrating the image features of the multiple instances according to the position information of the multiple instances and the global image features.

4. The method according to claim 3, characterized in that The method further comprises: Generate multiple mask images based on the position information of the multiple instances; wherein the multiple mask images correspond one-to-one to multiple pixels in the target image; the size of each mask image is the same as the size of the target image, the mask image corresponding to the first pixel is the mask image corresponding to the instance to which the first pixel belongs, and the first pixel is any pixel in the target image; Generate global integration information based on the multiple mask images; The step of integrating the image features of the multiple instances according to the position information of the multiple instances included in the multiple information groups and the global image features to generate the target image includes: The target image is generated by integrating the image features of the multiple instances according to the position information of the multiple instances, the global integration information, and the global image features.

5. The method according to any one of claims 1 to 4, characterized in that The obtaining of multiple information groups includes: Receive multiple information groups input by the user.

6. The method according to any one of claims 1 to 4, characterized in that The obtaining of multiple information groups includes: Performing text parsing based on the global description information of the target image to obtain attribute information of the multiple instances; Instance positions are predicted based on the global description information of the target image to obtain position information of the multiple instances.

7. The method according to any one of claims 1 to 6, characterized in that Generating an image feature of an instance according to the attribute information of an instance contained in each information group of the plurality of information groups includes: The attribute information of the multiple instances is divided into multiple paths and inputted into an instance processing network, and the instance processing network generates features according to the attribute information of one instance to obtain an image feature of the instance; The attribute information of an instance is an input to the instance processing network.

8. The method according to any one of claims 1 to 7, characterized in that The step of integrating the image features of the plurality of instances according to the position information of the plurality of instances included in the plurality of information groups to generate a target image includes: The position information of the multiple instances and the image features of the multiple instances are input into a global integration network to obtain the target image.

9. The method according to any one of claims 1 to 6, characterized in that Generating an image feature of an instance according to the attribute information of an instance contained in each information group of the plurality of information groups includes: During each generation process of the diffusion network performing T1 generation processes, the first intermediate image generated last time is input as one path, and the attribute information of the multiple instances is input as multiple paths into the diffusion network, and the instance processing network in the diffusion network performs multiple feature generation processes to obtain image features of the multiple instances; wherein the first intermediate image generated last time and the attribute information of one instance are used for one feature generation, and the attribute information of one instance is input as one path; The step of integrating the image features of the plurality of instances according to the position information of the plurality of instances included in the plurality of information groups to generate a target image includes: During each generation process of the diffusion network performing T1 generation processes, inputting the position information of the multiple instances and the image features of the multiple instances into a global integration network in the diffusion network to obtain a first intermediate feature; Inputting the first intermediate feature into other networks in the diffusion network to obtain a first intermediate image; wherein the global integration network is located after the instance processing network, and the other networks are located after the global integration network; The target image is determined based on a first intermediate image obtained by performing a T1-th generation process on the diffusion network.

10. The method according to claim 11, characterized in that Determining the target image based on the first intermediate image obtained by performing the T1th generation process on the diffusion network includes: During each subsequent T2 generation processes performed by the diffusion network, the second intermediate image generated last is input as one path and the attribute information of the plurality of instances is input as one path into the diffusion network, and the instance processing network performs feature generation based on the second intermediate image generated last and the attribute information of the plurality of instances to obtain second intermediate features; Inputting the second intermediate feature into the other network to obtain a second intermediate image; The second intermediate image obtained by performing the T-th generation process on the diffusion network is used as the target image; wherein T is the sum of T1 and T2.

11. The method according to claim 10, characterized in that Determining the target image based on the first intermediate image obtained by performing the T1th generation process on the diffusion network includes: The first intermediate image obtained by performing the T1th generation process on the diffusion network is used as the target image.

12. The method according to any one of claims 1 to 11, characterized in that The attribute information includes at least one of the following: an image or text; The location information includes at least one of the following: an image or text.

13. The method according to any one of claims 1 to 12, characterized in that The position information includes at least one of the following: instance outline information and instance bounding box information.

14. An image generating device, characterized in that: The device comprises: An acquisition module, configured to acquire a plurality of information groups, wherein each information group includes attribute information and location information of an instance; A first generating module, configured to generate an image feature of an instance according to attribute information of an instance contained in each information group of the plurality of information groups; The integration module is used to integrate the image features of the multiple instances according to the position information of the multiple instances included in the multiple information groups to generate a target image.

15. The device according to claim 14, characterized in that The device further comprises: a second generation module, configured to generate a plurality of mask images based on the position information of the plurality of instances; wherein the plurality of mask images correspond one-to-one to a plurality of pixels in the target image; the size of each mask image is the same as the size of the target image; the mask image corresponding to a first pixel is the mask image corresponding to the instance to which the first pixel belongs, and the first pixel is any pixel in the target image; and generate global integration information based on the plurality of mask images; The integration module is specifically configured to integrate the image features of the multiple instances according to the position information of the multiple instances and the global integration information to generate the target image.

16. The device according to claim 14, characterized in that The acquisition module is further used to obtain global description information of the target image; The device further comprises: A third generating module is used to generate the global image feature according to the global description information; The integration module is specifically configured to integrate the image features of the multiple instances according to the position information of the multiple instances and the global image features to generate the target image.

17. The device according to any one of claims 14 to 16, characterized in that The acquisition module is specifically configured to receive multiple information groups input by a user.

18. The device according to any one of claims 14 to 16, characterized in that The acquisition module is specifically used to perform text parsing based on the global description information of the target image to obtain attribute information of the multiple instances; and perform instance position prediction based on the global description information of the target image to obtain position information of the multiple instances.

19. An electronic device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 13.

20. A chip, characterized in that: The method comprises one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the method according to any one of claims 1 to 13 are performed.

21. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program runs on a computer or a processor, the computer or the processor executes the method according to any one of claims 1 to 13.

22. A computer program product, characterized in that The computer program product comprises computer instructions, which, when executed by a computer or a processor, cause the steps of the method according to any one of claims 1 to 13 to be performed.

Images