Multi-object image generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a multi-object image generation method. The multi-object image generation method comprises the steps of obtaining a first image generation instruction of each to-be-generated object, a segmentation image containing each to-be-generated object and a control image containing each to-be-generated object; performing first image generation processing based on the control chart and the first image generation instruction to obtain a first hidden feature map; second image generation processing is carried out based on the segmentation map and the first hidden feature map, a second hidden feature map is obtained, and the second hidden feature map corresponds to all the first image generation instructions; and decoding the second hidden feature map to obtain a target image containing a plurality of target generation objects. The individual features corresponding to the first image generation indication are obtained through the first image generation processing, and the overall features corresponding to the first image generation indication are obtained through the second image generation processing, so that the generation of the target object is more accurate, and the accuracy of multi-object image generation is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method, device, electronic device and storage medium for generating a multi-object image. Background Art

[0002] In the application field of image generation, AIGC (AI-Generated Content) is often unable to precisely control complex images and scenes. Especially for multi-object image generation, even with very detailed prompt words, it is difficult for the generated scene to completely correspond to the description of the prompt words. For example, if you want to generate a picture that contains football, basketball, volleyball, glass ball, and metal ball at the same time, if you directly enter "football, basketball, volleyball, glass ball, metal ball" in the prompt words, the final generated image will produce a confusing result. It may appear that each ball has a part of the characteristics of a ball, or the position of each ball cannot be controlled. Therefore, the existing multi-object image generation has the problem of poor generation accuracy. Summary of the invention

[0003] The embodiment of the present invention provides a multi-object image generation method, which aims to solve the problem of poor generation accuracy in the existing multi-object image generation. The first image generation processing is performed respectively through the first image generation instructions of each object to be generated and the control map containing all the objects to be generated, and the first latent feature map corresponding to each first image generation instruction is obtained. The second image generation processing is performed using the segmentation map containing each object to be generated and each first latent feature map to obtain the second latent feature map containing each target generation object. By decoding the second latent feature map, a target image containing each target generation object is obtained. The individual features corresponding to the first image generation instruction are obtained through the first image generation processing, and the overall features corresponding to the first image generation instruction are obtained through the second image generation processing, so that the generation of the target object is more accurate, and the accuracy of multi-object image generation is improved.

[0004] In a first aspect, an embodiment of the present invention provides a method for generating a multi-object image, the method comprising:

[0005] Acquire a first image generation instruction of each object to be generated, a segmentation map containing each of the objects to be generated, and a control map containing each of the objects to be generated;

[0006] Performing a first image generation process based on the control map and the first image generation instruction to obtain a first latent feature map, each of the first latent feature maps corresponding to one first image generation instruction;

[0007] Performing a second image generation process based on the segmentation map and the first latent feature map to obtain a second latent feature map, where the second latent feature map corresponds to all the first image generation instructions;

[0008] The second latent feature map is decoded to obtain a target image including a plurality of target generation objects, each target generation object corresponding to one of the first image generation instructions.

[0009] Optionally, the step of obtaining a first image generation instruction for each object to be generated, a segmentation map containing each object to be generated, and a control map containing each object to be generated includes:

[0010] Acquire a first image generation instruction for each object to be generated and a reference image including each of the objects to be generated;

[0011] Performing a first image processing on the reference image to obtain segmented regions of each of the objects to be generated, and determining a segmented image containing each of the objects to be generated based on the segmented regions;

[0012] A second image processing is performed on the reference image to obtain control sub-images in different dimensions, and a control image including each of the objects to be generated is determined based on the control sub-images.

[0013] Optionally, performing a first image generation process based on the control map and the first image generation instruction to obtain a first latent feature map includes:

[0014] The control map and each of the first image generation instructions are respectively subjected to a first diffusion process to obtain a plurality of first latent feature maps corresponding to the first image generation instructions.

[0015] Optionally, performing a second image generation process based on the segmentation map and the first latent feature map to obtain a second latent feature map includes:

[0016] Performing feature segmentation on the first latent feature map through the segmentation map to obtain segmentation features corresponding to the objects to be generated;

[0017] Splicing a plurality of the segmentation features to obtain a splicing feature map;

[0018] A second diffusion process is performed on the concatenated feature map to obtain a second latent feature map.

[0019] Optionally, performing feature segmentation on the first latent feature map by using the segmentation map to obtain a segmentation feature map corresponding one-to-one to the object to be generated includes:

[0020] Determining the segmentation area of ​​each of the objects to be generated in the segmentation map;

[0021] The first latent feature map is subjected to feature segmentation through the segmented area to obtain segmentation features corresponding one-to-one to the object to be generated.

[0022] Optionally, performing a second diffusion process on the concatenated feature map to obtain a second latent feature map includes:

[0023] Performing noise addition processing on the spliced ​​feature map to obtain a noise feature map;

[0024] A second diffusion process is performed on the noise feature map to obtain a second latent feature map.

[0025] Optionally, performing a second diffusion process on the noise feature map to obtain a second latent feature map includes:

[0026] determining a second image generation instruction based on the first image generation instruction;

[0027] The second image generation indication and the noise feature map are subjected to a second diffusion process to obtain a second latent feature map.

[0028] In a second aspect, an embodiment of the present invention further provides a multi-object image generation device, the multi-object image generation device comprising:

[0029] An acquisition module, used for acquiring a first image generation instruction of each object to be generated, a segmentation map containing each object to be generated, and a control map containing each object to be generated;

[0030] A first processing module, configured to perform first image generation processing based on the control map and the first image generation instruction to obtain first latent feature maps, each of the first latent feature maps corresponding to one first image generation instruction;

[0031] A second processing module, configured to perform a second image generation process based on the segmentation map and the first latent feature map to obtain a second latent feature map, wherein the second latent feature map corresponds to all the first image generation instructions;

[0032] The third processing module is used to decode the second latent feature map to obtain a target image including multiple target generation objects, each target generation object corresponds to one of the first image generation instructions.

[0033] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the multi-object image generation method provided in an embodiment of the present invention when executing the computer program.

[0034] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in the multi-object image generation method provided in the embodiment of the invention are implemented.

[0035] In an embodiment of the present invention, a first image generation indication of each object to be generated, a segmentation map containing each object to be generated, and a control map containing each object to be generated are obtained; a first image generation process is performed based on the control map and the first image generation indication to obtain a first latent feature map, each first latent feature map corresponds to a first image generation indication; a second image generation process is performed based on the segmentation map and the first latent feature map to obtain a second latent feature map, the second latent feature map corresponds to all the first image generation indications; the second latent feature map is decoded to obtain a target image containing multiple target generation objects, each target generation object corresponds to a first image generation indication. The first image generation processing is performed separately through the first image generation indication of each object to be generated and the control map containing all the objects to be generated, and the first latent feature map corresponding to each first image generation indication is obtained. The second image generation processing is performed using the segmentation map containing each object to be generated and the first latent feature map to obtain the second latent feature map containing each target generation object. By decoding the second latent feature map, the target image containing each target generation object is obtained. The individual features corresponding to the first image generation indication are obtained through the first image generation processing, and the overall features corresponding to the first image generation indication are obtained through the second image generation processing, so that the generation of the target object is more accurate and the accuracy of multi-object image generation is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 is a flow chart of a multi-object image generation method provided by an embodiment of the present invention;

[0038] Figure 2 It is a schematic diagram of the principle of the traditional diffusion model;

[0039] Figure 3 It is a schematic diagram of the principle of a stable diffusion provided by an embodiment of the present invention;

[0040] Figure 4is a schematic diagram of the principle of a diffusion model provided by an embodiment of the present invention;

[0041] Figure 5 is a schematic diagram of another diffusion model provided by an embodiment of the present invention;

[0042] Figure 6 is a structural schematic diagram of a multi-object image generating device provided in an embodiment of the present invention;

[0043] Figure 7 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] like Figure 1 As shown, Figure 1 1 is a method flow chart of a method for generating a multi-object image provided by an embodiment of the present invention. The method for generating a multi-object image comprises the steps of:

[0046] 101. Obtain a first image generation instruction of each object to be generated, a segmentation map containing each object to be generated, and a control map containing each object to be generated.

[0047] In an embodiment of the present invention, the multi-object image generation method can be deployed on an image generation platform, and the image generation platform is constructed based on a server or a server cluster. The image generation platform can respond to the user's image generation request, and generate a corresponding image according to the user's image generation request according to the multi-object image generation method deployed in the image generation platform, and return it to the user. The user's image generation request includes a first image generation instruction for each object to be generated, a segmentation map containing each object to be generated, and a control map containing each object to be generated. Specifically, the user can input the object to be generated, the first image generation instruction corresponding to the object to be generated, the segmentation map containing each object to be generated, and the control map containing each object to be generated through the interactive interface of the user terminal, and the user terminal packages the object to be generated, the first image generation instruction for the object to be generated, the segmentation map containing each object to be generated, and the control map containing each object to be generated into an image generation request and uploads it to the image generation platform.

[0048] The above-mentioned objects to be generated may be objects that the user wants to generate, and the number of the above-mentioned objects to be generated may be multiple. The above-mentioned objects to be generated may include foreground objects and background objects. The above-mentioned foreground objects may be entity objects, such as entities such as people, trees, balls, vehicles, buildings, etc. The number of the above-mentioned foreground objects may be multiple, and they may be entities of the same type or different types. The above-mentioned entities of the same type may include footballs, basketballs, volleyballs, glass balls, metal balls, etc., and the above-mentioned entities of different types may include people, trees, balls, vehicles, buildings, etc. The above-mentioned background objects may be scene objects, such as scenes such as beaches, indoors, and on grass.

[0049] Each object to be generated corresponds to a first image generation instruction, which is used to describe the object that the user wants to generate. For example, if the user wants to generate images including football, basketball, volleyball, glass ball and metal ball, the first image generation instruction may correspond to an image generation instruction for generating football, an image generation instruction for generating basketball, an image generation instruction for generating volleyball, an image generation instruction for generating glass ball and an image generation instruction for generating metal ball. When the object to be generated is a football, the corresponding first image generation instruction is an image generation instruction for generating football; when the object to be generated is a basketball, the corresponding first image generation instruction is an image generation instruction for generating basketball; when the object to be generated is a volleyball, the corresponding first image generation instruction is an image generation instruction for generating volleyball; when the object to be generated is a glass ball, the corresponding first image generation instruction is an image generation instruction for generating glass ball; and when the object to be generated is a metal ball, the corresponding first image generation instruction is an image generation instruction for generating metal ball.

[0050] The segmentation map includes the segmentation areas of each object to be generated. Specifically, the segmentation map may also be called a mask map. The segmentation areas of different objects are represented by different mask values. Each segmentation area of ​​an object corresponds to a mask value, so that the segmentation areas of different objects are distinguished.

[0051] The control map contains control features of each object to be generated, such as the edge contour of the object to be generated, the depth value of the object to be generated, the line drawing of the object to be generated, the posture of the object to be generated, the segmented area of ​​the object to be generated, etc. Specifically, the control map can be one or more control sub-maps including the edge map, depth map, line drawing map, posture map, segmentation map, etc. of each object to be generated. Among them, the edge map contains the edge contour of each object to be generated, the depth map contains the depth value of each object to be generated, the line drawing map contains the line drawing of each object to be generated, the posture map contains the posture of each object to be generated, and the segmentation map contains the segmented area of ​​each object to be generated. The control map can be used to control the generation position of each object to be generated.

[0052] 102. Perform a first image generation process based on the control graph and the first image generation instruction to obtain a first latent feature graph.

[0053] In an embodiment of the present invention, in the first image generation process, the generation position of each object to be generated can be controlled by a control diagram, and the color, shape and texture corresponding to each object to be generated can be generated by the first image generation instruction to obtain multiple first latent feature maps, each first latent feature map corresponding to a first image generation instruction.

[0054] For example, the three objects to be generated are maple tree, willow tree, peach tree from left to right, and the first image generation instructions correspond to red maple tree, green willow tree, and pink peach tree respectively. The generation positions of maple tree, willow tree, and peach tree are controlled by the control diagram. The first image generation instruction "red maple tree" generates three red maple trees from left to right, and the first latent feature map corresponding to "red maple tree" is obtained; the first image generation instruction "green willow tree" generates three green willow trees from left to right, and the first latent feature map corresponding to "green willow tree" is obtained; the first image generation instruction "pink peach tree" generates three pink peach trees from left to right, and the first latent feature map corresponding to "pink peach tree" is obtained.

[0055] In a possible embodiment, after obtaining multiple first latent feature maps, the multiple first latent feature maps can be directly decoded to obtain multiple generated images, each of which corresponds to a first image generation instruction, and each generated image includes multiple generated objects generated according to a first image generation instruction, and the multiple generated objects have similar features. For example, the first latent feature map corresponding to the above-mentioned "red maple tree" is decoded to obtain a generated image containing three "red maple trees"; the first latent feature map corresponding to the above-mentioned "green willow tree" is decoded to obtain a generated image containing three "green willow trees"; the first latent feature map corresponding to the above-mentioned "pink peach tree" is decoded to obtain a generated image containing three "pink peach trees".

[0056] In an embodiment of the present invention, the first image processing can be performed by a first image generation model. The first image generation model can be a non-decoding part of an image generation model such as a generative model and a diffusion model, and specifically can be a feature extraction part of an image generation model such as a generative model and a diffusion model. Only by using the feature extraction part, the control map and the first image generation instruction are encoded into a latent space to obtain a first latent feature map. It should be noted that the above-mentioned image generation models such as the generative model and the diffusion model can generate images according to the first image generation prompt to obtain a generated image corresponding to the first image generation prompt. The above-mentioned generative model can be an image generation model based on a GAN (Generative Adversarial Networks) network, and the above-mentioned diffusion model can be an image generation model based on stable diffusion. It should be noted that the above-mentioned image generation model based on the GAN network and the image generation model based on stable diffusion can be existing models that have been trained and do not need to be trained again.

[0057] 103. Perform a second image generation process based on the segmentation map and the first latent feature map to obtain a second latent feature map.

[0058] In an embodiment of the present invention, the above-mentioned segmentation map may include segmentation areas of each object to be generated, and each object to be generated corresponds to a segmentation area. Since each object to be generated corresponds to a first image generation indication, each object to be generated corresponds to a first latent feature map, and the second latent feature map corresponds to all first image generation indications.

[0059] The first latent feature map corresponding to each object to be generated can be subjected to feature extraction through the segmented area of ​​each object to be generated in the segmentation map to obtain the segmentation features of each object to be generated, and the second image generation process is performed based on the segmentation features of each object to be generated to obtain the second latent feature map.

[0060] The second image processing can be performed by a second image generation model, and the second image generation model can be a non-decoding part of an image generation model such as a generative model or a diffusion model. The second image generation model can be a feature extraction part of an image generation model such as a generative model or a diffusion model. Only by using the feature extraction part, the segmentation map and the first latent feature map are further encoded into a latent space to obtain a second latent feature map. Specifically, the feature extraction part can be used to further encode the segmentation features of each object to be generated into a latent space to obtain a second latent feature map.

[0061] It should be noted that the second image generation model may be the same as the first image generation model, or may be a different image generation model.

[0062] 104. Decode the second latent feature map to obtain a target image including multiple target generation objects.

[0063] In an embodiment of the present invention, the second latent feature map can be decoded by a decoder to obtain a target image containing multiple target generation objects. The target generation object corresponds to the first image generation instruction, and one target generation object corresponds to one first image generation instruction. It can be understood that the target generation object is the object to be generated by the first image generation instruction, and the object to be generated is the object intended to be generated by the user. The user adjusts the input first image generation instruction according to the object intended to be generated, and obtains the target data corresponding to the first image generation instruction through the above image generation method.

[0064] The decoder corresponds to the second image generation model. It should be noted that the first image generation model and the second image generation model can be understood as an encoder, through which the first image generation instruction and the control map are encoded into a first latent feature map, the first latent feature map and the segmentation map are encoded into a second latent feature map, and the second latent feature map is decoded into a target image through a decoder corresponding to the encoder.

[0065] In an embodiment of the present invention, a first image generation indication of each object to be generated, a segmentation map containing each object to be generated, and a control map containing each object to be generated are obtained; a first image generation process is performed based on the control map and the first image generation indication to obtain a first latent feature map, each first latent feature map corresponds to a first image generation indication; a second image generation process is performed based on the segmentation map and the first latent feature map to obtain a second latent feature map, the second latent feature map corresponds to all the first image generation indications; the second latent feature map is decoded to obtain a target image containing multiple target generation objects, each target generation object corresponds to a first image generation indication. The first image generation processing is performed separately through the first image generation indication of each object to be generated and the control map containing all the objects to be generated, and the first latent feature map corresponding to each first image generation indication is obtained. The second image generation processing is performed using the segmentation map containing each object to be generated and the first latent feature map to obtain the second latent feature map containing each target generation object. By decoding the second latent feature map, the target image containing each target generation object is obtained. The individual features corresponding to the first image generation indication are obtained through the first image generation processing, and the overall features corresponding to the first image generation indication are obtained through the second image generation processing, so that the generation of the target object is more accurate and the accuracy of multi-object image generation is improved.

[0066] Optionally, in the step of obtaining a first image generation indication of each object to be generated, a segmentation map containing each of the objects to be generated, and a control map containing each object to be generated, a first image generation indication of each object to be generated and a reference map containing each object to be generated can be obtained; a first image processing is performed on the reference map to obtain a segmentation area of ​​each object to be generated, and a segmentation map containing each object to be generated is determined based on the segmentation area; a second image processing is performed on the reference map to obtain control submaps in different dimensions, and a control map containing each object to be generated is determined based on the control submap.

[0067] In the embodiment of the present invention, the image generation request of the user includes a first image generation instruction of each object to be generated and a reference image containing each object to be generated.

[0068] The first image generation instruction may be a first image generation instruction of a word vector coding type, or a first image generation instruction of a non-word vector coding type, and the non-word vector coding type may include non-coding types such as text and speech. When the type of the first image generation instruction is a non-word vector coding type, the first image generation instruction is word vector coded to obtain a first image generation instruction of a word vector coding type.

[0069] Specifically, the user can input the object to be generated, the first image generation instruction corresponding to the object to be generated, and the reference image containing each object to be generated through the interactive interface of the user terminal. The user terminal packages the object to be generated, the first image generation instruction of the object to be generated, and the reference image containing each object to be generated into an image generation request and uploads it to the image generation platform. More specifically, the user can input a first image generation instruction for each object to be generated through the interactive interface of the user terminal. The above-mentioned reference image can be a hand-drawn picture, a rendering, or other image that can contain all the objects to be generated by the user. For example, the user's objects to be generated are maple trees, willow trees, and peach trees. For maple trees, the first image generation instruction input by the user is "red maple tree", for willow trees, the first image generation instruction input by the user is "green willow tree", and for peach trees, the first image generation instruction input by the user is "pink peach tree", and the reference image includes at least three trees.

[0070] After receiving the image generation request, the image generation platform parses the image generation request to obtain the first image generation instructions of each object to be generated and the reference image containing each object to be generated.

[0071] The first image processing may be an image segmentation processing, and the image segmentation processing may be performed through a reference image to obtain segmented areas of foreground objects and background objects, and the object to be generated may be matched with the segmented areas of the foreground objects and the background objects to obtain the segmented areas of the object to be generated, and the segmented areas of the non-objects to be generated may be merged into the segmented areas of the background objects, and the segmented areas include the foreground areas and the background areas. The reference image may be subjected to image segmentation processing through an image segmentation algorithm, and the image segmentation algorithm may be an image segmentation algorithm such as a threshold segmentation method, an edge segmentation method, a region segmentation method, and a morphological segmentation method. In the segmentation image, different objects to be generated may correspond to different mask values, and the mask values ​​are used to distinguish or mark different objects to be generated.

[0072] The above-mentioned second image processing can be at least one image processing such as edge detection, depth detection, line draft conversion, posture detection, etc., and can perform edge detection on the reference image to obtain the edge contour of the object to be generated, and combine them based on the edge contours of each object to be generated to obtain an edge map containing each object to be generated; perform depth detection on the reference image to obtain the depth value of the object to be generated, and perform pixel mapping based on the depth value of the object to be generated to obtain a depth map containing each object to be generated; perform line draft conversion on the reference image to obtain the line draft of the object to be generated, and combine them based on the line draft of the object to be generated to obtain a line draft map containing each object to be generated; perform posture detection on the reference image to obtain the posture of the object to be generated, and combine them based on the postures of each object to be generated to obtain a posture map containing each object to be generated.

[0073] After performing the first image processing and the second image processing on the reference image, control sub-images such as the edge map, depth map, line drawing map, posture map, segmentation map, etc. containing each object to be generated can be obtained, and at least one of the above sub-images can be determined as a control map containing each object to be generated.

[0074] Optionally, in the step of performing first image generation processing based on the control map and the first image generation indication to obtain a first latent feature map, the control map and each first image generation indication may be subjected to a first diffusion processing respectively to obtain a plurality of first latent feature maps corresponding to the first image generation indication.

[0075] In the embodiment of the present invention, the first diffusion processing may be performed on the control graph and each first image generation instruction respectively through stable diffusion, and the first diffusion processing may be performed on different first image generation instructions together with the same control graph.

[0076] It should be noted that if Figure 2 As shown, Figure 2This is a schematic diagram of the principle of the traditional diffusion model. The random seed is encoded into the latent space through the encoder to obtain the latent feature latent data. At the same time, there is also a decoder that can decode the latent data in the latent space into an image to obtain the target image. Stable diffusion is a latent diffusion model. Its diffusion process occurs in the latent space, so it is faster than the traditional diffusion model. Figure 3 As shown, Figure 3 It is a schematic diagram of the principle of a stable diffusion provided by an embodiment of the present invention. In the stable diffusion in the embodiment of the present invention, a random seed is encoded into a latent space by an encoder to obtain latent feature latent data, and the latent feature latent data is subjected to noise processing (noise) and denoising (denoise) to obtain a feature map of the same size as the original image, and the feature map is decoded into an image by a decoder (decoder) to obtain a target image.

[0077] Specifically, Figure 4 As shown, Figure 4 It is a schematic diagram of the principle of a diffusion model provided by an embodiment of the present invention. Based on the above-mentioned stable diffusion, the diffusion model sets multiple processing pipelines (stable diffusion pipelines), each processing pipeline includes a diffusion layer diffusion and a control layer controlnet, and one processing pipeline implements a diffusion process. Latent1 represents the input image of the first processing pipeline, latent2 represents the input image of the first processing pipeline, and latent3 represents the input image of the first processing pipeline. The above-mentioned input image is a random image, which can also be called a random seed, and can be a random image in the system database, and does not require user input.

[0078] certainly, Figure 4 The number of processing pipelines illustrated in the figure can be adjusted according to actual needs. The embodiment of the present invention does not limit the number of processing pipelines. For example, the number of processing pipelines can be set to be the same as the number of objects to be generated. For each processing pipeline, the input of each processing pipeline is a first image generation indication prompt, a control graph controlimage and a random graph latent, wherein the first image generation indication prompt and the random graph latent are directly input into the diffusion layer diffusion, and the control graph controlimage is input into the diffusion layer diffusion through the control layer controlnet. Figure 4In the first processing pipeline, the first diffusion processing is performed on the first image generation indication prompt1, the control graph control image and the random graph latent1, and the first latent feature map denoise latent1 corresponding to the first image generation indication prompt1 is obtained. The first diffusion processing is performed on the first image generation indication prompt2, the control graph control image and the random graph latent2 in the second processing pipeline, and the first latent feature map denoise latent2 corresponding to the first image generation indication prompt2 is obtained. The first diffusion processing is performed on the first image generation indication prompt3, the control graph control image and the random graph latent3 in the third processing pipeline, and the first latent feature map denoise latent3 corresponding to the first image generation indication prompt3 is obtained.

[0079] It should be noted that if Figure 5 As shown, Figure 5 It is a schematic diagram of the principle of another diffusion model provided by an embodiment of the present invention. After obtaining the first latent feature map denoise latent1, the first latent feature map denoise latent2 and the first latent feature map denoise latent3, the first latent feature map denoise latent1, the first latent feature map denoise latent2 and the first latent feature map denoise latent3 can be decoded respectively by a stable diffusion decoder to obtain a generated image corresponding to the first latent feature map denoise latent1, a generated image corresponding to the first latent feature map denoise latent2 and a generated image corresponding to the first latent feature map denoise latent3 respectively.

[0080] Optionally, in the step of performing a second image generation process based on the segmentation map and the first latent feature map to obtain a second latent feature map, feature segmentation can be performed on the first latent feature map through the segmentation map to obtain segmentation features that correspond one-to-one to the object to be generated; multiple segmentation features are spliced ​​to obtain a spliced ​​feature map; and a second diffusion process is performed on the spliced ​​feature map to obtain a second latent feature map.

[0081] In an embodiment of the present invention, the segmentation map includes various objects to be generated, each object to be generated corresponds to a first latent feature map, and the segmentation map can be mapped to each first latent feature map, and each object to be generated is segmented from the corresponding first latent feature map to obtain a segmentation feature corresponding to the object to be generated, and each object to be generated corresponds to a segmentation feature. Since the segmentation feature is segmented out in the corresponding first latent feature map, the segmentation feature of the object to be generated also has indication information corresponding to the first image generation indication.

[0082] After obtaining a plurality of segmentation features, the plurality of segmentation features are spliced ​​according to the positions of the objects to be generated in the segmentation map to obtain a spliced ​​feature map, wherein the spliced ​​feature map includes the segmentation features of the plurality of objects to be generated.

[0083] Furthermore, after obtaining the spliced ​​feature map, a second diffusion process can be performed on the spliced ​​feature map through stable diffusion to obtain a second latent feature map. It should be noted that in the second diffusion process, the input image generation indication and control map may not be required.

[0084] exist Figure 4 and Figure 5 In the figure, the first latent feature map denoise latent1, the first latent feature map denoise latent2 and the first latent feature map denoise latent3 are feature segmented through the segmentation map seg_map and then spliced ​​to obtain the spliced ​​feature map latent0, and then the second diffusion processing is performed on the spliced ​​feature map latent0 through stable diffusion to obtain the second latent feature map.

[0085] Optionally, in the step of performing feature segmentation on the first latent feature map through the segmentation map to obtain a segmentation feature map corresponding one-to-one to the object to be generated, the segmentation area of ​​each object to be generated can be determined in the segmentation map; and feature segmentation is performed on the first latent feature map through the segmentation area to obtain segmentation features corresponding one-to-one to the object to be generated.

[0086] In an embodiment of the present invention, the above-mentioned segmentation map includes segmentation areas of each object to be generated, and the segmentation areas of different objects to be generated correspond to different mask values. The segmentation map is scaled to the same size as the first latent feature map, and the scaled segmentation areas are mapped to the corresponding first latent feature map according to the objects to be generated. The area corresponding to the scaled segmentation area in the first latent feature map is extracted, and the extracted area is determined as the segmentation feature of the corresponding object to be generated. The segmentation features are spliced ​​according to the scaled segmentation map to obtain a spliced ​​feature map of the same size as the first latent feature map.

[0087] like Figure 5As shown, the first latent feature map denoise latent1 corresponds to the object to be generated on the left in the segmentation map seg_map, the first latent feature map denoise latent2 corresponds to the object to be generated in the middle of the segmentation map seg_map, and the first latent feature map denoise latent3 corresponds to the object to be generated on the right in the segmentation map seg_map. The first latent feature map denoise latent1 is segmented by the object to be generated on the left in the segmentation map seg_map to obtain the segmentation features of the object to be generated on the left, the first latent feature map denoise latent2 is segmented by the object to be generated in the middle of the segmentation map seg_map to obtain the segmentation features of the object to be generated in the middle, and the first latent feature map denoise latent1 is segmented by the object to be generated on the right in the segmentation map seg_map to obtain the segmentation features of the object to be generated on the right. The segmentation features of the object to be generated on the left, the segmentation features of the object to be generated in the middle, and the segmentation features of the object to be generated on the right are spliced ​​to obtain the spliced ​​feature map latent0.

[0088] Optionally, in the step of performing a second diffusion process on the concatenated feature map to obtain a second latent feature map, a noise addition process may be performed on the concatenated feature map to obtain a noise feature map; and a second diffusion process may be performed on the noise feature map to obtain a second latent feature map.

[0089] In an embodiment of the present invention, after obtaining the spliced ​​feature map, noise addition processing can be performed on the spliced ​​feature map to increase the size of the spliced ​​feature map to obtain a noise feature map, and the noise feature map can be subjected to a second diffusion processing through stable diffusion to obtain a second latent feature map.

[0090] like Figure 4 As shown, after obtaining the spliced ​​feature map latent0, the spliced ​​feature map latent0 is subjected to noise addition processing to obtain the noise feature map noise latent0, the noise feature map noise latent0 is input into diffusion for second diffusion processing to obtain the second latent feature map denoise latent0, the second latent feature map denoise latent0 is decoded by the stable diffusion decoder to obtain the final image, and the above final image is determined as the target image.

[0091] Furthermore, the above-mentioned noise addition process may be a noise addition process method used when training stable diffusion, and the number of noise additions may be determined according to actual needs.

[0092] Optionally, in the step of performing a second diffusion process on the noise feature map to obtain a second latent feature map, the second image generation indication can be determined based on the first image generation indication; and the second image generation indication and the noise feature map are subjected to a second diffusion process to obtain a second latent feature map.

[0093] In an embodiment of the present invention, multiple first image generation instructions can be integrated to obtain a second image generation instruction, the second image generation instruction includes all or part of the first image generation instructions, and the second image generation instruction and the noise feature map are subjected to a second diffusion process to obtain a second latent feature map. In this way, the guiding role of the image generation instruction for image generation can be further strengthened, so that the generated target object is closer to the user's intention.

[0094] In a possible embodiment, a second diffusion process may be performed based on the second image generation instruction, the control map and the noise feature map to obtain a second latent feature map. The guiding role of the image generation instruction on image generation can be further enhanced, while the control effect of the control map on the target object can be enhanced, so that the generated target object is closer to the user's intention.

[0095] Optionally, the embodiment of the present invention also provides another image generation method, specifically combining Figure 3 and Figure 4 To explain, in order to obtain a more controllable generation result, a control layer controlnet structure is adopted. The control layer controlnet can use one or more of the edge map, depth map, line drawing map, posture map, etc. as control input to make the edge, depth, line drawing, posture, etc. of the stable diffusion output image consistent with the control input.

[0096] Even with controlnet, if the scene is too complex, and there are many and independent descriptive words, it is difficult to obtain the expected generated image. Therefore, the embodiment of the present invention also proposes to use different prompt words as conditions to obtain multiple results, retain the latent space feature maps corresponding to these results, respectively splice the feature maps generated by the prompts of different image generation instructions to generate a new feature map z0, and then add a specific number of steps of noise to this feature map and diffuse it to obtain the final result. The specific process is as follows:

[0097] Given a controlnet input, the input is a control image, which can be multiple or one. In addition, a segmentation map seg_map corresponding to the input is also required for subsequent feature stitching. In addition, the segmentation map seg_map itself can also be used as a control input. Regardless of whether the segmentation map is used as the input of the controlnet, the segmentation map seg_map is required for latent feature stitching. For example, prepare a picture of 3 circles as the input of the canny controlnet (edge ​​map). In addition, each circle is used as a segmentation area, plus the background area, and a segmentation map containing 4 segmentation areas can be obtained. It can be determined that the user's intention is to get an image in which the first ball is a football, the second ball is a basketball, the third ball is a glass ball, and the background is a grass.

[0098] The stable diffusion pipeline is run several times, the number of times depends on the number of segmented regions. The control image input to each pipeline remains the same, and the image generation prompt for each input is independent of each other. For example, three circle images are used as the input of the canny controlnet, but the image generation prompt uses "football", "basketball", and "glass ball" respectively. In addition, there is an optional pipeline for generating background. The model for generating background can be generated without using the controlnet. You can directly input the image generation prompt of the desired background effect, such as "grassland". Of course, you can also use the result of any previous generation as the background without generating an additional background.

[0099] The latent feature maps generated by different image generation prompts are cropped according to the segmentation map. For example, the latent feature map guided by football only takes the area of ​​the first circle, the latent feature map guided by basketball only takes the area of ​​the second circle, the latent feature map guided by glass ball takes the area of ​​the third circle, and the background area uses the latent feature map guided by grassland. Then, a new latent feature map latent0 can be spliced ​​together. Furthermore, if the new feature map is directly used to decode the image, there may be disharmony, especially the splicing edge will be distorted, so the noise and denoise process is required.

[0100] Add noise to the spliced ​​feature map latent0, and the adding method is consistent with the method used for training stable diffusion. Pass the spliced ​​feature map noise latent0 with added noise through a new stable diffusion pipeline, and you can get a generated image that meets expectations in each area. It is worth mentioning that the number of steps for adding noise should not be too large, otherwise the feature map will be too close to the noise map, causing overfitting, so that the regenerated image has nothing to do with the feature map. Through the above embodiment, you can get an image of a football, a basketball and a glass ball placed on the grass. This can avoid the problem of being unable to determine the position of each ball, or each ball having some of the features of all balls at the same time.

[0101] By implementing the present invention, operations can be performed in the latent feature space generated by the diffusion model, and latent features generated by multiple different prompts or even different models can be combined. At the same time, noise is added to degrade the combined latent feature map and then diffused to obtain the final result, effectively avoiding the unreasonable situation caused by direct patchwork. The final effect achieved is that users can use different prompts and controlnet strategies for different areas, and then integrate all the results to obtain a natural and reasonable effect. Under this process, image generation is more controllable.

[0102] like Figure 6 As shown, an embodiment of the present invention provides a multi-object image generation device, the multi-object image generation device comprising:

[0103] An acquisition module 601 is used to acquire a first image generation instruction of each object to be generated, a segmentation map containing each object to be generated, and a control map containing each object to be generated;

[0104] A first processing module 602 is configured to perform first image generation processing based on the control map and the first image generation instruction to obtain first latent feature maps, where each first latent feature map corresponds to one first image generation instruction;

[0105] A second processing module 603 is configured to perform a second image generation process based on the segmentation map and the first latent feature map to obtain a second latent feature map, where the second latent feature map corresponds to all the first image generation instructions;

[0106] The third processing module 604 is used to decode the second latent feature map to obtain a target image including multiple target generation objects, each target generation object corresponds to one of the first image generation instructions.

[0107] Optionally, the acquisition module 601 includes:

[0108] An acquisition submodule, used to acquire a first image generation instruction of each object to be generated and a reference image including each of the objects to be generated;

[0109] A first processing submodule, configured to perform a first image processing on the reference image to obtain segmented regions of each of the objects to be generated, and determine a segmented image containing each of the objects to be generated based on the segmented regions;

[0110] The second processing submodule is used to perform a second image processing on the reference image to obtain control sub-images in different dimensions, and determine a control image including each of the objects to be generated based on the control sub-images.

[0111] Optionally, the first processing module 602 is further used to perform first diffusion processing on the control map and each of the first image generation instructions respectively to obtain multiple first latent feature maps corresponding to the first image generation instructions.

[0112] Optionally, the second processing module 603 includes:

[0113] A segmentation submodule, used to perform feature segmentation on the first latent feature map through the segmentation map to obtain segmentation features corresponding to the object to be generated;

[0114] A splicing submodule, used for splicing a plurality of the segmentation features to obtain a splicing feature map;

[0115] The third processing submodule is used to perform a second diffusion process on the spliced ​​feature map to obtain a second latent feature map.

[0116] Optionally, the segmentation submodule includes:

[0117] A determination unit, configured to determine the segmentation area of ​​each of the objects to be generated in the segmentation map;

[0118] The segmentation unit is used to perform feature segmentation on the first latent feature map through the segmentation area to obtain segmentation features corresponding to the object to be generated.

[0119] Optionally, the third processing submodule includes:

[0120] A noise adding unit, used for performing noise adding processing on the spliced ​​feature map to obtain a noise feature map;

[0121] A processing unit is used to perform a second diffusion process on the noise feature map to obtain a second latent feature map.

[0122] Optionally, the processing unit includes:

[0123] a determining subunit, configured to determine a second image generation instruction based on the first image generation instruction;

[0124] The processing subunit is used to perform a second diffusion process on the second image generation indication and the noise feature map to obtain a second latent feature map.

[0125] The multi-object image generation device provided in the embodiment of the present invention can implement each process implemented by the multi-object image generation method in the above method embodiment and can achieve the same beneficial effects. To avoid repetition, it will not be described here.

[0126] See also Figure 7 , Figure 7 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 7 As shown, it includes: a memory 702, a processor 701, and a computer program of a multi-object image generation method stored in the memory 702 and executable on the processor 701, wherein:

[0127] The processor 701 is used to call the computer program stored in the memory 702 to perform the following steps:

[0128] Acquire a first image generation instruction of each object to be generated, a segmentation map containing each of the objects to be generated, and a control map containing each of the objects to be generated;

[0129] Performing a first image generation process based on the control map and the first image generation instruction to obtain a first latent feature map, each of the first latent feature maps corresponding to one first image generation instruction;

[0130] Performing a second image generation process based on the segmentation map and the first latent feature map to obtain a second latent feature map, where the second latent feature map corresponds to all the first image generation instructions;

[0131] The second latent feature map is decoded to obtain a target image including a plurality of target generation objects, each target generation object corresponding to one of the first image generation instructions.

[0132] Optionally, the acquiring of the first image generation instruction of each object to be generated, the segmentation map containing each object to be generated, and the control map containing each object to be generated executed by the processor 701 includes:

[0133] Acquire a first image generation instruction for each object to be generated and a reference image including each of the objects to be generated;

[0134] Performing a first image processing on the reference image to obtain segmented regions of each of the objects to be generated, and determining a segmented image containing each of the objects to be generated based on the segmented regions;

[0135] A second image processing is performed on the reference image to obtain control sub-images in different dimensions, and a control image including each of the objects to be generated is determined based on the control sub-images.

[0136] Optionally, the processor 701 performs the first image generation processing based on the control map and the first image generation instruction to obtain the first latent feature map, including:

[0137] The control map and each of the first image generation instructions are respectively subjected to a first diffusion process to obtain a plurality of first latent feature maps corresponding to the first image generation instructions.

[0138] Optionally, the processor 701 performs the second image generation process based on the segmentation map and the first latent feature map to obtain the second latent feature map, including:

[0139] Performing feature segmentation on the first latent feature map through the segmentation map to obtain segmentation features corresponding to the objects to be generated;

[0140] Splicing a plurality of the segmentation features to obtain a splicing feature map;

[0141] A second diffusion process is performed on the concatenated feature map to obtain a second latent feature map.

[0142] Optionally, the processor 701 performs feature segmentation on the first latent feature map by using the segmentation map to obtain a segmentation feature map corresponding one-to-one to the object to be generated, including:

[0143] Determining the segmentation area of ​​each of the objects to be generated in the segmentation map;

[0144] The first latent feature map is subjected to feature segmentation through the segmented area to obtain segmentation features corresponding one-to-one to the object to be generated.

[0145] Optionally, the processor 701 performs a second diffusion process on the concatenated feature map to obtain a second latent feature map, including:

[0146] Performing noise addition processing on the spliced ​​feature map to obtain a noise feature map;

[0147] A second diffusion process is performed on the noise feature map to obtain a second latent feature map.

[0148] Optionally, the processor 701 performs a second diffusion process on the noise feature map to obtain a second latent feature map, including:

[0149] determining a second image generation instruction based on the first image generation instruction;

[0150] The second image generation indication and the noise feature map are subjected to a second diffusion process to obtain a second latent feature map.

[0151] The electronic device provided by the embodiment of the present invention can implement each process implemented by the multi-object image generation method in the above method embodiment, and can achieve the same beneficial effect. To avoid repetition, it will not be described here.

[0152] An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the computer program implements the various processes of the multi-object image generation method or the application-side multi-object image generation method provided in the embodiment of the present invention, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0153] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0154] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A method for generating a multi-object image, characterized in that: include: Acquire a first image generation instruction of each object to be generated, a segmentation map containing each of the objects to be generated, and a control map containing each of the objects to be generated; Performing a first image generation process based on the control map and the first image generation instruction to obtain a first latent feature map, each of the first latent feature maps corresponding to one first image generation instruction; Performing a second image generation process based on the segmentation map and the first latent feature map to obtain a second latent feature map, where the second latent feature map corresponds to all the first image generation instructions; The second latent feature map is decoded to obtain a target image including a plurality of target generation objects, each target generation object corresponding to one of the first image generation instructions.

2. The method according to claim 1, characterized in that The step of obtaining a first image generation instruction of each object to be generated, a segmentation map containing each object to be generated, and a control map containing each object to be generated includes: Acquire a first image generation instruction for each object to be generated and a reference image including each of the objects to be generated; Performing a first image processing on the reference image to obtain segmented regions of each of the objects to be generated, and determining a segmented image containing each of the objects to be generated based on the segmented regions; A second image processing is performed on the reference image to obtain control sub-images in different dimensions, and a control image including each of the objects to be generated is determined based on the control sub-images.

3. The method according to claim 1, characterized in that The performing a first image generation process based on the control graph and the first image generation instruction to obtain a first latent feature graph includes: The control map and each of the first image generation instructions are respectively subjected to a first diffusion process to obtain a plurality of first latent feature maps corresponding to the first image generation instructions.

4. The method according to claim 1, characterized in that The performing a second image generation process based on the segmentation map and the first latent feature map to obtain a second latent feature map includes: Performing feature segmentation on the first latent feature map through the segmentation map to obtain segmentation features corresponding to the objects to be generated; Splicing a plurality of the segmentation features to obtain a splicing feature map; A second diffusion process is performed on the concatenated feature map to obtain a second latent feature map.

5. The method according to claim 4, characterized in that The step of performing feature segmentation on the first latent feature map by using the segmentation map to obtain a segmentation feature map corresponding to the object to be generated comprises: Determining the segmentation area of ​​each of the objects to be generated in the segmentation map; The first latent feature map is subjected to feature segmentation through the segmented area to obtain segmentation features corresponding one-to-one to the object to be generated.

6. The method according to claim 4, characterized in that The step of performing a second diffusion process on the concatenated feature map to obtain a second latent feature map includes: Performing noise addition processing on the spliced ​​feature map to obtain a noise feature map; A second diffusion process is performed on the noise feature map to obtain a second latent feature map.

7. The method according to claim 6, characterized in that The performing a second diffusion process on the noise feature map to obtain a second latent feature map includes: determining a second image generation instruction based on the first image generation instruction; The second image generation indication and the noise feature map are subjected to a second diffusion process to obtain a second latent feature map.

8. A multi-object image generation device, characterized in that: The multi-object image generating device comprises: An acquisition module, used for acquiring a first image generation instruction of each object to be generated, a segmentation map containing each object to be generated, and a control map containing each object to be generated; A first processing module, configured to perform first image generation processing based on the control map and the first image generation instruction to obtain first latent feature maps, each of the first latent feature maps corresponding to one first image generation instruction; A second processing module, configured to perform a second image generation process based on the segmentation map and the first latent feature map to obtain a second latent feature map, wherein the second latent feature map corresponds to all the first image generation instructions; The third processing module is used to decode the second latent feature map to obtain a target image including multiple target generation objects, each target generation object corresponds to one of the first image generation instructions.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps in the multi-object image generation method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the multi-object image generation method according to any one of claims 1 to 7 are implemented.

Images