Image generation method and apparatus, electronic device, storage medium, and program

Abstract

The present disclosure provides an image generation method, device, electronic equipment, storage medium and program. A specific embodiment of the method comprises: obtaining a target guidance information tensor for guiding the generation of a target image; arranging the target guidance information tensor into a target object instance set; determining a target complexity of the target object instance set; calling a target noise schedule and a target time step matching the target complexity; using a trained diffusion probability model to denoise a preset random noise image according to the target noise schedule and the target time step, and generating a target slot refinement patch corresponding to each target object instance in the denoised random noise image to obtain the target image. This embodiment can accurately generate images of complex scenes.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of artificial intelligence, and in particular to an image generation method and device, electronic equipment, storage medium and program. BACKGROUND

[0002] In the process of generating an image by an artificial intelligence model, in the current way of generating an image according to input guidance information, attention is often focused on generating an image of a single form. In the process of generating such an image, the guidance information, target relationship, occlusion relationship and structural consistency input to the artificial intelligence model are usually simple.

[0003] However, when a more complex image needs to be generated, more complex guidance information needs to be input, such as multi-target relationship, multi-occlusion level, global layout and light consistency. It is often expected to generate a more complex image. However, the current way of generating an image is usually difficult to generate a more accurate and high-quality image in a complex scene, i.e., a complex guidance information situation. For example, in a complex scene, problems such as object boundary rupture, element crowding, relationship conflict and light inconsistency are prone to occur. SUMMARY

[0004] Therefore, embodiments of the present disclosure provide an image generation method, device, electronic equipment, storage medium and program.

[0005] In a first aspect, embodiments of the present disclosure provide an image generation method, which comprises:

[0006] obtaining target guidance information tensor for guiding the generation of a target image;

[0007] organizing the target guidance information tensor into a target object instance set, the target object instance set comprising at least one multi-modal described target object instance;

[0008] determining a target complexity of the target object instance set based on each target object instance;

[0009] retrieving a target noise schedule and a target time step matching the target complexity, the target noise schedule representing the denoising intensity in each denoising step, and the target time step representing the strategy of allocating denoising steps in the denoising process;

[0010] using a diffusion probability model trained to denoise a preset random noise image according to the target noise schedule and the target time step, and generating a target slot refinement patch corresponding to each target object instance in the denoised random noise image to obtain the target image.

[0011] In some optional embodiments, the target guidance information tensor at least comprises a two-dimensional layout frame and a semantic mask under the same coordinate and scale.

[0012] Correspondingly, the target guidance information tensor is sorted into a target object instance set, including:

[0013] Under the same coordinate and scale, the preset target region is segmented and de-coincided using a semantic mask and a two-dimensional layout box to obtain at least one multi-modal description of a target object instance;

[0014] Each target object instance is grouped into a target object instance set.

[0015] In some optional embodiments, based on each target object instance, a target complexity of the target object instance set is determined, including:

[0016] The instance quantity of the target object instance in the target object instance set, the occlusion level index of each target object instance occluding each other, the layout dispersion between each target object instance, the depth variance of the depth data of each target object instance, and the category entropy of the category of each target object instance are determined.

[0017] The instance quantity, the occlusion level index, the layout dispersion, the depth variance, and the category entropy are weighted and summed to obtain the target complexity.

[0018] In some optional embodiments, the trained diffusion probability model includes a trained global layout diffusion sub-model and a trained entity slot diffusion sub-model, the target guidance information tensor includes a scene graph relationship representing an occlusion ordered relationship between each target object instance; and

[0019] Correspondingly, the preset random noise image is denoised according to the target noise schedule and the target time step using the trained diffusion probability model, and a target slot refinement patch corresponding to each target object instance is generated in the denoised random noise image to obtain a target image, including:

[0020] The preset random noise image is denoised according to the target noise schedule and the target time step using the trained global layout diffusion sub-model to obtain a global target coarse denoising image corresponding to the target guidance information tensor;

[0021] The processing order of each target slot is determined according to the occlusion ordered relationship in the scene graph relationship, and each target slot is used to describe the position of the corresponding target object instance in the global target coarse denoising image;

[0022] Using the trained entity slot diffusion sub-model, a target slot refinement patch corresponding to the current target slot is generated according to the processing order, and the current target slot refinement patch is superimposed with the current base image to obtain a next base image, until the target slot refinement patch corresponding to each target slot is completed superimposed to obtain the target image;

[0023] The first base image is obtained by superimposing a global target coarse denoising image and a target slot refinement patch corresponding to a first target slot in the processing sequence.

[0024] In some optional embodiments, before denoising the preset random noise image according to the target noise schedule and the target time step using the trained diffusion probability model, the method further comprises:

[0025] Obtaining a plurality of preset training images, and a training instance set corresponding to each training image and a training noise image corresponding to each training image, each training instance set comprising at least one multi-modal described training object instance;

[0026] Denoising each training noise image using the diffusion probability model to be trained, and generating a training slot refinement patch corresponding to each training object instance in each denoised training noise image, to obtain a corresponding reconstructed image;

[0027] Determining a hybrid loss value between each training image and the corresponding reconstructed image, the hybrid loss value representing the difference between each training image and the corresponding reconstructed image;

[0028] Determining whether the hybrid loss value is less than or equal to a preset loss threshold value after continuous multiple rounds of training;

[0029] In response to determining that the hybrid loss value is not less than or equal to the preset loss threshold value after continuous multiple rounds of training, adjusting the model parameters of the diffusion probability model to be trained based on the hybrid loss value, and using the adjusted diffusion probability model to perform the next round of training until the hybrid loss value is less than or equal to the preset loss threshold value after continuous multiple rounds of training, to obtain the trained diffusion probability model.

[0030] In some optional embodiments, before denoising each training noise image using the diffusion probability model to be trained, the method further comprises:

[0031] Obtaining a training complexity corresponding to each training image, a global calibration coarse denoising image corresponding to each training image, and a calibration slot corresponding to each training image;

[0032] Injecting noise into each global calibration coarse denoising image to obtain a corresponding training noise image;

[0033] Retrieving a training noise schedule and a training time step matching the corresponding training complexity;

[0034] The diffusion probability model to be trained comprises a global layout diffusion sub-model to be trained and an entity slot diffusion sub-model to be trained;

[0035] Correspondingly, each training noise image is denoised by using the diffusion probability model to be trained, and a training slot refinement patch corresponding to each training object instance is generated in each denoised training noise image, to obtain a corresponding reconstructed image, including:

[0036] The corresponding training noise image is denoised by using the global layout diffusion sub-model to be trained according to the corresponding training noise schedule and the corresponding training time step, to obtain a corresponding global training coarse denoising image;

[0037] Each training slot refinement patch corresponding to each calibrated slot is generated by using the entity slot diffusion sub-model to be trained, and the corresponding global training coarse denoising image is taken as a substrate, and each corresponding training slot refinement patch is superimposed, to obtain a corresponding reconstructed image.

[0038] In some optional embodiments, after the preset plurality of training images are obtained, the method further includes: obtaining a scene graph relationship of each training image;

[0039] Correspondingly, a mixed loss value between each training image and the corresponding reconstructed image is determined, including:

[0040] A structural consistency loss value is determined by using each global calibrated coarse denoising image and the corresponding global training coarse denoising image;

[0041] A reconstruction consistency loss value is determined by using each training image and the corresponding reconstructed image;

[0042] A boundary continuity loss value is determined by using a gradient of each reconstructed image and a gradient of each corresponding training slot refinement patch;

[0043] A relationship consistency loss value is determined by using a scene graph relationship of each training image and each reconstructed object instance in the corresponding reconstructed image;

[0044] The structural consistency loss value, the reconstruction consistency loss value, the boundary continuity loss value and the relationship consistency loss value are weighted and summed to obtain the mixed loss value.

[0045] In a second aspect, embodiments of the present disclosure provide an image generation device, which includes:

[0046] An acquisition module configured to acquire target guidance information tensor for guiding generation of a target image;

[0047] An arrangement module configured to arrange the target guidance information tensor into a target object instance set, the target object instance set including at least one multi-modal described target object instance;

[0048] A complexity determination module configured to determine a target complexity of the target object instance set based on each target object instance;

[0049] The calling module is configured to call a target noise schedule and a target time step matching the target complexity, the target noise schedule representing a denoising intensity in each denoising step, and the target time step representing a strategy of allocating the denoising step in the denoising process;

[0050] The generating module is configured to use the trained diffusion probability model to denoise the preset random noise image according to the target noise schedule and the target time step, and generate target slot refinement patches corresponding to each target object instance in the denoised random noise image, to obtain a target image.

[0051] In some optional embodiments, the target guidance information tensor at least includes a two-dimensional layout frame and a semantic mask at the same coordinate and scale; accordingly, the sorting module is further configured to:

[0052] segment and de-coincide the preset target region using the semantic mask and the two-dimensional layout frame at the same coordinate and scale, to obtain at least one multi-modal description of the target object instance;

[0053] Each target object instance is grouped into a target object instance set.

[0054] In some optional embodiments, the complexity determination module is further configured to:

[0055] determine an instance number of the target object instance in the target object instance set, an occlusion level index of mutual occlusion of each target object instance, a layout dispersion between the target object instances, a depth variance of depth data of each target object instance, and a category entropy of a category of each target object instance;

[0056] perform weighted summation on the instance number, the occlusion level index, the layout dispersion, the depth variance, and the category entropy, to obtain the target complexity.

[0057] In some optional embodiments, the trained diffusion probability model includes a trained global layout diffusion sub-model and a trained entity slot diffusion sub-model, and the target guidance information tensor includes a scene graph relationship representing an occlusion order relationship between the target object instances.

[0058] Accordingly, the generating module is further configured to:

[0059] use the trained global layout diffusion sub-model to denoise the preset random noise image according to the target noise schedule and the target time step, to obtain a global target coarse denoising image corresponding to the target guidance information tensor;

[0060] The processing order of each target slot is determined according to the occlusion order in the scene graph relationship, and each target slot is used to describe the position of the corresponding target object instance in the global target coarse denoising image.

[0061] The trained entity slot diffusion sub-model is used to generate the target slot refinement patch corresponding to the current target slot according to the processing order, and the current target slot refinement patch is superimposed on the current base image to obtain the next base image, until the target slot refinement patches corresponding to all target slots are superimposed to obtain the target image.

[0062] The first base image is obtained by superimposing the global target coarse denoising image and the target slot refinement patch corresponding to the first target slot in the processing order.

[0063] In some optional embodiments, the generating device further includes a training module configured to:

[0064] Before denoising the preset random noise image according to the target noise schedule and the target time step using the trained diffusion probability model, the following steps are performed:

[0065] A plurality of preset training images, a training instance set corresponding to each training image, and a training noise image corresponding to each training image are obtained, and each training instance set includes at least one training object instance described in multiple modalities.

[0066] Each training object instance is denoised using the trained diffusion probability model, and a training slot refinement patch corresponding to each training object instance is generated in each denoised training noise image to obtain a corresponding reconstructed image.

[0067] A mixed loss value between each training image and the corresponding reconstructed image is determined, and the mixed loss value represents the difference between each training image and the corresponding reconstructed image.

[0068] It is determined whether the mixed loss value is less than or equal to a preset loss threshold value after continuous multiple rounds of training.

[0069] In response to determining that the mixed loss value is not less than or equal to the preset loss threshold value after continuous multiple rounds of training, the model parameters of the trained diffusion probability model are adjusted based on the mixed loss value, and the next round of training is performed using the adjusted diffusion probability model until the mixed loss value is less than or equal to the preset loss threshold value after continuous multiple rounds of training, thereby obtaining the trained diffusion probability model.

[0070] Before denoising each training noise image using the trained diffusion probability model, the following steps can be performed:

[0071] The training complexity corresponding to each training image, the global calibration coarse denoising image corresponding to each training image, and the calibration slot corresponding to each training image are obtained.

[0072] injecting noise to each global calibration coarse denoising image to obtain a corresponding training noise image;

[0073] retrieve a training noise schedule and a training time step matching the corresponding training complexity;

[0074] The diffusion probability model to be trained includes a global layout diffusion sub-model to be trained and an entity slot diffusion sub-model to be trained;

[0075] Correspondingly, each training noise image is denoised by using the diffusion probability model to be trained, and a training slot refinement patch corresponding to each training object instance is generated in each denoised training noise image to obtain a corresponding reconstructed image, including:

[0076] The corresponding training noise image is denoised by using the global layout diffusion sub-model to be trained according to the corresponding training noise schedule and the corresponding training time step to obtain a corresponding global training coarse denoising image;

[0077] Each training slot refinement patch corresponding to each calibration slot is generated by using the entity slot diffusion sub-model to be trained, and the corresponding global training coarse denoising image is taken as a substrate, and each corresponding training slot refinement patch is superimposed to obtain a corresponding reconstructed image.

[0078] In some optional embodiments, after obtaining the plurality of preset training images, a scene graph relationship of each training image is obtained;

[0079] Correspondingly, a mixed loss value between each training image and the corresponding reconstructed image is determined, including:

[0080] A structural consistency loss value is determined by using each global calibration coarse denoising image and the corresponding global training coarse denoising image;

[0081] A reconstruction consistency loss value is determined by using each training image and the corresponding reconstructed image;

[0082] A boundary continuity loss value is determined by using a gradient of each reconstructed image and a gradient of each corresponding training slot refinement patch;

[0083] A relationship consistency loss value is determined by using a scene graph relationship of each training image and each reconstructed object instance in the corresponding reconstructed image;

[0084] The structural consistency loss value, the reconstruction consistency loss value, the boundary continuity loss value and the relationship consistency loss value are weighted and summed to obtain the mixed loss value.

[0085] In a third aspect, embodiments of the present disclosure provide an electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, which when executed by the one or more processors, cause the one or more processors to implement the method as described in any implementation manner of the first aspect.

[0086] In a fourth aspect, embodiments of the present disclosure provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by one or more processors, implements the method as described in any implementation manner of the first aspect.

[0087] In a fifth aspect, embodiments of the present disclosure provide a computer program product comprising computer programs / instructions, which when executed by a processor, implement the method as described in any implementation manner of the first aspect.

[0088] To solve the problem that it is difficult to generate high-quality images in a complex scene, i.e., a complex guidance information case, the image generation method, device, electronic device, storage medium and program provided by the embodiments of the present disclosure align and fuse different types of guidance information under a unified coordinate and scale, thereby providing a structured and scale-unified target guidance information tensor for subsequent processing, ensuring consistency between different guidance information, providing a basis for generating a target image that meets the guidance information, and effectively avoiding the problem of deviation in the generated target image due to the non-uniform data format or reference of the input guidance information.

[0089] Further, by quantitatively analyzing the instance quantity of the target object instances in the target object instance set, the occlusion level index of each target object instance, the layout divergence between the target object instances, the depth variance of the depth data of each target object instance, and the category entropy of the category of each target object instance, a target complexity that is adapted to the target object instance set can be generated, and the complexity of the target image to be generated can be determined.

[0090] Further, based on the target complexity, a target noise schedule and a target time step suitable for generating the current target image can be accurately adapted, and the self-adaptive effect of the complexity can be achieved, so that for a simple scene, the generation can be quickly and efficiently completed, and for a complex scene, more computing resources and iteration steps are allocated to focus on processing complex structures and occlusion relationships, thereby significantly improving the intelligent and optimal level of the overall generation efficiency while ensuring the generation quality. BRIEF DESCRIPTION OF DRAWINGS

[0091] Other features, objects, and advantages of the present disclosure will become more apparent from the following detailed description of non-limiting embodiments, read in conjunction with the accompanying drawings. The drawings are only for the purpose of illustrating the specific embodiments and are not to be construed as limiting the present disclosure. In the drawings:

[0092] Figure 1 An exemplary system architecture diagram to which an embodiment of the present disclosure can be applied;

[0093] Figure 2 A flowchart of an embodiment of an image generation method according to the present disclosure;

[0094] Figure 3 A decomposition flowchart of an embodiment of model training according to the present disclosure;

[0095] Figure 4 A structural schematic diagram of an embodiment of an image generation apparatus according to the present disclosure;

[0096] Figure 5 A structural schematic diagram of a computer system of an electronic device suitable for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0097] The present disclosure will be further described below in conjunction with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related application, and not to limit the application. In addition, it should be noted that, for the sake of description, only the parts related to the application are shown in the drawings.

[0098] The applicant found in the research process that the related image generation method is prone to problems such as object boundary breakage, element crowding, relationship conflict, and inconsistent lighting in complex scenes, and it is difficult to simultaneously consider the step relationship and local details by using the fixed noise schedule and single-scale denoising.

[0099] In view of the above shortcomings, the present disclosure uses a double sub-model of a global layout diffusion sub-model and an entity slot diffusion sub-model that has completed training, and introduces a noise schedule and time step distribution that adapts to complexity, gives more iterations to high-complexity samples at key stages, further uses slot refinement patches and cooperates with fusion weights to ensure the continuity of multi-object boundaries and local consistency.

[0100] It should be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict. The present disclosure will be described in detail below with reference to the drawings and in conjunction with the embodiments.

[0101] Figure 1 An exemplary system architecture 100 is shown, to which embodiments of the image generation method, apparatus, electronic device, storage medium, and program of the present disclosure can be applied.

[0102] As shown in Figure 1 System architecture 100 can include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. Network 104 is a medium for providing communication links between first terminal device 101, second terminal device 102, third terminal device 103, and server 105. Network 104 can include various connection types, such as wired, wireless communication links, or fiber optic cables, and the like.

[0103] A user can use first terminal device 101, second terminal device 102, and third terminal device 103 to interact with server 105 through network 104 to receive or send messages, and the like. First terminal device 101, second terminal device 102, and third terminal device 103 can be installed with various communication client applications, such as applications for processing images (e.g., applications including models for denoising images and / or applications for generating models of images, and the like), natural language processing applications, large model applications, speech recognition applications, short video social applications, audio and video conference applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, and the like.

[0104] First terminal device 101, second terminal device 102, and third terminal device 103 can be hardware or software. When first terminal device 101, second terminal device 102, and third terminal device 103 are hardware, they can be various electronic devices with sound acquisition devices (e.g., microphones), video acquisition devices (e.g., cameras), and display screens, including but not limited to smartphones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, and the like. When first terminal device 101, second terminal device 102, and third terminal device 103 are software, they can be installed in the terminal devices listed above. They can be implemented as multiple software or software modules (e.g., related services for providing task management), or as a single software or software module. No specific limitation is made herein.

[0105] In some cases, the image generation method provided by the present disclosure can be executed by the first terminal device 101, the second terminal device 102, and the third terminal device 103, and correspondingly, the image generation apparatus can be arranged in the first terminal device 101, the second terminal device 102, and the third terminal device 103. At this time, the system architecture 100 can also not include the server 105.

[0106] In some cases, the image generation method provided by the present disclosure can be executed by the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105 together, for example, the steps of “obtaining a target guidance information tensor for guiding the generation of a target image”, “organizing the target guidance information tensor into a target object instance set”, and “determining a target complexity of the target object instance set based on each target object instance” can be executed by the first terminal device 101, the second terminal device 102, and the third terminal device 103, and the steps of “calling a target noise schedule and a target time step matching the target complexity” and “using the diffusion probability model trained to denoise a preset random noise image according to the target noise schedule and the target time step, and generating a target slot refinement patch corresponding to each target object instance in the denoised random noise image to obtain a target image” can be executed by the server 105. The present disclosure does not make any limitation in this regard. Correspondingly, the image generation apparatus can also be arranged in the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105, respectively.

[0107] In some cases, the image generation method provided by the present disclosure can be executed by the server 105, and correspondingly, the image generation apparatus can also be arranged in the server 105, at this time, the system architecture 100 can also not include the first terminal device 101, the second terminal device 102, and the third terminal device 103.

[0108] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (for example, for providing distributed services), or as a single software or software module. No specific limitation is made herein.

[0109] It should be understood that Figure 1 The number of terminal devices, networks, and servers in the system architecture 100 is only illustrative. According to the needs of implementation, there can be any number of terminal devices, networks, and servers.

[0110] Continuing to refer to the system architecture 100 Figure 2FIG. 2 shows a flow 200 of one embodiment of the image generation method of the present disclosure, which comprises the following steps 201-205:

[0111] Step 201, obtaining a target guidance information tensor for guiding the generation of a target image.

[0112] Before starting to generate the image, a set of guidance information input by a user can be received, and a guidance information tensor is generated by using the set of guidance information, which is used to guide the generation of the target image.

[0113] The set of guidance information comprises one or more guidance information, each of which is related information, condition, data and / or requirement about the target image in anticipation input by the user, and is used to guide the generation of the target image, which can be, for example, a text description, a scene graph relationship, a two-dimensional layout frame, a semantic mask, depth data and camera parameters.

[0114] The text description can be, for example, “apples are placed on the table”; the scene graph relationship is used to represent the occlusion order relationship between each target object instance in anticipation, each target object instance in anticipation being an object that needs to be included in the target image in anticipation; the two-dimensional layout frame is used to represent the position of each target object instance in anticipation in the target image in anticipation, which can be, for example, a range frame represented by coordinates; the semantic mask is used to represent the outline and shape of each target object instance in anticipation; the depth data is used to represent the three-dimensional information of the target image in anticipation; and the camera parameters represent the parameters of a virtual camera, wherein the virtual camera represents a camera that takes the target image in anticipation as an image taken.

[0115] In some other cases, the scene graph relationship, the two-dimensional layout frame, the semantic mask, the depth data and the camera parameters can also be obtained from the text description.

[0116] In this step, based on the obtained set of guidance information, each guidance information in the set of guidance information can be aligned and fused to obtain the corresponding target guidance information tensor.

[0117] Specifically, the basic processing can be performed based on each guidance information.

[0118] For example, for the text description, it can be encoded into a fixed-length semantic representation, which can be a vector representation, and the fixed length can be, for example, 512.

[0119] For the scene graph relationship, it can be encoded as an adjacency representation, which can be in the form of a matrix tensor and specifically characterizes the occlusion order relationship between each target object instance in the expectation, where the category of the occlusion order relationship can be fixed and no less than, for example, 8 categories, and the category of the occlusion order relationship between instances can be, for example, instance i is above instance j; the empty position exceeding the upper limit of the number of target object instances can be filled with 0, and the number of target object instances can be set to an upper limit, for example, 64.

[0120] In a specific example, a matrix tensor A can be first constructed, with a size of (upper limit of the number of target object instances, upper limit of the number of target object instances, category of occlusion order relationship).

[0121] Further, for instance i and instance j, if there is a relationship r, then A[i, j, r]=1, otherwise 0. The part less than 64 instances is filled with 0.

[0122] Accordingly, the matrix tensor can be obtained.

[0123] For the two-dimensional layout frame, it can be rasterized into a guide grid of edges and centers, the number of grid channels can be set to, for example, 4, and a Gaussian kernel can be used for smoothing, and the kernel radius can be, for example, 3 pixels.

[0124] Specifically, the two-dimensional layout frame can be represented as [x_min, y_min, x_max, y_max], which can be normalized to the [0, 1] interval relative to the preset target resolution.

[0125] Further, two channels with the same resolution as the preset target reference image are created for each frame, one of which represents a two-dimensional Gaussian distribution drawn at the center point of the frame, with a peak value of 1, and the other represents a line drawn on the four edges of the frame, with a value of 1.

[0126] Further, the maximum values of the center map and the edge map of all instances are taken respectively to obtain two global channels, and a Gaussian kernel is used to smooth the two global channels. In the case where the target resolution is 1024x1024 pixels, a two-dimensional layout frame tensor with a shape of (1024, 1024, 2) can be obtained.

[0127] For the semantic mask, the two-dimensional layout box and the depth data, the depth data can be re-projected to the plane of the reference image, so that the depth data is aligned with the reference image at the pixel level; and the semantic mask and the two-dimensional layout box are aligned to the plane of the reference image, and the semantic mask, the two-dimensional layout box and the depth data are size-specified and value-normalized with the preset reference image, the normalization interval can be fixed as 0 to 1, and the normalization coefficient is recorded, wherein the reference image can be a carrier of the reference coordinate system and scale, and is used to provide a unified geometric reference, so as to ensure that all different types of guidance information such as text description, two-dimensional layout box, semantic mask and depth data are aligned under the same coordinate space and resolution, so that corresponding structured tensor data can be obtained.

[0128] For the camera parameters, the internal and external parameters of the virtual camera can be calibrated according to the camera parameters, and a unified camera coordinate system and target resolution are determined, the target resolution is a pre-set image size, for example, a pre-set pixel value of width and height, all guidance information received, except for the text description, can be converted and aligned to the target resolution, and the target resolution can be set to, for example, 1024x1024 pixels.

[0129] Further, the reference view angle is determined under the unified camera coordinate system and target resolution, and if there is data of other view angles, it is transformed to the reference view angle.

[0130] Further, target object instance parsing can be performed from the text description, so as to determine the name and / or semantic representation of each target object instance in the expected target image, and each target object instance name and / or semantic representation is numbered and used as an instance index of each target object instance.

[0131] Based on this, a one-to-one correspondence between the scene graph relationship and the target instance index can be established.

[0132] Further, missing window detection and outlier detection can be performed on each channel of the scene graph relationship, the two-dimensional layout box, the semantic mask, the depth data and the camera parameters, wherein the minimum connected area of the missing window can be set to 9 pixels; the outliers can be detected by using the median deviation of the adjacent 16 neighborhoods, for example, pixels exceeding 3 times the median deviation can be marked as outliers.

[0133] In the case of detecting a missing window, the completion confidence of the missing window and the outlier part can be calculated, based on which the missing window and the outlier can be completed in geometric consistency. Specifically, local plane fitting of a 3x3 neighborhood can be used and the depth change is limited to no more than 5% per pixel. Further, semantic consistency completion can be performed. Specifically, a closing operation with a radius of 2 pixels can be performed on the mask and the instance number is kept unchanged. Further, learning-based completion can be performed. Specifically, a pre-trained network can be used to refine the texture and the boundary, and the part with a completion confidence lower than 0.7 is recorded as a low-confidence area and the position and area ratio are recorded in the preset metadata.

[0134] Based on the above basic processing, the vector representation of the text description after the basic processing, the matrix tensor of the scene graph relationship, the semantic mask, the two-dimensional layout frame, the depth data and the camera parameter corresponding to the structured tensor can be aligned and fused into a target guidance information tensor.

[0135] Specifically, the semantic representation and the relationship channel group of the adjacency representation after the basic processing, the two-dimensional layout frame, the semantic mask, the depth data and the camera parameter can be stacked in a fixed channel order and in the same memory layout to obtain the target guidance information tensor.

[0136] Step 202, arranging the target guidance information tensor into a target object instance set.

[0137] Based on the target guidance information tensor determined in the foregoing step 201, the specific description of each target object instance in the expectation can be determined, and each specific target object instance can be grouped into a target object instance set.

[0138] Specifically, the semantic mask channel and the two-dimensional layout frame channel in the target guidance information tensor can be used to perform instance segmentation on a preset target region in the same coordinate and scale as the target guidance information tensor, to obtain an initial object instance. The target region can be a position marked as needing to generate a foreground object on the plane of the reference image, i.e., a pixel region occupied by the initial object instance.

[0139] Further, each initial object instance can be de-duplicated and merged by setting a de-duplication intersection over union threshold, for example, 0.5. The repeated frames with an intersection over union greater than or equal to the intersection over union threshold, for example, greater than or equal to 0.5, are merged into a single object instance. Thus, the merged object instance and the initial object instance that does not need to be merged can be used as each specific target object instance, and each target object instance can be grouped into an object instance set.

[0140] Step 203, determining the target complexity of the target object instance set based on each target object instance.

[0141] Based on the object instance set obtained in the foregoing step 202, a target complexity of the object instance set can be determined based on each target object instance therein, wherein the target complexity is used to represent a complexity degree of a target image expected to be generated, for example, wherein each target object instance and an ordered relationship of mutual position occlusion.

[0142] Specifically, an instance quantity of the target object instances in the target object instance set, an occlusion level indicator of mutual occlusion of the target object instances, a layout dispersion between the target object instances, a depth variance of depth data of the target object instances, a category entropy of categories of the target object instances, and the like can be determined and weighted, so as to obtain the target complexity.

[0143] In the process of determining the instance quantity of the target object instances in the target object instance set, the instance quantity of the target object instances can be counted based on instance indexes.

[0144] In the process of determining the occlusion level indicator of mutual occlusion of the target object instances, for the target object instances that mutually occlude each other in the plane of the reference image, two target object instances with an overlapping area greater than or equal to 16 pixels can be determined to have a front-back relationship.

[0145] Specifically, the absolute value of the difference between the depth data greater than or equal to 1% of the global depth data range can be used as two target object instances that can be distinguished in front and back, wherein the target object instance as the foreground, i.e., the target object instance closer, can be the target object instance with smaller depth data.

[0146] Accordingly, an occlusion ordered relationship between each target object instance can be generated, and the number of other target object instances that occlude each target object instance, i.e., the number of other target object instances that occlude the target object instance, can be counted, so as to obtain an instance-level occlusion level.

[0147] Further, a mean value of all instance-level occlusion levels can be calculated and used as an image-level occlusion level indicator.

[0148] In the process of determining the layout dispersion between the target object instances, the centroid coordinates of the target object instances on the plane of the reference image can be determined, and a set of the centroid coordinates can be formed, and the discrete degree of the distribution of the target object instances can be calculated based on the set of the centroid coordinates.

[0149] Specifically, a standard deviation measurement of variance and square root of each centroid coordinate can be performed, and the layout dispersion can be obtained by normalizing the specific size of the target object instance.

[0150] In the process of determining the depth variance of the depth data of each target object instance, the overall magnitude of the change of the depth data can be calculated in the range of the depth data to obtain the depth variance. In some cases, after the overall magnitude is determined, outliers can also be determined and removed. The outliers can be, for example, depth data greater than or less than three times the median.

[0151] In the process of determining the category entropy of the category of each target object instance, a category label representing the object category of the corresponding target object instance can be set in the semantic mask in advance. The category label and the instance index are used to count the frequency of the appearance of the object category. According to the frequency of the object category, the object category is summarized into a 64-grid statistical table to obtain a category frequency distribution. The information entropy of the distribution is further calculated by taking the base as 2. According to this, the information entropy can be divided by the maximum entropy value when the 64 grids are uniformly distributed, and the division result is normalized to make the normalized result fall within the interval of 0 to 1 to obtain the category entropy.

[0152] Based on this, the instance number of the target object instance in the target object instance set, the occlusion level index of each target object instance, the layout divergence between each target object instance, the depth variance of the depth data of each target object instance, and the category entropy of the category of each target object instance can be linearly scaled to a pre-set scaling interval, such as a scaling interval of 0 to 1.

[0153] Further, the instance number, the occlusion level index, the layout divergence, the depth variance, and the category entropy after linear scaling are weighted according to a pre-set equal weight to obtain a target complexity, wherein the equal weight can be, for example, 0.2.

[0154] In this step, in the case where the scaling interval is 0 to 1 and the equal weight is 0.2, the target complexity obtained is also in the interval of 0 to 1. According to this, the different target complexities determined can be divided into low complexity, medium complexity, and high complexity.

[0155] Specifically, the target complexity less than 0.3 can be determined as low complexity, the target complexity greater than or equal to 0.3 and less than 0.7 can be determined as medium complexity, and the target complexity greater than 0.7 can be determined as high complexity.

[0156] The metadata can be pre-constructed to record relevant data in the process of generating the target image. According to this, a corresponding relationship can be established between the target complexity determined and the corresponding target guidance information tensor, and the target complexity, the target guidance information tensor, and the corresponding relationship can be recorded in the metadata.

[0157] Step 204, call the target noise schedule and the target time step matching the target complexity.

[0158] Based on the target complexity determined in the foregoing step 203, a target noise schedule and a target time step can be retrieved according to a preset mapping rule, where the target noise schedule refers to the degree of each denoising in the process of obtaining the target image after denoising the pure noise picture multiple times, i.e., the denoising intensity in each denoising step, and the target time step refers to the total number of denoising, i.e., the number of denoising steps, in the process of obtaining the target image after denoising the pure noise picture multiple times.

[0159] It can be seen that the target noise schedule and the target time step specifically give the denoising strategy in the denoising process.

[0160] Specifically, after determining the target complexity, the current mapping rule can be loaded, where the mapping rule specifically represents the corresponding matching relationship between the target complexity and the target noise schedule and the target time step.

[0161] For example, the mapping relationship between the target noise schedule and the target complexity can be: in the case of low complexity, the intensity at the starting stage is 0.60, the intensity at the terminal stage is 0.02, and the density distribution is set to 30% in the high noise stage, 40% in the medium noise stage, and 30% in the low noise stage; in the case of medium complexity, the intensity at the starting stage is 0.75, the intensity at the terminal stage is 0.02, and the density distribution is set to 40% in the high noise stage, 35% in the medium noise stage, and 25% in the low noise stage; in the case of high complexity, the intensity at the starting stage is 0.90, the intensity at the terminal stage is 0.02, and the density distribution is set to 50% in the high noise stage, 30% in the medium noise stage, and 20% in the low noise stage.

[0162] For example, the mapping relationship between the target time step and the target complexity can be: in the case of low complexity, the total number of steps is set to 40, the middle stage is set to uniform, and the end stage is set to refinement iteration 1 time; in the case of medium complexity, the total number of steps is set to 52, the high stage is set to refinement, the middle stage is set to uniform, and the end stage is set to refinement iteration 1 time; in the case of high complexity, the total number of steps is set to 64, the high stage is set to refinement, the middle stage is set to uniform, and the end stage is set to refinement iteration 2 times.

[0163] Based on this, after loading the mapping rule, the target noise schedule and the target time step corresponding to the target complexity can be determined according to the corresponding matching relationship represented by the mapping rule.

[0164] At step 205, the preset random noise image is denoised according to the target noise schedule and the target time step using the diffusion probability model trained, and target slot refined patches corresponding to each target object instance are generated in the denoised random noise image to obtain a target image.

[0165] Based on the target noise schedule and the target time step determined in the foregoing step 204, the diffusion probability model trained can be used to denoise the noise image according to the target noise schedule and the target time step, so as to generate a global target rough denoised image close to the target image, and on the basis of the global target rough denoising, generate patches described by each target object instance, that is, target slot refined patches, so as to obtain the target image.

[0166] The noise image can be randomly generated; the trained diffusion probability model includes a trained global layout diffusion sub-model and a trained entity slot diffusion sub-model, and the diffusion probability model can further include a feature encoder shared by the global layout diffusion sub-model and the entity slot diffusion sub-model.

[0167] In the process of generating the target image using the trained global layout diffusion sub-model and the trained entity slot diffusion sub-model, the current generation parameter set can be loaded first, and the generation parameter set includes respective model parameters of the trained global layout diffusion sub-model and the trained entity slot diffusion sub-model.

[0168] Based on the global layout diffusion sub-model loaded with the completed generation parameter set, the target guidance information tensor and the target object instance set can be input into the global layout diffusion sub-model, and the trained global layout diffusion sub-model can denoise the pre-set random noise image according to the description of the target guidance information tensor to the expected target image.

[0169] Further, in the process of denoising by the trained global layout diffusion sub-model, the denoising can be performed according to the denoising strategy of the target noise schedule and the target time step matched with the target complexity determined above, and consistency guidance can be added in each step of denoising.

[0170] Specifically, a weight table of the consistency guidance item can be pre-set, so as to set different intensities of the consistency guidance item in the high noise stage, the medium noise stage and the low noise stage.

[0171] For example, the intensity of the consistency guidance item is set to 0.5 in the high noise stage, the intensity of the consistency guidance item is set to 1.0 in the medium noise stage, and the intensity of the consistency guidance item is set to 1.5 in the low noise stage.

[0172] The consistency guidance item can include, for example, differentiable guidance of geometric consistency, physical feasibility and semantic relationship consistency.

[0173] The differentiable guidance of geometric consistency can be used to maintain the boundary continuity of each target object instance and the gradient alignment of depth data, the differentiable guidance of physical feasibility can be used to avoid unreasonable penetration and contact of each target object instance, and the differentiable guidance of semantic relationship consistency can be used to meet the constraints of the scene relationship represented in the target guidance information tensor and each target object instance.

[0174] Based on the intensity of the consistency guidance items set at different stages, the global structure and occlusion order relationship can be prioritized in the high-noise stage, the main texture and large outline can be prioritized in the medium-noise stage, and the details and boundary transitions can be prioritized in the low-noise stage through refinement iterations.

[0175] Based on this, after completing the denoising process, a coarse-level representation close to the target image from a global perspective can be obtained, that is, a global target coarse denoising image.

[0176] In the process of obtaining the coarse-level representation, that is, the global target coarse denoising image, the trained global layout diffusion sub-model can be used to obtain the global target coarse denoising image in the low-resolution space.

[0177] Further, after obtaining the coarse-level representation, a synthesis buffer can be constructed, and the current base image is saved in the synthesis buffer, and the obtained coarse-level representation is used as the initial base image.

[0178] Based on this, the preset target slot, target guidance information tensor and target object instance set can be input into the trained entity slot diffusion sub-model, and the trained entity slot diffusion sub-model can be used to generate a target image.

[0179] The target slot is used to represent the position of each target object instance in the target region, that is, the two-dimensional layout frame, used to generate the target image.

[0180] Specifically, the trained entity slot diffusion sub-model can determine the scene graph relationship from the target guidance information tensor. Since the scene graph relationship specifically represents the occlusion order relationship between each target object instance, based on the occlusion order relationship between each target object instance and the target slot representing the position of each target object instance, the front and rear position relationship between each target slot can be determined, and thus the processing order of each target slot, that is, the slot sequence, can be determined. For example, the order from back to front can be used as the processing order.

[0181] Based on this, in the process of processing each target slot, the trained entity slot diffusion sub-model can output the target slot refinement patch and the fusion weight of the target slot.

[0182] The target slot refined patch is a predicted tile at each preset target slot.

[0183] Further, the target slot refined patch is pixel-level weighted superimposed with the slot mask range of the corresponding target slot in the current base image in the synthesis buffer, to obtain an updated base image.

[0184] In this process, when the slot masks of different target slots overlap, the weighted sum is performed according to the rule that the weight sum in the overlapping area is 1, and feathering with a width of 2 pixels and color consistency correction are performed on the boundary neighborhood of the superimposed area after each superimposition.

[0185] Further, the boundary continuity score and the occlusion order score of the superimposed area can be calculated, and the boundary continuity score threshold and the occlusion order score threshold can be set in advance, for example, both of which can be 0.95.

[0186] Accordingly, if the boundary continuity score is less than the boundary continuity score threshold, or the occlusion order score is less than the occlusion order score threshold, the target slot is iterated twice again for low-noise refinement and pixel-level weighted superimposition is performed again, and if the boundary continuity score is still less than the boundary continuity score threshold, or the occlusion order score is still less than the occlusion order score threshold, the target slot can be marked as a problem slot.

[0187] Further, if the boundary continuity score is greater than or equal to the boundary continuity score threshold, and the occlusion order score is greater than or equal to the occlusion order score threshold, the pixel-level weighted superimposition of the current target slot is completed and the threshold check is completed.

[0188] Based on this, the image obtained after the current weighted superimposition can be used as the next base image for updating the current base image, and it is written into the synthesis buffer for weighted superimposition of the next target slot.

[0189] When performing pixel-level weighted superimposition for the first time, the global target coarse denoising image currently serving as the base image can be retrieved from the synthesis buffer for weighted superimposition.

[0190] Further, after completing the weighted superimposition of all target slots, the expected target image can be obtained.

[0191] Based on this, by aligning and fusing different types of guidance information under the same coordinates and scales, a structured and scale-unified target guidance information tensor is provided for subsequent processing, thereby ensuring the consistency between different guidance information and providing a basis for generating a target image that meets the guidance information, effectively avoiding the problem of deviation in the generated target image due to the non-uniform data format or reference of the input guidance information.

[0192] Further, by quantitatively analyzing the instance number of the target object instances in the target object instance set, the occlusion level index of each target object instance occluding each other, the layout divergence between each target object instance, the depth variance of the depth data of each target object instance, and the class entropy of the class of each target object instance, a target complexity that is adapted to the target object instance set can be generated, and the complexity of the target image to be generated can be determined.

[0193] Further, based on the target complexity, the target noise schedule and the target time step suitable for generating the current target image can be accurately adapted, and the self-adaptive effect of complexity can be realized, so that for a simple scene, the generation can be quickly and efficiently completed, and for a complex scene, more computing resources and iteration steps are allocated to focus on processing complex structures and occlusion relationships, thereby significantly improving the intelligent and optimal level of overall generation efficiency while ensuring generation quality.

[0194] Before generating the target image using the trained diffusion probability model, the trained diffusion probability model needs to be trained using the training data set.

[0195] Specifically, with reference to Figure 3 which shows a flow 300 of one embodiment of training a diffusion probability model of the present disclosure. The flow 300 includes the following steps 301 to 305:

[0196] Step 301, obtaining a plurality of preset training images, and a training instance set corresponding to each training image and a corresponding training noise image.

[0197] Before training the trained diffusion probability model, a training data set can be collected, which includes a plurality of collected training images, a training object instance set corresponding to each training image, and a corresponding training guidance information set, and the training guidance information tensor is obtained from the training guidance information set.

[0198] Among them, the training guidance information set includes one or more training guidance information, each training guidance information is used to guide the generation of a target image, which can be, for example, a text description, a scene graph relationship, a two-dimensional layout frame, a semantic mask, depth data and camera parameters.

[0199] Further, based on the obtained training guidance information set, each training guidance information in the training guidance information set can be aligned and fused to obtain the corresponding training guidance information tensor.

[0200] Specifically, the basic processing can be performed based on each guidance information first.

[0201] For example, for the text description, it can be encoded into a fixed-length semantic representation.

[0202] For the scene graph relationship, it can be encoded into an adjacency representation and a one-to-one correspondence between the training instance index.

[0203] For the two-dimensional layout box, it can be rasterized into a guide grid of edges and centers.

[0204] For the semantic mask, two-dimensional layout box and depth data, the depth data can be re-projected to the plane of the reference image, aligning the depth data with the reference image at the pixel level; and the semantic mask and two-dimensional layout box are aligned to the plane of the reference image, and the semantic mask, two-dimensional layout box and depth data are size-specified and value-normalized with the preset reference image.

[0205] For the camera parameters, the internal and external parameters of the virtual camera can be calibrated according to the camera parameters, at least 25 calibration images are used in the calibration process, and the root mean square of the residual error is not higher than 0.5 pixels.

[0206] Further, a unified camera coordinate system and a target resolution are determined, and a reference view angle is determined under the unified camera coordinate system and the target resolution. If there is data of other view angles, it is transformed to the reference view angle.

[0207] Further, each training instance set includes at least one multi-modal description of a training object instance, and the name and / or semantic representation of each training object instance can be numbered and used as the training instance index of each target object instance.

[0208] Based on this, a one-to-one correspondence between the scene graph relationship and the training instance index can be established.

[0209] Further, the detection of missing windows and the detection of outliers can be performed on each channel of the scene graph relationship, two-dimensional layout box, semantic mask, depth data and camera parameters.

[0210] In the case of detecting missing windows, the completion confidence of the missing window and the outlier part can be calculated, and accordingly, the missing window and the outlier can be geometrically consistent; further, semantic consistency completion and learning completion can be performed, specifically, a pre-trained network can be used to refine the texture and boundary, and the part with a completion confidence lower than 0.7 is recorded as a low-confidence area.

[0211] Based on the above basic processing, the training guidance information set after the basic processing can be aligned and fused into a training guidance information tensor.

[0212] Specifically, the relationship channel group, the two-dimensional layout frame, the semantic mask, the depth data and the camera parameter of the basic processed semantic representation and the adjacency representation can be stacked in the same memory layout according to a fixed channel order to obtain a training guide information tensor.

[0213] Further, based on the obtained training images, the training guide information tensors corresponding to the training images and the training reference images corresponding to the training guide information tensors, the training guide information tensors and the training reference images can be time stamped and aligned in a coordinate reference at a target resolution.

[0214] Further, the training reference images can be mapped into corresponding low-resolution target representations in a quarter scale downsampling manner, and the training reference images can be calibrated into global calibration coarse denoising images corresponding to the training reference images.

[0215] Based on this, noise can be injected into each low-resolution target representation, i.e., the global calibration coarse denoising image, to obtain a corresponding training noise image.

[0216] Step 302, using the diffusion probability model to be trained to denoise each training noise image, and generating a training slot refinement patch corresponding to each training object instance in each denoised training noise image to obtain a corresponding reconstructed image.

[0217] Based on the training guide information tensors determined in step 301 and the corresponding training object instance sets, the diffusion probability model to be trained can be used to denoise each training noise image, and a corresponding reconstructed image can be generated based on each denoised training noise image.

[0218] In this step, before denoising training using the diffusion probability model to be trained, the training complexity of each training image and the corresponding calibrated calibration slot can be determined, and accordingly, the corresponding matching training noise schedule and training time step can be determined according to the training complexity.

[0219] Specifically, the instance number of the training object instances in the training object instance set, the occlusion level index of each training object instance, the layout divergence between each training object instance, the depth variance of the depth data of each training object instance, and the class entropy of the class of each training object instance can be determined and weighted to obtain the training complexity.

[0220] For any one training image, in the process of determining the instance number of the training object instances in the training object instance set, the instance number of each training object instance can be counted based on the instance index.

[0221] For any one training image, in the process of determining the occlusion level indicators of the mutually occluded training object instances, for the training object instances mutually occluded in the plane of the training reference image, the front-back relationship of two training object instances with an overlapping area greater than or equal to 16 pixels can be determined.

[0222] Specifically, the absolute value of the difference between the depth data greater than or equal to 1% of the global depth data range can be used as the two training object instances that can be distinguished in front and back, and the training object instance in front, i.e., the training object instance closer, can be the training object instance with smaller depth data, by using the channel of the depth data in the training guidance information tensor and the camera parameters.

[0223] Accordingly, the occlusion order relationship between each training object instance can be generated, and the number of levels of each training object instance occluded by other training object instances, i.e., the number of other training object instances occluding the training object instance, can be counted, so that the instance-level occlusion level can be obtained.

[0224] Further, the mean of all instance-level occlusion levels can be calculated and used as the image-level occlusion level indicator of the training image.

[0225] For any one training image, in the process of determining the layout divergence between the training object instances, the centroid coordinates of the training object instances in the plane of the training reference image can be determined, the centroid coordinates are grouped into a set, and the dispersion degree of the distribution of the training object instances is calculated by using the centroid coordinate set, and is used as the layout divergence of the corresponding training image.

[0226] For any one training image, in the process of determining the depth variance of the depth data of the training object instances, the overall amplitude of the change of the depth data in the range of the depth data can be calculated to obtain the depth variance, and in some cases, after the overall amplitude is determined, outliers can be determined and removed.

[0227] For any one training image, in the process of determining the class entropy of the class of the training object instances, a class label can be set in advance in the semantic mask of the training object instance, the class label represents the object class of the corresponding training object instance, and the frequency of the appearance of the object class is counted by the class label and the instance index. Accordingly, according to the frequency of the object class, the object class is summarized into a 64-grid statistical table to obtain the class frequency distribution, and the information entropy of the distribution is further calculated by taking 2 as the base. Accordingly, the information entropy can be divided by the maximum entropy value when the 64 grids are uniformly distributed, and the division result is normalized to fall within the interval of 0 to 1 to obtain the class entropy.

[0228] Based on this, the instance quantity of the training object instance in the training object instance set, the occlusion level index of each training object instance occluding each other, the layout divergence between each training object instance, the depth variance of the depth data of each training object instance, and the category entropy of the category of each training object instance can be linearly scaled into a pre-set scaling interval, and the linearly scaled instance quantity, occlusion level index, layout divergence, depth variance and category entropy are weighted according to a pre-set equal weight to obtain the training complexity.

[0229] Among them, the training metadata can be pre-constructed to record the related data in the process of training using the training image, according to which the determined training complexity can be constructed with the corresponding training guidance information tensor and the training image, and the training complexity, the training guidance information tensor, the training image and their corresponding relationship are recorded in the metadata.

[0230] Further, based on the determined training complexity, a pre-set mapping rule can be loaded, so that the training noise schedule and the training time step corresponding to the training complexity can be determined according to the corresponding matching relationship represented by the mapping rule.

[0231] Based on this, the diffusion probability model to be trained can be denoised.

[0232] Among them, the diffusion probability model to be trained includes a global layout diffusion sub-model to be trained and an entity slot diffusion sub-model to be trained.

[0233] In the process of denoising the diffusion probability model to be trained, the target guidance information tensor and the target object instance set can be input into the global layout diffusion sub-model to be trained, and the global layout diffusion sub-model to be trained can denoise the training noise image according to the description of the training image in the training guidance information tensor.

[0234] Further, in the process of denoising the global layout diffusion sub-model to be trained, the denoising can be performed according to the denoising strategy of the noise schedule to be trained and the time step to be trained corresponding to the training complexity determined above, and consistency guidance can be added in each step of denoising.

[0235] Specifically, the consistency guidance items can be set to different intensities in the high noise stage, the medium noise stage and the low noise stage according to the pre-set weight table of the consistency guidance items.

[0236] Among them, the consistency guidance items can include, for example, differentiable guidance of geometric consistency, physical feasibility and semantic relationship consistency.

[0237] Based on this, after the denoising process is completed, a global training rough denoising image close to the training image from a global perspective can be obtained.

[0238] In the process of obtaining the global training rough denoising image, the global layout diffusion sub-model to be trained can be used to obtain the prediction in the low-resolution space.

[0239] Further, after obtaining the global training rough denoising image, a training buffer can be constructed, and the current global training rough denoising image is saved in the training buffer.

[0240] Based on this, the preset each calibration slot, training guide information tensor and training object instance set can be input into the entity slot diffusion sub-model to be trained, and the entity slot diffusion sub-model to be trained is used to generate a reconstructed image.

[0241] Each training slot is used to represent the position of each training object instance in the two-dimensional layout frame of the corresponding training image.

[0242] Specifically, the entity slot diffusion sub-model to be trained can determine the scene graph relationship from the training guide information tensor, determine the front and rear position relationship between each training slot based on the occlusion order relationship between each training object instance represented in the scene graph relationship, and determine the processing order of each training slot, i.e., the slot sequence, based on the training slot representing the position of each training object instance.

[0243] Based on this, in the process of processing each training slot, the trained entity slot diffusion sub-model can output the training slot refinement patch and the fusion weight of the training slot.

[0244] Each training slot refinement patch is a predicted tile at the corresponding training slot.

[0245] Further, by pixel-level weighted superposition of the training slot refinement patch and the slot mask range of the corresponding training slot in the current global training rough denoising image in the training buffer, the next global training rough denoising image for replacing the current global training rough denoising image can be obtained.

[0246] In this process, when the slot masks of different training slots overlap, the weighted sum is performed according to the rule that the weight sum in the overlapping region is 1, and the boundary neighborhood of the superposition region is feathered with a width of 2 pixels and color consistency correction is performed after each superposition.

[0247] Further, the boundary continuity score and the occlusion order score of the superimposed region can be calculated, and the boundary continuity score threshold and the occlusion order score threshold can be preset, wherein the score thresholds during training of the diffusion probability model can be the same as or different from the score thresholds of the trained diffusion probability model.

[0248] Accordingly, if the boundary continuity score is less than the boundary continuity score threshold or the occlusion order score is less than the occlusion order score threshold, the training slot is subjected to two iterations of low-noise refinement and pixel-level weighted superimposition again, and if the boundary continuity score is still less than the boundary continuity score threshold or the occlusion order score is still less than the occlusion order score threshold, the training slot can be marked as a problem slot.

[0249] Further, if the boundary continuity score is greater than or equal to the boundary continuity score threshold and the occlusion order score is greater than or equal to the occlusion order score threshold, the pixel-level weighted superimposition of the current training slot and the threshold check are completed.

[0250] Based on this, the image obtained after the current weighted superimposition can be used as the next global training coarse denoising image for updating the current global training coarse denoising image, and written into the training buffer for weighted superimposition of the next training slot.

[0251] Further, for any training image, after the weighted superimposition of all training slots is completed, a reconstructed image can be obtained.

[0252] Step 303, determine the hybrid loss value between each training image and the corresponding reconstructed image.

[0253] Based on the reconstructed image corresponding to each training image and the corresponding global training coarse denoising image determined in the foregoing step 302, the structural consistency loss value, the reconstruction consistency loss value, the boundary continuity loss value, and the relationship consistency loss value can be determined, and then the hybrid loss value can be determined using the structural consistency loss value, the reconstruction consistency loss value, the boundary continuity loss value, and the relationship consistency loss value, so that whether the trained diffusion probability model is trained can be determined according to the hybrid loss value in the following steps.

[0254] Specifically, for each training image, in the process of determining the structural consistency loss value, the global calibration coarse denoising image and the corresponding global training coarse denoising image can be aligned to the same low-resolution space, and a structural similarity loss value can be further calculated.

[0255] Specifically, the structural similarity loss value can be calculated according to the following formula (1).

[0256] (1)

[0257] wherein, denotes a structural consistency loss, C denotes a global training coarse de-noising image; T denotes a global calibrated coarse de-noising image; denotes a preset spatial gradient operator; SSIM denotes a preset structural similarity index with a fixed window and a fixed kernel; mean denotes taking a mean value in a pixel domain; denotes taking an element-wise absolute value; ε denotes a small positive number for avoiding zero denominator and is set as a fixed constant.

[0258] wherein, in formula (1), the size of the fixed window can be set as 11x11, the fixed kernel is a Gaussian distribution, the gradient adopts a 3x3 spatial gradient operator, and the pixel-level difference is averaged on the whole image. In order to avoid numerical instability, ε can be a stable constant of 0.000001 added at the denominator.

[0259] For each training image, in the process of calculating the reconstruction consistency loss value, the reconstructed image and the training reference image can be compared in the plane of the training reference image, wherein the pixel-level error can be suppressed by using a smooth penalty term, and in order to further suppress outliers, a stable constant of 0.001 can also be introduced and an average value is taken; at the same time, the perceptual feature difference of the reconstructed image and the training reference image is extracted, the perceptual feature can be given by a perceptual encoder of a frozen parameter, in some examples, the channel features of the 3rd layer and the 5th layer can be taken, and the absolute difference is calculated and averaged respectively, the result of taking the average value and the result of calculating the absolute difference and taking the average value are added, and the reconstruction consistency loss can be obtained.

[0260] Specifically, formula (2) shown as follows can be used for calculation:

[0261] (2)

[0262] wherein, denotes a reconstruction consistency loss value, R denotes a reconstructed image; I denotes a training reference image; denotes a Charbonnier penalty term for suppressing outlier errors; denotes a feature extraction function of a perceptual encoder for freezing current parameters; denotes an element-wise absolute value sum; denotes a fixed small positive constant.

[0263] In the process of calculating the boundary continuity loss value, for each training image, the gradient of the fused reconstructed image can be calculated, and the gradient of each training slot refinement patch is pixel-level weighted according to the fusion weight, the fusion weight is non-negative and pixel-wise summed to 1 in the overlapping area, and the boundary continuity loss can be obtained by further averaging the reconstructed image gradient and each training slot refinement patch on the whole image.

[0264] Specifically, the formula (3) shown below can be used for calculation:

[0265] (3)

[0266] wherein, denotes the boundary continuity loss value, and K denotes the number of training slots; denotes the training slot refinement patch of the kth training slot; denotes the fusion weight corresponding to the kth training slot, and R denotes the reconstructed image obtained by step-by-step synthesis according to the fusion weight; denotes the spatial gradient operator.

[0267] In the process of calculating the relationship consistency loss value, for each training image, based on the scene graph relationship, for each pair of training object instances with a positional relationship, it can be judged whether the positional relationship is established on the reconstructed image to obtain a probability distribution of the relationship establishment, and the probability distribution is compared with the calibration distribution obtained by pre-training guidance information tensor labeling, and then the relative entropy measurement can be used as the relationship consistency loss, wherein, in order to avoid numerical problems caused by logarithmic operation, a smoothing constant of 0.000001 can be introduced for the position with a probability of 0, and the measurement of the relationship triple set obtained by analysis in all training guidance information tensors is averaged, so that the relationship consistency loss value can be obtained.

[0268] Specifically, the formula (4) shown below can be used for calculation:

[0269] (4)

[0270] wherein, denotes the relationship consistency loss value, and ε denotes the relationship triple set, wherein, the elements of each set of relationship triples include training object instance i, training object instance j and relationship type r, denotes the probability distribution of the relationship establishment output by the relationship determination head frozen with the current parameters on the generated reconstructed image; denotes the calibration distribution obtained according to the training guidance information tensor labeling, denotes the calculation of the KL divergence; denotes the number of relationship triples.

[0271] Based on this, after determining the above four loss values of each training image, each loss value can be divided by the stable constant of the current training round mean plus 0.001, so as to obtain the dimensionless normalization result, and the four dimensionless normalization results are averaged according to equal weight to obtain the joint loss.

[0272] Specifically, weighting can be performed according to formula (5) as shown below to obtain a hybrid loss value:

[0273] (5)

[0274] wherein, indicates that the mean is calculated in the current training round; stopgrad(*) indicates the operation of blocking gradient backpropagation to avoid that the scaling coefficient is learned by the network; ζ indicates a small positive number to stabilize the normalization.

[0275] Step 304, determining whether the hybrid loss value is less than or equal to the preset loss threshold value in continuous multiple training rounds.

[0276] Based on the hybrid loss value determined in the above step 303, it can be determined based on the preset loss threshold value that the current training round and the previous multiple continuous training rounds are less than or equal to the loss threshold value.

[0277] Wherein, the continuous rounds can be, for example, 4 rounds, that is, in addition to the current training round, it is determined whether the hybrid loss value of 5 continuous training rounds is less than or equal to the loss threshold value.

[0278] Step 305, in response to determining that the hybrid loss value is not less than or equal to the preset loss threshold value in continuous multiple training rounds, adjusting the model parameters of the diffusion probability model to be trained based on the hybrid loss value, and using the adjusted diffusion probability model to perform the next round of training until the hybrid loss value is less than or equal to the preset loss threshold value in continuous multiple training rounds, obtaining the trained diffusion probability model.

[0279] Based on the determination of the foregoing step 304, if the hybrid loss value in the current training round is less than the loss threshold value, or the hybrid loss values of the previous continuous multiple training rounds are not all less than or equal to the loss threshold value, the model parameters of the diffusion probability model to be trained are adjusted according to the hybrid loss value of the current round.

[0280] Further, each training image, corresponding training guidance information tensor and corresponding training object instance are input into the adjusted diffusion probability model, so as to perform the next round of training. ​​​​​

[0281] Based on this, it can be judged again whether the hybrid loss value of the current continuous multiple rounds of training is less than or equal to the loss threshold.

[0282] If it is judged that the hybrid loss value of the continuous multiple rounds of training is less than or equal to the loss threshold, the diffusion probability model of the completed training can be obtained.

[0283] Based on this, the difference between the reconstructed image and the training image is quantitatively evaluated by calculating the hybrid loss value, so that the model can gradually train the ability to recover the reconstructed image that meets the training guidance information tensor from noise through iterative learning. The hybrid loss value determined by the structural consistency loss value, the reconstruction consistency loss value, the boundary continuity loss value and the relationship consistency loss value can comprehensively reflect the current performance of the diffusion probability model from multiple aspects.

[0284] In the above embodiment of the present disclosure, through the cooperation of the global layout diffusion sub-model and the entity slot diffusion sub-model, a coarse representation is first generated to stabilize the global whole and the occlusion ordered relationship, and then the target slot refinement patch is synthesized step by step according to the fusion weight, and the guidance of geometric consistency, physical feasibility and semantic relationship consistency is cooperated, which significantly reduces the object boundary fracture, element crowding, relationship conflict and inconsistent illumination, and ensures the continuity of the structure of the complex scene.

[0285] In the above embodiment of the present disclosure, the target noise schedule and the target time step are set according to the target complexity score. In the case of high complexity, more detailed time steps and more steps are allocated to important stages, and in the case of low complexity, invalid calculations are reduced for non-critical stages, the generation delay is shortened and the success rate is improved.

[0286] In the above embodiment of the present disclosure, the target guidance information tensor is used to align the text description, scene graph relationship, two-dimensional layout frame, semantic mask, depth data and camera parameter, and the object level control is provided for the target slot refinement patch and the fusion weight.

[0287] Further reference Figure 4 , as an implementation of the method shown in the above figures, the present disclosure provides an embodiment of an image generation device, which corresponds to the method embodiment shown in Figure 2 , and the device can be applied in various electronic devices.

[0288] As shown in Figure 4 , the image generation device 400 of the present embodiment comprises an acquisition module 401, an arrangement module 402, a complexity determination module 403, a calling module 404 and a generation module 405.

[0289] The acquisition module 401 is configured to acquire a target guidance information tensor for guiding the generation of a target image.

[0290] The collation module 402 is configured to collate the target guidance information tensor into a target object instance set, the target object instance set including at least one multi-modal description target object instance;

[0291] The complexity determination module 403 is configured to determine a target complexity of the target object instance set based on each target object instance;

[0292] The calling module 404 is configured to call a target noise schedule and a target time step matching the target complexity, the target noise schedule representing a denoising intensity in each denoising step, and the target time step representing a strategy of allocating denoising steps in the denoising process;

[0293] The generation module 405 is configured to use the diffusion probability model trained to denoise a preset random noise image according to the target noise schedule and the target time step, and generate a target slot refinement patch corresponding to each target object instance in the denoised random noise image to obtain a target image.

[0294] In the embodiment, the specific processing of the acquisition module 401, the collation module 402, the complexity determination module 403, the calling module 404 and the generation module 405 and the technical effects brought by the same can be respectively referred to the related descriptions of the steps 201, 202, 203, 204 and 205 in the corresponding embodiments, which will not be repeated here. Figure 2 The corresponding embodiments of the steps 201, 202, 203, 204 and 205, which will not be repeated here.

[0295] In some optional embodiments, the target guidance information tensor at least includes a two-dimensional layout frame and a semantic mask under the same coordinate and scale; accordingly, the collation module 402 is further configured to:

[0296] Under the same coordinate and scale, the preset target region is segmented and de-coincided using the semantic mask and the two-dimensional layout frame to obtain at least one multi-modal description target object instance;

[0297] Each target object instance is grouped into a target object instance set.

[0298] In some optional embodiments, the complexity determination module 403 is further configured to:

[0299] determine an instance number of the target object instance in the target object instance set, an occlusion level index of each target object instance occluding each other, a layout divergence between each target object instance, a depth variance of depth data of each target object instance, and a category entropy of a category of each target object instance;

[0300] The instance number, the occlusion level index, the layout divergence, the depth variance and the category entropy are weighted and summed to obtain the target complexity.

[0301] In some optional embodiments, the trained diffusion probability model comprises a trained global layout diffusion sub-model and a trained entity slot diffusion sub-model, and the target guidance information tensor comprises a scene graph relationship representing an occlusion order relationship between target object instances;

[0302] Correspondingly, the generation module 405 is further configured to:

[0303] de-noise the preset random noise image according to the target noise schedule and the target time step using the trained global layout diffusion sub-model to obtain a global target coarse de-noised image corresponding to the target guidance information tensor;

[0304] determine a processing order of each target slot according to the occlusion order relationship in the scene graph relationship, each target slot being used to describe a position of a corresponding target object instance in the global target coarse de-noised image;

[0305] generate a target slot refinement patch corresponding to a current target slot according to the processing order using the trained entity slot diffusion sub-model, and superimpose the current target slot refinement patch and a current base image to obtain a next base image, until target slot refinement patches corresponding to all target slots are superimposed to obtain a target image;

[0306] wherein the first base image is obtained by superimposing the global target coarse de-noised image and a target slot refinement patch corresponding to a first target slot in the processing order.

[0307] In some optional embodiments, the generation apparatus 400 further comprises a training module 406 configured to:

[0308] Before de-noising the preset random noise image according to the target noise schedule and the target time step using the trained diffusion probability model, the following steps are performed:

[0309] obtain a plurality of preset training images, and a training instance set corresponding to each training image and a training noise image corresponding to each training image, each training instance set comprising at least one training object instance described in multiple modalities;

[0310] de-noise each training noise image using the diffusion probability model to be trained, and generate training slot refinement patches corresponding to each training object instance in each de-noised training noise image to obtain a corresponding reconstructed image;

[0311] determine a hybrid loss value between each training image and the corresponding reconstructed image, the hybrid loss value representing a difference degree between each training image and the corresponding reconstructed image;

[0312] determining whether the mixed loss value is less than or equal to a preset loss threshold value in continuous multiple rounds of training;

[0313] In response to determining that the mixed loss value is not less than or equal to the preset loss threshold value in continuous multiple rounds of training, adjusting model parameters of the diffusion probability model to be trained based on the mixed loss value, and performing the next round of training using the adjusted diffusion probability model until the mixed loss value is less than or equal to the preset loss threshold value in continuous multiple rounds of training, thereby obtaining the trained diffusion probability model.

[0314] Before denoising each training noise image using the diffusion probability model to be trained, the following steps can be performed:

[0315] Obtaining a training complexity corresponding to each training image, a globally calibrated coarse denoising image corresponding thereto, and a calibration slot corresponding thereto;

[0316] Injecting noise into each globally calibrated coarse denoising image to obtain a corresponding training noise image;

[0317] Retrieving a training noise schedule and a training time step that match the corresponding training complexity;

[0318] The diffusion probability model to be trained includes a global layout diffusion sub-model to be trained and an entity slot diffusion sub-model to be trained;

[0319] Correspondingly, denoising each training noise image using the diffusion probability model to be trained, and generating a training slot refinement patch corresponding to each training object instance in each denoised training noise image to obtain a corresponding reconstructed image, including:

[0320] Denoising the corresponding training noise image according to the corresponding training noise schedule and the corresponding training time step using the global layout diffusion sub-model to be trained to obtain a corresponding global training coarse denoising image;

[0321] Generating a training slot refinement patch corresponding to each calibration slot using the entity slot diffusion sub-model to be trained, and superimposing the corresponding training slot refinement patch on the corresponding global training coarse denoising image to obtain a corresponding reconstructed image.

[0322] In some optional embodiments, after obtaining a plurality of preset training images, the scene graph relationship of each training image is obtained;

[0323] Correspondingly, determining the mixed loss value between each training image and the corresponding reconstructed image, including:

[0324] Determining a structural consistency loss value using each globally calibrated coarse denoising image and the corresponding global training coarse denoising image;

[0325] determine a reconstruction consistency loss value by using each training image and a corresponding reconstructed image;

[0326] determine a boundary continuity loss value by using a gradient of each reconstructed image and a corresponding gradient of each training slot refinement patch;

[0327] determine a relationship consistency loss value by using a scene graph relationship of each training image and a corresponding reconstructed object instance in each reconstructed image;

[0328] perform weighted summation on the structure consistency loss value, the reconstruction consistency loss value, the boundary continuity loss value, and the relationship consistency loss value to obtain a hybrid loss value.

[0329] It should be noted that the implementation details and technical effects of each unit in the image generation apparatus provided by the embodiments of the present disclosure can refer to the descriptions of other embodiments in the present disclosure, and will not be described here.

[0330] Reference will now be made to the following description Figure 5 which shows a schematic diagram of a computer system 500 suitable for implementing an electronic device of the present disclosure. Figure 5 The computer system 500 shown is merely one example. It should not be understood in any way as a limitation on the scope of use or functionality of embodiments of the present disclosure.

[0331] As shown in Figure 5 , the computer system 500 can include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 502 or loaded from a storage device 508 into a random access memory (RAM) 503. Various programs and data required for the operation of the computer system 500 are also stored in the RAM 503. The processing device 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0332] In general, the following devices can be connected to the I / O interface 505: input devices 506 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, etc.; output devices 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 508 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 509. The communication devices 509 can allow the computer system 500 to communicate wirelessly or by wire with other devices to exchange data. Although Figure 5 The computer system 500 is shown with various devices, but it should be understood that not all of the shown devices are required to be implemented or present. More or fewer devices can alternatively be implemented or present.

[0333] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.

[0334] Note that the computer readable medium described above in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination thereof. The computer readable storage medium may, for example, be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, the computer readable signal medium can include a data signal carried in a baseband or as part of a carrier wave, in which the computer readable program code is carried. Such a propagated data signal can take any of a variety of forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. The computer readable signal medium can also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer readable medium can be transmitted by any suitable medium, including but not limited to wire, cable, RF (radio frequency), or the like, or any suitable combination of the above.

[0335] The computer readable medium described above can be included in the electronic device described above; or can exist separately from the electronic device and not be assembled into the electronic device.

[0336] The computer readable medium described above carries one or more programs, which, when executed by the electronic device, cause the electronic device to implement the task management method shown in the embodiments and optional implementation manners thereof. Figure 2 The computer readable medium described above carries one or more programs, which, when executed by the electronic device, cause the electronic device to implement the task management method shown in the embodiments and optional implementation manners thereof.

[0337] Computer program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0338] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a procedure, or a part of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in some cases, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations of blocks in the block diagrams and / or flow diagrams, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0339] The units or modules described in the embodiments of the present disclosure can be implemented by software, or by hardware. In some cases, the name of the unit or module does not constitute a limitation on the unit itself, for example, the acquisition module can also be described as a "module for acquiring a target guidance information tensor for guiding the generation of a target image".

[0340] The above description is merely that of the preferred embodiments of the present disclosure and a description of the technical principles of the present disclosure. It should be understood by those skilled in the art that the disclosed scope of the present disclosure is not limited to the technical solutions with the specific combinations of the above technical features, and should also cover other technical solutions formed by the combinations of the above technical features or their equivalent features without departing from the concept of the above disclosure. For example, the technical solutions formed by the mutual replacement of the above features and the technical features with similar functions disclosed in the present disclosure (but not limited to) without departing from the concept of the above disclosure.

Claims

1. An image generation method, characterized in that, include: Obtain the target guidance information tensor used to guide the generation of the target image; The target guidance information tensor is organized into a set of target object instances, the set of target object instances including at least one target object instance with a multimodal description; Based on each target object instance, determine the target complexity of the set of target object instances; Retrieve a target noise schedule and a target time step that match the target complexity. The target noise schedule represents the denoising intensity within each denoising step, and the target time step represents the strategy for allocating denoising steps during the denoising process. The trained diffusion probability model is used to denoise the preset random noise image according to the target noise schedule and the target time step, and a target slot refinement patch corresponding to each target object instance is generated in the denoised random noise image to obtain the target image. The completed diffusion probability model includes a completed global layout diffusion sub-model and a completed entity slot diffusion sub-model, and the target guidance information tensor includes a scene graph relationship representing the occlusion order relationship between each target object instance. The step of using the trained diffusion probability model to denoise the preset random noise image according to the target noise schedule and the target time step, and generating target slot refinement patches corresponding to each target object instance in the denoised random noise image to obtain the target image, includes: The trained global layout diffusion sub-model is used to denoise the preset random noise image according to the target noise schedule and the target time step to obtain a global target coarse denoised image corresponding to the target guidance information tensor. The processing order of each target slot is determined according to the occlusion order relationship in the scene graph relationship. Each target slot is used to describe the position of the corresponding target object instance in the global target coarse denoising image. Using the trained entity slot diffusion sub-model, a target slot refinement patch corresponding to the current target slot is generated according to the processing order. The current target slot refinement patch is then superimposed on the current base image to obtain the next base image. This process continues until the target slot refinement patches corresponding to each target slot are superimposed to obtain the target image. The first base image is obtained by superimposing the global target coarse denoised image with the target slot refinement patch corresponding to the first target slot in the processing order.

2. The method according to claim 1, characterized in that, The target guidance information tensor includes at least a two-dimensional layout box and a semantic mask under the same coordinates and scale; And, the step of organizing the target guidance information tensor into a set of target object instances includes: Under the same coordinates and scale, the semantic mask and the two-dimensional layout box are used to segment and deduplicatize the preset target region to obtain at least one multimodal target object instance; The target object instances are combined into the target object instance set.

3. The method according to claim 1, characterized in that, Determining the target complexity of the set of target object instances based on each target object instance includes: Determine the number of target object instances in the target object instance set, the occlusion level index of mutual occlusion among target object instances, the layout divergence among target object instances, the depth variance of depth data of each target object instance, and the category entropy of the category of each target object instance; The target complexity is obtained by weighted summation of the number of instances, the occlusion level index, the layout divergence, the depth variance, and the class entropy.

4. The method according to claim 1, characterized in that, Before denoising the preset random noise image using the trained diffusion probability model according to the target noise schedule and the target time step, the method further includes: Obtain multiple preset training images, as well as a set of training instances corresponding to each training image and a corresponding training noise image. Each set of training instances includes at least one training object instance with multimodal description. The diffusion probability model to be trained is used to denoise each training noise image, and training slot refinement patches corresponding to each training object instance are generated in each denoised training noise image to obtain the corresponding reconstructed image. Determine the mixing loss value between each training image and its corresponding reconstructed image, whereby the mixing loss value represents the degree of difference between each training image and its corresponding reconstructed image; Determine whether the mixed loss value is less than or equal to a preset loss threshold for multiple consecutive training rounds; In response to the determination that the mixed loss value is less than or equal to a preset loss threshold for multiple consecutive training rounds, the model parameters of the diffusion probability model to be trained are adjusted based on the mixed loss value, and the adjusted diffusion probability model is used for the next round of training until the mixed loss value is less than or equal to the preset loss threshold for multiple consecutive training rounds, thus obtaining a completed diffusion probability model.

5. The method according to claim 4, characterized in that, Before denoising each training noisy image using the diffusion probability model to be trained, the method further includes: Obtain the training complexity, the corresponding global calibration coarse denoising image, and the corresponding calibration slot for each training image; Noise is injected into each globally calibrated coarsely denoised image to obtain the corresponding training noise image; Retrieve the training noise schedule and training time step that match the corresponding training complexity; The diffusion probability model to be trained includes a global layout diffusion sub-model to be trained and an entity slot diffusion sub-model to be trained. The process of denoising each training noise image using the diffusion probability model to be trained, and generating training slot refinement patches corresponding to each training object instance in each denoised training noise image to obtain the corresponding reconstructed image, includes: The global layout diffusion sub-model to be trained is used to denoise the corresponding training noise image according to the corresponding training noise schedule and the corresponding training time step to obtain the corresponding global training coarse denoised image. The training slot refinement patches corresponding to each calibrated slot are generated using the entity slot diffusion sub-model to be trained. The corresponding local training coarse denoised image is used as the base, and the corresponding training slot refinement patches are superimposed to obtain the corresponding reconstructed image. After acquiring the preset multiple training images, the method further includes: acquiring the scene graph relationship of each training image; And, determining the mixing loss value between each training image and the corresponding reconstructed image includes: The structural consistency loss value is determined by using each globally calibrated coarse denoised image and the corresponding globally trained coarse denoised image. The reconstruction consistency loss value is determined using each training image and its corresponding reconstructed image. The boundary continuity loss value is determined by using the gradient of each reconstructed image and the gradient of the corresponding training slot thinning patch. The relationship consistency loss value is determined by using the scene graph relationships of each training image and the corresponding reconstructed object instances in the reconstructed image; The mixed loss value is obtained by weighted summing of the structural consistency loss value, the reconstruction consistency loss value, the boundary continuity loss value, and the relation consistency loss value.

6. An image generation apparatus, characterized in that, include: The acquisition module is configured to acquire a target guidance information tensor used to guide the generation of the target image; The sorting module is configured to sort the target guidance information tensor into a set of target object instances, the set of target object instances including at least one target object instance with a multimodal description; The complexity determination module is configured to determine the target complexity of the set of target object instances based on each target object instance; The retrieval module is configured to retrieve a target noise schedule and a target time step that match the target complexity, wherein the target noise schedule represents the denoising intensity within each denoising step, and the target time step represents the strategy for allocating denoising steps during the denoising process. The generation module is configured to use a trained diffusion probability model to denoise a preset random noise image according to the target noise schedule and the target time step, and generate target slot refinement patches corresponding to each target object instance in the denoised random noise image to obtain the target image. The completed diffusion probability model includes a completed global layout diffusion sub-model and a completed entity slot diffusion sub-model, and the target guidance information tensor includes a scene graph relationship representing the occlusion order relationship between each target object instance. The step of using the trained diffusion probability model to denoise the preset random noise image according to the target noise schedule and the target time step, and generating target slot refinement patches corresponding to each target object instance in the denoised random noise image to obtain the target image, includes: The trained global layout diffusion sub-model is used to denoise the preset random noise image according to the target noise schedule and the target time step to obtain a global target coarse denoised image corresponding to the target guidance information tensor. The processing order of each target slot is determined according to the occlusion order relationship in the scene graph relationship. Each target slot is used to describe the position of the corresponding target object instance in the global target coarse denoising image. Using the trained entity slot diffusion sub-model, a target slot refinement patch corresponding to the current target slot is generated according to the processing order. The current target slot refinement patch is then superimposed on the current base image to obtain the next base image. This process continues until the target slot refinement patches corresponding to each target slot are superimposed to obtain the target image. The first base image is obtained by superimposing the global target coarse denoised image with the target slot refinement patch corresponding to the first target slot in the processing order.

7. An electronic device, characterized in that, include: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, It stores a computer program thereon, wherein the computer program, when executed by one or more processors, implements the method as described in any one of claims 1 to 5.

9. A computer program product, characterized in that, It includes computer program instructions, wherein when the computer program instructions are executed on a computer, the computer causes the computer to perform the method as described in any one of claims 1 to 5.

Images