Image generation method and device based on iterative focusing refinement, medium and product

Abstract

The invention discloses an image generation method and device based on iteration focusing refinement, a medium and a product, and relates to the technical field of image processing, and the method comprises the steps: obtaining a condition image of an initial generated image and an input condition image; comparing the condition image of the initially generated image with the input condition image, and determining a focusing area; constructing a mask image according to the focusing area; inputting the mask image and the input condition image into the trained FousNet to obtain an iterated generated image, and generating a corresponding condition image of the generated image according to the iterated generated image; and the focused area after iteration is obtained again based on the condition image of the generated image after iteration, the focused area is iterated for multiple times to determine the final generated image, and the quality of the generated image is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image generation method, device, medium and product based on iterative focused refinement (IFR). Background Art

[0002] With the rapid development of computer vision and image generation technologies, deep learning-based image generation methods have been widely used in multiple fields, such as text-to-image generation, layout-to-image generation, and image restoration. These methods typically rely on generative models such as generative adversarial networks (GANs), variational autoencoders (VAEs), or diffusion models, and are able to generate high-quality images based on input control conditions (such as text descriptions and layout information).

[0003] Existing conditional diffusion models, such as ControlNet++, have achieved significant progress in image generation tasks. By introducing additional conditional branches, these models can better match generated results with control conditions. However, ensuring consistency between generated results and control images remains a challenge, resulting in low quality generated images. Summary of the Invention

[0004] The purpose of this application is to provide an image generation method, device, medium and product based on iterative focus refinement, which can improve the quality of generated images.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides an image generation method based on iterative focus refinement, comprising:

[0007] Obtain the conditional image of the initial generated image and the input conditional image;

[0008] Comparing the conditional image of the initial generated image with the input conditional image to determine a focus area; the focus area is an inconsistent area;

[0009] constructing a mask image according to the focus area;

[0010] Inputting the mask image and the input condition image into the trained FocusNet to obtain an iterative generated image, and generating a conditional image of the iterative generated image according to the iterative generated image;

[0011] The conditional image of the iteratively generated image is used to replace the conditional image of the initial generated image, and the process returns to the step of "comparing the conditional image of the initial generated image with the input conditional image to determine a focused area; the focused area is an inconsistent area", and the non-focused areas other than the focused area are kept unchanged to obtain the iterative focused area, until the iteration stop condition is met, and the final generated image is determined.

[0012] Optionally, comparing the conditional image of the initially generated image with the input conditional image to determine a focus area specifically includes:

[0013] Segmenting the conditional image of the initial generated image and the input conditional image respectively to obtain a plurality of first segmentation blocks and a plurality of second segmentation blocks; each of the first segmentation blocks and the second segmentation block includes a plurality of pixel blocks;

[0014] Calculating an inconsistency parameter between a first segmented block and a second segmented block at the same position; the inconsistency parameter is a mean square error of pixel values or a structural similarity index;

[0015] The focus area is screened out based on the inconsistent parameters of the first segmentation block and the second segmentation block at all the same positions.

[0016] Optionally, before inputting the mask image and the input condition image into a trained FocusNet to obtain an iterative generated image, the image generation method based on iterative focus refinement further includes:

[0017] Acquire a data set; the data set includes a number of sample condition images and sample mask images and a generated image corresponding to each sample condition image;

[0018] The data set is used to train FocusNet to obtain a trained FocusNet.

[0019] Optionally, constructing a mask image according to the focus area specifically includes:

[0020] The focus area is subjected to binary processing to obtain a mask image, wherein an area with a mask value of 1 represents a focus area.

[0021] Optionally, generating a conditional image of the iterative generated image according to the iterative generated image specifically includes:

[0022] The conditional extraction network is used to generate a conditional image of the iterative generated image based on the iterative generated image.

[0023] Optionally, the conditional extraction network is a generative model.

[0024] Optionally, the image generation method based on iterative focus refinement further includes:

[0025] During the iteration process, the FocusCache mechanism is introduced; the FocusCache mechanism is used to accelerate the iteration process.

[0026] In a second aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned image generation method based on iterative focus refinement.

[0027] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned image generation method based on iterative focus refinement.

[0028] In a fourth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements any one of the above-described image generation methods based on iterative focus refinement.

[0029] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0030] The present application provides an image generation method, device, medium, and product based on iterative focus refinement. The method determines inconsistent focus areas in the two images by comparing the conditional image of the generated image with the input conditional image, constructs a mask image based on the focus areas, uses the mask image and the input conditional image as inputs of a trained FocusNet, and re-obtains the generated image after iteration. The focus area is re-determined based on the conditional image of the generated image after iteration and the input conditional image to obtain the focus area after iteration. The process is repeated, and the inconsistent focus area is gradually improved through multiple iterations. Compared with the existing conditional diffusion model, which attempts to match all details of the control image in a single generation process, which may lead to inconsistencies between the generated image and the control image, the present application can gradually improve the consistency between the generated image and the control image, that is, it iterates the focus area multiple times, and can iteratively and dynamically identify the most inconsistent areas in the generated image after iteration, gradually reduce the mismatch between the generated image and the input conditional image, and improve the consistency between the generated image and the input conditional image, thereby gradually improving the quality of the generated image. The method can be used in the image generation method in the conditional diffusion model. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 This is a diagram of an application environment of an image generation method based on iterative focus refinement in one embodiment of the present application;

[0033] Figure 2 A schematic flow chart of an image generation method based on iterative focus refinement provided in one embodiment of the present application;

[0034] Figure 3 A schematic diagram of a module flow of an image generation method based on iterative focus refinement provided in one embodiment of the present application;

[0035] Figure 4 A schematic diagram illustrating the principle of a focus area identification module provided in one embodiment of the present application;

[0036] Figure 5 A schematic diagram illustrating the principle of a focus area refinement module provided in one embodiment of the present application;

[0037] Figure 6 A schematic diagram illustrating the principle of FocusNet provided in one embodiment of the present application;

[0038] Figure 7 A schematic diagram illustrating the principle of an iterative focus refinement module provided in one embodiment of the present application;

[0039] Figure 8 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0041] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0042] The image generation method based on iterative focus refinement provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the conditional image of the initial generated image to be processed and the input conditional image to the server 104. After receiving the conditional image of the initial generated image to be processed and the input conditional image, the server 104 compares the conditional image of the initial generated image with the input conditional image to determine the focus area, constructs a mask image based on the focus area, inputs the mask image and the input conditional image into the trained FocusNet, obtains the iterative generated image, and generates the corresponding conditional image of the generated image based on the iterative generated image. Based on the conditional image of the iterative generated image, the iterative focus area is obtained again, and the focus area is determined by multiple iterations. The server 104 can feedback the obtained final generated image to the terminal 102. In addition, in some embodiments, the image generation method based on iterative focus refinement can also be implemented independently by the server 104 or the terminal 102. For example, the terminal 102 can directly perform iterative focus refinement processing on the conditional image and the input conditional image of the initial generated image to be processed, or the server 104 can obtain the conditional image and the input conditional image of the initial generated image to be processed from the data storage system, and perform iterative focus refinement processing on the conditional image and the input conditional image of the initial generated image to be processed.

[0043] Terminal 102 may include, but is not limited to, various desktop computers, laptops, smartphones, tablet computers, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, and smart car devices. Portable wearable devices may include smart watches, smart bracelets, and head-mounted devices. Server 104 may be implemented as a standalone server or a server cluster consisting of multiple servers, or may be a cloud server.

[0044] In an exemplary embodiment, Figure 2 As shown, an image generation method based on iterative focus refinement is provided. The method is executed by a computer device, specifically a computer device such as a terminal or a server, or a terminal and a server. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate the process, including the following steps 201 to 205.

[0045] Step 201: Obtain a conditional image of an initially generated image and an input conditional image.

[0046] Step 202 : Compare the conditional image of the initially generated image with the input conditional image to determine a focus area; the focus area is an inconsistent area.

[0047] Step 203: construct a mask image according to the focus area.

[0048] Step 204 : Input the mask image and the input condition image into the trained FocusNet to obtain an iterative generated image, and generate a conditional image of the iterative generated image based on the iterative generated image.

[0049] In step 205, the conditional image of the iteratively generated image is used to replace the conditional image of the initial generated image. The process returns to the step of "comparing the conditional image of the initial generated image with the input conditional image to determine a focused area; the focused area is an inconsistent area." The non-focused areas other than the focused area are kept unchanged to obtain the iterative focused area until the iteration stop condition is met, thereby determining the final generated image.

[0050] Implement the above steps 201 to 205, determine the inconsistent focus areas in the two images by using the conditional image of the generated image and the input conditional image, construct a mask image based on the focus areas, use the mask image and the input conditional image as the input of the trained FocusNet, and re-obtain the generated image after iteration. Re-determine the focus area based on the conditional image of the generated image after iteration and the input conditional image to obtain the focus area after iteration, repeat this process, and gradually improve the inconsistent focus area through multiple iterations. Compared with the existing conditional diffusion model, which attempts to match all the details of the control image in a single generation process, which may lead to inconsistencies between the generated image and the control image, the present application can gradually improve the consistency between the generated image and the control image, that is, it iterate the focus area multiple times, and can dynamically identify and improve the most inconsistent areas in the generated image, thereby gradually improving the quality of the generated image. It can be used in the image generation method in the conditional diffusion model to significantly improve the consistency between the generated image and the control image. In addition, the present application introduces the FocusCache caching mechanism to reduce redundant calculations by caching high-order features, thereby significantly improving the iteration efficiency.

[0051] like Figure 3 As shown, the present embodiment provides a focus region identification module, a focus region refinement module, an iterative focus refinement module, and an image generation module.

[0052] like Figure 4As shown, in the focus area identification module, the mean squared error (MSE) or structural similarity index (SSIM) between the conditional image of the generated image and the input conditional image is calculated to identify the most inconsistent area and mark it as the focus area. After each iteration, the MSE or structural similarity index between the improved conditional image of the generated image and the input conditional image is calculated to evaluate the improvement effect, and the focus area of the next iteration is adjusted according to the evaluation result.

[0053] like Figure 5 As shown in the figure, in the focus area refinement module, a binary mask is constructed based on the identified focus area, and the mask image and the input condition image are input into FocusNet to regenerate the focus area, which is recorded as the focus area after iteration. At the same time, the non-focus area remains unchanged, and an updated mask image is constructed based on the focus area after iteration. Figure 6 As shown, FocusNet takes the updated mask image and the input conditional image as input, re-obtains the generated image after iteration, and then obtains the conditional image of the generated image after iteration through the conditional extraction network. FocusNet is a specially designed network model for processing mask images. It generates image content that is more in line with the input conditions by reinitializing the potential representation of the focus area. FocusNet includes a downsampling module, a multiplication module, a VAE encoder, a splicing module, and a generated image determination module; the generated image determination module includes a conditional network and a UNet network; the conditional network consists of a downsampling module, an attention module, and is initialized by zero convolution; the UNet network includes an encoder and a decoder. The encoder consists of a series of convolutional layers, activation functions (usually ReLU activation functions), and maximum pooling layers. The decoder is symmetrical with the encoder and consists of a series of convolutional layers, activation functions, and upsampling layers (usually transposed convolutions).

[0054] like Figure 7 As shown, in the iterative focus refinement module, the mismatch between the generated image and the input condition image is gradually reduced through multiple iterations, and finally a high-quality generation effect is achieved.

[0055] In the image generation module, Figure 7 As shown in the legend on the far right, if the preset maximum number of iterations is reached, the final generated image is output.

[0056] Prior to step 202, the image generation method based on iterative focus refinement also includes a FocusNet training phase. The training phase may include: obtaining a dataset; the dataset includes a number of sample condition images and sample mask images, and a generated image corresponding to each sample condition image; and training FocusNet using the dataset to obtain a trained FocusNet. The specific steps are as follows:

[0057] Step 301: In the training phase, the sample input condition graph I c It can be obtained from the COCO-2017 dataset repository in Hugging Face. Sample mask image I m The mask ratio is randomly initialized with mask ratio ∈ [0, 1]. The multiplication module converts the sample mask image I m And sample input condition diagram I c Multiply the corresponding positions to obtain the masked conditional graph I mc .

[0058] Step 302: During the training phase, the masked conditional graph I mc After being encoded into the latent space by the VAE encoder, the latent variable z of the mask conditional graph is obtained. mc At the same time, the sample mask image is downsampled by the downsampling module to the downsampled mask I m' , to match the size of the latent variables.

[0059] Step 303: During the training phase, the splicing module performs the noise-containing latent variable z img , the hidden variable z of the mask conditional graph mc and the downsampled mask I m' Perform splicing to obtain the spliced latent variable, denoted as z cat , as the input of the conditional network.

[0060] Step 304: During the training phase, the concatenated latent variable z cat Is input to the conditional network, the conditional network concatenates the latent variable z cat The downsampling process is performed and input into the middle layer and upsampling layer of the UNet network through zero convolution. The goal of the conditional network is to use conditional information to guide the generation process. The conditional network learns how to adjust the features in the generation process according to the input conditional information (such as text description, target image, etc.) so that the generated image or content meets the specified conditions. Through the processing of the conditional network, a feature representation that reflects the influence of the conditional information is obtained. The latent variable z containing noise img is input into the frozen UNet network, which uses the noisy latent variable z imgAs input, combined with the conditional information passed by the conditional network, the noise image is predicted (generated image). At this time, the noise image contains the conditional information of the conditional network.

[0061] After obtaining the trained FocusNet in the training phase, the inference phase begins.

[0062] The method uses a conditional diffusion model to generate an input conditional image. In step 201, the initial generated image is the output result obtained by using the conditional diffusion model, and a conditional extraction network is used to perform conditional extraction on the initial generated image to obtain a conditional image of the initial generated image.

[0063] The above step 202 may include the following steps 401 to 403 .

[0064] Step 401: Segment the conditional image of the initial generated image and the input conditional image respectively to obtain a plurality of first segmentation blocks and a plurality of second segmentation blocks; each of the first segmentation block and the second segmentation block includes a plurality of pixel blocks;

[0065] Step 402: Calculate the inconsistency parameter of the first segmented block and the second segmented block at the same position; the inconsistency parameter is the mean square error of pixel values or a structural similarity index;

[0066] Step 403: Filter out the focus area based on the inconsistent parameters of all first segmented blocks and second segmented blocks with the same position.

[0067] In the inference phase, the input conditional image c gt and the conditional image c of the generated image out are input into the focus area recognition module, and the input condition image c gt and the conditional image c of the generated image out Divide into multiple non-overlapping blocks of set pixel size. gt and the conditional image c of the generated image out The first segmentation block and the second segmentation block each have a size of 16×16 pixels. The non-overlapping blocks obtained by segmenting the conditional image of the initial generated image are recorded as the first segmentation block R. out , the non-overlapping blocks obtained by segmenting the input conditional image are recorded as the second segmentation block R gt .

[0068] When the mean square error of pixel values is used as the inconsistency parameter, for each pair of second segmentation blocks R with the same position in the image, gt and the first segmentation block R out , calculate the mean square error between them, the calculation formula is as follows:

[0069]

[0070] Among them, MSE(R gt ,R out ) is the second segmentation block R gt and the first segmentation block R out Mean square error of pixel values, c gt [i,j] and c out [i, j] represents the pixel value of the input conditional image and the conditional image of the initial generated image at position (i, j), respectively.

[0071] When the structural similarity index is used as the inconsistency parameter, the calculation formula is as follows:

[0072]

[0073] In the formula, SSIM(R gt ,R out ) represents the second segmentation block R gt and the first segmentation block R out The structural similarity index between gt and μ out The second segmentation block R gt and the first segmentation block R out The average brightness, σ gt ,σ out The second segmentation block R gt and the first segmentation block R out Standard deviation of pixels, σ gt,out The second segmentation block R gt and the first segmentation block R out Covariance of pixels; C1, C2 are constants used to avoid the denominator being zero. Usually C1 = (K1L) 2 and C2=(K2L) 2 , where K1 = -0.01 and K2 = -0.03, L is the dynamic range, for example, for an 8-bit image, L = -255.

[0074] In another exemplary embodiment of the present application, the above step 203 specifically includes: performing binary processing on the focus area to obtain a mask image, wherein the area with a mask value of 1 represents the focus area.

[0075] In the inference phase, a binary mask image M is generated based on the identified focus area:

[0076]

[0077] Where M[i,j] represents the mask value at position [i,j] in the conditional image of the generated image, i and j are the row and column numbers of the position in the conditional image of the generated image, respectively. Focused region represents the focused region, and if(i,j)∈focused region means that if position [i,j] belongs to the focused region, the mask value of that position is set to 1, otherwise the mask value of that position is set to 0. Mask M is used in the subsequent focus region refinement step, where the region with a value of 1 indicates the focused region that needs to be regenerated.

[0078] In the inference phase, the trained FocusNet is used to regenerate the focused area while keeping the non-focused area unchanged. gt Input to the trained FocusNet. The trained FocusNet will update the hidden variable z according to the following rules:

[0079]

[0080] Among them, z i represents the updated hidden variable corresponding to the current iteration, i represents the number of current iterations, M is the binary mask in the mask image, z new Refers to the newly initialized hidden variable, Z prev Represents the latent variables saved in the previous iteration, ⊙ represents element-by-element multiplication; focused regions latents represents the update of the focused area, that is, in each iteration, the focused area will be updated according to the mask part M in the mask image and the newly initialized latent variable z new To update; other regionslatents represents the update of non-focused regions, that is, the non-focused regions will be updated by the latent variables z saved in the previous iteration. prev and the non-masked part (1-M) of the mask image to update.

[0081] The downsampling module downsamples the mask image to obtain a downsampled mask. The multiplication module multiplies the input conditional image with the mask image to obtain the masked conditional image. The VAE encoder encodes the masked conditional image into the latent space to obtain the latent variables of the masked conditional image. The concatenation module concatenates the downsampled mask, the latent variables of the masked conditional image, and the noisy latent variables to obtain concatenated latent variables. These concatenated latent variables are input to the conditional network, which downsamples them and, through zero convolution, feeds them into the intermediate and upsampling layers of the UNet network. The goal of the conditional network is to use conditional information to guide the generation process. The conditional network learns how to adjust features in the generation process based on the input conditional information (such as text descriptions or target images) so that the generated image or content meets the specified conditions. The processing of the conditional network results in a feature representation that reflects the influence of the conditional information. The UNet network takes the latent variable zimg as input and, combined with the conditional information transmitted by the conditional network, predicts the noisy image, which is the generated image output by the UNet after iteration. The noise image at this time contains the conditional information of the conditional network.

[0082] In the inference stage, according to the updated latent variable z obtained above i , use the VAE decoder to decode the updated latent variables to generate the generated image after iteration, and the generated image after iteration is the intermediate image image i The generated image after iteration is applied to the conditional extraction network to obtain the conditional image of the generated image after iteration, that is, a new conditional graph is obtained. During the iteration process, the conditional extraction network can be used as the model G of the adversarial generative network, that is, the generative model.

[0083] Then, in the above step 204, generating a conditional image of the iterative generated image according to the iterative generated image specifically includes: using a condition extraction network to generate a conditional image of the iterative generated image according to the iterative generated image.

[0084] By iteratively executing the operations from step 202 to step 204 above, the areas in the generated image that are inconsistent with the input condition image are gradually refined. The iteration stopping condition is that the current number of iterations reaches the preset maximum number of iterations. When the maximum number of iterations is reached, the iteration stops to obtain the final generated image.

[0085] The image generation method based on iterative focus refinement provided by this application also includes: introducing a FocusCache mechanism during the iteration process; utilizing the FocusCache mechanism to accelerate the iteration process; the FocusCache mechanism caches high-order features in each iteration and reuses the high-order features in subsequent iterations to avoid repeated calculations, thereby significantly improving the iteration speed. For example, the updated features of the non-focused area in the current iteration can be saved and directly used in the next iteration, accelerating the process through the FocusCache mechanism, that is, using the FocusCache mechanism (caching mechanism) to reduce redundant calculations and improve iteration efficiency.

[0086] The present application also provides an application scenario, which applies the above-mentioned image generation method based on iterative focus refinement. Specifically: the image generation method based on iterative focus refinement provided in this embodiment can be applied in the image generation scenario. The image generation scenario includes an image acquisition link and an iterative focus refinement link; the conditional image of the initial generated image to be processed and the input conditional image enter the iterative focus refinement link from the image acquisition link, and the corresponding final generated image is obtained through human-computer collaboration. The image generation method based on iterative focus refinement provided in this embodiment belongs to the iterative focus refinement link. Specifically, in the iterative focus refinement link process for the video, the conditional image of the initial generated image and the input conditional image can be compared to determine the focus area, and a mask image is constructed according to the focus area. The mask image and the input conditional image are input into the trained FocusNet to obtain the generated image after iteration, and the conditional image of the corresponding generated image is generated according to the generated image after iteration. The focus area after iteration is obtained again based on the conditional image of the generated image after iteration, and the final generated image is determined by multiple iterations of the focus area.

[0087] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 8As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image generation data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image generation method based on iterative focus refinement is implemented.

[0088] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0089] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0090] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0091] In an exemplary embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0092] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0093] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0094] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0095] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0096] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. An image generation method based on iterative focus refinement, characterized in that: The image generation method based on iterative focus refinement includes: Obtain the conditional image of the initial generated image and the input conditional image; Comparing the conditional image of the initial generated image with the input conditional image to determine a focus area; the focus area is an inconsistent area; constructing a mask image according to the focus area; Inputting the mask image and the input condition image into the trained FocusNet to obtain an iterative generated image, and generating a conditional image of the iterative generated image according to the iterative generated image; The conditional image of the iteratively generated image is used to replace the conditional image of the initial generated image. The process returns to the step of "comparing the conditional image of the initial generated image with the input conditional image to determine a focused area; the focused area is an inconsistent area." The non-focused areas other than the focused area remain unchanged to obtain the iterative focused area. The process continues until the iteration stopping condition is met, and the final generated image is determined.

2. The image generation method based on iterative focus refinement according to claim 1, characterized in that: Comparing the conditional image of the initial generated image with the input conditional image to determine a focus area specifically includes: Segmenting the conditional image of the initial generated image and the input conditional image respectively to obtain a plurality of first segmentation blocks and a plurality of second segmentation blocks; each of the first segmentation blocks and the second segmentation block includes a plurality of pixel blocks; Calculating an inconsistency parameter between a first segmented block and a second segmented block at the same position; the inconsistency parameter is a mean square error of pixel values or a structural similarity index; The focus area is screened out based on the inconsistent parameters of the first segmentation block and the second segmentation block at all the same positions.

3. The image generation method based on iterative focus refinement according to claim 1, characterized in that: Before inputting the mask image and the input condition image into the trained FocusNet to obtain the iterative generated image, the image generation method based on iterative focus refinement further includes: Acquire a data set; the data set includes a number of sample condition images and sample mask images and a generated image corresponding to each sample condition image; The data set is used to train FocusNet to obtain a trained FocusNet.

4. The image generation method based on iterative focus refinement according to claim 1, characterized in that: Constructing a mask image according to the focus area specifically includes: The focus area is subjected to binary processing to obtain a mask image, wherein an area with a mask value of 1 represents a focus area.

5. The image generation method based on iterative focus refinement according to claim 1, characterized in that: Generate a conditional image of the iterative generated image according to the iterative generated image, specifically including: The conditional extraction network is used to generate a conditional image of the iterative generated image based on the iterative generated image.

6. The image generation method based on iterative focus refinement according to claim 1, characterized in that: The conditional extraction network is a generative model.

7. The image generation method based on iterative focus refinement according to claim 1, characterized in that: The image generation method based on iterative focus refinement also includes: During the iteration process, the FocusCache mechanism is introduced; the FocusCache mechanism is used to accelerate the iteration process.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image generation method based on iterative focus refinement according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image generation method based on iterative focus refinement according to any one of claims 1 to 7 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the image generation method based on iterative focus refinement according to any one of claims 1 to 6 is implemented.

Images