Context-enhanced image generation method, and model training method and system

Abstract

The present invention provides a context-enhanced image generation method, and a model training method and system, comprising: acquiring a training image data set; using a preset noise-adding mechanism to perform multi-timestep noise-adding on a training image, wherein a variance of Gaussian noise added at each timestep depends on a current timestep and progressively increases until a clean training image is transformed into standard Gaussian noise so as to obtain a noisy image at each timestep; using a random masking mechanism to generate a mask, and according to the mask, discarding a masked region of the noisy image and discarding an unmasked region of the clean image; and on the basis of reconstructing a discarded region using a non-discarded noisy image and the clean image, training a context-enhanced image generation model so as to obtain a trained image generation model. The present invention effectively improves the context comprehension capabilities of image generation methods, improves image generation quality, and achieves high-resolution diversified image generation.

Description

Context-enhanced image generation methods, model training methods, and systems Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a context-enhanced image generation method, model training method and system, as well as a medium and terminal. Background Technology

[0002] With the rapid development of artificial intelligence, computer vision technology is becoming increasingly popular and widely applied in people's daily lives and production activities. Among them, diffusion models, as a new and reliable image generation technique, have been applied to various generation fields. Through a process of forward denoising and backward denoising, diffusion models can gradually achieve controllable generation of images or videos from Gaussian noise. Specifically, the diffusion model uses a progressive forward denoising process to simulate gradually transforming a real image into standard Gaussian noise, and then uses a learnable denoising network to progressively denoise the standard Gaussian noise in the backward denoising process, thereby generating a realistic image. Therefore, this denoising network learns and predicts noise on images with different levels of noise. Due to the differences in the added noise, the network model often lacks an understanding of the consistency of the generated content, thus generating various counterfactual images and affecting the practical application of image generation technology.

[0003] To address this issue, some research methods utilize mask-reconstruction tasks to enable denoising networks to learn contextual information, thereby enhancing their deep understanding of image content. Specifically, the mask-reconstruction task divides the input noisy image into blocks and discards some noisy blocks. It then reconstructs the discarded noisy blocks using the remaining noisy blocks. Since the process of discarding noisy blocks is random, the mask-reconstruction task allows the denoising network to learn and understand different parts of the image content, enhancing its ability to utilize contextual information. In 2023, Gao et al. disclosed MDT in their paper "Masked diffusion transformer is a strong image synthesizer" at the 2023 IEEE / CVF International Conference on Computer Vision. MDT utilizes undisturbed noisy image patches to predict the noise of discarded patches and uses a side-interpolator between the denoising encoder and decoder to reduce the poor generalization problem caused by discarded patches during model training. In 2024, Zheng et al. disclosed MaskDiT in their paper "Fast Training of Diffusion Models with Masked Transformers" at Transactions on Machine Learning Research. MaskDiT uses undisturbed noisy image patches to predict clean discarded image patches, aiming to increase the model's understanding of different content during training. Also in 2024, Zhu et al. disclosed SD-DiT in their paper "SD-DiT: Unleashing the Power of Self-supervised Discrimination in Diffusion Transformer" at the 2024 IEEE / CVF Conference on Computer Vision and Pattern Recognition. SD-DiT uses undisturbed noisy image patches and a discriminative network to increase the model's discriminative ability during training. However, since the mask-reconstruction task is performed on a noisy image, as the degree of noise addition gradually increases in the forward process, the noisy image gradually tends to standard Gaussian noise, the contextual information is gradually destroyed, the learnable contextual content gradually decreases, the ability of the denoising network to extract contextual information is limited, and the contextual consistency of the generated image is affected. Summary of the Invention

[0004] In view of the deficiencies in the prior art, the purpose of this invention is to provide a context-enhanced image generation method, system, medium, and electronic device.

[0005] According to a first aspect of the present invention, a method for training a context-enhanced image generation model is provided, comprising:

[0006] Obtain the training image dataset;

[0007] The images in the training image set are subjected to multi-step noise addition using a preset noise addition mechanism to obtain noisy images at each step;

[0008] A mask is generated using a random masking mechanism, and the masked regions of the noisy image are discarded and the unmasked regions of the noisy image are discarded based on the mask.

[0009] The dropped regions are reconstructed using both the noisy and unnoisy images, and the context-enhanced image generation model is trained to obtain a well-trained image generation model.

[0010] Preferably, the step of performing multi-step noise addition on the images of the training image set using a preset noise addition mechanism to obtain noisy images at each step includes:

[0011] Gaussian noise is gradually added to the images in the training image set. The variance of the Gaussian noise added at each step depends on the current step number. The variance gradually increases with the number of steps until the un-noised image is converted into standard Gaussian noise.

[0012] Preferably, the step of generating a mask using a random masking mechanism, and discarding masked regions in the noisy image and unmasked regions in the unnoisy image based on the mask, includes:

[0013] A mask is randomly generated using a mask generation mechanism to represent the discarded and non-discarded regions in a noisy image;

[0014] The masked region of the noisy image is replaced with zero value according to the mask, while the unmasked region remains unchanged, and the masked region of the noisy image is discarded.

[0015] The unmasked region of the noiseless image is replaced with a zero value according to the mask, while the masked region remains unchanged, and the unmasked region of the noiseless image is discarded.

[0016] Preferably, the trained image generation model performs multi-step denoising on the input noisy image to gradually restore it to a noisy image.

[0017] Preferably, the image generation model includes a block embedding network, an encoder, and a decoder;

[0018] The block embedding network divides the input noisy image into blocks to obtain block embeddings.

[0019] The encoder obtains intermediate features based on the block embedding of the noisy image;

[0020] The decoder obtains the denoised restored image based on the intermediate features of the noisy image.

[0021] Preferably, the encoder and the decoder have a symmetrical structure, and both the encoder and the decoder include multiple Transformer blocks, each of which includes:

[0022] The normalization layer normalizes the block embedding of the noisy image, limiting the range of the interactive features of the subsequent output to between 0 and 1.

[0023] A multi-head self-attention layer extracts and fuses features from different normalized block embeddings to obtain interactive features;

[0024] The forward propagation layer integrates all interaction features.

[0025] Preferably, the step of reconstructing the discarded region using the noisy image without discarding and the unnoisy image, and training the context-enhanced image generation model to obtain a trained image generation model includes:

[0026] The input masked noisy image and the unnoisy image are divided into multiple blocks, and the block embedding of the noisy image and the unnoisy image are obtained through the block embedding network of the image generation model.

[0027] The encoder of the image generation model obtains the intermediate features of the noisy image based on the block embedding of the noisy image;

[0028] The composite intermediate feature is obtained by splicing the intermediate features of the noisy image with the mask and the block embedding of the uncluttered image;

[0029] The image generation model's decoder obtains a denoised composite generated image through the composite intermediate features.

[0030] The original generated image is obtained by using the un-noiseed image and the first shared network;

[0031] A denoised generated image is obtained by using the noisy image and a second shared network;

[0032] Using the mask, the reconstruction loss function L is calculated based on the undiscarded regions of the denoised composite generated image and the undiscarded image. clean ;

[0033] Using the mask, the noise loss function L is calculated based on the discarded regions of the denoised composite image and the discarded regions of the noisy image. noise ;

[0034] Using the mask, a first constrained loss function L is calculated based on the undiscarded regions of the denoised composite generated image and the undiscarded regions of the generated original image. con1 ;

[0035] Using the mask, a second constrained loss function L is calculated based on the discarded regions of the denoised composite generated image and the discarded regions of the denoised generated image. con2 ;

[0036] The reconstruction loss function, the noise-adding loss function, the first constraint loss function, and the second constraint loss function are weighted to obtain the overall loss function L, specifically: L = L clean +λ1·L noise +λ2·L con1 +λ3·L con2

[0037] Wherein, λ1 represents the weighted weight of the reconstruction loss function, λ2 represents the weighted weight of the first constraint loss function, and λ3 represents the weighted weight of the second constraint loss function;

[0038] The model parameters are updated using the backpropagation algorithm based on the overall loss function.

[0039] Preferably, the first shared network and the decoder of the image generation model have the same structure and share some parameters; the second shared network and the decoder of the image generation model have the same structure and share some parameters.

[0040] According to a second aspect of the present invention, a context-enhanced image generation model training system is provided, comprising:

[0041] Original image acquisition module: acquires the training image dataset, which includes image samples of different types;

[0042] The noise-adding module uses a preset noise-adding mechanism to add noise to the images in the training image set in multiple steps to obtain the noise-adding images at each step;

[0043] The mask discarding module generates a mask using a random masking mechanism, and discards the masked regions of the noisy image and the unmasked regions of the noisy image according to the mask.

[0044] The training module reconstructs the discarded regions using both the noisy and unnoisy images and trains the context-enhanced image generation model to obtain a well-trained image generation model.

[0045] According to a third aspect of the present invention, a context-enhanced image generation method is provided, wherein an image to be processed is obtained, input into a context-enhanced image generation model, and a denoised generated image is obtained; wherein the context-enhanced image generation model is trained by any of the context-enhanced image generation model training methods or the context-enhanced image generation model training system described herein.

[0046] Preferably, the image generation model starts with standard Gaussian noise in the image to be processed and gradually denoises it. Specifically, the denoised restored image output at each step is used as the input denoised image for the next step, and the denoised restored image output at the last step is the finally obtained generated image.

[0047] According to a fourth aspect of the present invention, a terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can be used to describe any of the methods described herein, or to run the system described herein.

[0048] According to a fifth aspect of the invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, can be used to perform any of the methods described herein, or to run the system described herein.

[0049] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0050] The context-enhanced image generation method, model training method, and system in this invention reconstruct the discarded region using the undiscarded noisy image and the original image, thereby improving the context modeling capability of the image generation model, reducing inconsistent generation cases, and demonstrating strong adaptability to various image categories, achieving low-cost and highly robust image generation.

[0051] The context-enhanced image generation method, model training method, and system in this embodiment of the invention introduce a context-enhanced image generation model composed of Transformer blocks. Through a normalization layer, a multi-head self-attention layer, and a forward propagation layer, features of different blocks are extracted and fused, interactive features are extracted, and the contextual information of different blocks of the original image is learned, thereby enhancing the content consistency of the generated image.

[0052] The context-enhanced image generation method, model training method, and system in this invention employ large-scale, high-resolution, and diverse natural images to train a context-consistent image generation model, achieving more realistic and content-consistent high-resolution image generation under both unconstrained and conditional conditions. Attached Figure Description

[0053] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0054] Figure 1 is a flowchart of a context-enhanced image generation model training method according to an embodiment of the present invention;

[0055] Figure 2 is a schematic diagram of the structure of a context-enhanced image generation model in a preferred embodiment of the present invention;

[0056] Figure 3 is a flowchart of a method for training a context-enhanced image generation model according to a preferred embodiment of the present invention.

[0057] Figure 4 is a schematic diagram of the structure of a context-enhanced image generation model training system according to an embodiment of the present invention;

[0058] Figure 5 is a generation result diagram of the context-enhanced image generation model in a specific embodiment of the present invention. Detailed Implementation

[0059] The present disclosure will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present disclosure, but do not limit the present disclosure in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present disclosure. These all fall within the protection scope of the present disclosure.

[0060] In one embodiment of the present invention, a method for training a context-enhanced image generation model is provided, as shown in Figure 1. The main steps are as follows:

[0061] S11, Obtain the training image dataset;

[0062] S12, using a preset noise-adding mechanism, perform multi-step noise addition on the images of the training image set obtained in S11 to obtain the noise-added images at each step;

[0063] S13, generate a mask using a random masking mechanism, and discard the masked region of the noisy image obtained in S12 and discard the unmasked region of the noisy image according to the mask;

[0064] S14: Based on the noisy image and the unnoisy image processed by S13, the discarded region is reconstructed, and the context-enhanced image generation model is trained to obtain the trained image generation model.

[0065] The above embodiments utilize the undiscarded noisy image and the original image to reconstruct the discarded region, which improves the context modeling capability of the image generation model, reduces inconsistent generation cases, has strong adaptability to various types of images, and achieves low-cost and highly robust image generation.

[0066] In a preferred embodiment of the present invention, step S11 is performed to obtain a training image dataset, which includes image samples of different types. Specifically, each image sample is a high-resolution image sample {x0, y0}, where x0 is the input high-resolution image and y0 is the image's category label, used for conditional generation.

[0067] In a preferred embodiment of the present invention, step S12 is performed, in which a preset noise-adding mechanism is used to add noise to the high-resolution image to different degrees. In this embodiment, Gaussian noise is gradually added to the images in the training image set, and the variance of the Gaussian noise added at each step depends on the current step number. This variance gradually increases with the increase of the number of steps until the un-noised image is converted into standard Gaussian noise.

[0068] Furthermore, in some specific embodiments, for the t-th forward propagation, Gaussian noise n ~ N(0; t) is used. 2 I) Add noise to obtain the noisy image x t = x0 + n.

[0069] In a preferred embodiment of the present invention, step S13 involves generating a mask using a random masking mechanism, and then discarding masked regions from the noisy image and unmasked regions from the original image based on the mask. The specific process is as follows:

[0070] S131, a mask is randomly generated using a mask generation mechanism to represent the discarded and non-discarded regions in the noisy image;

[0071] S132, based on the mask generated in S131, replace the masked region of the noisy image with zero values, leave the unmasked region unchanged, and discard the masked region of the noisy image.

[0072] S133: Replace the unmasked region of the noisy image with zero values ​​according to the mask generated in S131, while keeping the mask region unchanged, and discard the unmasked region of the noisy image.

[0073] Furthermore, in some embodiments, a mask m∈{0,1} is randomly generated using a mask generation mechanism. N 0 represents regions that are not discarded, and 1 represents regions that are discarded. Based on this mask m, the noisy image x after discarding is obtained. t = (x0+n)⊙(1-m) and the original image x′0=x0⊙m, where ⊙ represents element-wise multiplication.

[0074] In the foregoing embodiments, the ability to extract contextual features is improved by performing masking and discarding operations. By using the unmasked regions in the unmasked image to predict the masked regions in the noisy image, the correlation between the unmasked regions and the masked regions is learned, thereby enhancing the understanding of image contextual information.

[0075] In a preferred embodiment of the present invention, a preferred structure for a context-enhanced image generation model is provided, comprising a convolutional network, an encoder, and a decoder. Generally, the input image is processed by the convolutional network to generate block embeddings. The encoder encodes the block embeddings to obtain intermediate features, and the decoder decodes the intermediate features to obtain the generated image. The trained context-enhanced image generation model performs multi-step denoising on the input noisy image, gradually restoring it to a denoised image.

[0076] Furthermore, in some preferred embodiments, the encoder and decoder of the context-enhanced image generation model have a symmetrical structure, and both the encoder and decoder consist of multiple Transformer blocks. Each Transformer block includes a normalization layer, a multi-head self-attention layer, and a forward propagation layer. The normalization layer normalizes the block embeddings of the noisy image, limiting the range of the subsequent output interaction features to between 0 and 1; the multi-head self-attention layer extracts and fuses features from the different normalized block embeddings to extract interaction features; and the forward propagation layer processes and fuses all interaction features.

[0077] In another preferred embodiment of the present invention, step S14 is implemented by training a preset context-enhanced image generation model using a high-resolution image data training set, and reconstructing the discarded region using the undiscarded noisy image and the original image to determine the context-enhanced image generation model. As shown in Figure 2, the preset context-enhanced image generation model training process includes random mask discarding, first and second shared network constraints, and reconstruction loss constraints.

[0078] The baseline model uses a noisy image as input and consists of an encoder and a decoder. The initial parameters of the network are randomly generated and optimized according to the reconstruction loss function, the denoising loss function, the first constraint loss function, and the second constraint loss function.

[0079] As shown in Figure 3, in some preferred embodiments, a reconstruction loss function and multiple constraint loss functions are used to constrain the generation effect, optimize the preset context-enhanced image model parameters, and determine the context-enhanced image model, including S31 to S35, as follows:

[0080] S31, input the unmasked original image into the first shared network, and output the generated original image.

[0081] In some specific embodiments, a first shared network is utilized. Feature extraction and reconstruction are performed on the unmasked original image to obtain the generated original image.

[0082] S32, input the unmasked noisy image into the second shared network, and output the denoised generated image.

[0083] In some specific embodiments, a second shared network is utilized. Feature extraction and reconstruction are performed on the unmasked noisy image to obtain the generated original image.

[0084] S33, input the masked and noisy image into the encoder of the image generation model to obtain the intermediate features of the noisy image, and concatenate the intermediate features with the block embeddings of the original image to obtain the composite intermediate features.

[0085] In some specific embodiments, the encoder E of the image generation model is used to extract features from the masked noisy image to obtain the intermediate features z1=E((x0+n)⊙(1-m)), and the intermediate features and the block embeddings of the original image are concatenated to obtain the composite intermediate features z=concat(z1,x0⊙m).

[0086] S34, input the composite intermediate features into the decoder of the image generation model to obtain the denoised composite generated image.

[0087] In some specific embodiments, the decoder D of the image generation model is used to decode and reconstruct the composite intermediate features to obtain a denoised composite generated image D((x0+n)⊙(1-m),x0⊙m).

[0088] It should be noted that the order of S31, S32, S33, and S34 is not unique. In other embodiments, steps S33 and S34 can be performed first, followed by S31 and S32.

[0089] S35, using the original generated image, the denoised generated image, and the denoised composite generated image, determine the reconstruction loss function, the noise loss function, the first constraint loss function, and the second constraint loss function.

[0090] Using a mask, the reconstruction loss function L is calculated based on the undiscarded regions of the denoised composite generated image and the undiscarded regions of the original image. clean ;

[0091] in, This indicates that sample x0 follows the distribution of real image data. This indicates that the noise n follows a pattern with a mean of 0 and a variance of t.2 The distribution is Gaussian, and θ represents the parameters of the Transformer network;

[0092] Using a mask, the noise loss function L is calculated based on the discarded regions of the denoised composite image and the discarded regions of the noisy image. noise ;

[0093] Using a mask, the first constrained loss function L is calculated based on the undiscarded regions of the denoised composite generated image and the undiscarded regions of the generated original image. con1 ;

[0094] Using a mask, the second constrained loss function L is calculated based on the discarded regions of the denoised composite generated image and the discarded regions of the denoised generated image. con2 ;

[0095] The overall loss function is obtained by weighting the reconstruction loss function, the noise addition loss function, the first constraint loss function, and the second constraint loss function, specifically: L = L clean +λ1·L noise +λ2·L con1 +λ3·L con2

[0096] Where L represents the overall loss function, L clean Let L represent the reconstruction loss function. noise Let L represent the noise loss function. con1 L represents the first constraint loss function. con Let λ1 represent the weighted weight of the reconstruction loss function, λ2 represent the weighted weight of the first constraint loss function, and λ3 represent the weighted weight of the second constraint loss function.

[0097] Based on the overall loss function, gradient descent is used to optimize the parameters of the context-enhanced image generation model until the parameters of the context-enhanced image generation model converge, thus forming the context-enhanced image generation model.

[0098] The above embodiments acquire a high-resolution original image dataset; add noise to the high-resolution images to different degrees using a preset noise-adding mechanism; generate a mask using a random masking mechanism, and discard masked regions in the noisy images and unmasked regions in the original images based on the mask; use the undiscarded noisy images and the original images as input to determine a context-enhanced image generation model; train the preset context-enhanced image generation model using a high-resolution image data training set, and reconstruct the discarded regions using the undiscarded noisy images and the original images. This improves the context modeling capability of the image generation model, reduces inconsistent generation cases, has strong adaptability to various image categories, and achieves low-cost, highly robust image generation.

[0099] Based on the same inventive concept, other embodiments of the present invention provide a context-enhanced image generation model training system, as shown in Figure 4, including:

[0100] Original image acquisition module 110: Acquires a training image dataset, wherein the training image dataset includes image samples of different types;

[0101] The noise-adding module 120 uses a preset noise-adding mechanism to add noise to the images in the training image set in multiple steps to obtain the noise-adding images at each step;

[0102] The mask discarding module 130 generates a mask using a random masking mechanism, and discards the masked region of the noisy image and the unmasked region of the noisy image according to the mask.

[0103] Training module 140 trains the context-enhanced image generation model by reconstructing the discarded region using the noisy image without discarding and the noisy image without discarding, thus obtaining a trained image generation model.

[0104] The specific implementation techniques of each module / unit in the above examples of the present invention can be referred to the steps of the context-enhanced image generation model training method in the above embodiments, and will not be repeated here.

[0105] Based on the same inventive concept, in another embodiment of the present invention, a context-enhanced image generation method is provided, which obtains an image to be processed, inputs it into a context-enhanced image generation model, and obtains a denoised generated image; the context-enhanced image generation model is trained by any of the context-enhanced image generation model training methods or the context-enhanced image generation model training system described above.

[0106] Furthermore, the image generation model starts with standard Gaussian noise in the image to be processed and gradually denoises it. Specifically, the denoised and restored image output at each step is used as the input denoised image for the next step, and the denoised and restored image output at the last step is the final generated image.

[0107] To verify the feasibility and effectiveness of the context-enhanced image generation method, model training method, and system in the above embodiments, some specific embodiments of the present invention publicly collect and acquire some high-resolution and diverse natural image data ImageNet as training and testing datasets. The ImageNet dataset contains 14,197,122 images from 1,000 categories, and each image has a resolution of 224×224.

[0108] This embodiment demonstrates experimental validation on the ImageNet dataset. Two test protocols are used: unconditional generation and category-conditional generation. Each test protocol evaluates the difference between its generated images and real images.

[0109] Specifically, experiments were conducted on the ImageNet dataset. The evaluation metrics for the model included FID, sFID, IS, Prec., and Rec. The smaller the FID and sFID, and the larger the IS, Prec., and Rec., the better the model's performance.

[0110] The specific comparison results are shown in Tables 1 and 2 below. Table 1 is a comparison of the image generation quality of the first test protocol on the ImageNet dataset in an embodiment of the present invention. Table 1 shows that the image generation model (MC-DiT) of the embodiment of the present invention generates images that are closer to real images, and the generated images have higher accuracy in classification, achieving superior generation quality.

[0111] Table 2 shows a comparison of the classification accuracy results of a second test protocol on the ImageNet dataset for an embodiment of the present invention. Table 2 shows that the image generation model (MC-DiT) of the embodiment of the present invention can still achieve the best results under the constraints of category conditions, and the generated images have higher accuracy in classification, achieving better generation quality.

[0112] Table 1

[0113] Table 1: Comparison with state-of-the-art approaches for ImageNet-256×256 class conditional image generation.Bold font represents the best result.'-G'means using classifier-free guidance.

[0114] Table 2

[0115] Table 2: Comparison with state-of-the-art approaches for ImageNet-512×512 class conditional image generation.The bold font represents the best performance.

[0116] In some specific embodiments, as shown in Figure 5, the context-enhanced image generation model achieves more realistic actions and forms, greatly improves the consistency of generated content, and can achieve accurate generation for each category.

[0117] The context-enhanced image generation method in the above specific embodiments can achieve smaller differences from real images in both unclassified and classified conditions, indicating that the context-enhanced image generation model of the present invention can guarantee the consistency and realism of the generated content for different categories, thus having a wider range of application scenarios.

[0118] Based on the same inventive concept, in other embodiments of the present invention, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to perform the above-described method or to run the above-described system.

[0119] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs, functional modules, etc. that implement the above methods), computer instructions, etc., and the above-mentioned computer programs, computer instructions, etc., may be partitioned and stored in one or more memories.

[0120] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0121] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0122] Based on the same inventive concept, in other embodiments of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, can be used to perform the above-described method or to run the above-described system.

[0123] Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of computer programs from one place to another. Storage media can be any available medium accessible to a general-purpose or special-purpose computer. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a user device. Of course, the processor and storage medium can also exist as discrete components in a communication device.

[0124] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0125] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0126] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0127] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0128] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for training a context-enhanced image generation model, characterized in that, include: Obtain the training image dataset; The images in the training image set are subjected to multi-step noise addition using a preset noise addition mechanism to obtain noisy images at each step; A mask is generated using a random masking mechanism, and the masked regions of the noisy image are discarded and the unmasked regions of the noisy image are discarded based on the mask. The dropped regions are reconstructed using both the noisy and unnoisy images, and the context-enhanced image generation model is trained to obtain a well-trained image generation model.

2. The context-enhanced image generation model training method according to claim 1, characterized in that, The step of applying noise to the images in the training image set in multiple steps using a preset noise-adding mechanism to obtain noisy images at each step includes: Gaussian noise is gradually added to the images in the training image set. The variance of the Gaussian noise added at each step depends on the current step number. The variance gradually increases with the number of steps until the un-noised image is converted into standard Gaussian noise.

3. The context-enhanced image generation model training method according to claim 1, characterized in that, The step of generating a mask using a random masking mechanism, and discarding masked regions from the noisy image and unmasked regions from the unnoisy image based on the mask, includes: A mask is randomly generated using a mask generation mechanism to represent the discarded and non-discarded regions in a noisy image; The masked region of the noisy image is replaced with zero value according to the mask, while the unmasked region remains unchanged, and the masked region of the noisy image is discarded. The unmasked region of the noiseless image is replaced with a zero value according to the mask, while the masked region remains unchanged, and the unmasked region of the noiseless image is discarded.

4. The context-enhanced image generation model training method according to claim 1, characterized in that, The trained image generation model performs multi-step denoising on the input noisy image, gradually restoring it to a denoised image.

5. The context-enhanced image generation model training method according to claim 4, characterized in that, The image generation model includes a block embedding network, an encoder, and a decoder; The block embedding network divides the input noisy image into blocks to obtain block embeddings. The encoder obtains intermediate features based on the block embedding of the noisy image; The decoder obtains the denoised restored image based on the intermediate features of the noisy image.

6. The context-enhanced image generation model training method according to claim 5, characterized in that, The encoder and the decoder have a symmetrical structure, and both the encoder and the decoder include multiple Transformer blocks, each of which includes: The normalization layer normalizes the block embedding of the noisy image, limiting the range of the interactive features of the subsequent output to between 0 and 1. A multi-head self-attention layer extracts and fuses features from different normalized block embeddings to obtain interactive features; The forward propagation layer integrates all interaction features.

7. The context-enhanced image generation model training method according to claim 1, characterized in that, The process of reconstructing the discarded region using both the noisy and unnoisy images, and then training the context-enhanced image generation model to obtain a trained image generation model includes: The input masked noisy image and the unnoisy image are divided into multiple blocks, and the block embedding of the noisy image and the unnoisy image are obtained through the block embedding network of the image generation model. The encoder of the image generation model obtains the intermediate features of the noisy image based on the block embedding of the noisy image; The composite intermediate feature is obtained by splicing the intermediate features of the noisy image with the mask and the block embedding of the uncluttered image; The image generation model's decoder obtains a denoised composite generated image through the composite intermediate features. The original generated image is obtained by using the un-noiseed image and the first shared network; A denoised generated image is obtained by using the noisy image and a second shared network; Using the mask, the reconstruction loss function L is calculated based on the undiscarded regions of the denoised composite generated image and the undiscarded image. clean ; Using the mask, the noise loss function L is calculated based on the discarded regions of the denoised composite image and the discarded regions of the noisy image. noise ; Using the mask, a first constrained loss function L is calculated based on the undiscarded regions of the denoised composite generated image and the undiscarded regions of the generated original image. con1 ; Using the mask, a second constrained loss function L is calculated based on the discarded regions of the denoised composite generated image and the discarded regions of the denoised generated image. con2 ; The reconstruction loss function, the noise-adding loss function, the first constraint loss function, and the second constraint loss function are weighted to obtain the overall loss function L, specifically: L=L clean +λ1·L noise +λ2·L con1 +λ3·L Con2 Wherein, λ1 represents the weighted weight of the reconstruction loss function, λ2 represents the weighted weight of the first constraint loss function, and λ3 represents the weighted weight of the second constraint loss function; The model parameters are updated using the backpropagation algorithm based on the overall loss function.

8. The context-enhanced image generation model training method according to claim 7, characterized in that, The first shared network and the decoder of the image generation model have the same structure and share some parameters; the second shared network and the decoder of the image generation model have the same structure and share some parameters.

9. A context-enhanced image generation model training system, characterized in that, include: Original image acquisition module: acquires the training image dataset, which includes image samples of different types; The noise-adding module uses a preset noise-adding mechanism to add noise to the images in the training image set in multiple steps to obtain the noise-adding images at each step; The mask discarding module generates a mask using a random masking mechanism, and discards the masked regions of the noisy image and the unmasked regions of the noisy image according to the mask. The training module reconstructs the discarded regions using both the noisy and unnoisy images and trains the context-enhanced image generation model to obtain a well-trained image generation model.

10. A context-enhanced image generation method, characterized in that, The image to be processed is obtained and input into the context-enhanced image generation model to obtain a denoised generated image; the context-enhanced image generation model is trained by the method of any one of claims 1 to 8 or the system of claim 9.

11. The context-enhanced image generation method according to claim 10, characterized in that, The image generation model starts with standard Gaussian noise in the image to be processed and gradually removes noise, specifically as follows: The denoised restored image output from each step serves as the input denoised image for the next step, and the denoised restored image output from the last step is the final generated image.

12. A terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it can be used to execute the method of any one of claims 1-8 and 10-11, or to run the system of claim 9.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program can be used to perform the method of any one of claims 1-8, 10-11, or to run the system of claim 9.

Images