Diffusion model training method and apparatus, server, and storage medium
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA UNITED NETWORK COMM GRP CO LTD
- Filing Date
- 2026-02-13
- Publication Date
- 2026-06-05
AI Technical Summary
In existing technologies, training diffusion models with low-to-medium resolution images has limited application scenarios, and when processing high-resolution images, issues such as server memory overflow and computation timeouts are prone to occur.
By acquiring high-resolution images, semantic segmentation is performed to determine editable and non-editable regions, a latent space mask matrix is constructed, a pre-defined diffusion model is used to process latent space variables and tensors, a loss function is determined and the diffusion model is trained, an image reconstruction loss function is constructed, and finally a diffusion model suitable for high-resolution images is obtained.
It achieves stable operation of the diffusion model in high-resolution image processing, reduces the computational pressure on the server and the deployment threshold, requires no additional computing resources, and is seamlessly compatible with existing models.
Smart Images
Figure CN122156371A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a diffusion model training method, apparatus, server and storage medium. Background Technology
[0002] Diffusion models are a type of generative AI model that generates data by progressively adding noise to the data and then reversing the process. During training, they learn to predict noise; during inference, they start with random noise and generate data through multiple denoising steps. They are particularly effective in the image domain, used to generate high-quality images.
[0003] In existing technologies, traditional diffusion model training methods involve a diffusion model consisting of an encoder, a decoder, and a series of denoising modules. The modules in the diffusion model are trained using original low-to-medium resolution images to obtain a trained diffusion model.
[0004] However, the existing technology of training each module in the diffusion model using raw low-to-medium resolution images has limited application scenarios. When the trained model processes high-resolution images, it is prone to problems such as server memory overflow and computation timeout. Summary of the Invention
[0005] This application provides a diffusion model training method, apparatus, server, and storage medium to address the limitation of training modules in a diffusion model using raw low-to-medium resolution images. When the trained model processes high-resolution images, issues such as server memory overflow and computation timeouts can easily occur.
[0006] Firstly, this application provides a diffusion model training method applied to a server, comprising:
[0007] Acquire a high-resolution image of a preset size for model training, and determine the corresponding tensor based on the high-resolution image;
[0008] Semantic segmentation is performed on the high-resolution image to determine the editing region corresponding to the preset editing target, and the region in the high-resolution image other than the editing region is determined as the non-editing region;
[0009] Construct a corresponding latent space mask matrix based on the edited region and the non-edited region;
[0010] The latent space variables at each time step are determined based on the tensor and multiple preset parameters;
[0011] The latent space variables and the tensor are input into a preset diffusion model for processing to obtain the corresponding model output.
[0012] The loss function corresponding to the edited region is determined based on the preset transformation algorithm, the model output, and the latent space mask matrix.
[0013] Construct an image reconstruction loss function for the non-editable region, and determine the final loss function based on the loss function and the image reconstruction loss function;
[0014] The preset diffusion model is trained according to the final loss function to obtain a trained diffusion model, which is used for image editing of high-resolution images.
[0015] In one possible design, determining the corresponding tensor based on the high-resolution image includes: dividing the high-resolution image into multiple low-resolution images according to a preset partitioning size and spatial rearrangement method; encoding each low-resolution image using a preset encoder to obtain an image latent space vector corresponding to the target dimension; and stacking the latent space vectors of each image to obtain the corresponding tensor.
[0016] In one possible design, dividing the high-resolution image into multiple low-resolution images according to a preset partitioning size and spatial rearrangement method includes: dividing the high-resolution image into a corresponding number of non-overlapping image blocks according to the preset partitioning size; acquiring pixels from multiple locations in each image block; and constructing multiple low-resolution images based on the pixels in each image block according to the spatial rearrangement method.
[0017] In one possible design, constructing the corresponding latent space mask matrix based on the edited region and the non-edited region includes: constructing an initial latent space mask matrix based on the edited region and the non-edited region; and assigning values to different positions of the initial latent space mask matrix according to preset requirements to obtain the latent space mask matrix.
[0018] In one possible design, determining the loss function corresponding to the edited region based on the preset transformation algorithm, the model output, and the latent space mask matrix includes: performing high-frequency and low-frequency component separation processing on the model output using the preset transformation algorithm to obtain multiple components; and determining the loss function corresponding to the edited region based on the latent space mask matrix, the multiple components, and the preset diffusion model standard paradigm.
[0019] In one possible design, the preset transformation algorithm includes a scaling function and a wavelet function; correspondingly, the step of separating high-frequency and low-frequency components of the model output through the preset transformation algorithm to obtain multiple components includes: determining the corresponding wavelet transform function based on the scaling function and the wavelet function; and separating high-frequency and low-frequency components of the model output based on the wavelet transform function to obtain multiple components.
[0020] In one possible design, after determining the latent space variables at each time step based on the tensor and multiple preset parameters, the design further includes: in a self-attention mechanism, sharing the query matrix, key matrix, and value matrix of the latent space variables to adapt to the corresponding latent space dimension.
[0021] Secondly, this application provides a diffusion model training device applied to a server, comprising:
[0022] The first determining module is used to acquire a high-resolution image of a preset size for model training, and to determine the corresponding tensor based on the high-resolution image.
[0023] The segmentation module is used to perform semantic segmentation on the high-resolution image to determine the editing region corresponding to the preset editing target, and to determine the region in the high-resolution image other than the editing region as the non-editing region.
[0024] The construction module is used to construct a corresponding latent space mask matrix based on the edited region and the non-edited region;
[0025] The second determining module is used to determine the latent space variables at each time step based on the tensor and multiple preset parameters;
[0026] The input module is used to input the latent space variables and the tensor into a preset diffusion model for processing, so as to obtain the corresponding model output quantity;
[0027] The third determining module is used to determine the loss function corresponding to the edited region based on the preset transformation algorithm, the model output, and the latent space mask matrix.
[0028] The fourth determining module is used to construct the image reconstruction loss function for the non-edited region, and determine the final loss function based on the loss function and the image reconstruction loss function;
[0029] The training module is used to train the preset diffusion model according to the final loss function to obtain a trained diffusion model, which is used for image editing of high-resolution images.
[0030] Thirdly, this application provides a server, including: at least one processor and a memory;
[0031] The memory stores computer-executed instructions;
[0032] The at least one processor executes the computer execution instructions stored in the memory, causing the at least one processor to perform the diffusion model training method as described in the first aspect and various possible designs of the first aspect.
[0033] Fourthly, this application provides a computer storage medium storing computer execution instructions, which, when executed by a processor, implement the diffusion model training method described in the first aspect and various possible designs of the first aspect.
[0034] The diffusion model training method, apparatus, server, and storage medium provided in this application acquire a high-resolution image of a preset size for model training and determine the corresponding tensor based on the high-resolution image; perform semantic segmentation on the high-resolution image to determine the edit region corresponding to the preset editing target, and determine the region in the high-resolution image other than the edit region as the non-edit region; construct the corresponding latent space mask matrix based on the edit region and the non-edit region; determine the latent space variables at each time step based on the tensor and multiple preset parameters; input the latent space variables and tensor into a preset diffusion model for processing to obtain the corresponding model output; determine the loss function corresponding to the edit region based on the preset transformation algorithm, the model output, and the latent space mask matrix; and construct the non-edit region... The image reconstruction loss function for the edited region is determined, and the final loss function is determined based on the loss function and the image reconstruction loss function. The preset diffusion model is trained based on the final loss function to obtain a trained diffusion model. The trained diffusion model is used for image editing of high-resolution images. The tensor determined by the high-resolution image is input into the diffusion model for processing to obtain the model output. The loss function for the edited region is determined based on the model output and the latent space mask matrix, and integrated with the image reconstruction loss function for the non-edited region to obtain the final loss function, which is used to train the diffusion model. This allows the trained diffusion model to process high-resolution images without additional computing resources, thereby ensuring stable server operation and greatly reducing the deployment threshold. Attached Figure Description
[0035] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0036] Figure 1This is a schematic diagram illustrating an application scenario of the diffusion model training method provided in the embodiments of this application;
[0037] Figure 2 A flowchart illustrating the diffusion model training method provided in this application embodiment. Figure 1 ;
[0038] Figure 3 A flowchart illustrating the diffusion model training method provided in this application embodiment. Figure 2 ;
[0039] Figure 4 This is a schematic diagram of the structure of the diffusion model training device provided in the embodiments of this application;
[0040] Figure 5 A schematic diagram of the hardware structure of the server provided in an embodiment of this application;
[0041] Figure 6 A schematic diagram of the latent space mask matrix provided in an embodiment of this application;
[0042] Figure 7 This is a schematic diagram of the tensor generation process provided in an embodiment of this application. Detailed Implementation
[0043] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0044] Diffusion models are a type of generative AI model that generates data by progressively adding noise and then reversing the process. During training, they learn to predict noise; during inference, they start with random noise and generate data through multiple denoising steps. They are particularly effective in the image domain, used to generate high-quality images. In existing technologies, traditional diffusion model training methods consist of an encoder, decoder, and a series of denoising modules. These modules are trained using raw, low-to-medium resolution images to obtain a trained diffusion model. However, this method of training modules using raw, low-to-medium resolution images has limitations in application scenarios. When the trained model processes high-resolution images, it is prone to server memory overflow and computation timeouts.
[0045] To address the aforementioned technical problems, this application proposes the following technical concept: Considering high-resolution images used for model training, the inventors determine tensors, edited regions, and non-edited regions based on the high-resolution images, construct latent space mask matrices based on the edited and non-edited regions, determine latent space variables at each time step based on the tensors and their parameters, process the latent space variables and tensors using a preset diffusion model to obtain the model output, determine the loss function corresponding to the edited region based on the preset transformation algorithm, the model output, and the latent space mask matrix, determine the final loss function based on the loss function and the image reconstruction loss function corresponding to the non-edited region, and train the preset diffusion model to obtain a trained diffusion model, enabling the server to still operate stably when the model processes high-resolution images.
[0046] Figure 1 This is a schematic diagram illustrating an application scenario of the diffusion model training method provided in this application embodiment.
[0047] like Figure 1 As shown, the scenario includes a display terminal 101 and a server 102.
[0048] The display terminal 101 can be a display screen or a personal computer or other terminal.
[0049] Server 102 can be a standalone server or a cluster of multiple servers.
[0050] Server 102 acquires a high-resolution image of a preset size for model training and determines the corresponding tensor based on the high-resolution image; performs semantic segmentation on the high-resolution image to determine the edit region corresponding to the preset editing target, and defines the regions in the high-resolution image other than the edit region as non-edit regions; constructs the corresponding latent space mask matrix based on the edit region and non-edit region; determines the latent space variables at each time step based on the tensor and multiple preset parameters; inputs the latent space variables and tensor into a preset diffusion model for processing to obtain the corresponding model output; determines the loss function corresponding to the edit region based on the preset transformation algorithm, the model output, and the latent space mask matrix; constructs the image reconstruction loss function for the non-edit region, and determines the final loss function based on the loss function and the image reconstruction loss function; trains the preset diffusion model based on the final loss function to obtain the trained diffusion model, and outputs a prompt indicating that the diffusion model training is complete to the display terminal 101. The following detailed embodiments illustrate this process.
[0051] Figure 2 A flowchart illustrating the diffusion model training method provided in this application embodiment. Figure 1 The execution entity in this embodiment can be Figure 1 The server in the illustrated embodiment is not specifically limited in this embodiment. Figure 2 As shown, the method includes:
[0052] S201: Obtain a high-resolution image of a preset size for model training, and determine the corresponding tensor based on the high-resolution image.
[0053] For example, the preset dimensions are length and width respectively. .
[0054] In this embodiment, the high-resolution image can be any of the following: a 4K image, an 8K image, or a 10K image, or other high-resolution images.
[0055] For example, a high-resolution image is a 4k image I.
[0056] For example, the tensor is .
[0057] S202: Perform semantic segmentation on the high-resolution image to determine the editing region corresponding to the preset editing target, and determine the region in the high-resolution image other than the editing region as the non-editing region.
[0058] Specifically, semantic segmentation is performed on image I to obtain a mask image. This is used to determine the editing area corresponding to the preset editing target, and to identify the area in the high-resolution image other than the editing area as the non-editing area.
[0059] The preset editing targets can be tasks such as adding, deleting, modifying, and replacing.
[0060] S203: Construct the corresponding latent space mask matrix based on the edited and non-edited regions.
[0061] Specifically, step S203 includes steps a~b:
[0062] Step a: Construct the corresponding initial latent space mask matrix based on the edited and non-edited regions.
[0063] Specifically, the initial latent space mask matrix M0 in the corresponding self-attention mechanism is constructed based on the edited region and the non-edited region.
[0064] Step b: Assign values to different positions of the initial latent space mask matrix according to the preset requirements to obtain the latent space mask matrix.
[0065] For example, the latent space mask matrix is M.
[0066] In this embodiment, the preset requirement is as follows: Requirement 1: The non-editable region needs to remain completely consistent before and after editing. To ensure the stability of the non-editable region, it does not need to receive information from the edited region. Therefore, attention interaction between any token i in the non-editable region and any token j in the edited region needs to be prohibited. express.
[0067] Requirement two: the edited area needs to ensure a perfect transition between the generated result and the non-edited area. Therefore, it is necessary to obtain certain information from the non-edited area, such as lighting, and use... express.
[0068] Requirement 3: Within each of the two regions, all information about other tokens within that region is needed. express.
[0069] In this embodiment, according to preset requirements, values are assigned to different positions of the initial latent space mask matrix to obtain the latent space mask matrix. The processing formula is as follows:
[0070]
[0071] For example, a schematic diagram of the latent space mask matrix can be found in [link to diagram]. Figure 6 As shown.
[0072] In the schematic diagram of the latent space mask matrix, p_e represents the token from the edit region; p_n represents the token from the non-edit region.
[0073] S204: Determine the latent space variables at each time step based on the tensor and multiple preset parameters.
[0074] In this embodiment, multiple preset parameters include hyperparameters in the diffusion process and a standard normal distribution.
[0075] In this embodiment, the formula for calculating the latent space variables at each time step based on the tensor and multiple preset parameters is as follows:
[0076]
[0077] In the formula, These are implicit space variables; Let be the hyperparameter of the diffusion process, and t be the time corresponding to the time. It is a tensor; It follows a standard normal distribution.
[0078] Furthermore, after step S204, the method further includes: in the self-attention mechanism, sharing the query matrix, key matrix, and value matrix of the latent space variables to adapt to the corresponding latent space dimensions.
[0079] in, Having the same dimension, the low-resolution images obtained in the tensor calculation corresponding to S201 are approximately substitutes for each other. Therefore, in the self-attention mechanism, the query matrix, key matrix, and value matrix, i.e., Q, K, and V matrices, can also be shared to adapt to the corresponding latent space dimension.
[0080] Furthermore, during training, each time the Q, K, and V matrices of the attention mechanism are calculated, they are randomly selected from... Take one from the last dimension, that is .
[0081] S205: Input the latent space variables and tensors into the preset diffusion model for processing to obtain the corresponding model output.
[0082] In this implementation, the latent space variables and tensors are input into a preset diffusion model for processing, and the calculation formula for the corresponding model output is as follows:
[0083]
[0084] In the formula, For model output, These are model parameters; These are latent space variables; t corresponds to time t. It follows a standard normal distribution; It is a tensor.
[0085] S206: Determine the loss function corresponding to the edit region based on the preset transformation algorithm, model output, and latent space mask matrix.
[0086] Specifically, step S206 includes steps a~b:
[0087] Step a: Using a preset transformation algorithm, separate the high-frequency components and low-frequency components of the model output to obtain multiple components.
[0088] The preset transformation algorithm is the two-dimensional discrete Haar wavelet transform algorithm.
[0089] In this embodiment, the preset transformation algorithm includes a scaling function and a wavelet function; correspondingly, step a specifically includes steps a1 to a2:
[0090] Step a1: Determine the corresponding wavelet transform function based on the scaling function and the wavelet function.
[0091] In this embodiment, the scaling function is The wavelet function is .
[0092] Since determining the corresponding wavelet transform function based on the scaling function and the wavelet function is quite cumbersome, the wavelet transform function can be directly given as follows: .
[0093] Step a2: Based on the wavelet transform function, separate the high-frequency components and low-frequency components of the model output to obtain multiple components.
[0094] Specifically, according to the wavelet transform function ,right High-frequency components and low-frequency components are separated to obtain multiple components LL, LH, HL and HH.
[0095] Among them, the low-frequency component mainly represents the spatially varied and highly correlated basis signals in the image. It determines the overall perceptual attributes that the human visual system first captures, covering the macroscopic structure at the scene level, the appearance representation at the object level, and the global understanding at the semantic level.
[0096] Among them, the high-frequency component focuses on the fine features of dramatic changes in local areas of the encoded image, accurately capturing key visual cues that are most easily lost during downsampling or smoothing, such as sharp boundaries, complex textures, and easily overlooked content such as facial details and hand contours.
[0097] Step b: Determine the loss function corresponding to the editing region based on the latent space mask matrix, multiple components, and the preset diffusion model standard paradigm.
[0098] In this embodiment, the formula for calculating the loss function corresponding to the edited region, based on the latent space mask matrix, multiple components, and a preset diffusion model standard paradigm, is as follows:
[0099]
[0100] In the formula, The loss function corresponding to the edited region; As a weighted term; Used to represent multiple components It is a wavelet transform function; M is used to represent the editing region, where M is the latent space mask matrix; This is the standard paradigm of the diffusion model, which has undergone wavelet transform processing.
[0101] in, Equivalent to LL, LH, HL and HH.
[0102] S207: Construct the image reconstruction loss function for the non-editable region, and determine the final loss function based on the loss function and the image reconstruction loss function.
[0103] For example, the image reconstruction loss function for constructing the non-editable region is: .
[0104] In this embodiment, the formula for calculating the final loss function based on the loss function and the image reconstruction loss function is as follows:
[0105]
[0106] In the formula, This is the final loss function.
[0107] S208: Train the preset diffusion model according to the final loss function to obtain a trained diffusion model, which is then used for image editing of high-resolution images.
[0108] Specifically, after determining the loss between the predicted noise and the actual noise, the gradient of the loss function is calculated through backpropagation, and the model parameters of the preset diffusion model are adjusted using an optimizer until the model converges to obtain a trained diffusion model.
[0109] In summary, the diffusion model training method provided in this embodiment acquires a high-resolution image of a preset size for model training and determines the corresponding tensor based on the high-resolution image; performs semantic segmentation on the high-resolution image to determine the edit region corresponding to the preset editing target, and determines the region in the high-resolution image other than the edit region as the non-edit region; constructs the corresponding latent space mask matrix based on the edit region and the non-edit region; determines the latent space variables at each time step based on the tensor and multiple preset parameters; inputs the latent space variables and tensor into the preset diffusion model for processing to obtain the corresponding model output; determines the loss function corresponding to the edit region based on the preset transformation algorithm, the model output, and the latent space mask matrix; and constructs the non-edit region... The image reconstruction loss function is used to determine the final loss function based on the image reconstruction loss function. A pre-defined diffusion model is then trained using this final loss function to obtain a trained diffusion model. This trained model is used for image editing of high-resolution images. A tensor determined by the high-resolution image is input into the diffusion model for processing, yielding the model output. The loss function for the edited region, determined based on the model output and the latent space mask matrix, is integrated with the image reconstruction loss function for the non-edited region to obtain the final loss function, which is then used to train the diffusion model. This allows the trained diffusion model to process high-resolution images without additional computational resources, ensuring stable server operation and significantly lowering the deployment threshold. Furthermore, the trained diffusion model requires no structural modifications and is seamlessly compatible with all current mainstream multivariate autoencoders. Its latent space dimension remains unchanged, and the computational overhead is negligible, enabling it to quickly adapt to existing editing models at zero cost.
[0110] In addition, the diffusion model training method provided in this embodiment avoids a surge in computation by sharing the query matrix, key matrix and value matrix of latent space variables in the self-attention mechanism to adapt to the corresponding latent space dimension, thereby reducing the processing pressure on the subsequent server.
[0111] In addition, the diffusion model training method provided in this embodiment can receive information from non-edited areas, such as lighting, through the edited area, but the non-edited area completely blocks the information from the edited area, thus achieving seamless connection of image boundaries and protection of regional content.
[0112] In addition, the diffusion model training method provided in this embodiment determines the corresponding wavelet transform function based on the scaling function and the wavelet function; based on the wavelet transform function, the high-frequency components and low-frequency components of the model output are separated to obtain multiple components. By applying the wavelet transform function to the latent space variables corresponding to the editing region, the aim is to retain the low-frequency components while emphasizing the high-frequency components, so as to enhance the fine details and rich texture of the image generated in the editing region.
[0113] Furthermore, the diffusion model training method provided in this embodiment, from the perspective of the training and deployment process of the diffusion model, can further strengthen the consistency constraint of the non-editing region through super-resolution reconstruction loss.
[0114] Figure 3 A flowchart illustrating the diffusion model training method provided in this application embodiment. Figure 2 In the embodiments of this application, in Figure 2 Based on the provided embodiments, a detailed explanation is given of the specific implementation method for determining the corresponding tensor from the high-resolution image in S201. For example... Figure 3 As shown, the method includes:
[0115] S301: Divide a high-resolution image into multiple low-resolution images according to a preset division size and spatial rearrangement method.
[0116] Specifically, step S301 includes steps a to c:
[0117] Step a: Divide the high-resolution image into a corresponding number of non-overlapping image blocks according to the preset division size.
[0118] In this embodiment, the preset division size is: High-resolution images .
[0119] For example, m=n=2; the high-resolution image is a 4k image.
[0120] For example, according to a preset division size, the high-resolution image is divided into a corresponding number of H / 2×W / 2 non-overlapping image blocks.
[0121] Step b: Collect pixels from multiple locations in each image block.
[0122] In this embodiment, the multiple orientations are upper left, upper right, lower left, and lower right.
[0123] Step c: According to the spatial rearrangement method, construct multiple low-resolution images based on the pixels in each image block.
[0124] Specifically, according to the spatial rearrangement method, m×n low-resolution images with lengths and widths of H' and W' are constructed based on each pixel in each image block.
[0125] in, , .
[0126] For example, multiple low-resolution images are four images with a width and height of approximately 1024, which can be represented as: .
[0127] Furthermore, when m and n are within a certain range, all low-resolution images are approximately identical and can be considered approximations of the original image. Therefore, in the process of calculating the attention mechanism, there is no need for dimensional adaptation; simply copying m×n times in the last dimension is sufficient. This avoids a surge in computational complexity during self-attention calculations and allows for perfect compatibility with existing image editing models.
[0128] S302: Encode each low-resolution image using a preset encoder to obtain the image latent space vector corresponding to the target dimension.
[0129] In this embodiment, the preset encoder can be a variational autoencoder (VAE) encoder or other encoders.
[0130] For example, the target dimension is .
[0131] For example, each image's latent space vector consists of four latent space coding vectors, namely... .
[0132] in, .
[0133] S303: Stack the latent space vectors of each image to obtain the corresponding tensor.
[0134] Specifically, the latent space vectors of each image are stacked in the fourth dimension to obtain the corresponding tensor.
[0135] For example, the tensor is .
[0136] For example, the process of steps S301 to S302 can be detailed as follows: Figure 7 The diagram shows the tensor generation process.
[0137] In summary, the diffusion model training method provided in this embodiment divides a high-resolution image into multiple low-resolution images according to a preset partitioning size and spatial rearrangement method; encodes each low-resolution image using a preset encoder to obtain the corresponding image latent space vector; and stacks the latent space vectors of each image to obtain the corresponding tensor. By converting a high-resolution image into multiple low-resolution images, the efficiency and stability of subsequent diffusion model training are improved, key structural information is preserved, and computational resource consumption is reduced.
[0138] In addition, the diffusion model training method provided in this embodiment encodes each low-resolution image through a preset encoder to obtain the image latent space vector of the corresponding target dimension. This method has almost no information loss compared with the latent space vector obtained by directly encoding the high-resolution image. It also avoids the loss of image information caused by increasing the compression ratio of the variational autoencoder, and there is no need to retrain an adapted variational autoencoder for the high-resolution image.
[0139] In addition, the diffusion model training method provided in this embodiment can reduce the amount of computation while preserving high-resolution image information to the greatest extent through spatial rearrangement.
[0140] Figure 4 This is a schematic diagram of the diffusion model training device provided in an embodiment of this application. Figure 4 As shown, the diffusion model training device includes: a first determining module 401, a segmentation module 402, a construction module 403, a second determining module 404, an input module 405, a third determining module 406, a fourth determining module 407, and a training module 408.
[0141] The first determining module 401 is used to acquire a high-resolution image of a preset size for model training, and to determine the corresponding tensor based on the high-resolution image.
[0142] The segmentation module 402 is used to perform semantic segmentation on the high-resolution image to determine the editing region corresponding to the preset editing target, and to determine the region in the high-resolution image other than the editing region as the non-editing region.
[0143] Module 403 is used to construct the corresponding latent space mask matrix based on the edited area and the non-edited area;
[0144] The second determining module 404 is used to determine the latent space variables at each time step based on the tensor and multiple preset parameters;
[0145] The input module 405 is used to input latent space variables and tensors into a preset diffusion model for processing in order to obtain the corresponding model output quantities;
[0146] The third determining module 406 is used to determine the loss function corresponding to the edit region based on the preset transformation algorithm, the model output and the latent space mask matrix.
[0147] The fourth determining module 407 is used to construct the image reconstruction loss function for the non-editable region and determine the final loss function based on the loss function and the image reconstruction loss function;
[0148] The training module 408 is used to train the preset diffusion model according to the final loss function to obtain the trained diffusion model, which is used for image editing of high-resolution images.
[0149] In one possible implementation, the first determining module 401 specifically includes:
[0150] The partitioning unit is used to divide a high-resolution image into multiple low-resolution images according to a preset partitioning size and spatial rearrangement method.
[0151] The encoding unit is used to encode each low-resolution image through a preset encoder to obtain the image latent space vector of the corresponding target dimension.
[0152] Stacking units are used to stack the latent space vectors of each image to obtain the corresponding tensor.
[0153] In one possible implementation, the division of units specifically includes:
[0154] A partitioning unit is used to divide a high-resolution image into a corresponding number of non-overlapping image blocks according to a preset partitioning size;
[0155] The acquisition unit is used to acquire pixels from multiple locations in each image block;
[0156] The constitutive unit is used to construct multiple low-resolution images based on the pixels in each image block according to a spatial rearrangement method.
[0157] In one possible implementation, module 403 specifically includes:
[0158] The construction unit is used to construct the corresponding initial latent space mask matrix based on the edited and non-edited regions;
[0159] The assignment unit is used to assign values to different positions of the initial latent space mask matrix according to preset requirements, so as to obtain the latent space mask matrix.
[0160] In one possible implementation, the third determining module 406 specifically includes:
[0161] The separation unit is used to separate the high-frequency components and low-frequency components of the model output through a preset transformation algorithm to obtain multiple components.
[0162] The determination unit is used to determine the loss function corresponding to the editing region based on the latent space mask matrix, multiple components, and the preset diffusion model standard paradigm.
[0163] In one possible implementation, the preset transformation algorithm includes a scaling function and a wavelet function; correspondingly, the separation unit specifically includes:
[0164] The unit is used to determine the corresponding wavelet transform function based on the scaling function and the wavelet function.
[0165] The analysis unit is used to separate the high-frequency and low-frequency components of the model output based on the wavelet transform function to obtain multiple components.
[0166] In one possible implementation, the device further includes:
[0167] The shared module is used in the self-attention mechanism to share the query matrix, key matrix, and value matrix of latent space variables to adapt to the corresponding latent space dimensions.
[0168] The apparatus provided in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effects are similar, and will not be described again here.
[0169] Figure 5 This is a schematic diagram of the hardware structure of the server provided in an embodiment of this application. Figure 5 As shown, the server in this embodiment includes a processor 501 and a memory 502; the memory stores computer execution instructions; at least one processor executes the computer execution instructions stored in the memory, causing at least one processor to execute the diffusion model training method described above.
[0170] Alternatively, the memory 502 can be either standalone or integrated with the processor 501.
[0171] When the memory 502 is configured independently, the server also includes a bus 503 for connecting the memory 502 and the processor 501.
[0172] This application also provides a computer storage medium storing computer execution instructions. When the processor executes the computer execution instructions, the above-described diffusion model training method is implemented.
[0173] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described diffusion model training method.
[0174] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.
[0175] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.
[0176] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.
[0177] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.
[0178] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.
[0179] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.
[0180] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.
[0181] The aforementioned storage media can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage media can be any available medium accessible to general-purpose or special-purpose computers.
[0182] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. Both the processor and the storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic device or host device.
[0183] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.
[0184] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.
Claims
1. A diffusion model training method, characterized in that, Applied to servers, including: Acquire a high-resolution image of a preset size for model training, and determine the corresponding tensor based on the high-resolution image; Semantic segmentation is performed on the high-resolution image to determine the editing region corresponding to the preset editing target, and the region in the high-resolution image other than the editing region is determined as the non-editing region; Construct a corresponding latent space mask matrix based on the edited region and the non-edited region; The latent space variables at each time step are determined based on the tensor and multiple preset parameters; The latent space variables and the tensor are input into a preset diffusion model for processing to obtain the corresponding model output. The loss function corresponding to the edited region is determined based on the preset transformation algorithm, the model output, and the latent space mask matrix. Construct an image reconstruction loss function for the non-editable region, and determine the final loss function based on the loss function and the image reconstruction loss function; The preset diffusion model is trained according to the final loss function to obtain a trained diffusion model, which is used for image editing of high-resolution images.
2. The method according to claim 1, characterized in that, Determining the corresponding tensor based on the high-resolution image includes: The high-resolution image is divided into multiple low-resolution images according to a preset division size and spatial rearrangement method; By using a preset encoder, each low-resolution image is encoded to obtain the image latent space vector corresponding to the target dimension; The latent space vectors of each image are stacked to obtain the corresponding tensor.
3. The method according to claim 2, characterized in that, The process of dividing the high-resolution image into multiple low-resolution images according to a preset partitioning size and spatial rearrangement method includes: According to the preset division size, the high-resolution image is divided into a corresponding number of non-overlapping image blocks; Collect pixels from multiple locations in each image block; According to the spatial rearrangement method, multiple low-resolution images are constructed based on each pixel in each image block.
4. The method according to claim 1, characterized in that, The step of constructing a corresponding latent space mask matrix based on the edited region and the non-edited region includes: Construct an initial latent space mask matrix based on the edited region and the non-edited region; According to preset requirements, values are assigned to different positions of the initial latent space mask matrix to obtain the latent space mask matrix.
5. The method according to claim 1, characterized in that, The step of determining the loss function corresponding to the edited region based on the preset transformation algorithm, the model output, and the latent space mask matrix includes: The model output is processed by separating high-frequency and low-frequency components using a preset transformation algorithm to obtain multiple components. The loss function corresponding to the edited region is determined based on the latent space mask matrix, the multiple components, and the preset diffusion model standard paradigm.
6. The method according to claim 5, characterized in that, The preset transformation algorithm includes a scaling function and a wavelet function; Accordingly, the model output is processed by separating high-frequency and low-frequency components using a preset transformation algorithm to obtain multiple components, including: Determine the corresponding wavelet transform function based on the scaling function and the wavelet function; Based on the wavelet transform function, the high-frequency and low-frequency components of the model output are separated to obtain multiple components.
7. The method according to any one of claims 1 to 6, characterized in that, After determining the latent space variables at each time step based on the tensor and multiple preset parameters, the method further includes: In the self-attention mechanism, the query matrix, key matrix, and value matrix of the latent space variables are shared to adapt to the corresponding latent space dimensions.
8. A diffusion model training device, characterized in that, Applied to servers, including: The first determining module is used to acquire a high-resolution image of a preset size for model training, and to determine the corresponding tensor based on the high-resolution image. The segmentation module is used to perform semantic segmentation on the high-resolution image to determine the editing region corresponding to the preset editing target, and to determine the region in the high-resolution image other than the editing region as the non-editing region. The construction module is used to construct a corresponding latent space mask matrix based on the edited region and the non-edited region; The second determining module is used to determine the latent space variables at each time step based on the tensor and multiple preset parameters; The input module is used to input the latent space variables and the tensor into a preset diffusion model for processing, so as to obtain the corresponding model output quantity; The third determining module is used to determine the loss function corresponding to the edited region based on the preset transformation algorithm, the model output, and the latent space mask matrix. The fourth determining module is used to construct the image reconstruction loss function for the non-edited region, and determine the final loss function based on the loss function and the image reconstruction loss function; The training module is used to train the preset diffusion model according to the final loss function to obtain a trained diffusion model, which is used for image editing of high-resolution images.
9. A server, characterized in that, include: At least one processor and memory; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the diffusion model training method as described in any one of claims 1 to 7.
10. A computer storage medium, characterized in that, The computer storage medium stores computer execution instructions, and when the processor executes the computer execution instructions, it implements the diffusion model training method as described in any one of claims 1 to 7.