A model training method, an image migration method, a computing device and a storage medium

Abstract

The present disclosure provides a model training method, the model comprising an encoder and a diffusion transformer, the method comprising: generating, by the encoder, a first source domain feature of a training image; obtaining a first prompt feature from a source domain to a target domain and a second prompt feature from the target domain to the source domain; inputting the first source domain feature and the first prompt feature into the diffusion transformer to obtain a first target domain feature; inputting the first target domain feature and the second prompt feature into the diffusion transformer to obtain a second source domain feature; calculating a loss function according to a difference between the first source domain feature and the second source domain feature; and training the model according to the loss function. The present disclosure also provides a method for migrating an image by using an image migration model trained by the model training method, and a corresponding computing device and storage medium.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision, and more particularly to a model training method, an image transfer method, a computing device, and a storage medium. Background Technology

[0002] Image transfer models generate new images based on original images using prompts. Current image transfer models, such as diffusion models, are generative models based on Markov chains. They generate data samples by simulating a forward diffusion process from clear to noisy data and a backward diffusion process from noisy data back to clear data. Diffusion models often add Gaussian noise to the data during the forward process and train a neural network during the backward process, recovering the original data through multiple iterations. This approach is computationally expensive, slow in generation, and requires inference time, making it difficult to apply to scenarios requiring real-time interaction. Furthermore, the multi-step sampling process can lead to high deployment complexity. Therefore, a faster and more practical image transfer method is needed. Summary of the Invention

[0003] Embodiments of this disclosure provide a model training method, an image transfer method, a computing device, and a storage medium to at least solve the aforementioned problems.

[0004] To achieve the above objectives, the embodiments of this disclosure adopt the following technical solutions:

[0005] A first aspect of this disclosure provides a model training method, comprising:

[0006] A model training method, wherein the model includes an encoder and a diffusion converter, the method comprising:

[0007] The encoder is used to generate the first source domain features of the training images;

[0008] Obtain the first cue feature from the source domain to the target domain, and the second cue feature from the target domain to the source domain;

[0009] The first source domain feature and the first cue feature are input into the diffusion converter to obtain the first target domain feature;

[0010] The first target domain feature and the second cue feature are input into the diffusion converter to obtain the second source domain feature;

[0011] Calculate the loss function based on the difference between the first source domain features and the second source domain features; and

[0012] The model is trained based on the loss function.

[0013] A second aspect of this disclosure provides an image migration method, comprising:

[0014] The image transfer model is trained using the model training method described above;

[0015] The image to be processed and the prompt information are input into the trained image transfer model to obtain a new image that conforms to the prompt information.

[0016] A fourth aspect of this disclosure provides a computing device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor; wherein, when the processor runs the computer program, it performs the method described above.

[0017] A fifth aspect of this disclosure provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the method described above.

[0018] The technical solution provided in this disclosure obtains a one-step image transfer model without denoising by fine-tuning a pre-trained model. Furthermore, by training the model through the calculation of the latent space feature loss function, instead of restoring it to the image layer, the computational cost of model training is reduced, thus improving training efficiency. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the network structure of an image transfer model provided in an embodiment of the present disclosure;

[0021] Figure 2 A flowchart of a model training method 200 provided in this embodiment of the disclosure;

[0022] Figure 3 A schematic diagram illustrating a new model paradigm provided by an embodiment of this disclosure;

[0023] Figure 4 A flowchart of an encoder training method 100 provided in this embodiment of the disclosure;

[0024] Figure 5 This is a structural block diagram of a computing device 500 provided in an embodiment of the present disclosure. Detailed Implementation

[0025] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this disclosure.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] Figure 1 A schematic diagram of an image transfer model according to an embodiment of this disclosure is shown. In some embodiments, the image transfer model is, for example, a flow model. A flow model is a generative model based on optimal transport theory, which generates data samples by constructing a straight path from noise to data. A flow model is, for example, a rectified flow model. Figure 1 As shown, image transfer models (such as streaming models) include an image encoder, a cue encoder, a diffusion transformer (DiT), and an image decoder.

[0028] Image encoders are used to extract image features from input images. One type of image encoder is the Variational Autoencoder (VAE), a deep learning model that generates and encodes data. A VAE transforms the input data into a distribution in a latent space through a probabilistic encoding mechanism. Decoding can then be performed within this latent space to generate an approximate version of the original data. In video generation tasks, VAEs can be used to compress data and reduce the computational overhead of the generative model.

[0029] The prompt encoder is used to extract features (such as text vectors) from the prompt information, which can be referred to as prompt features. The prompt information in this embodiment can be text prompts, voice prompts, image prompts, etc., and this disclosure does not limit this. A voice prompt might be, for example, playing a voice message such as "Convert the image to an anime style." An image prompt could be an example image of an image style, used to convert the input image into a new image consistent with the style of that example image. Each prompt information format has a corresponding encoder used to extract the features of that prompt information, thus obtaining the prompt features.

[0030] The diffusion transformer (DiT) generates new image features that match the cue information based on image features and cue features. The image decoder decodes these new image features to obtain the corresponding new image. The input to the diffusion transformer includes image features of the input image and cue features of the cue information; the output includes new image features, which, after decoding, yield the transformed image from the input image. The diffusion transformer utilizes a self-attention mechanism to capture global and long-range dependencies in the image. By dividing the image into a series of blocks and processing these blocks in multiple Transformer layers, DiT can more effectively integrate local and global information during the diffusion process, thereby generating high-quality images.

[0031] If the input image is a brown horse image, and the prompt is "Generate a zebra from a brown horse," then the model can obtain an image that converts the brown horse into a zebra based on the input image and the prompt. Alternatively, the model can perform reverse reasoning: if the input image is a converted zebra image, and the prompt is "Generate a brown horse from a zebra," then the model can obtain an image that converts the zebra into a brown horse based on the input image and the prompt.

[0032] In some embodiments, the time steps of the original image and the noise can be input into a pre-trained image transfer model. After obtaining the predicted noise, the noise and the original image are denoised in one step according to the denoising diffusion probabilistic models (DDPM). However, this method may be difficult to converge during model fine-tuning, affecting the image transfer effect.

[0033] Therefore, this disclosure provides an alternative model training method and a novel single-step image-to-image transfer paradigm that directly utilizes the output of the diffusion converter of the pre-trained model to generate the target image, thereby bypassing the denoising process. This method enables the image transfer model to converge more easily during fine-tuning and breaks away from the inverse denoising process of traditional diffusion models.

[0034] Figure 2A flowchart of a model training method 200 according to another embodiment of the present disclosure is shown, such as a flowchart of an image transfer model training method, adapted to be executed on a computing device. Figure 2 As shown, the method includes:

[0035] Step S210: Generate the first source domain features of the training image using the encoder.

[0036] Step S220: Obtain the first cue feature from the source domain to the target domain, and the second cue feature from the target domain to the source domain. For example, the second cue is the reverse of the second cue; for instance, the first cue is to generate a zebra from a brown horse, and the second cue is to generate a brown horse from a zebra. The first cue is to convert a real image into an animated image, and the second cue is to convert an animated image into a real image.

[0037] Step S230: Input the first source domain features and the first cue features into the diffusion converter to obtain the first target domain features. This is the forward inference process of the model.

[0038] Step S240: Input the first target domain features and the second cue features into the diffusion converter to obtain the second source domain features. This is the reverse reasoning process of the model.

[0039] Step S250: Calculate the loss function based on the first difference between the first source domain features and the second source domain features; and

[0040] Step S260: Train a model based on the loss function, such as training an image transfer model. Further, a diffusion converter can be trained based on this loss function, thereby training the flow model.

[0041] Generally, the training set of an image transfer model includes training images and ground truth images corresponding to the training images, also known as target images. The target image can be the image after the training images have been processed according to the corresponding prompts, or it can be an image close to the processed image. For example, if the similarity of image features between an image in the target domain and the processed image is greater than or equal to a predetermined threshold, then that image in the target domain can be used as the ground truth image, i.e., the target image. The training set includes a source domain and a target domain, each of which can contain multiple images. A source domain image can be converted into a target domain image based on a first prompt, and a target domain image can be converted into a source domain image based on a second prompt. The source domain and target domain are relative concepts; training images can be selected from the source domain or the target domain. After the encoder extracts features from the source domain image, it obtains the source domain features of that image; after extracting features from the target domain image, it obtains the target domain features of that image.

[0042] Here, we first define the image transfer task based on a pre-trained flow model from the perspective of the flow path in the latent space of the flow model. Suppose we need to transform an image from the source domain p(a) to the target domain p(b), we can randomly select an image from the source domain p(a). Considering that many datasets in the source and target domains may be unpaired, therefore The corresponding image in the target domain may be unknown. It can be... The corresponding optimal transformed image (i.e., the target image) is defined as Then, in the latent space after compression by the image encoder, and Corresponding to and

[0043] Therefore, this disclosure achieves [the desired result] by utilizing adversarial learning objectives and fine-tuning a pre-trained model. arrive The transformation is described in the diagram. Here, a and b represent the source and target domains, respectively, and p represents the distribution. The data in the source and target domains can each conform to a specific distribution, such as a Gaussian distribution. x and y represent images, z represents features, and the subscripts 0 and 1 represent the time step of the flow model. The time step t is a variable used to parameterize the continuous interpolation path of data points between the noise and data distributions. The flow model constructs a linearly interpolated flow path x. t = (1-t)·z+t·x, t∈[0,1], where x t Z represents the data point at time step t, and z represents randomly sampled noise data. The values ​​0 and 1 at time step t represent the starting point (noise) and the ending point (target data), respectively.

[0044] The flow model learns an approximate straight-line path from the Gaussian distribution to the image distribution. Therefore, the difference between the pre-trained flow in the pre-trained model and the target flow under the constraint of the adversarial learning objective is the key to the convergence of the model during the fine-tuning process. Figure 3 The flow paths in the latent space of the ordinary paradigm (a), the inversion paradigm (b), and the novel model paradigm (c) of this disclosure embodiment are shown. Here, N(0,1) represents a noise domain that follows a normal distribution, and the source domain distribution p(a) and the target domain distribution p(b) in the latent space are close to each other. arrive It is a pre-trained stream. arrive It is the training flow, in the ordinary paradigm and the inversion paradigm. arrive It is the initial flow. arrive It is the flow of the target.

[0045] It can be seen that, for the general paradigm, the first source domain image Using a time step t=0 as input, a predicted target domain image is generated using a standard one-step denoising process, and features from the second target domain image are analyzed. Image features of the first target domain Construct adversarial learning objectives between them, initial flow Fitting the target flow under this constraint At this point, the direction of the pre-training flow is almost orthogonal to the direction of the target flow, making it difficult for the training process to converge.

[0046] For the inversion paradigm, the time step of the reverse step is defined as t. inv First, start with the input image from the first source domain. Pushing backwards one step to Then, a denoising step is performed to obtain the predicted target domain image. During the training process of the adversarial learning objective, the initial flow is also constrained to fit the target domain. Since the inversion and denoising steps are optimized simultaneously, the direction of the target flow forms a streamline. Furthermore, because the two steps are trained concurrently, the difference between the pre-trained flow and the target flow is smaller, making the training process easier to converge.

[0047] Furthermore, as the time step increases, although the regression objective remains the straight path direction from noise to a clean image at all time steps, the noise level in the direct output of the pre-trained model gradually decreases. Especially when the time step approaches t=1, the pre-trained model's predictions... The closest image to the clean image z1, rather than the difference between the image and the noise.

[0048] Therefore, this disclosure provides a new model paradigm that directly uses the output of the DiT module. As the predicted value of the target domain image This means using the module's output features to replace the image, instead of decoding it through a decoder, which is simple and efficient. Here, θ represents the model parameters, and v... θ It is the parameterized output function of the model. The DiT module, based on the parameter θ, performs the output on the sample z1 at the first time step. a The vector of predicted output is used to further deduce the original image or latent variables. This adversarial learning objective acts upon... and Force the initial flow from Approaching This method is easier to learn because Can be regarded as Its direction aligns with the direction from noise to the image. The parallelogram in the new model paradigm represents the model output. and input features near.

[0049] In some embodiments, this disclosure designs a cycle consistency loss in the latent space. For example... Figure 1 As shown in (a), firstly, the features of the first source domain image are... The input is fed into DiT driven by the first cue (i.e., positive cue) to obtain the predicted first target domain image features. Then, the image features of the first target domain are... The input is fed into DiT driven by a second cue (i.e., a reverse cue) to obtain the predicted second source domain image features. The first source domain image features of the original input The second source domain image features obtained by two DiT transforms, one forward and one inverse. This constitutes the latent space cycle consistency loss.

[0050] Similarly, the image features of the second target domain are... The input is fed into DiT driven by the second cue (i.e., the reverse cue) to obtain the predicted fourth source domain image features. Then, the features of the fourth source domain image are... The input is fed into DiT driven by the first cue (i.e., positive cue) to obtain the predicted fourth target domain image features. The second target domain image features of the original input and the fourth target domain image features obtained after two DiT transforms (inverse and forward directions) are then compared. This constitutes the latent space cycle consistency loss.

[0051] That is, the latent space cycle consistency loss includes at least one of the following: the difference between the predicted and true values ​​of the first source domain image features after two DiT operations (i.e., the difference between the transformed second source domain image features and the first source domain image features), and the difference between the predicted and true values ​​of the second target domain image features after two DiT operations (i.e., the difference between the transformed fourth target domain image features and the second target domain image features). In some embodiments, the formula for calculating the latent space cycle consistency loss includes:

[0052]

[0053] Among them, DiT a→b The representative model predicts domain b from domain a, DiT b→a The representative model predicts domain a from domain b, E a E represents the expectation of an image from domain a. b This represents the expectation of an image from domain b, where the first term represents a feature from the first source domain. After two iterations (forward and reverse), the L1 loss is calculated using the predicted and true values. The second term represents the feature of the second target domain. The L1 loss is calculated by comparing the predicted and actual values ​​after two iterations.

[0054] In other embodiments, this disclosure also incorporates a latent space identity loss in the latent space, such as... Figure 1 As shown in (a), when the second target domain image features are... When input into DiT driven by the first cue, the output is the third target domain image feature. It should be as close as possible to the features of the second target domain. Similarly, when the first source domain image features When input into DiT driven by the second cue, the output is the third source domain image features. It should be as close as possible to the features of the first source domain. The two should be as close as possible, meaning the difference between them should be less than or equal to a predetermined threshold.

[0055] Therefore, method 200 also includes:

[0056] A1: Determine the target image corresponding to the training image. The target image is the image in the target domain corresponding to the transformed image of the training image after processing according to the first prompt.

[0057] A2: Second target domain features of the target image generated using the encoder;

[0058] A3: Input the second target domain features and the first cue features into the diffusion converter to obtain the third target domain features. Then the loss function also includes the second difference between the second target domain features and the third target domain features.

[0059] A4: If the first source domain feature and the second cue feature are input into the diffusion converter to obtain the third source domain feature, then the loss function also includes the third difference between the first source domain feature and the third source domain feature.

[0060] That is, the latent space identity loss includes at least one of the following: the difference between the predicted value (third source domain feature) and the true value (first source domain feature) of the first source domain feature after passing through inverse DiT, and the difference between the predicted value (third target domain feature) and the true value (second target domain feature) of the second target domain feature after passing through forward DiT.

[0061] In some embodiments, the formula for calculating latent space identity loss may include:

[0062]

[0063] In some other embodiments, method 200 further includes:

[0064] If the discriminator determines whether the generated image features come from the source domain or the target domain, the loss function also includes the adversarial loss function of the discriminator.

[0065] Here, adversarial loss is used for supervised training. The discriminator is configured according to Vision-Aided GAN, using a backbone network to extract image features and a simple multilayer perceptron to classify the image features as real or fake. The adversarial loss includes:

[0066]

[0067] Among them, D a and D b These are discriminators used to determine whether an image originates from domain a and domain b, respectively. a→b and G b→a These are generators that convert an image in domain a to an image in domain b, and vice versa.

[0068] In summary, the loss functions used for training image transfer models include at least one of the following: latent space consistency loss, latent space consistency loss, and adversarial loss. These loss functions can enhance content preservation on unpaired datasets while reducing the high memory consumption caused by large pre-trained models. The formulas for the loss functions include:

[0069] L=λ cyc ·L cyc +λ idt ·L idt +λ gan ·L gan

[0070] Where, λ cyc , λ idt and λ gan These are the weighting coefficients for the three types of losses. Those skilled in the art can set the values ​​of each weight as needed, and this disclosure does not impose any restrictions on this.

[0071] In some embodiments, step S260 of training the model according to the loss function includes:

[0072] The parameters of the encoder and decoder are frozen, and the parameters of the diffusion converter are adjusted according to the loss function to train the image transfer model.

[0073] Of course, it should be understood that the parameters of the entire image transfer model can be adjusted to train the model without freezing the parameters of the encoder and decoder.

[0074] The training images are divided into multiple image patches. Adjusting the parameters of the diffusion transducer includes tweaking the parameters of its position encoding layer, which determines the location information of each image patch. Additionally, adjusting the parameters of the diffusion transducer also involves fine-tuning the linear layers within each DiT block. These linear layers map the input data to a new feature space, extract different features from the input data through learning the weight matrix, and then aggregate this information before passing it to the next layer.

[0075] This disclosure also discloses another embodiment of an image transfer method, the method comprising:

[0076] The image transfer model is trained using the model training method described above;

[0077] The image to be processed and the prompt information are input into the trained image transfer model to obtain a new image that matches the prompt information.

[0078] Specifically, the image encoder module of the image transfer model extracts image features from the image to be processed, the prompt encoder extracts prompt features from the prompt information, the image features and prompt features are input into the diffusion converter to obtain output features, and the decoder is used to decode the output features to obtain the corresponding new image. In some embodiments, the image transfer model is a streaming model, such as a rectified model.

[0079] As can be seen, the embodiments of this disclosure employ a novel model paradigm to achieve convergence when fine-tuning the pre-trained image transfer model, enabling a one-step image transfer model without denoising through fine-tuning the pre-trained streaming model. Furthermore, when applying to specific pre-trained models, the embodiments of this disclosure employ latent space cycle consistency loss and latent space identity loss, enhancing image transfer capabilities on unpaired datasets while reducing GPU memory usage during training. Ultimately, the image transfer model fine-tuned using the new model paradigm achieves state-of-the-art performance in multiple image transfer tasks. For example, the SD3.5-Large-Turbo model can be fine-tuned and trained on the LHQ2Shinkai dataset to obtain an image transfer model capable of converting real landscape images into landscape images in the style of Makoto Shinkai anime.

[0080] As mentioned above, this disclosure allows for the training of an encoder first, and then the training of a diffusion converter based on the trained encoder, thereby training an image transfer model. Figure 4 A flowchart of an encoder training method 400 according to an embodiment of the present disclosure is provided. Figure 4 As shown, method 400 includes:

[0081] Step S410: Extract image features. If the feature height is H, the width is W, and the number of channels is K, then the feature has H*W pixel bits and H*W*K feature bits. Each pixel bit corresponds to K channel bits. These parameters are all positive integers. Divide the K channels into G groups, and combine the features of each group in the height and width dimensions to obtain G feature blocks.

[0082] Step S420: Divide the image features into multiple feature blocks along the channel dimension. For example, if the feature height H is 6, the width W is 8, and the number of channels is 8, divide the 8 channels into 2 groups, each group including 4 channels, to obtain 2 corresponding image blocks, namely the feature blocks corresponding to channels 1-4 and the feature blocks corresponding to channels 5-8.

[0083] Step S430: Calculate the entropy constraint of the feature based on the encoding entropy of each feature block.

[0084] The basic idea of ​​entropy coding is to utilize the statistical properties of data to make the average length of the encoded codeword approach the limit of information entropy, thereby reducing redundancy between data and achieving efficient data compression. Encoding entropy uses the concept of entropy from information theory for data compression. Encoding entropy represents the probability of the output code appearing. A variable-length code table is used to encode the source codeword, making the average length of the encoded codeword close to the information entropy, thus achieving efficient compression. The encoding is binary; typically, if the feature value of each feature bit is greater than 0, it is encoded as 1, and vice versa.

[0085] S440: Train the encoder based on the entropy constraints of the features. Entropy constraints represent the conditions that the encoding entropy should satisfy. The purpose of entropy constraints is to maximize the entropy of the probability distribution while satisfying known information. Therefore, the encoder can be trained by continuously optimizing relevant parameters such as weights and biases in the neural network based on these entropy constraints.

[0086] In some embodiments, calculating the entropy constraint of the feature based on the encoding entropy of each feature block includes:

[0087] The entropy constraint of a feature is calculated based on the coding entropy of each feature bit within the feature and the average coding entropy of each feature block.

[0088] Furthermore, the entropy constraints of the features include:

[0089]

[0090] Where H, W, and K represent the feature height, width, and number of channels, respectively; B represents the number of images; G represents the number of groups; i represents the i-th channel; j represents the j-th pixel; m represents the m-th frame; g represents the g-th group; H(x) represents the output encoding entropy; ρ is the similarity function; z represents the feature value at the corresponding position; q is the step function; and the bin function is used to convert integers into binary strings. g Represents the eigenvalues ​​of the g-th group. Let represent the feature value of the j-th pixel in the i-th channel. The first term after the equals sign represents the sum of the coding entropy of each feature bit, and the second term represents the average coding entropy of each feature block. The first term of the formula minimizes the information entropy of all codes, making the feature values ​​as far away from 0 as possible, thus making the coding more deterministic. The second term increases the information entropy of the average code, making the equivalent codebook more balanced and avoiding the problem of invalid coding.

[0091] Alternatively, the encoding entropy, step function, and bin function can be expressed as the following expressions:

[0092] H(x) = -x·log(x)

[0093] q(x) = sgn(x)

[0094] bin:int→bin

[0095] ρ: cosine similarity

[0096] In some other embodiments, this disclosure may also adjust the order of feature channels, i.e., shuffle or reshuffle the feature channels, to achieve a balance of encoded features among different groups. The formula for calculating the entropy constraint can then be further expressed as:

[0097]

[0098] Where, φ(z) g This includes the feature value of the g-th group after adjusting the feature channels. φ represents the operation of adjusting the feature channels, i.e., the shuffling operation.

[0099] Furthermore, the order of multiple channels can be adjusted randomly, that is, the feature channels are randomized. For example, if the initial feature channel order is 1-8, after one random adjustment, the feature channel order becomes 3, 2, 5, 6, 1, 7, 4, 8. Of course, they can also be adjusted according to a predetermined order, such as each channel moving one or more channels backward within the feature volume each time.

[0100] In one implementation, an entropy constraint can be calculated once based on the initial features, and then the entropy constraint can be calculated again for each of the adjusted features after adjusting the initial feature channels individually. Method 400 may also include the following steps:

[0101] S450: Adjust the order of multiple channels in the feature to obtain a new feature;

[0102] S460: Divide the new feature into multiple new feature blocks along the channel dimension.

[0103] S470: Calculate the entropy constraint of the new feature based on the encoding entropy of each new feature block; and

[0104] S480: Retrain the encoder based on the entropy constraints of the new features.

[0105] Repeat steps S450-S480 until a predetermined number of adjustments is reached or the total loss function meets a predetermined condition. That is, in step S440, the encoder was initially trained; the entropy constraints of the new features were generated using this initially trained encoder; and the encoder was retrained based on the entropy constraints of the new features to obtain the re-optimized encoder parameters. In other words, the encoder is retrained based on the encoder parameters obtained from the previous training.

[0106] The above provides a method for achieving balanced group coding by shuffling feature channels. In other embodiments, this disclosure can also achieve balanced group coding by dividing the code into multiple groups, i.e., using multiple grouping methods to obtain multiple grouping results, and calculating an entropy constraint based on each grouping result. Each encoding group can be a regular grouping, as close as possible to the average grouping. Of course, each encoding grouping can also be random, with different groups including different numbers of channels.

[0107] In another implementation, embodiments of this disclosure may also train the encoder and decoder simultaneously, and train the diffusion converter based on the trained encoder and decoder.

[0108] Therefore, step S440, training the encoder based on the entropy constraints of the features, includes:

[0109] The encoder and decoder are trained simultaneously based on the entropy constraints of the features and the reconstruction loss of the decoder.

[0110] The reconstruction loss of the decoder includes the difference between the image decoded by the decoder and the original image. This includes the differences in all pixel values ​​and all channel values ​​between the original and output images, specifically:

[0111]

[0112] Where B represents the number of images in the training set, and each image has C feature channels (such as red, yellow, and green channels, grayscale and texture channels, etc.). Let I be the true value of the k-th channel of the j-th pixel in the m-th frame. mjkThis is the predicted value for the k-th channel of the j-th pixel in the m-th frame. Training stops when the predetermined number of iterations or the loss function is less than or equal to a predetermined threshold.

[0113] Then, by summing or weighted summing the entropy constraint of the encoder and the reconstruction loss of the decoder, the loss function for training the encoder and decoder can be obtained. The encoder and decoder are trained simultaneously based on this loss function. Both the encoder and decoder are neural network models, and those skilled in the art can set the hyperparameters of these two models as needed. This disclosure does not impose any restrictions on this.

[0114] In another implementation, embodiments of this disclosure can train the encoder and the diffusion converter simultaneously. In this case, steps S410-S440 are still performed, except that the total loss function for model training includes the encoder's entropy constraint and at least one of the following for the diffusion converter: latent space consistency loss, latent space identity loss, and adversarial loss. Calculating this total loss function allows for the simultaneous training of the encoder and the diffusion converter.

[0115] Furthermore, embodiments of this disclosure can simultaneously train the encoder, decoder, and diffusion converter. In this case, steps S410-S440 are still performed, except that the total loss function for model training includes the encoder's entropy constraint, the decoder's reconstruction loss, and at least one of the following for the diffusion converter: latent space consistency loss, latent space identity loss, and adversarial loss. Calculating this total loss function allows for the simultaneous training of the encoder, decoder, and diffusion converter.

[0116] According to another embodiment of this disclosure, an image encoding method is also provided, comprising:

[0117] The encoder trained using the encoder training method described above is used to generate compressed codes for the image.

[0118] According to another embodiment of this disclosure, an image transfer method is also provided, comprising:

[0119] The encoder trained using the encoder training method described above is used to train the diffusion converter, thereby training the image transfer model; and

[0120] The input image is transformed into a new image based on the trained image transfer model.

[0121] Figure 5An illustration shows a machine in an example form of a computing device 500, which may be a device for performing model training methods, image transfer methods, etc. Within this computing device, an instruction set, when executed, and / or processing logic, when initiated, may cause the machine to perform any or more of the methods described and / or claimed herein. In alternative embodiments, the machine operates as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine may operate as a server or client machine in a server-client network environment, or as a peer-to-peer (or distributed) network environment. The machine may be a personal computer (PC), laptop computer, tablet computing system, personal digital assistant (PDA), cellular phone, smartphone, web application, set-top box (STB), network router, switch, or bridge, or any machine capable of executing an instruction set (sequentially or otherwise) specifying actions to be taken by the machine, or initiating processing logic. Furthermore, although only a single machine is illustrated, the term "machine" may also be understood to include any collection of machines that individually or jointly execute one or more sets of instructions (or multiple sets of instructions) for performing any one or more of the methods described and / or required herein.

[0122] Example computing device 500 may include a data processor 502 (e.g., a system-on-a-chip (SoC), a general-purpose processing core, a graphics core, and optional other processing logic) and a memory 504 (e.g., RAM) capable of communicating with each other via a bus 506 or other data transmission system. Computing device 500 may also include various input / output (I / O) devices and / or interfaces 510, such as a touchscreen display, an audio jack, a voice interface, and an optional network interface 512. In an example embodiment, network interface 512 may include one or more radio transceivers configured to communicate with any one or more standard wireless and / or cellular protocols or access technologies (e.g., second-generation (2G), 2.5G, third-generation (3G), fourth-generation (4G), and next-generation radio access for cellular systems, Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), LTE, CDMA2000, WLAN, Wireless Router (WR) mesh, etc.). Network interface 512 can also be configured to be used with a variety of other wired and / or wireless communication protocols, including TCP / IP, UDP, SIP, SMS, RTP, WAP, CDMA, TDMA, UMTS, UWB, WiFi, WiMax, Bluetooth, IEEE 802.11x, etc. Essentially, network interface 512 can virtually include or support any wired and / or wireless communication and data processing mechanism through which information / data can be propagated between computing device 500 and another computing or communication system via network 514.

[0123] Memory 504 may represent a machine-readable medium (or computer-readable storage medium) storing one or more instruction sets, software, firmware, or other processing logic (e.g., logic 508) that implements any one or more of the methods or functions described and / or claimed herein. During execution by computing device 500, logic 508 or a portion thereof may also be wholly or at least partially located within processor 502. Thus, memory 504 and processor 502 may also constitute a machine-readable medium (or computer-readable storage medium). Logic 508 or a portion thereof may also be configured as processing logic or logic, at least a portion of which is partially implemented in hardware. Logic 508 or a portion thereof may also be transmitted or received via network 514 via network interface 512. While the machine-readable medium (or computer-readable storage medium) of the example embodiments may be a single type of medium, the term "machine-readable medium" (or computer-readable storage medium) should be understood to include a single non-transitory medium or multiple non-transitory media (e.g., centralized or distributed databases and / or associated caches and computing systems) storing one or more instruction sets. The term "machine-readable medium" (or computer-readable storage medium) can also be understood to include any non-transitory medium capable of storing, encoding, or carrying a set of instructions for machine execution and causing the machine to perform any one or more methods of various embodiments, or capable of storing, encoding, or carrying data structures utilized or associated with such set of instructions. The term "machine-readable medium" (or computer-readable storage medium) can therefore be understood to include, but is not limited to, solid-state memory, optical media, and magnetic media.

[0124] The disclosed and other embodiments, modules, and functional operations described in this document can be implemented in digital electronic circuit systems, or in computer software, firmware, or hardware (including the structures disclosed in this document and their structural equivalents), or combinations thereof. The disclosed and other embodiments can be implemented as one or more computer program products, that is, one or more modules of computer program instructions encoded on a computer-readable medium for execution by a data processing apparatus or for controlling the operation of such data processing apparatus. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of material influencing machine-readable propagation signals, or combinations thereof. The term "data processing apparatus" encompasses all means, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program under investigation, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or combinations thereof. Propagation signals are artificially generated signals, such as machine-generated electrical, optical, or electromagnetic signals, generated to encode information to be transmitted to a suitable receiver device.

[0125] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any programming language (including compiled or interpreted languages), and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), or as a single file dedicated to the program under development, or as multiple collaborative files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program can be deployed to execute on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected via a communication network.

[0126] The processes and logic flows described in this document can be executed by one or more programmable processors that execute one or more computer programs to perform functions by manipulating input data and generating outputs. The processes and logic flows can also be executed by special-purpose logic circuit systems (e.g., FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits)), and the devices can also be implemented as special-purpose logic circuits (e.g., FPGAs or ASICs).

[0127] Processors suitable for executing computer programs include, for example, both general-purpose microprocessors and special-purpose microprocessors, as well as any one or more processors in any kind of digital computer. Typically, a processor receives instructions and data from read-only memory or random access memory, or both. Essential components of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer also includes one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or the computer is operatively coupled to receive data from or transfer data to or from such mass storage devices, or both. However, a computer does not necessarily need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated into a special-purpose logic circuit system.

[0128] While this disclosure contains numerous details, these details should not be construed as limiting the scope of any of this disclosure or any content that may be claimed, but rather as a description of features that may be implemented for specific embodiments of a particular disclosure. Certain features described in the context of individual embodiments in this disclosure may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and initially or even equally claimed, in some cases one or more features from the claimed combination may be removed from the combination, and the claimed combination may be for sub-combinations or variations thereof.

[0129] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the specific order shown or in a sequential order, or as requiring all illustrated operations to achieve the desired result. Furthermore, the separation of various system components in the embodiments described in this disclosure should not be construed as requiring such separation in all embodiments. Only some implementations and examples have been described, and other implementations, enhancements, and variations may be made based on what is described and illustrated in this disclosure.

[0130] The embodiments described herein are intended to provide a general understanding of the structure of various embodiments, and are not intended to be used as a complete description of all elements and features of components and systems that may utilize the structures described herein. Many other embodiments will be apparent to those skilled in the art upon review of the descriptions provided herein. Other embodiments can be utilized and derived, allowing for structural and logical substitutions and changes without departing from the scope of this disclosure. The figures herein are representative only and may not be drawn to scale. Some scales may be enlarged, while others may be minimized. Therefore, the specification and figures are to be considered illustrative rather than restrictive.

[0131] Some embodiments implement functionality in two or more specific interconnected hardware modules or devices, wherein control and data signals are communicated between and through the modules, or as part of an application-specific integrated circuit. Therefore, the example system is applicable to software, firmware, and hardware implementations.

[0132] While exemplary embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the exemplary discussions above are not intended to be exhaustive or to limit this disclosure to the specific forms disclosed. Many modifications and variations are possible in light of the above teachings. Therefore, the subject matter disclosed should not be limited to any single embodiment or example described herein, but should be interpreted in accordance with the breadth and scope of the appended claims.

Claims

1. A model training method, the model comprising an encoder and a diffusion converter, the method comprising: The encoder is used to generate the first source domain features of the training images; Obtain the first cue feature from the source domain to the target domain, and the second cue feature from the target domain to the source domain; The first source domain feature and the first cue feature are input into the diffusion converter to obtain the first target domain feature; The first target domain feature and the second cue feature are input into the diffusion converter to obtain the second source domain feature; Calculate the loss function based on the difference between the first source domain features and the second source domain features; and The model is trained based on the loss function.

2. The method of claim 1, further comprising: Determine the target image corresponding to the training image; The second target domain features of the target image are generated using an encoder; The second target domain feature and the first prompt feature are input into the diffusion converter to obtain the third target domain feature; The loss function further includes the difference between the second target domain features and the third target domain features.

3. The method of claim 1, further comprising: Determine the target image corresponding to the training image; The second target domain features of the target image are generated using an encoder. The first source domain feature and the second cue feature are input into the diffusion converter to obtain the third source domain feature; The loss function further includes the difference between the first source domain features and the third source domain features.

4. The method of claim 1, further comprising: The discriminator determines whether the generated image features come from the source domain or the target domain. The loss function also includes the adversarial loss function of the discriminator.

5. The method of claim 1, wherein, The loss function includes at least one of the following: latent space consistency loss, latent space consistency loss, and adversarial loss.

6. The method of claim 1, wherein, The model further includes a decoder, and training the model according to the loss function includes: The parameters of the encoder and decoder are frozen, and the parameters of the diffusion converter are adjusted according to the loss function to train the model.

7. The method of claim 6, wherein, The training image is divided into multiple image blocks, and adjusting the parameters of the diffusion converter includes adjusting the position encoding layer parameters of the diffusion converter, the position encoding layer being used to determine the position information of each image block.

8. The method as described in claim 2 or 3, wherein, The target image is the image corresponding to the transformed image of the training image after processing according to the first prompt in the target domain.

9. An image transfer method, comprising: The image transfer model is trained using the model training method described in any one of claims 1-8; The image to be processed and the prompt information are input into the trained image transfer model to obtain a new image that conforms to the prompt information.

10. A computing device, comprising: Processor, memory, and computer programs stored in memory and capable of running on the processor; When the processor runs the computer program, it performs the method according to any one of claims 1-9.

11. A computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the method of any one of claims 1-9.

Images