A method for accelerating image generation based on second-order Sinkhorn gradient flow of neural networks

Abstract

This invention discloses an accelerated image generation method based on neural network second-order Sinkhorn gradient flow. The invention utilizes a neural second-order Sinkhorn gradient flow 2-NSGF model to parameterize the time-varying momentum field related to the Sinkhorn divergence in the second-order Watson-Stanstein gradient flow, thereby achieving efficient inference and simulation. By utilizing source and target distribution samples, the 2-NSGF model can generate and refine simulation data in real time, reducing the computational complexity of traditional methods. Furthermore, by introducing a two-stage 2-NSGF++ model, the efficiency of processing high-dimensional datasets is further optimized, improving simulation accuracy and speed. This invention provides an efficient data processing technique applicable to various applications in machine learning, such as image generation, style transfer, and audio-text conversion. This invention significantly improves image generation efficiency; eliminates reliance on kernel functions, reducing complexity; and enhances sample quality and generation stability.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence image generation models, and in particular to an image generation method based on second-order Sinkhorn gradient flow of neural networks to accelerate image generation. Background Technology

[0002] It involves optimizing generative models, particularly for applications on high-dimensional image datasets, with the aim of improving the efficiency and quality of the image generation process.

[0003] Generative models are a type of machine learning model used to learn distributions from data and generate new, similar data samples. They are widely used in tasks such as image generation, text generation, data augmentation, and style transfer. Common generative models include generative adversarial networks (GANs), variational autoencoders (VAEs), and the recently developed diffusion models. GANs generate high-quality data by training two adversarial neural networks, a generator and a discriminator. VAEs compress input data into a latent space using an encoder and then reconstruct it using a decoder, thereby generating new samples. These models have achieved significant results in multiple fields, driving the progress of artificial intelligence.

[0004] Generative models based on neural network ordinary differential equations and stochastic differential equations have made significant progress in the field of machine learning in recent years. Notable examples include diffusion models (Wijmans JG, Baker RW. The solution-diffusion model: a review[J]. Journal of membrane science, 1995, 107(1-2): 1-21.)(Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models[J]. Advances in neural information processing systems, 2020, 33: 6840-6851.)(Song Y, Sohl-Dickstein J, Kingma DP, et al. Score-based generative modeling through stochastic differential equations[J]. arXiv preprint arXiv: 2011. 13456, 2020.) and flow matching models (Lipman Y, Chen RTQ, Ben-Hamu H, et al. Flow matching for generative modeling[J]. arXiv preprint (arXiv:2210.02747,2022.)(Liu X, Gong C, Liu Q. Flowstraight and fast: Learning to generate and transfer data with rectified flow[J].arXiv preprint arXiv:2209.03003,2022.). Diffusion models and flow-matching models provide a natural way to model the evolution of data distribution by simulating continuous dynamic systems. These methods have many advantages in generative models, but also face some challenges and drawbacks.

[0005] Diffusion models generate data through progressive denoising. The basic idea is to gradually add noise to the data until it transforms into a Gaussian distribution, and then recover the original data through a reverse process of progressive denoising. These models include diffusion probability models and score-matching models. Diffusion models perform well in image generation tasks, producing high-quality and detailed images. They have a solid theoretical foundation, are modeled through explicit probabilistic processes, and have good interpretability. Through progressive denoising, diffusion models effectively handle noise and uncertainty in the data. Flow-matching models, as an improvement on diffusion models, use neural networks to match straight-line flows, significantly reducing the number of steps required for generation. While maintaining high-quality generated images, flow-matching models greatly reduce computational costs.

[0006] However, the stepwise denoising process of diffusion models requires significant computation, and the inference process can be very time-consuming. Due to the need for multiple denoising steps, the inference speed of diffusion models is slow, making it difficult to meet the requirements of real-time applications. The model training and inference processes are complex and require careful design and parameter tuning.

[0007] In recent years, gradient flow-based generative models have been increasingly widely used in machine learning, especially in tasks such as image generation, style transfer, and data augmentation. Wasserstein gradient flow, as a powerful optimization tool, has been extensively used to solve optimization problems in the Wasserstein probability space. However, traditional Wasserstein gradient flow methods are computationally complex and costly in high-dimensional spaces, especially when frequent transport map calculations are required, which limits their widespread use in practical applications. To overcome these challenges, researchers have proposed various methods to approximate and accelerate the computation of Wasserstein gradient flow.

[0008] In the development of gradient flow-based generative models, neural networks have been widely used to approximate complex parts of the Wasserstein gradient flow, such as time-dependent transport maps. Neural networks can learn and approximate transport maps using samples from the target distribution during training, thus simplifying computation. However, existing methods still face challenges in computational complexity and training stability in high-dimensional spaces.

[0009] Furthermore, commonly used distribution metrics for gradient flow include f-divergence (such as Kullback-Leibler and Jensen-Shannon divergence), maximum mean difference, and slice-Wotherstein distance. These metrics guide generative models to optimize from the source distribution to the target distribution, but they may perform poorly in high-dimensional spaces and are computationally complex. To overcome these problems, Sinkhorn divergence has been proposed as an efficient metric. By entropy regularizing the Wortherstein distance, Sinkhorn divergence can be computationally efficiently achieved through the Sinkhorn algorithm, making it highly promising for generative models.

[0010] Based on a review of existing generative model literature, diffusion models and flow matching models are the two methods closest to the present invention in the prior art.

[0011] Diffusion models generate data through progressive denoising. The basic idea is to gradually add noise to the data until it transforms into a Gaussian distribution, and then gradually denoise again to recover the original data. This process can be divided into two stages. First, the forward process: starting with the original data, Gaussian noise is gradually added until it becomes a pure noise distribution. This process is implemented using a multi-step Markov chain, where a certain amount of noise is added at each step, making the data distribution gradually more random and uncertain. Second, the reverse process: starting with pure noise, a trained denoising network is used to gradually remove the noise, ultimately recovering a clear image. In this process, the denoising network attempts to reduce the impact of noise at each step, gradually restoring the data to its initial clear state. Diffusion models perform well in image generation tasks, generating high-quality and detailed images. They have a solid theoretical foundation, modeled through explicit probabilistic processes, and have good interpretability. Through progressive denoising, diffusion models can effectively handle noise and uncertainty in the data. However, the progressive denoising process of diffusion models requires significant computation, especially in high-resolution image generation, where the inference process can be very time-consuming. Because it requires multiple steps of denoising, the diffusion model has a slow inference speed, making it difficult to meet the needs of real-time applications.

[0012] Flow matching models, as an improvement on diffusion models, use neural networks to match straight-line flows, significantly reducing the number of steps required for generation. In flow matching models, the goal is to learn an optimal flow path from the initial distribution to the target distribution, rather than progressively denoising. Its core idea is to learn a mapping function through optimization, allowing the generation of target data from initial random noise to be completed in fewer steps. Flow matching models learn the optimal transport mapping through neural networks, directly transforming the initial noise distribution into the target data distribution. This method reduces the number of steps required for generation, thus significantly improving efficiency.

[0013] During training, the momentum field is optimized by minimizing the difference between generated and target samples. This method not only maintains the high quality of the generated images but also significantly reduces computational cost. In recent years, this type of model has achieved significant success in image generation tasks, such as the application of Stability AI in image generation, demonstrating its powerful ability to generate high-resolution images. Flow matching models significantly reduce computational cost while maintaining the high quality of generated images. This method not only improves generation efficiency but also retains the advantages of diffusion models in handling noise and uncertainty.

[0014] While diffusion and flow matching models perform well in generating high-quality images, they also have some significant drawbacks and limitations.

[0015] First, the stepwise denoising process of the diffusion model requires a significant amount of computation. Generating a high-quality image often requires hundreds or even thousands of denoising steps, each involving complex calculations and forward propagation of the neural network, making the computational cost extremely high. Especially when generating high-resolution images, the inference process can be very time-consuming, as each frame requires multiple computational steps to remove noise and restore clear details. This high computational demand for stepwise denoising not only increases generation time but also places higher demands on the performance of the hardware.

[0016] Secondly, due to the need for multi-step denoising, the inference speed of diffusion models is slow, making it difficult to meet the demands of real-time applications. In many practical applications, such as video generation, real-time image processing, and augmented reality, generation speed is a critical factor. The multi-step inference process of diffusion models makes them difficult to apply in these applications requiring rapid responses. Although the generated images are of high quality, their inference speed becomes a major bottleneck in scenarios requiring immediate feedback.

[0017] Furthermore, the training and inference processes of diffusion models are complex, requiring careful design and hyperparameter tuning. During training, the noise addition and denoising processes need fine-tuning to ensure the model effectively learns the characteristics of the data distribution. This includes selecting appropriate noise levels, designing effective denoising network structures, and optimizing hyperparameters during training. All of these require extensive experimentation and experience, increasing the difficulty of model development.

[0018] While flow matching models improve the efficiency of diffusion models to some extent, they also have some limitations. Flow matching models rely on neural networks to learn the optimal flow path from the initial distribution to the target distribution, a process that also requires substantial data and computational resources. Furthermore, during the learning process, flow matching models need to optimize the momentum field, which involves complex mathematical operations and large amounts of training data to ensure the model accurately simulates the data evolution. This reliance on computational resources and data means that flow matching models still face high computational costs and long training times when dealing with large-scale datasets.

[0019] Stream matching models may also face some challenges in practical applications. For example, the training and inference processes of the model rely on high-quality data; if the data quality is low or the distribution is highly variable, the model's performance may be affected. Furthermore, although the inference process of stream matching models is faster than traditional diffusion models, it may still not meet the requirements for real-time processing in some extremely high-resolution and high-complexity applications.

[0020] In summary, while diffusion and flow matching models offer significant advantages in generating high-quality images, their high computational cost, complex training and inference processes, and performance limitations in real-time applications restrict their widespread adoption. These drawbacks and shortcomings present opportunities for the development of new technologies. Through further optimization and improvement, these limitations can be overcome, enhancing the efficiency and practicality of generative models.

[0021] Disadvantages and limitations of the diffusion model:

[0022] High computational requirements:

[0023] The stepwise denoising process requires a lot of computation, with each step requiring complex calculations and forward propagation of neural networks.

[0024] High-resolution image generation is computationally more expensive, and the inference process is very time-consuming.

[0025] Slow reasoning speed:

[0026] Multi-step denoising processes result in slow inference speeds, making it difficult to meet the needs of real-time applications such as video generation and augmented reality applications.

[0027] Complex training and parameter tuning:

[0028] The noise addition and denoising processes require fine-tuning, including selecting the noise intensity, designing the denoising network structure, and optimizing the training hyperparameters.

[0029] The training process requires a lot of experiments and experience, which increases the difficulty of model development.

[0030] Disadvantages and limitations of flow matching models:

[0031] Relies on large amounts of data and computing resources:

[0032] The neural network needs to learn the optimal flow path from the initial distribution to the target distribution, which involves complex mathematical operations and a large amount of training data.

[0033] The training process is computationally expensive and takes a long time.

[0034] Challenges in practical applications:

[0035] The model relies on high-quality data; when the data quality is low or the distribution varies greatly, the model's performance will be affected.

[0036] In some extremely high-resolution and highly complex applications, although the inference speed is faster than the traditional diffusion model, it may still not meet the requirements for real-time processing.

[0037] Diffusion models and flow matching models, as existing technologies, are the closest technological foundations for this invention. They provide important methodological and theoretical support for generating high-quality images, but this invention achieves significant improvements in generation efficiency and computational cost by further optimizing these techniques. Summary of the Invention

[0038] The purpose of this invention is to address the shortcomings of existing technologies by providing a method for accelerating image generation based on second-order Sinkhorn gradient flow in neural networks.

[0039] The objective of this invention is achieved through the following technical solution: a method for accelerating image generation based on second-order Sinkhorn gradient flow of neural networks, comprising the following steps:

[0040] Data preparation involves acquiring the target image dataset and an initial image dataset of the same resolution (either noisy or initial) obtained through random sampling.

[0041] Based on the target image dataset and the initial image dataset, a discretized image second-order Sinckhorn gradient flow is constructed; the gradient flow includes second-order Sinckhorn gradient solving and iteration, mini-batch second-order Sinckhorn gradient flow technology and trajectory pooling technology, wherein the second-order Sinckhorn gradient solving and iteration includes solving the Sinckhorn gradient of the current intermediate image with respect to the target image and iteratively updating the current intermediate image;

[0042] The neural network second-order Sinckhorn gradient flow model is trained using experience playback technology and momentum field matching technology; a specified momentum field is fitted by solving the optimization objective of the neural network; the input and output of the neural network second-order Sinckhorn gradient flow model are: input is the current intermediate image, the current corresponding time t, the velocity field corresponding to the image, and the momentum field corresponding to the image; output is the gradient of the current Sinckhorn divergence;

[0043] A stage transition time predictor is used to predict the optimal time point from the current state of the model to the intermediate state of the general model, thereby enhancing the model's adaptability and response speed in accelerating the generation of the general model.

[0044] Image generation is accelerated using second-order Sinckhorn gradient flow. The generation method includes using a neural network second-order Sinckhorn gradient flow model to quickly generate intermediate images near the target flow, predicting the optimal time point of the intermediate images, and matching images generated by general diffusion-based generation models or flow matching models.

[0045] Furthermore, the target image dataset is a large-scale image database that is publicly available or private in reality; the initial image dataset is a noisy image of the same resolution sampled by a random method or an initial image dataset of the same resolution.

[0046] Further, the construction of the discretized image second-order Sinckhorn gradient flow specifically involves: using randomly sampled Gaussian noise or random sampling of the initial image dataset of the same resolution, matching it with the sampling of the target image dataset to form a mini-batch noise-image pair; randomly setting the initial velocity field; using a Sinckhorn divergence calculation library to calculate the Sinckhorn divergence of the mini-batch noise-image pair, and performing gradient solving to obtain the gradient of the current Sinckhorn divergence as the current momentum field; using the current momentum field to update the current image velocity field, and updating the current image through the updated velocity field to form a new mini-batch noise-image pair; storing the momentum field and velocity field corresponding to the current mini-batch noise; and repeating the above steps.

[0047] Furthermore, the construction of the second-order Sinckhorn gradient flow of the discretized image further includes:

[0048] Mini-batch noise-image pairing is constructed by matching samples from the target image dataset with randomly sampled Gaussian noise or random samples from an initial image dataset of the same resolution. in, The superscript i indicates the i-th pair, and the subscript t indicates the time sequence number. This represents the intermediate image, where t=0 indicates... This is the initial image, t=1 indicates It is the target image. This indicates that there are n pairs that constitute a mini-batch of noise-image pairings;

[0049] The Sinckhorst divergence of mini-batch noise-image pairings is calculated using the geomloss.SamplesLoss function from the GeomLoss library. The function is used as follows: Input: Current image; Target: [Image Name] Parameters are set as loss = sinkhorn; output Sinkhorn divergence. Where S represents the Sinckhorn divergence, since regardless of whether the input is the initial image or an intermediate image, the target image... Nothing changes; abbreviations are used. The gradient is calculated using the Autograd function from the PyTorch library. The function is used as follows: input the Sinckhorn divergence, output the gradient of the Sinckhorn divergence. As the current momentum field, in which Indicates the gradient sign;

[0050] Update the current velocity field using the current momentum field:

[0051]

[0052] in, This is the updated velocity field; the superscript 'i' indicates that the intermediate image corresponding to the velocity field is... Represents the current velocity field, with the subscript t. * ,t represents the intermediate time; ∈ represents the velocity field update rate; γ represents the original velocity field momentum retention parameter, which is a manually set hyperparameter. Both control the velocity field update rate; ^ indicates that small batch updates are currently being used.

[0053] Update the current image using the current velocity field:

[0054]

[0055] in, This is the updated image. This is the current middle image, where the superscript i represents the i intermediate images. It is the updated velocity field, and σ is the rate at which the image is updated, which is a hyperparameter set manually.

[0056] Store the momentum field and velocity field corresponding to the current mini-batch noise; store the current triplet.

[0057] From the initial image or random noise Begin by repeating the above steps to obtain a sequence of triples. Where m represents the number of times the above steps are repeated to construct the second-order Sinckhorn gradient flow of the discretized image;

[0058] Repeatedly select the initial image or random noise Begin reconstructing the second-order Sinckhorn gradient flow of the discretized image, and iterate multiple times to build the trajectory pool.

[0059] Furthermore, the training of the second-order Sinckhorn gradient flow model of the neural network is specifically as follows: using the empirical replay technique, training is performed using the constructed trajectory pool without introducing additional second-order Sinckhorn gradient flow data; using the momentum field matching technique, the U-net neural network is used to approximate the real-time update of the momentum field to simulate the second-order Sinckhorn gradient flow to obtain the neural network second-order Sinckhorn gradient flow model.

[0060] Furthermore, the trained neural network second-order Sinkhorn gradient flow model further includes:

[0061] The empirical replay technique is used, and the constructed trajectory pool is used for training without introducing additional second-order Sinkhorn gradient flow data.

[0062] Momentum field matching technology is employed, and momentum field matching targets are randomly selected to minimize the prediction error of the momentum field through neural network parameterization technology; the mathematical expression of the matching target is:

[0063]

[0064] Where θ represents the network parameters, t represents the intermediate time, T represents the upper bound of the time, and x ~ μ t E represents the expectation derived from a time-dependent intermediate distribution of the image, where E represents the average of multiple calculations. It's training loss. This refers to a neural network that has been trained and optimized through backpropagation of the neural network's loss function.

[0065] The model is trained using Loss backpropagation gradient descent with the Adam optimizer and a properly set learning rate.

[0066] Furthermore, the stage transition time predictor, which predicts the optimal time point from the current state to the intermediate state of the general model, specifically involves:

[0067] Map any input image x to the correct time t on which other general flow-matching-based generative models depend, so that the images generated by the current neural network's second-order Sinkhorn gradient flow conform to the intermediate image distribution P of the general flow-matching-based generative model. t Even if we get x~P t Using neural networks Prediction time t; during the optimization process, the training loss L(φ) includes the loss from P. t Sample X t And minimize t φ (X t The distance between ) and t ensures X t Accurate simulation of P t Time Predictor The training uses the following regression loss:

[0068]

[0069] Where θ represents the network parameters, T represents the upper bound of time, and x t ~P t This indicates that an intermediate image has been selected;

[0070] The model is trained using Loss backpropagation gradient descent with the Adam optimizer and a properly set learning rate.

[0071] Furthermore, the method of accelerating image generation using second-order Sinkhorn gradient flow specifically involves:

[0072] Initial noise is generated randomly or initial images are sampled randomly from the initial image dataset. and a randomly set initial velocity field

[0073] An intermediate image close to the target image distribution is generated using a second-order Sinkhorn gradient flow model of a neural network: velocity field iteration guided by momentum field prediction and subsequent image updates are performed using a function evaluation of less than 10 steps.

[0074]

[0075] in, From Intermediate image of the departure point. From The actual intermediate velocity field, It is the momentum field predicted by the second-order Sinkhorn gradient flow model of the neural network, ∈ represents the velocity field update rate, γ represents the original velocity field momentum retention parameter, and σ is the rate that controls the image update, which is a manually set hyperparameter.

[0076] The present invention also provides an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the described method for accelerating image generation based on second-order Sinkhorn gradient flow of neural networks.

[0077] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the described method for accelerating image generation based on second-order Sinkhorn gradient flow of neural networks.

[0078] The beneficial effects of this invention are as follows:

[0079] Significantly improved generation efficiency: Compared with the baseline method, the 2-NSGF++ model uses a two-stage process to first quickly approximate the high-dimensional image manifold, and then finely adjust it along a simple straight line, reducing the computation time and resources required for generation.

[0080] No kernel function required, reducing complexity: This method is based on Sinkhorn gradient flow, which does not require a kernel function, thus avoiding the performance degradation of kernel methods in high-dimensional data and improving the applicability and efficiency of the model.

[0081] Improved sample quality and generation stability: As the sample size increases, the approximation of the velocity field gradually converges to the true velocity field, and the generated sample quality is better than the existing neural WGF method, and it shows higher stability and reliability on multiple datasets. Attached Figure Description

[0082] Figure 1 This is a schematic diagram of the dynamic simulation and optimization algorithm for the momentum field;

[0083] Figure 2 This is a comparison chart of the 2-NSGF++ and FM methods;

[0084] Figure 3 This is a schematic diagram of the 2-NSGF++ generation process;

[0085] Figure 4 This is a schematic diagram of the 2-NSGF training process. Detailed Implementation

[0086] To make the objectives, technical solutions, and technical effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings.

[0087] like Figure 1 As shown, an embodiment of the present invention provides a method for accelerating image generation based on second-order Sinkhorn gradient flow of neural networks, comprising:

[0088] S1. Data preparation: Obtain the target image dataset and an initial image dataset of the same resolution (either noisy or initial) obtained through random sampling. The target image dataset is a large-scale, publicly available or private image database; the initial image dataset is a noisy image of the same resolution or an initial image dataset of the same resolution obtained through random sampling.

[0089] S2. Based on the target image dataset and the initial image dataset, construct a discretized image second-order Sinckhorn gradient flow. The gradient flow includes second-order Sinckhorn gradient solving and iteration, mini-batch second-order Sinckhorn gradient flow technology, and trajectory pooling technology. The second-order Sinckhorn gradient solving and iteration includes solving the Sinckhorn gradient of the current intermediate image with respect to the target image and iteratively updating the current intermediate image. Specifically, constructing the discretized image second-order Sinckhorn gradient flow involves: matching randomly sampled Gaussian noise or random sampling of the initial image dataset with samples from the target image dataset to form a mini-batch noise-image pair; randomly setting the initial velocity field; calculating the Sinckhorn divergence of the mini-batch noise-image pair using a Sinckhorn divergence calculation library and solving for the gradient to obtain the gradient of the current Sinckhorn divergence as the current momentum field; updating the current image velocity field using the current momentum field, and updating the current image using the updated velocity field to form a new mini-batch noise-image pair; storing the momentum field and velocity field corresponding to the current mini-batch noise; and repeating the above steps.

[0090] The construction of the second-order Sinckhorn gradient flow for discretized images is based on the following theoretical basis:

[0091] Constructing a second-order Sinckhorn gradient flow and its finite particle form:

[0092] The probability distribution space can be expressed by the following formula: upper functional Optimization problem: in, μ represents the distribution of the optimized object or the initial distribution. * Indicates the target distribution. Indicates the selected divergence. The divergence between the distribution of the optimization object and the target distribution is represented by the following formula: The difference between distributions is measured; to solve the above optimization problem, a functional in Wasserstein space is used. The descending flow is described by the following continuity equation:

[0093]

[0094] in, It is a momentum field that defines the direction of position transition.

[0095] The second-order Hamiltonian gradient flow is expressed by the following formula:

[0096]

[0097] in, This represents the Hamiltonian momentum variable, while Represents Hamiltonian potential energy; where Hamiltonian momentum variable Φ t For μ t In a second-order Hamiltonian gradient flow system, the change in momentum is applied to the momentum rather than directly to the momentum field.

[0098] Define the Sinckhorn divergence, given two probability measures. Entropy-regularized 2-Watsonstein distance The definition is as follows:

[0099]

[0100] Where Π(μ,v) represents the set of all probabilities coupled to π with marginal distributions of μ and v, and ε>0 is the regularization coefficient. Represents the product measure, i.e. Represent π and KL divergence between them; construct the Sinkhorn objective based on the definition of Sinkhorn divergence. Where μ * Represent the target distribution; according to the first variation of the Sinkhorn target, we obtain:

[0101]

[0102] in, and f μ,μ They are - Part of the momentum, pairing Called of -potential, similarly (f) μ,μ ,g μ,μ ) is called of - Potential; by setting the momentum field in the second-order Hamiltonian gradient flow. Construct a second-order Sinkhorn-Warsenstein gradient flow; finally, consider a distribution with a finite particle composition. A generative model suitable for real-world applications is shown in the following formula:

[0103]

[0104] The method of using neural networks to approximate the real-time update of the momentum field simulation of the second-order Sinckhorn gradient flow also includes:

[0105] Real-time updates of Sinckhorn gradient flow:

[0106] Based on During the process of driving the update of discrete systems, if Distribution At that time, the gradient flow stops changing as it approaches the target distribution; according to the continuity equation... Discretization:

[0107]

[0108] For time-dependent particle distributions uniquely determined by initial particle distribution, initial velocity, and momentum field To be updated in real time, among which The momentum field is determined by predictions from a neural network.

[0109] Furthermore, the construction of the second-order Sinckhorn gradient flow of the discretized image further includes:

[0110] Mini-batch noise-image pairing is constructed by matching samples from the target image dataset with randomly sampled Gaussian noise or random samples from an initial image dataset of the same resolution. in, The superscript i indicates the i-th pair, and the subscript t indicates the time sequence number. This represents the intermediate image, where t=0 indicates... This is the initial image, t=1 indicates It is the target image. This indicates that there are n pairs that constitute a mini-batch of noise-image pairs.

[0111] The Sinckhorst divergence of mini-batch noise-image pairings is calculated using the geomloss.SamplesLoss function from the GeomLoss library. The function is used as follows: Input: Current image; Target: [Image Name] Parameters are set as loss = sinkhorn; output Sinkhorn divergence. Where S represents the Sinckhorn divergence, since regardless of whether the input is the initial image or an intermediate image, the target image... Nothing changes; abbreviations are used. The gradient is calculated using the Autograd function from the PyTorch library. The function is used as follows: input the Sinckhorn divergence, output the gradient of the Sinckhorn divergence. As the current momentum field, in which This represents the gradient symbol.

[0112] Update the current velocity field using the current momentum field:

[0113]

[0114] in, This is the updated velocity field; the superscript 'i' indicates that the intermediate image corresponding to the velocity field is... Represents the current velocity field, with the subscript t. * ,t represents the intermediate time; ∈ represents the velocity field update rate; γ represents the original velocity field momentum retention parameter, which is a manually set hyperparameter that controls the velocity field update rate; ^ indicates that small batch updates are currently being used.

[0115] Update the current image using the current velocity field:

[0116]

[0117] in, This is the updated image. This is the current middle image, where the superscript i represents the i intermediate images. It is the updated velocity field, and σ is the rate at which the image is updated; it is a hyperparameter that is set manually.

[0118] Store the momentum field and velocity field corresponding to the current mini-batch noise; store the current triplet.

[0119] From the initial image or random noise Begin by repeating the above steps to obtain a sequence of triples. Where m represents the number of times the above steps are repeated to construct the second-order Sinckhorn gradient flow of the discretized image.

[0120] Repeatedly select the initial image or random noise Begin reconstructing the second-order Sinckhorn gradient flow of the discretized image, and iterate multiple times to build the trajectory pool.

[0121] S3. A second-order Sinckhorn gradient flow model of a neural network is trained using empirical replay technology and momentum field matching technology; a specified momentum field is fitted by solving the optimization objective of the neural network; the input and output of the second-order Sinckhorn gradient flow model of the neural network are: input is the current intermediate image, the current corresponding time t, the velocity field corresponding to the image, and the momentum field corresponding to the image; output is the gradient of the current Sinckhorn divergence. Further, the training of the second-order Sinckhorn gradient flow model of the neural network specifically involves: using empirical replay technology, training with the constructed trajectory pool without introducing additional second-order Sinckhorn gradient flow data; using momentum field matching technology, a U-net neural network is used to approximate the real-time update of the momentum field to simulate the second-order Sinckhorn gradient flow, thus obtaining the neural network second-order Sinckhorn gradient flow model.

[0122] The trained neural network second-order Sinkhorn gradient flow model further includes:

[0123] We employ an experience replay technique and use the constructed trajectory pool for training without introducing additional second-order Sinkhorn gradient flow data.

[0124] Momentum field matching technology is employed, and momentum field matching targets are randomly selected to minimize the prediction error of the momentum field through neural network parameterization technology; the mathematical expression of the matching target is:

[0125]

[0126] Where θ represents the network parameters, t represents the intermediate time, T represents the upper bound of the time, and x ~ μ t E represents the expectation derived from a time-dependent intermediate distribution of the image, where E represents the average of multiple calculations. It's training loss. This refers to a neural network that has been trained and optimized through backpropagation using the Loss algorithm.

[0127] The model is trained using Loss backpropagation gradient descent with the Adam optimizer and a properly set learning rate.

[0128] Furthermore, the stage transition time predictor, which predicts the optimal time point from the current state to the intermediate state of the general model, specifically involves:

[0129] Map any input image x to the correct time t of other general models to identify the most likely P. t Make x ~ P t Using neural networks Prediction time t; during the optimization process, the training loss L(φ) includes the loss from P. t Sample X t And minimize t φ (X t The distance between ) and t ensures X t Accurate simulation of P t Time Predictor The training uses the following regression loss:

[0130]

[0131] Where θ represents the network parameters, T represents the upper bound of time, and x t ~P t This indicates that the intermediate image is selected.

[0132] The model is trained using Loss backpropagation gradient descent with the Adam optimizer and a properly set learning rate.

[0133] S4. Employ a stage transition time predictor to predict the optimal time point from the current state of the model to the intermediate state of the general model, thereby enhancing the model's adaptability and response speed in accelerating the generation of the general model.

[0134] S5. Accelerate image generation using second-order Sinckhorn gradient flow, wherein the generation method includes using a neural network second-order Sinckhorn gradient flow model to quickly generate intermediate images near the target flow, predict the optimal time point of the intermediate images, and match images generated by general diffusion-based generation models or flow matching models.

[0135] The method of accelerating image generation using second-order Sinckhorn gradient flow specifically involves:

[0136] Initial noise is generated randomly or initial images are sampled randomly from the initial image dataset. and a randomly set initial velocity field

[0137] An intermediate image close to the target image distribution is generated using a second-order Sinkhorn gradient flow model of a neural network: velocity field iteration guided by momentum field prediction and subsequent image updates are performed using a function evaluation of less than 10 steps.

[0138]

[0139] in, From Intermediate image of the departure point. From The actual intermediate velocity field, It is the momentum field predicted by the second-order Sinkhorn gradient flow model of the neural network, ∈ represents the velocity field update rate, γ represents the original velocity field momentum retention parameter, and σ is the rate that controls the image update, which is a manually set hyperparameter.

[0140] Based on the above theory, the following is the process for accelerating the second-order Sinkhorn gradient flow of the neural network to generate a generative model (2-NSGF++):

[0141] Neural Sinkhorn Gradient Flow++ is a model that combines neural Sinkhorn gradient flow with existing generative flow models to accelerate production. It introduces a two-stage 2-NSGF++ algorithm to enhance generation efficiency. It uses neural Sinkhorn gradient flow to quickly approximate the image manifold, and then uses a simple process to refine the samples.

[0142] The 2-NSGF++ model consists of the following three components:

[0143] 1.2-NSGF model:

[0144] like Figure 1 As shown, the 2-NSGF model is the foundation of this system, utilizing a neural network to dynamically simulate the Sinckhorn gradient flow. This model focuses specifically on optimizing the probability distribution in the Wasserstein space to minimize the entropy-regularized Wasserstein distance between the target distribution and the current distribution.

[0145] Simulation of gradient flow: Based on the initial and target distributions of the input, a neural network is used to calculate the momentum field at each location in the gradient flow. This momentum field not only reflects the direction of movement of each point in the distribution but also optimizes the movement rate of these points.

[0146] Momentum field update: Real-time update of the momentum field is achieved by using the definition of Sinkhorn divergence and gradient descent method to adjust the momentum field at each time step to ensure effective approximation to the target distribution.

[0147] Numerical method: The continuous equations are discretized using the explicit Euler method, which ensures the stability and efficiency of the calculation.

[0148] 2. Neural Straight Flow (NSF):

[0149] The NSF model is an advanced version of 2-NSGF, specifically designed to further refine and optimize the generated images based on gradient flow.

[0150] Linear flow process: In a linear flow, each sample transitions from the initial state to the target state along a simplified dynamic path, which is determined by the directly computed optimal gradient flow.

[0151] Model training: The NSF model is trained using momentum field matching technology to ensure that the transformation of each sample is highly accurate and consistent with the linear flow theory, thereby significantly improving the generation speed and effect.

[0152] 3. Stage transition time predictor:

[0153] The stage transition time predictor is a key technology used to determine when to switch the generation process from the 2-NSGF model to the NSF model to achieve an optimal balance between efficiency and effectiveness.

[0154] Predictive Model: This module uses machine learning techniques to predict the optimal switching time and dynamically adjusts it based on real-time data.

[0155] Optimization objective: The training objective is to minimize the difference between the predicted time and the actual optimal switching time, using regression analysis to optimize the predictor parameters.

[0156] Implementation results: It can significantly reduce unnecessary calculations, while ensuring a smooth transition in the generation process, and improving the overall response speed and output quality of the generation system.

[0157] Implementation Examples

[0158] like Figure 4 As shown, this method first constructs a training trajectory pool based on a small batch of second-order Sinkhorn gradient flow, and then uses a momentum matching algorithm to train the neural network second-order Sinkhorn gradient flow model through neural network fitting.

[0159] like Figure 3 As shown, an embodiment of the present invention is implemented on a machine equipped with an NVIDIA 4090 graphics processor and 256GB of memory.

[0160] like Figure 2 As shown, this implementation example demonstrates the significant advantages of the novel neural network second-order Sinkhorn gradient flow accelerated image generation model (2-NSGF++) in the following key aspects by comparing the traditional flow matching model (FM) with the novel model:

[0161] Accelerate the generation process:

[0162] 2-NSGF++ utilizes a stage transition time predictor to reduce the necessary iteration steps, significantly accelerating image generation. Comparative data shows that, at the same generation quality, 2-NSGF++ typically requires only a fraction of the steps of traditional models.

[0163] Improve generation quality:

[0164] In high-resolution image generation tasks, such as the CelebA-HQ dataset, images generated by 2-NSGF++ are visually sharper and more realistic. This is thanks to the model's effective preservation of detail and control of noise when processing large-scale datasets.

[0165] Optimize resource usage:

[0166] By using mini-batch processing and trajectory pooling techniques, 2-NSGF++ optimizes the use of memory and computing resources, enabling efficient image generation even in resource-constrained environments.

[0167] Stability and adaptability:

[0168] The new time predictor enhances the model's stability and adaptability in variable environments, ensuring consistent quality throughout the generation process.

[0169] Reduce training time:

[0170] The introduction of mini-batch gradient flow and experience replay reduces the total number of iterations required during the training phase, thereby shortening the time span from training to deployment.

[0171] This invention also provides an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the image generation method based on second-order Sinkhorn gradient flow of neural networks as described in any embodiment.

[0172] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the image generation method based on second-order Sinkhorn gradient flow of neural networks as described in any embodiment.

[0173] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

Claims

1. A method for accelerating image generation based on second-order Sinckhorn gradient flow of neural networks, characterized in that, Includes the following steps: Data preparation involves acquiring the target image dataset and an initial image dataset of the same resolution (either noisy or initial) obtained through random sampling. Based on the target image dataset and the initial image dataset, a discretized image second-order Sinckhorn gradient flow is constructed; the gradient flow includes second-order Sinckhorn gradient solving and iteration, mini-batch second-order Sinckhorn gradient flow technology and trajectory pooling technology, wherein the second-order Sinckhorn gradient solving and iteration includes solving the Sinckhorn gradient of the current intermediate image with respect to the target image and iteratively updating the current intermediate image; Specifically, the construction of the discretized image second-order Sinkhorn gradient flow involves: matching randomly sampled Gaussian noise or random sampling of an initial image dataset of the same resolution with samples of the target image dataset to form mini-batch noise-image pairs; randomly setting an initial velocity field; calculating the Sinkhorn divergence of the mini-batch noise-image pairs using a Sinkhorn divergence calculation library and solving for the gradient to obtain the gradient of the current Sinkhorn divergence as the current momentum field; updating the current image velocity field using the current momentum field, and updating the current image using the updated velocity field to form a new mini-batch noise-image pair; storing the momentum field and velocity field corresponding to the current mini-batch noise; repeating the above steps until the amount of data required to train the neural network second-order Sinkhorn gradient flow model is reached. Mini-batch noise-image pairing is constructed by matching samples from the target image dataset with randomly sampled Gaussian noise or random samples from an initial image dataset of the same resolution. in, The superscript i indicates the i-th pair, and the subscript t indicates the time sequence number. This represents the intermediate image, where t=0 indicates... This is the initial image, t=1 indicates It is the target image. This indicates that there are n pairs that constitute a mini-batch of noise-image pairings; The Sinckhorst divergence of mini-batch noise-image pairings is calculated using the geomloss.SamplesLoss function from the GeomLoss library. The function is used as follows: Input: Current image; Target: [Image Name] Parameters are set as loss = sinkhorn; output Sinkhorn divergence. Where S represents the Sinckhorn divergence, since regardless of whether the input is the initial image or an intermediate image, the target image... Nothing changes; abbreviations are used. The gradient is calculated using the Autograd function from the PyTorch library. The function is used as follows: input the Sinckhorn divergence, output the gradient of the Sinckhorn divergence. As the current momentum field, in which Indicates the gradient sign; Update the current velocity field using the current momentum field: in, This is the updated velocity field; the superscript 'i' indicates that the intermediate image corresponding to the velocity field is... Represents the current velocity field, subscript t * ,t represents the intermediate time; ∈ represents the velocity field update rate; γ represents the original velocity field momentum retention parameter, which is a manually set hyperparameter; ^ indicates that small batch updates are currently being used. Update the current image using the current velocity field: in, This is the updated image. This is the current middle image, where the superscript i represents the i intermediate images. It is the updated velocity field, and σ is the rate at which the image is updated, which is a hyperparameter set manually. Store the momentum field and velocity field corresponding to the current mini-batch noise; store the current triplet. From the initial image or random noise Begin by repeating the above steps to obtain a sequence of triples. Where m represents the number of times the above steps are repeated to construct the second-order Sinckhorn gradient flow of the discretized image; Repeatedly select the initial image or random noise Begin reconstructing the second-order Sinckhorn gradient flow of the discretized image, and iterate multiple times to build the trajectory pool; The neural network second-order Sinckhorn gradient flow model is trained using experience playback technology and momentum field matching technology; a specified momentum field is fitted by solving the optimization objective of the neural network; the input and output of the neural network second-order Sinckhorn gradient flow model are: input is the current intermediate image, the current corresponding time t, the velocity field corresponding to the image, and the momentum field corresponding to the image; output is the gradient of the current Sinckhorn divergence; A stage transition time predictor is used to predict the optimal time point from the current state of the model to the intermediate state of the general model, thereby enhancing the model's adaptability and response speed in accelerating the generation of the general model. Image generation is accelerated using second-order Sinckhorn gradient flow. The generation method includes using a neural network second-order Sinckhorn gradient flow model to quickly generate intermediate images near the target flow, predicting the optimal time point of the intermediate images, and matching images generated by general diffusion-based generation models or flow matching models.

2. The image generation acceleration method based on second-order Sinckhorn gradient flow of neural networks according to claim 1, characterized in that, The target image dataset is a large-scale image database that is publicly available or private in reality; the initial image dataset is a noisy image of the same resolution sampled by a random method or an initial image dataset of the same resolution.

3. The image generation acceleration method based on second-order Sinckhorn gradient flow of neural networks according to claim 1, characterized in that, The training of the second-order Sinckhorn gradient flow model of the neural network is specifically carried out as follows: using the empirical replay technique, the constructed trajectory pool is used for training without introducing additional second-order Sinckhorn gradient flow data; using the momentum field matching technique, the U-net neural network is used to approximate the momentum field to simulate the real-time update of the second-order Sinckhorn gradient flow to obtain the neural network second-order Sinckhorn gradient flow model.

4. The image generation acceleration method based on second-order Sinckhorn gradient flow of neural networks according to claim 3, characterized in that, The trained neural network second-order Sinkhorn gradient flow model further includes: The empirical replay technique is used, and the constructed trajectory pool is used for training without introducing additional second-order Sinkhorn gradient flow data. Momentum field matching technology is employed, and momentum field matching targets are randomly selected to minimize the prediction error of the momentum field through neural network parameterization technology; the mathematical expression of the matching target is: Where θ represents the network parameters, t represents the intermediate time, T represents the upper bound of the time, and x ~ μ t E represents the expectation derived from a time-dependent intermediate distribution of the image, i.e., the average of multiple calculations. It's training loss. This refers to a neural network that has been trained and optimized through backpropagation using the Loss algorithm. The model is trained using Loss backpropagation gradient descent with the Adam optimizer and a properly set learning rate.

5. The image generation acceleration method based on second-order Sinckhorn gradient flow of neural networks according to claim 1, characterized in that, The stage transition time predictor is used to predict the optimal time point from the current state to the intermediate state of the general model. Map any input image x to the correct time t on which other general flow-matching-based generative models depend, so that the images generated by the current neural network's second-order Sinkhorn gradient flow conform to the intermediate image distribution P of the general flow-matching-based generative model. t Even if we get x~P t Using neural networks Prediction time t; during the optimization process, the training loss L(φ) includes the loss from P. t Sample X t And minimize t φ (X t The distance between ) and t ensures X t Accurate simulation of P t Time Predictor The training uses the following regression loss: Where θ represents the network parameters, T represents the upper bound of time, and x t ~P t This indicates that an intermediate image has been selected; The model is trained using Loss backpropagation gradient descent with the Adam optimizer and a properly set learning rate.

6. The image generation acceleration method based on second-order Sinckhorn gradient flow of neural network according to claim 1, characterized in that, The method of accelerating image generation using second-order Sinckhorn gradient flow specifically involves: Initial noise is generated randomly or initial images are sampled randomly from the initial image dataset. and a randomly set initial velocity field An intermediate image close to the target image distribution is generated using a second-order Sinkhorn gradient flow model of a neural network: velocity field iteration guided by momentum field prediction and subsequent image updates are performed using a function evaluation of less than 10 steps. in, From Intermediate image of the departure point. From The actual intermediate velocity field, It is the momentum field predicted by the second-order Sinkhorn gradient flow model of the neural network, ∈ represents the velocity field update rate, γ represents the original velocity field momentum retention parameter, and σ is the rate that controls the image update, which is a manually set hyperparameter.

7. An electronic device comprising a memory and a processor, characterized in that, The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the image generation method based on the second-order Sinckhorn gradient flow of a neural network as described in any one of claims 1-6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the image generation method based on the second-order Sinkhorn gradient flow of a neural network as described in any one of claims 1-6.

Images