Second-order Sinkhohn gradient flow acceleration image generation method based on neural network

Abstract

The invention discloses a second-order Sinkhon gradient flow acceleration image generation method based on a neural network. According to the method, the neural second-order Sinkhohn gradient flow 2-NSGF model is utilized to parameterize the time variation momentum field related to the Sinkhohn divergence in the second-order Wosenstein gradient flow, so that efficient inference and simulation are realized. Through utilization of source and target distribution samples, the 2-NSGF model can generate and refine simulation data in real time, and the calculation complexity of a traditional method is reduced. In addition, by introducing a two-stage 2-NSGF + + model, the efficiency of processing a high-dimensional data set is further optimized, and the simulation precision and speed are improved. The invention provides an efficient data processing technology which is suitable for various applications in the field of machine learning, such as image generation, style migration, audio-text conversion and the like. According to the invention, the image generation efficiency is obviously improved; a kernel function is not needed, so that the complexity is reduced; and the sample quality and the generation stability are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence image generation models, and in particular to a method for accelerating image generation based on a neural network second-order Sinkhorn gradient flow. Background Art

[0002] It involves the optimization of generative models, especially with application on high-dimensional image datasets, aiming to improve the efficiency and quality of the image generation process.

[0003] Generative models are a type of model in machine learning that is used to learn distributions from data and generate new, similar data samples. They are widely used in tasks such as image generation, text generation, data augmentation, and style transfer. Common generative models include generative adversarial networks, variational autoencoders, and the recently developed diffusion model. Generative adversarial networks generate high-quality data by training two adversarial neural networks, the generator and the discriminator. Variational autoencoders compress input data into a latent space through an encoder and restore it through a decoder to generate new samples. These models have achieved remarkable results in many fields and promoted the advancement of artificial intelligence.

[0004] Generative models based on neural network ordinary differential equations and stochastic differential equations have made significant progress in the field of machine learning in recent years. Famous examples include diffusion models (Wijmans JG, Baker R W. The solution-diffusion model: a review [J]. Journal of membrane science, 1995, 107 (1-2): 1-21.) (Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models [J]. Advances inneural information processing systems, 2020, 33: 6840-6851.) (Song Y, Sohl-Dickstein J, Kingma DP, et al. Score-based generative modeling through stochastic differential equations [J]. arXiv preprint arXiv: 2011.13456, 2020.) and flow matching models (Lipman Y, Chen RTQ, Ben-Hamu H, et al. Flow matching for generative modeling [J]. arXiv preprint arXiv:2210.02747,2022.)(Liu X, Gong C, Liu Q.Flowstraight and fast:Learning to generate and transfer data with rectified flow[J].arXiv preprint arXiv:2209.03003,2022.). Diffusion models and flow matching models provide a natural way to model the evolution of data distribution by simulating continuous dynamic systems. These methods have many advantages in generative models, but also face some challenges and disadvantages.

[0005] Diffusion models generate data by gradually denoising. The basic idea is to gradually add noise to the data until it is transformed into a Gaussian distribution, and then gradually denoise the data to restore the original data through the reverse process. This type of model includes diffusion probability models and score matching models. Diffusion models perform well in image generation tasks and can generate high-quality and detailed images. They have a solid theoretical foundation and have good interpretability through clear probabilistic process modeling. Through gradual denoising, diffusion models can effectively handle noise and uncertainty in data. As an improvement on the diffusion model, the flow matching model uses a neural network to match straight-line flows, significantly reducing the number of steps required for generation. While maintaining the high quality of the generated images, the flow matching model greatly reduces the computational cost.

[0006] However, the step-by-step denoising process of the diffusion model requires a lot of computation, and the inference process can be very time-consuming. Due to the need for multi-step denoising, the inference speed of the diffusion model is slow and difficult to meet the needs of real-time applications. The model training and inference process is complex and requires careful design and parameter adjustment.

[0007] In recent years, the application of gradient flow-based generative models in machine learning has become increasingly widespread, especially in tasks such as image generation, style transfer, and data augmentation. As a powerful optimization tool, Wasserstein gradient flow has been widely used to solve optimization problems on Wasserstein probability space. However, the traditional Wasserstein gradient flow method is computationally complex and costly in high-dimensional space, especially when frequent transfer mapping calculations are required, which limits its widespread use in practical applications. To overcome these challenges, researchers have proposed a variety of methods to approximate and accelerate the calculation of Wasserstein gradient flow.

[0008] In the development of gradient flow-based generative models, neural networks are widely used to approximate complex parts of Wasserstein gradient flow, such as time-dependent transport maps. Through neural networks, samples of the target distribution can be used during training to learn and approximate the transport map, thereby simplifying the calculation. However, existing methods still face problems of computational complexity and training stability in high-dimensional space.

[0009] In addition, commonly used distribution metrics for gradient flow include f-divergence (such as Kullback-Leibler and Jensen-Shannon divergence), maximum mean difference, and slice-Wasserstein distance. These metrics are used to guide the generative model to optimize from the source distribution to the target distribution, but in high-dimensional space, these methods may perform poorly and have high computational complexity. To overcome these problems, the Sinkhorn divergence was proposed as an efficient metric. By entropy regularizing the Wasserstein distance, the Sinkhorn divergence can be efficiently calculated through the Sinkhorn algorithm, making it have great potential in generative models.

[0010] According to the research on the existing generative model literature, the diffusion model and the flow matching model are the two methods in the prior art that are closest to the present invention.

[0011] Diffusion models generate data by gradually denoising. The basic idea is to gradually add noise to the data until it is transformed into a Gaussian distribution, and then gradually denoise the data to restore the original data through a reverse process. This process can be divided into two stages. The first is the forward process, which starts from the original data and gradually adds Gaussian noise to it, so that it eventually transforms into a pure noise distribution. This process is implemented through a multi-step Markov chain, where a certain amount of noise is added at each step, making the data distribution gradually more random and uncertain. The second is the reverse process, which starts from pure noise and gradually removes the noise through a trained denoising network to finally restore a clear image. In this process, the denoising network tries to reduce the impact of noise at each step, so that the data gradually returns to its initial clear state. Diffusion models perform well in image generation tasks and can generate high-quality and detailed images. They have a solid theoretical foundation and are well interpretable through explicit probabilistic process modeling. Through gradual denoising, diffusion models can effectively handle noise and uncertainty in data. However, the gradual denoising process of diffusion models requires a lot of computation, and the inference process can be very time-consuming, especially when generating high-resolution images. Since multi-step denoising is required, the inference speed of the diffusion model is slow and cannot meet the needs of real-time applications.

[0012] As an improvement of the diffusion model, the flow matching model uses a neural network to match straight-line flows, significantly reducing the number of steps required for generation. In the flow matching model, the goal is to learn an optimal flow path from the initial distribution to the target distribution, rather than denoising step by step. The core idea is to optimize and learn a mapping function so that the process of generating target data from initial random noise can be completed in fewer steps. The flow matching model learns the optimal transmission mapping through a neural network, directly transforming the initial noise distribution into the target data distribution. This method reduces the steps required for generation, thereby greatly improving efficiency.

[0013] During the training process, the momentum field is optimized by minimizing the difference between the generated samples and the target samples. This method not only maintains the high quality of the generated images, but also significantly reduces the computational cost. In recent years, such models have achieved remarkable success in image generation tasks, such as the application of Stability AI in image generation, which demonstrates its powerful ability in generating high-resolution images. The flow matching model significantly reduces the computational cost while maintaining the high quality of the generated images. This method not only improves the generation efficiency, but also retains the advantages of the diffusion model in dealing with noise and uncertainty.

[0014] Although diffusion models and flow matching models perform well in generating high-quality images, they also have some significant shortcomings and deficiencies.

[0015] First, the step-by-step denoising process of the diffusion model requires a lot of computation. Generating a high-quality image often requires hundreds or even thousands of steps of denoising, and each step requires complex calculations and forward propagation of the neural network, which makes the computational cost very high. Especially when generating high-resolution images, the inference process can be very time-consuming, because each frame of the image needs to go through multiple calculation steps to remove noise and restore clear details. This high computational demand for step-by-step denoising not only increases the generation time, but also places higher requirements on the performance of the hardware device.

[0016] Secondly, due to the need for multi-step denoising, the diffusion model has a slow inference speed and is difficult to meet the needs of real-time applications. In many practical applications, such as video generation, real-time image processing, and augmented reality applications, generation speed is a key factor. The multi-step inference process of the diffusion model makes it difficult to play a role in these applications that require fast response. Although the generated image quality is high, its inference speed becomes a major bottleneck in scenarios that require instant feedback.

[0017] In addition, the training and reasoning process of the diffusion model is complex and requires careful design and parameter adjustment. During the training process, the noise addition and denoising process need to be fine-tuned to ensure that the model can effectively learn the characteristics of the data distribution. This includes selecting the appropriate noise intensity, designing an effective denoising network structure, and optimizing the hyperparameters during training. All of these require a lot of experiments and experience, which increases the difficulty of model development.

[0018] Although the flow matching model has improved the efficiency of the diffusion model to a certain extent, it also has some shortcomings. The flow matching model relies on neural networks to learn the optimal flow path from the initial distribution to the target distribution, and this process also requires a large amount of data and computing resources. In addition, during the learning process, the flow matching model needs to optimize the momentum field, which involves complex mathematical operations and a large amount of training data to ensure that the model can accurately simulate the evolution of the data. This training process that relies on a large amount of calculations and data means that the flow matching model still faces the problems of high computational cost and long training time when processing large-scale data sets.

[0019] The stream matching model may also face some challenges in practical applications. For example, the model training and reasoning process rely on high-quality data. If the data quality is not high or the distribution is very different, the performance of the model may be affected. In addition, although the reasoning process of the stream matching model is faster than the traditional diffusion model, it may still not meet the requirements of real-time processing in some extremely high-resolution and high-complexity applications.

[0020] In summary, although the diffusion model and flow matching model have significant advantages in generating high-quality images, their high computational cost, complex training and inference process, and performance in real-time applications limit their widespread application. These shortcomings and deficiencies provide opportunities for the development of new technologies, which can overcome these limitations and improve the efficiency and practicality of the generated models through further optimization and improvement.

[0021] Disadvantages and shortcomings of the diffusion model:

[0022] High computing requirements:

[0023] The step-by-step denoising process is computationally intensive, with each step requiring complex calculations and forward propagation of the neural network.

[0024] When high-resolution images are generated, the computational cost is higher and the inference process is very time-consuming.

[0025] Slow inference speed:

[0026] The multi-step denoising process results in slow inference speed, which is difficult to meet the needs of real-time applications such as video generation and augmented reality applications.

[0027] Complex training and parameter tuning:

[0028] The noise addition and denoising processes need to be finely controlled, including selecting noise intensity, designing denoising network structure, and optimizing training hyperparameters.

[0029] The training process requires a lot of experiments and experience, which increases the difficulty of model development.

[0030] Disadvantages and shortcomings of the flow matching model:

[0031] Reliance on large amounts of data and computing resources:

[0032] Neural networks are required to learn the optimal flow path from the initial distribution to the target distribution, which involves complex mathematical operations and a large amount of training data.

[0033] The training process is computationally expensive and takes a long time to train.

[0034] Challenges in practical applications:

[0035] Rely on high-quality data. When the data quality is not high or the distribution difference is large, the model performance will be affected.

[0036] In some extremely high-resolution and high-complexity applications, although the inference speed is faster than that of traditional diffusion models, it may still not meet the requirements of real-time processing.

[0037] As existing technologies, the diffusion model and the flow matching model are the closest technical foundations of the present invention. They provide important methods and theoretical support for generating high-quality images, but the present invention achieves significant improvements in generation efficiency and computational consumption by further optimizing these technologies. Summary of the invention

[0038] The object of the present invention is to provide a method for accelerating image generation based on a neural network second-order Sinkhorn gradient flow in view of the deficiencies in the prior art.

[0039] The object of the present invention is achieved through the following technical solution: a method for accelerating image generation based on a neural network second-order Sinkhorn gradient flow, comprising the following steps:

[0040] Data preparation: obtaining the target image dataset and the noise of the same resolution or the initial image dataset of the same resolution obtained by random sampling;

[0041] According to the target image data set and the initial image data set, a discretized image second-order Sinkhorn gradient flow is constructed; the gradient flow includes second-order Sinkhorn gradient solution and iteration, small-batch second-order Sinkhorn gradient flow technology and trajectory pool construction technology, wherein the second-order Sinkhorn gradient solution and iteration include Sinkhorn gradient solution of the current intermediate image for the target image and iterative update of the current intermediate image;

[0042] The neural network second-order Sinkhorn gradient flow model is trained using the experience replay technology and the momentum field matching technology; the specified momentum field is fitted by solving the neural network optimization target; the input and output of the neural network second-order Sinkhorn gradient flow model are: input the current intermediate image, the current corresponding time t, the image corresponding velocity field, and the image corresponding momentum field; output the gradient of the current Sinkhorn divergence;

[0043] A phase transition time predictor is used to predict the optimal time point for the model to change from the current state to the intermediate state of the general model, so as to enhance the adaptability and response speed of the model to accelerate the generation of the general model;

[0044] The second-order Sinkhorn gradient flow is used to accelerate image generation. The generation method includes using a neural network second-order Sinkhorn gradient flow model to quickly generate an intermediate image near a target flow, predicting the optimal time point of the intermediate image, and matching a general diffusion-based generation model or a flow matching model to generate an image.

[0045] Furthermore, the target image dataset is a large-scale image database that is public or private in reality; and the initial image dataset is a noise image of the same resolution sampled by a random method or an initial image dataset of the same resolution.

[0046] Furthermore, the construction of the second-order Sinkhorn gradient flow of the discretized image is specifically as follows: using randomly sampled Gaussian noise or random sampling of an initial image data set of the same resolution to match with the sampling of a target image data set to form a small batch noise-image pairing, and randomly setting an initial velocity field; using a Sinkhorn divergence calculation library to calculate the Sinkhorn divergence of the small batch noise-image pairing, and performing gradient solution to obtain the gradient of the current Sinkhorn divergence as the current momentum field; using the current momentum field to update the current image velocity field, and updating the current image through the updated velocity field to form a new small batch noise-image pairing; storing the momentum field and velocity field corresponding to the current small batch noise; and repeating the above steps.

[0047] Furthermore, the constructing of the discretized image second-order Sinkhorn gradient flow further comprises:

[0048] Use randomly sampled Gaussian noise or random sampling of the initial image dataset of the same resolution to match the sampling of the target image dataset to form a small batch of noise-image pairings in, The superscript i indicates the i-th pairing, and the subscript t indicates the time sequence number. represents the intermediate image, t=0 represents is the initial image, t=1 means is the target image, It means that there are n pairs in total that constitute a small batch of noise-image pairs;

[0049] The geomloss.SamplesLoss function in the Sinkhorn divergence calculation library GeomLoss is used to calculate the Sinkhorn divergence of small batch noise-image pairs. The specific use of the function is as follows: Input current image, target image Parameter setting loss = sinkhorn; output Sinkhorn divergence Where S represents the Sinkhorn divergence. Since no matter the input initial image or intermediate image, the target image No change occurs, use the shorthand method The Autograd function in the PyTorch library is used to solve the gradient. The specific use of the function is as follows: Input Sinkhorn divergence, output Sinkhorn divergence gradient As the current momentum field, Represents the gradient symbol;

[0050] Update the current velocity field using the current momentum field:

[0051]

[0052] in, is the updated velocity field, and the superscript i indicates that the intermediate image corresponding to the velocity field is Indicates the current velocity field, subscript t * ,t represents the intermediate time;∈ represents the velocity field update rate,γ represents the momentum retention parameter of the original velocity field, which is a manually set hyperparameter. Both of them control the velocity field update rate,^ represents that small batch updates are currently used;

[0053] Update the current image using the current velocity field:

[0054]

[0055] in, This is the updated image. is the current intermediate image, and the superscript i represents the i intermediate image. is the velocity field after update, σ controls the rate of image update and is a manually set hyperparameter;

[0056] Store the momentum field and velocity field corresponding to the current small batch noise: store the current triplet

[0057] From the initial image or random noise Start by repeating the above steps to get the triple sequence Where m means repeating the above steps m times to construct the second-order Sinkhorn gradient flow of the discretized image;

[0058] Repeatedly select the initial image or random noise Start to reconstruct the second-order Sinkhorn gradient flow of the discretized image, and loop multiple times to build the trajectory pool.

[0059] Furthermore, the training of the neural network second-order Sinkhorn gradient flow model is specifically as follows: using the experience replay technology, using the constructed trajectory pool for training, and not introducing additional second-order Sinkhorn gradient flow data; using the momentum field matching technology, using the U-net neural network to approximate the momentum field to simulate the real-time update of the second-order Sinkhorn gradient flow to obtain the neural network second-order Sinkhorn gradient flow model.

[0060] Furthermore, the training of the neural network second-order Sinkhorn gradient flow model further includes:

[0061] The experience replay technique is used to train the constructed trajectory pool without introducing additional second-order Sinkhorn gradient flow data;

[0062] The momentum field matching technology is used to randomly extract the momentum field matching target to minimize the prediction error of the momentum field through the neural network parameterization technology; the matching target is mathematically expressed as:

[0063]

[0064] Among them, θ represents the network parameter, t represents the intermediate time, T represents the upper limit of time, x~μ t It means that the image comes from the time-dependent intermediate distribution, E means to find the expectation, that is, to find the average after multiple calculations. It is training loss. Refers to a neural network that has been trained through neural network loss back propagation optimization;

[0065] Train the model to perform Loss backpropagation gradient descent, use the Adam optimizer, and set the learning rate.

[0066] Furthermore, the phase transition time predictor is used to predict the optimal time point for the model to change from the current state to the intermediate state of the general model:

[0067] Map any input image x to the correct time t that other general flow matching-based generative models depend on, so that the image generated by the second-order Sinkhorn gradient flow of the current neural network obeys the intermediate image distribution P of the general flow matching-based generative model t , even if x~P t ; Using Neural Networks Prediction time t; During the optimization process, the training loss L(φ) includes the t Extract sample X t And minimize t φ (X t ) and t, ensuring that X t Accurate simulation P t ; Time predictor The training uses the following regression loss:

[0068]

[0069] Among them, θ represents the network parameters, T represents the upper bound of time, and x t ~P t Indicates the selection of the middle image;

[0070] Train the model to perform Loss backpropagation gradient descent, use the Adam optimizer, and set the learning rate.

[0071] Furthermore, the use of the second-order Sinkhorn gradient flow to accelerate image generation is specifically as follows:

[0072] Randomly generate initial noise or randomly sample initial images from the initial image dataset and a randomly set initial velocity field

[0073] An intermediate image close to the target image distribution is generated using a neural network second-order Sinkhorn gradient flow model: velocity field iteration guided by momentum field prediction and subsequent image update using less than 10 steps of function evaluation:

[0074]

[0075] in, is from The starting intermediate image, is from In fact, the intermediate velocity field, It is the momentum field predicted by the neural network second-order Sinkhorn gradient flow model, ∈ represents the velocity field update rate, γ represents the momentum preservation parameter of the original velocity field, and σ controls the image update rate, which is an artificially set hyperparameter.

[0076] The present invention also provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the method for accelerating image generation based on the neural network second-order Sinkhorn gradient flow.

[0077] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method for accelerating image generation based on a neural network second-order Sinkhorn gradient flow is implemented.

[0078] The beneficial effects of the present invention are:

[0079] Significantly improved generation efficiency: Compared with the baseline method, the 2-NSGF++ model uses a two-stage process to quickly approximate the high-dimensional image manifold and then fine-tune it along a simple straight line, reducing the computational time and resources required for generation.

[0080] No need to rely on kernel functions, reducing complexity: This method is based on the Sinkhorn gradient flow and does not require kernel functions, avoiding the problem of performance degradation of kernel methods in high-dimensional data and improving the applicability and efficiency of the model.

[0081] Improved sample quality and generation stability: As the sample size increases, the velocity field approximation gradually converges to the true velocity field. The quality of the generated samples is better than the existing neural WGF method, and it shows higher stability and reliability on multiple datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 It is a schematic diagram of the dynamic simulation and optimization algorithm of the momentum field;

[0083] Figure 2 This is a comparison chart between 2-NSGF++ and FM methods;

[0084] Figure 3 It is a schematic diagram of the generation process of 2-NSGF++;

[0085] Figure 4 This is a schematic diagram of the 2-NSGF training process. DETAILED DESCRIPTION

[0086] In order to make the purpose, technical scheme and technical effect of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings.

[0087] like Figure 1 As shown, an embodiment of the present invention provides a method for accelerating image generation based on a neural network second-order Sinkhorn gradient flow, comprising:

[0088] S1. Data preparation: obtain the target image dataset and the same-resolution noise or the same-resolution initial image dataset obtained by random sampling. The target image dataset is a large-scale image database that is public or private in reality; the initial image dataset is the same-resolution noise image or the same-resolution initial image dataset obtained by random sampling.

[0089] S2. According to the target image data set and the initial image data set, a second-order Sinkhorn gradient flow for discretized images is constructed; the gradient flow includes second-order Sinkhorn gradient solution and iteration, small-batch second-order Sinkhorn gradient flow technology and trajectory pool construction technology, wherein the second-order Sinkhorn gradient solution and iteration include the Sinkhorn gradient solution of the current intermediate image for the target image and the iterative update of the current intermediate image. The construction of the second-order Sinkhorn gradient flow for discretized images is specifically as follows: using random sampling Gaussian noise or random sampling of the initial image data set with the same resolution to match the sampling of the target image data set to form a small-batch noise-image pairing, and randomly setting the initial velocity field; using the Sinkhorn divergence calculation library to calculate the Sinkhorn divergence of the small-batch noise-image pairing, and performing gradient solution to obtain the gradient of the current Sinkhorn divergence as the current momentum field; using the current momentum field to update the current image velocity field, and updating the current image through the updated velocity field to form a new small-batch noise-image pairing; storing the momentum field and velocity field corresponding to the current small-batch noise; repeating the above steps.

[0090] The construction of the second-order Sinkhorn gradient flow of the discretized image includes the following theoretical basis:

[0091] Construct a second-order Sinkhorn gradient flow and its finite particle form:

[0092] The probability distribution space is expressed as follows: Upper functional Optimization problem: in, represents the optimized object distribution or initial distribution, μ * represents the target distribution, represents the selected divergence, Indicates the distribution of the optimization object and the target distribution with divergence To solve the above optimization problem, we use the functional in Wasserstein space The downwelling flow is described by the following continuity equation:

[0093]

[0094] in, is a momentum field that defines the direction of position transfer.

[0095] The second-order Hamiltonian gradient flow is expressed by the following formula:

[0096]

[0097] in, represents the Hamiltonian momentum variable, and represents the Hamiltonian potential energy; where the Hamiltonian momentum variable Φ t μ t Momentum, in the second-order Hamiltonian gradient flow system, the change of the system is applied to the momentum instead of being directly used for the momentum field.

[0098] Define the Sinkhorn divergence, given two probability measures Entropy Regularized 2-Watsonstein Distance The definition is as follows:

[0099]

[0100] Among them, Π(μ,v) represents the set of all probability couplings π with μ,v as marginal distributions, ε>0 is the regularization coefficient, represents the product measure, that is, Represents π and The KL divergence between ; Based on the definition of Sinkhorn divergence, construct the Sinkhorn objective where μ * represents the target distribution; according to the first variation of the Sinkhorn objective, we get:

[0101]

[0102] in, and f μ,μ They are -Part of the potential, put the pair called of - potential, similarly (f μ,μ ,g μ,μ ) is called of -potential; by setting the momentum field in the second-order Hamiltonian gradient flow Constructing a second-order Sinkhorn-Watsonstein gradient flow; finally, considering the distribution of finite particle composition A generative model suitable for real-world applications is shown in the following formula:

[0103]

[0104] The real-time update of using a neural network to approximate the momentum field to simulate the second-order Sinkhorn gradient flow also includes:

[0105] Real-time update of Sinkhorn gradient flow:

[0106] In accordance with In the process of driving the discrete system update, if The distribution , the gradient flow stops changing when it approaches the target distribution; according to the continuity equation Discretization of :

[0107]

[0108] For a time-dependent particle distribution uniquely determined by the initial particle distribution, initial velocity, and momentum field Real-time updates, including Determined by the momentum field predicted by the neural network.

[0109] Furthermore, the constructing of the discretized image second-order Sinkhorn gradient flow further comprises:

[0110] Use randomly sampled Gaussian noise or random sampling of the initial image dataset of the same resolution to match the sampling of the target image dataset to form a small batch of noise-image pairings in, The superscript i indicates the i-th pairing, and the subscript t indicates the time sequence number. represents the intermediate image, t=0 represents is the initial image, t=1 means is the target image, It means that there are n pairs in total that constitute a small batch of noise-image pairs.

[0111] The geomloss.SamplesLoss function in the Sinkhorn divergence calculation library GeomLoss is used to calculate the Sinkhorn divergence of small batch noise-image pairs. The specific use of the function is as follows: Input current image, target image Parameter setting loss = sinkhorn; output Sinkhorn divergence Where S represents the Sinkhorn divergence. Since no matter the input initial image or intermediate image, the target image No change occurs, use the shorthand method The Autograd function in the PyTorch library is used to solve the gradient. The specific use of the function is as follows: Input Sinkhorn divergence, output Sinkhorn divergence gradient As the current momentum field, Represents the gradient symbol.

[0112] Update the current velocity field using the current momentum field:

[0113]

[0114] in, is the updated velocity field, and the superscript i indicates that the intermediate image corresponding to the velocity field is Indicates the current velocity field, subscript t * ,t represents the intermediate time; ∈ represents the velocity field update rate, γ represents the momentum retention parameter of the original velocity field, which is an artificially set hyperparameter to control the velocity field update rate, and ^ represents that small batch updates are currently used.

[0115] Update the current image using the current velocity field:

[0116]

[0117] in, This is the updated image. is the current intermediate image, and the superscript i represents the i intermediate image. is the velocity field after update, σ controls the rate of image update and is a manually set hyperparameter.

[0118] Store the momentum field and velocity field corresponding to the current small batch noise: store the current triplet

[0119] From the initial image or random noise Start by repeating the above steps to get the triple sequence Wherein, m means repeating the above steps m times to construct the second-order Sinkhorn gradient flow of the discretized image.

[0120] Repeatedly select the initial image or random noise Start to reconstruct the second-order Sinkhorn gradient flow of the discretized image, and loop multiple times to build the trajectory pool.

[0121] S3. Use experience replay technology and momentum field matching technology to train the neural network second-order Sinkhorn gradient flow model; solve and fit the specified momentum field by solving the neural network optimization target; the input and output of the neural network second-order Sinkhorn gradient flow model are: input the current intermediate image, the current corresponding time t, the image corresponding velocity field, the image corresponding momentum field; output the gradient of the current Sinkhorn divergence. Further, the training of the neural network second-order Sinkhorn gradient flow model is specifically as follows: adopt the experience replay technology, use the constructed trajectory pool for training, and do not introduce additional second-order Sinkhorn gradient flow data; adopt the momentum field matching technology, use the U-net neural network to approximate the momentum field to simulate the real-time update of the second-order Sinkhorn gradient flow to obtain the neural network second-order Sinkhorn gradient flow model.

[0122] The training neural network second-order Sinkhorn gradient flow model further includes:

[0123] The experience replay technique is adopted to use the constructed trajectory pool for training without introducing additional second-order Sinkhorn gradient flow data.

[0124] The momentum field matching technology is used to randomly extract the momentum field matching target to minimize the prediction error of the momentum field through the neural network parameterization technology; the matching target is mathematically expressed as:

[0125]

[0126] Among them, θ represents the network parameter, t represents the intermediate time, T represents the upper limit of time, x~μ t It means that the image comes from the time-dependent intermediate distribution, E means to find the expectation, that is, to find the average after multiple calculations. It is training loss. Refers to a neural network that has been trained through neural network loss back propagation optimization.

[0127] Train the model to perform Loss backpropagation gradient descent, use the Adam optimizer, and set the learning rate.

[0128] Furthermore, the phase transition time predictor is used to predict the optimal time point for the model to change from the current state to the intermediate state of the general model:

[0129] Map any input image x to the correct time t of other general models, thereby identifying the most likely P t So that x~P t ; Using Neural Networks Prediction time t; During the optimization process, the training loss L(φ) includes the t Extract sample X t And minimize t φ (X t ) and t, ensuring that X t Accurate simulation P t ; Time predictor The training uses the following regression loss:

[0130]

[0131] Among them, θ represents the network parameters, T represents the upper bound of time, and x t ~P t Indicates selecting the middle image.

[0132] Train the model to perform Loss backpropagation gradient descent, use the Adam optimizer, and set the learning rate.

[0133] S4. A phase transition time predictor is used to predict the optimal time point for the model to change from the current state to the intermediate state of the general model, so as to enhance the adaptability and response speed of the model to accelerate the generation of the general model.

[0134] S5. Use the second-order Sinkhorn gradient flow to accelerate image generation, the generation method includes using a neural network second-order Sinkhorn gradient flow model to quickly generate an intermediate image near the target flow, predicting the optimal time point of the intermediate image, and matching the general diffusion-based generation model or flow matching model to generate the image.

[0135] The use of the second-order Sinkhorn gradient flow to accelerate image generation is specifically as follows:

[0136] Randomly generate initial noise or randomly sample initial images from the initial image dataset and a randomly set initial velocity field

[0137] An intermediate image close to the target image distribution is generated using a neural network second-order Sinkhorn gradient flow model: velocity field iteration guided by momentum field prediction and subsequent image update using less than 10 steps of function evaluation:

[0138]

[0139] in, is from The starting intermediate image, is from In fact, the intermediate velocity field, It is the momentum field predicted by the neural network second-order Sinkhorn gradient flow model, ∈ represents the velocity field update rate, γ represents the momentum preservation parameter of the original velocity field, and σ controls the image update rate, which is an artificially set hyperparameter.

[0140] Based on the above theory, the following is the process of accelerating the neural second-order Sinkhorn gradient flow to generate the model (2-NSGF++):

[0141] Neural Sinkhorn Gradient Flow++ is a model that combines Neural Sinkhorn Gradient Flow with existing generative flow models to achieve accelerated production effects; introduces a two-stage 2-NSGF++ algorithm to enhance generation efficiency; uses Neural Sinkhorn Gradient Flow to quickly approximate the image manifold, and then uses a simple process to refine the samples.

[0142] The 2-NSGF++ model consists of the following three components:

[0143] 1.2-NSGF model:

[0144] like Figure 1 As shown in Figure 2, the 2-NSGF model is the basis of the system, which uses a neural network to simulate the dynamics of the Sinkhorn gradient flow. This model specifically focuses on the optimization of the probability distribution in the Wasserstein space to minimize the entropy regularized Wasserstein distance between the target distribution and the current distribution.

[0145] Simulation of gradient flow: Based on the input initial distribution and target distribution, the momentum field of each position in the gradient flow is calculated through a neural network. This momentum field not only reflects the moving direction of each point in the distribution, but also optimizes the moving rate of these points.

[0146] Momentum field update: Real-time update of the momentum field is achieved. The definition of Sinkhorn divergence and the gradient descent method are used to adjust the momentum field at each time step to ensure effective approach to the target distribution.

[0147] Numerical method: The explicit Euler method is used to discretize the continuous equations, which ensures the stability and efficiency of the calculation.

[0148] 2. Neural Straight Flow (NSF):

[0149] The NSF model is an advancement of 2-NSGF, specifically designed to further refine and optimize the generated images based on the gradient flow.

[0150] Straight-line flow process: In straight-line flow, each sample transitions from the initial state to the target state along a simplified dynamic path, which is determined by the directly calculated optimal gradient flow.

[0151] Model training: The NSF model is trained through momentum field matching technology to ensure that the conversion of each sample is highly accurate and consistent with the straight-line flow theory, which significantly improves the speed and effect of generation.

[0152] 3. Phase transition time predictor:

[0153] The phase transition time predictor is a key technique used to determine when to switch the generation process from the 2-NSGF model to the NSF model to achieve the optimal balance between efficiency and effect.

[0154] Predictive model: This module uses machine learning technology to predict the best switching time and dynamically adjusts based on real-time data.

[0155] Optimization objective: The training objective is to minimize the difference between the predicted time and the actual optimal switching time, using regression analysis to optimize the parameters of the predictor.

[0156] Implementation effect: It can significantly reduce unnecessary calculations while ensuring a smooth transition of the generation process and improving the response speed and output quality of the overall generation system.

[0157] Implementation Examples

[0158] like Figure 4 As shown, this method first constructs a training trajectory pool based on a small batch second-order Sinkhorn gradient flow, and then trains the neural network second-order Sinkhorn gradient flow model through a momentum matching algorithm for neural network fitting.

[0159] like Figure 3 As shown, an implementation example of the present invention is implemented on a machine equipped with an NVidia 4090 graphics processor and 256GB of memory.

[0160] like Figure 2 As shown, this implementation example compares the traditional flow matching model (FM) with the new neural network second-order Sinkhorn gradient flow accelerated image generation model (2-NSGF++), and demonstrates the significant advantages of 2-NSGF++ in the following key aspects:

[0161] Speed ​​up the build process:

[0162] 2-NSGF++ uses a stage transition time predictor to reduce the necessary iteration steps and significantly speed up image generation. Comparative data shows that 2-NSGF++ usually only requires a fraction of the number of steps of traditional models at the same generation quality.

[0163] Improve build quality:

[0164] In high-resolution image generation tasks, such as the CelebA-HQ dataset, the images generated by 2-NSGF++ are visually clearer and more realistic. This is due to the model's ability to preserve details and effectively control noise when processing large-scale datasets.

[0165] Optimize resource usage:

[0166] By using mini-batch processing and trajectory pooling techniques, 2-NSGF++ optimizes the use of memory and computing resources, enabling efficient image generation even in resource-constrained environments.

[0167] Stability and adaptability:

[0168] The temporal predictor newly enhances the model’s stability and adaptability in changing environments, ensuring consistent quality during the generation process.

[0169] Reduce training time:

[0170] The introduction of mini-batch gradient flow and experience replay reduces the total number of iterations required in the training phase, thereby shortening the time span from training to deployment.

[0171] An embodiment of the present invention also provides an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the method for accelerating image generation based on the second-order Sinkhorn gradient flow of a neural network as described in any embodiment.

[0172] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the method for accelerating image generation based on the second-order Sinkhorn gradient flow of a neural network as described in any embodiment is implemented.

[0173] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capability and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

Claims

1. A method for accelerating image generation based on a neural network second-order Sinkhorn gradient flow, characterized in that: The steps include: Data preparation: obtaining the target image dataset and the noise of the same resolution or the initial image dataset of the same resolution obtained by random sampling; According to the target image data set and the initial image data set, a discretized image second-order Sinkhorn gradient flow is constructed; the gradient flow includes second-order Sinkhorn gradient solution and iteration, small-batch second-order Sinkhorn gradient flow technology and trajectory pool construction technology, wherein the second-order Sinkhorn gradient solution and iteration include Sinkhorn gradient solution of the current intermediate image for the target image and iterative update of the current intermediate image; The neural network second-order Sinkhorn gradient flow model is trained using the experience replay technology and the momentum field matching technology; the specified momentum field is fitted by solving the neural network optimization target; the input and output of the neural network second-order Sinkhorn gradient flow model are: input the current intermediate image, the current corresponding time 0, the image corresponding to the velocity field, the image corresponding to the momentum field; output the gradient of the current Sinkhorn divergence; A phase transition time predictor is used to predict the optimal time point for the model to change from the current state to the intermediate state of the general model, so as to enhance the adaptability and response speed of the model to accelerate the generation of the general model; The second-order Sinkhorn gradient flow is used to accelerate image generation. The generation method includes using a neural network second-order Sinkhorn gradient flow model to quickly generate an intermediate image near a target flow, predicting the optimal time point of the intermediate image, and matching a general diffusion-based generation model or a flow matching model to generate an image.

2. The method for accelerating image generation based on neural network second-order Sinkhorn gradient flow according to claim 1 is characterized in that: The target image dataset is a large-scale image database that is public or private in reality; the initial image dataset is a noise image with the same resolution sampled by a random method or an initial image dataset with the same resolution.

3. The method for accelerating image generation based on neural network second-order Sinkhorn gradient flow according to claim 1, characterized in that: The method for constructing the second-order Sinkhorn gradient flow of the discretized image is specifically as follows: using randomly sampled Gaussian noise or random sampling of an initial image data set with the same resolution to match the sampling of a target image data set to form a small batch noise-image pairing, and randomly setting an initial velocity field; using a Sinkhorn divergence calculation library to calculate the Sinkhorn divergence of the small batch noise-image pairing, and performing gradient solution to obtain the gradient of the current Sinkhorn divergence as the current momentum field; using the current momentum field to update the current image velocity field, and updating the current image through the updated velocity field to form a new small batch noise-image pairing; storing the momentum field and velocity field corresponding to the current small batch noise; and repeating the above steps until the amount of data required for training the neural network second-order Sinkhorn gradient flow model is reached.

4. The method for accelerating image generation based on neural network second-order Sinkhorn gradient flow according to claim 3 is characterized in that: The constructing of the discretized image second-order Sinkhorn gradient flow further comprises: Use randomly sampled Gaussian noise or random sampling of the initial image dataset of the same resolution to match the sampling of the target image dataset to form a small batch of noise-image pairings in, The superscript i indicates the i-th pairing, and the subscript t indicates the time sequence number. represents the intermediate image, t=0 represents is the initial image, t=1 means is the target image, It means that there are n pairs in total that constitute a small batch of noise-image pairs; The geomloss.SamplesLoss function in the Sinkhorn divergence calculation library GeomLoss is used to calculate the Sinkhorn divergence of small batch noise-image pairs. The specific use of the function is as follows: Input current image, target image Parameter setting loss = sinkhorn; output Sinkhorn divergence Where S represents the Sinkhorn divergence. Since no matter the input initial image or intermediate image, the target image No change occurs, use the shorthand method The Autograd function in the PyTorch library is used to solve the gradient. The specific use of the function is as follows: Input Sinkhorn divergence, output Sinkhorn divergence gradient As the current momentum field, Represents the gradient symbol; Update the current velocity field using the current momentum field: in, is the updated velocity field, and the superscript i indicates that the intermediate image corresponding to the velocity field is Indicates the current velocity field, subscript t * ,t represents the intermediate time;∈ represents the velocity field update rate,γ represents the momentum retention parameter of the original velocity field, which is a manually set hyperparameter,^ represents that small batch updates are currently used; Update the current image using the current velocity field: in, This is the updated image. is the current intermediate image, and the superscript i represents the i intermediate image. is the velocity field after update, σ controls the rate of image update and is a manually set hyperparameter; Store the momentum field and velocity field corresponding to the current small batch noise: store the current triplet From the initial image or random noise Start by repeating the above steps to get the triple sequence Where m means repeating the above steps m times to construct the second-order Sinkhorn gradient flow of the discretized image; Repeatedly select the initial image or random noise Start to reconstruct the second-order Sinkhorn gradient flow of the discretized image, and loop multiple times to build the trajectory pool.

5. The method for accelerating image generation based on neural network second-order Sinkhorn gradient flow according to claim 1, characterized in that: The training neural network second-order Sinkhorn gradient flow model is specifically as follows: using the experience replay technology, using the constructed trajectory pool for training, and not introducing additional second-order Sinkhorn gradient flow data; using the momentum field matching technology, using the U-net neural network to approximate the momentum field to simulate the real-time update of the second-order Sinkhorn gradient flow to obtain the neural network second-order Sinkhorn gradient flow model.

6. The method for accelerating image generation based on neural network second-order Sinkhorn gradient flow according to claim 5, characterized in that: The training neural network second-order Sinkhorn gradient flow model further includes: The experience replay technique is used to train the constructed trajectory pool without introducing additional second-order Sinkhorn gradient flow data; The momentum field matching technology is used to randomly extract the momentum field matching target to minimize the prediction error of the momentum field through the neural network parameterization technology; the matching target is mathematically expressed as: Among them, θ represents the network parameter, t represents the intermediate time, T represents the upper limit of time, x~μ t It means that the image comes from the time-dependent intermediate distribution, E means to find the expectation, that is, to find the average after multiple calculations. It is training loss. Refers to a neural network that has been trained through neural network loss back propagation optimization; Train the model to perform Loss backpropagation gradient descent, use the Adam optimizer, and set the learning rate.

7. The method for accelerating image generation based on neural network second-order Sinkhorn gradient flow according to claim 1, characterized in that: The phase transition time predictor is used to predict the optimal time point from the current state of the model to the intermediate state of the general model: Map any input image x to the correct time t that other general flow matching-based generative models depend on, so that the image generated by the current neural network second-order Sinkhorn gradient flow obeys the intermediate image distribution P of the general flow matching-based generative model t , even if x~P t ; Using Neural Networks Prediction time t; During the optimization process, the training loss L(φ) includes the t Extract sample X t And minimize t φ (X t ) and t, ensuring that X t Accurate simulation P t ; Time predictor The training uses the following regression loss: Among them, θ represents the network parameters, T represents the upper bound of time, and x t ~P t Indicates the selection of the middle image; Train the model to perform Loss backpropagation gradient descent, use the Adam optimizer, and set the learning rate.

8. The method for accelerating image generation based on neural network second-order Sinkhorn gradient flow according to claim 1, characterized in that: The use of the second-order Sinkhorn gradient flow to accelerate image generation is specifically as follows: Randomly generate initial noise or randomly sample initial images from the initial image dataset and a randomly set initial velocity field An intermediate image close to the target image distribution is generated using a neural network second-order Sinkhorn gradient flow model: velocity field iteration guided by momentum field prediction and subsequent image update using less than 10 steps of function evaluation: in, is from The starting intermediate image, is from In fact, the intermediate velocity field, It is the momentum field predicted by the neural network second-order Sinkhorn gradient flow model, ∈ represents the velocity field update rate, γ represents the momentum preservation parameter of the original velocity field, and σ controls the image update rate, which is an artificially set hyperparameter.

9. An electronic device, comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the neural network second-order Sinkhorn gradient flow accelerated image generation method according to any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for accelerating image generation based on a neural network second-order Sinkhorn gradient flow as described in any one of claims 1 to 8 is implemented.

Images