A skip-coupling flow matching method for face image recovery

Abstract

The application discloses a kind of for face image recovery leap coupling flow matching method, it is related to face image recovery technical field.The application at least includes the following steps: S1: using lightweight recovery network to carry out preliminary recovery to LQ input, generate rough result as conditional information;S2: carry out coupling flow matching (CFM), the coupling flow matching path of construction, rate;S3: introduce shortcut constraint, so that model can predict the average speed between any two time points, rather than instantaneous speed on the traditional ODE trajectory, to form a "shortcut" path.The application proposes a kind of face recovery method based on coupling flow matching (CFM), utilizes the data dependency between LQ and HQ image, constructs more linear distribution transmission trajectory;By introducing shortcut constraint in CFM, so that model can predict the average speed between any two points, realizes current optimal performance under single-step reasoning condition.

Description

Technical Field

[0001] This invention relates to the field of face image restoration technology, specifically a step-coupled flow matching method for face image restoration. Background Technology

[0002] In recent years, generative methods, represented by diffusion models, have performed exceptionally well in this task, achieving remarkable results in both realism and detail fidelity. However, diffusion models typically require dozens of iterations, resulting in slow inference speed and high computational cost.

[0003] Flow matching, as an emerging generative model, is gradually attracting attention. Unlike diffusion models based on stochastic differential equations (SDEs), flow matching constructs the source distribution using ordinary differential equations (ODEs). To target distribution An approximate straight-line transmission path is derived. The velocity field determines the direction and rate of transmission, guiding the samples from the source distribution to the target distribution. By constructing the optimal transmission path, flow matching can effectively reduce the number of iterations required for inference.

[0004] FlowIE is the first model to use flow matching for image restoration; it constructs a model based on a Gaussian distribution. To high-quality image distribution The mapping achieved good recovery results. However, FlowIE assumes that the source sample With target sample The paths are independent of each other and connected only by simple linear interpolation. This assumption of independence leads to a large number of path intersections, causing the learned velocity field to be averaged across multiple conflicting directions. This results in the ODE trajectory deviating from the ideal straight line, necessitating an increase in the number of iterations to maintain recovery accuracy.

[0005] In summary, generative face reconstruction techniques have made significant progress in recent years, especially diffusion models and flow matching methods, which achieve image reconstruction by constructing a continuous-time mapping between two distributions. However, most existing methods use Gaussian noise as the starting distribution, ignoring the inherent data correlation between low-quality (LQ) and high-quality (HQ) images. This approach leads to an excessively curved transmission path, often requiring multiple iterative sampling steps, which affects inference efficiency. Therefore, a new solution is needed to address these issues. Summary of the Invention

[0006] The purpose of this invention is to provide a step-coupled flow matching method for face image restoration, so as to solve the technical problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a skip-step coupled flow matching method for face image restoration, comprising at least the following steps:

[0008] S1: Use lightweight network recovery The LQ input is initially recovered to generate a coarse result as conditional information, i.e., a lightweight recovery network is used. The generated preliminary restoration results serve as conditional information. ;

[0009] S2: Perform Coupled Flow Matching (CFM). Construct a Coupled Flow Matching path, set the source distribution to a low-quality image distribution instead of Gaussian noise, and the target distribution to a high-quality image distribution. Utilize the inherent correlation between LQ and HQ images to reduce path intersections and decrease trajectory curvature.

[0010] S3: Introducing shortcut constraints enables the model to predict the average velocity between any two time points, rather than the instantaneous velocity on the traditional ODE trajectory, thus forming a "shortcut" path. A hybrid training mechanism is adopted, combining standard flow matching loss and shortcut constraint loss for training.

[0011] Furthermore, S1 includes at least the following steps:

[0012] First, a lightweight pre-trained network is used. A preliminary recovery of the input yields a rough result;

[0013] The output of this rough result is then fed into a pre-trained variational autoencoder, which converts it into a latent representation, i.e., the conditional latent variables are... ;

[0014] Conditional latent variables It serves not only as the starting point of the transmission path but also as conditional information input into the velocity field prediction network.

[0015] Furthermore, the coupled flow matching path constructed in S2 includes at least the following steps:

[0016] conditional latent variables As a starting point, a small amount of Gaussian noise is added to form the actual starting point of the transmission path. ;

[0017] Simultaneously, the HQ image is encoded into a latent representation. As the end point of the path;

[0018] For each pair of data Construct a conditional probability path in the form of a straight line, where a point at any time t on the path is defined as:

[0019]

[0020] The speed corresponding to this path can be determined:

[0021]

[0022] However, if only given Then there are multiple possibilities. To produce the same , making It becomes a random variable;

[0023] A flow matching model is employed, which estimates the conditional expectation of the velocity by learning a neural network. That is to Averaging all possible velocities to approximate the true probability flow;

[0024] Specifically, the velocity field Regression was performed on randomly sampled LQ-HQ data. The speed of gaining experience To perform optimization, the loss function is:

[0025]

[0026] During the inference phase, from the LQ latent code with added noise... Depart, according to The defined ODE is iteratively updated until it is obtained. ;

[0027] Ultimately, Send to decoder This will generate a high-quality restored face image.

[0028] Furthermore, S3 includes at least the following steps:

[0029] velocity field Used to predict from the current state To the ideal next state The average speed, i.e. the shortcut direction, is updated using the following formula:

[0030]

[0031] when At this point, the shortcut direction degenerates into the instantaneous velocity field in the standard flow matching;

[0032] And when At the same time, the model learns to predict incremental movements with larger spans;

[0033] To achieve this goal, the self-consistency principle of shortcut constraints is utilized:

[0034] From state One of the departures A large step size should be equivalent to two consecutive steps. Small step size steps;

[0035] Therefore, the model predicts large step speeds. Should be in line with target speed Consistent with this, the target velocity is defined by the average velocity of two consecutive small steps:

[0036]

[0037] The corresponding training objective—shortcut constraint loss Defined as:

[0038]

[0039] in, This indicates expectations for different data.

[0040] Compared with the prior art, the beneficial effects of the present invention are:

[0041] 1. This invention proposes a face restoration method based on Coupled Stream Matching (CFM), which utilizes the data dependency between LQ and HQ images to construct a distributed transmission trajectory that is closer to a straight line;

[0042] 2. This invention introduces shortcut constraints into CFM, enabling the model to predict the average velocity between any two points, achieving the current optimal performance under single-step inference conditions. Attached Figure Description

[0043] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a comparative diagram of the present invention and the prior art;

[0045] Figure 2 This is a diagram of the overall architecture of the present invention;

[0046] Figure 3 This is the qualitative comparison result on the CelebA-Test dataset of this invention. Detailed Implementation

[0047] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0048] See Figure 1 , Figure 1 (a) For optimal transmission using linear interpolation, the independence assumption proposed in the background technique leads to a large number of path intersections; Figure 1 (b) The velocity field obtained by standard flow matching learning corresponds to the background technique proposed that causes the ODE trajectory to deviate from the ideal straight line.

[0049] To address the problems mentioned in the background art, this invention introduces data-dependent coupling to construct a straighter transmission path, proposing Coupled Stream Matching (CFM). Specifically, this invention sets the source distribution to a low-quality image distribution instead of Gaussian noise, and the target distribution to a high-quality image distribution, thereby utilizing the inherent correlation between LQ and HQ images to reduce path intersections and lower trajectory curvature. However, existing research has shown that CFM is still insufficient for single-step mapping, and the learned ODE trajectory is still not completely straight ( Figure 1 (c) is the velocity field learned by Coupled Flow Matching (CFM).

[0050] To achieve one-step inference, a shortcut constraint is introduced on top of CFM, leading to the proposal SCFlowFR. Its core idea is to predict any two time points. and The average velocity between them, rather than the instantaneous velocity on the traditional ODE trajectory, thus forming a "shortcut" path, such as... Figure 1 As shown in (d), this is the shortcut-enhanced coupled flow matching proposed in this invention.

[0051] The core objective of flow matching is to learn a parameterized velocity field. The velocity field distributes the source through ODE. Continuously transform into the target distribution In standard image generation flow matching [4][5], it is generally assumed that the source sample With target sample They are independent of each other. However, in image restoration tasks, there is a natural data correspondence between LQ and HQ images. Therefore,

[0052] This invention employs the CFM method, replacing the starting point with an LQ image instead of Gaussian noise, thereby establishing a more direct transmission path between the two. This approach helps reduce path crossings and effectively prevents excessive curvature of the ODE trajectory.

[0053] While theoretically a perfectly trained ODE can map the LQ distribution to the HQ distribution in continuous time, in practical applications, the finite number of discretization sampling steps introduces errors, causing the generated trajectory to deviate from the ideal path. To achieve efficient single-step mapping, this invention introduces a shortcut constraint to significantly reduce such discretization errors.

[0054] This shortcut strategy enhances the original model, enabling it to receive the current time step. Conditional latent variables and target step size As input, this allows the model to "predict" the curvature of the trajectory, thus enabling it to perform more accurate large-span jumps.

[0055] Specifically as follows:

[0056] See Figure 2 A skip-step coupled flow matching method for face image restoration includes at least the following steps:

[0057] S1: Use lightweight network recovery The LQ input is initially recovered to generate a coarse result as conditional information, i.e., a lightweight recovery network is used. The generated preliminary restoration results serve as conditional information. Used to guide the velocity field Learning;

[0058] S2: Perform Coupled Flow Matching (CFM). Construct a Coupled Flow Matching path, set the source distribution to a low-quality image distribution instead of Gaussian noise, and the target distribution to a high-quality image distribution. Utilize the inherent correlation between LQ and HQ images to reduce path intersections and decrease trajectory curvature.

[0059] S3: Introducing shortcut constraints enables the model to predict the average velocity between any two time points, rather than the instantaneous velocity on the traditional ODE trajectory, thus forming a "shortcut" path. A hybrid training mechanism is adopted, combining standard flow matching loss and shortcut constraint loss for training, ensuring that the model can learn an accurate LQ to HQ distribution mapping while maintaining the continuity and stability of inference at any step size.

[0060] S1 includes at least the following steps:

[0061] First, a lightweight pre-trained network is used. A preliminary recovery of the input yields a rough result;

[0062] The output of this rough result is then fed into a pre-trained variational autoencoder, which converts it into a latent representation, i.e., the conditional latent variables are... ;

[0063] Conditional latent variables Not only does it serve as the starting point of the transmission path, but it also acts as conditional information input to the velocity field prediction network. This design process takes into account the complexity of real-world face degradation and makes better use of the semantic information of the LQ input.

[0064] The coupled flow matching path constructed in S2 includes at least the following steps:

[0065] conditional latent variables As a starting point, a small amount of Gaussian noise is added to form the actual starting point of the transmission path. This ensures that the probability density is smooth and well-defined in the high-dimensional latent space;

[0066] Simultaneously, the HQ image is encoded into a latent representation. As the end point of the path;

[0067] To achieve efficient distributed transformation, the ideal transmission path should be as close to a straight line as possible for each pair of data. Construct a conditional probability path in the form of a straight line, where a point on the path at any time t is defined as:

[0068] (1)

[0069] The speed corresponding to this path can be determined: ;

[0070] However, if only given Then there are multiple possibilities. To produce the same , making To become a random variable;

[0071] A flow matching model is employed, which estimates the conditional expectation of the velocity by learning a neural network. That is to Averaging all possible velocities to approximate the true probability flow;

[0072] Specifically, the velocity field Regression was performed on randomly sampled LQ-HQ data. The speed of gaining experience To perform optimization, the loss function is:

[0073] (2)

[0074] During the inference phase, from the LQ latent code with added noise... Depart, according to The defined ODE is iteratively updated until it is obtained. ;

[0075] Ultimately, Send to decoder This will generate a high-quality restored face image.

[0076] S3 includes at least the following steps:

[0077] velocity field Used to predict from the current state To the ideal next state The average speed, i.e. the shortcut direction, is updated using the following formula:

[0078] (3)

[0079] when At this point, the shortcut direction degenerates into the instantaneous velocity field in the standard flow matching;

[0080] And when At the same time, the model learns to predict incremental movements with larger spans;

[0081] To achieve this goal, the self-consistency principle of shortcut constraints is utilized:

[0082] From state One of the departures A large step size should be equivalent to two consecutive steps. Small step size steps;

[0083] Therefore, the model predicts large step speeds. Should be in line with target speed Consistent with this, the target velocity is defined by the average velocity of two consecutive small steps:

[0084] (4)

[0085] The corresponding training objective—shortcut constraint loss Defined as:

[0086] (5)

[0087] in, This indicates expectations for different data.

[0088] In summary, the training method of this invention combines two objectives: when When, use the standard flow matching loss (Equation (2)); when In this case, the self-consistency loss described above is used. During training, each batch of data is divided into two subsets, which are used to optimize the two objectives respectively. This hybrid training mechanism ensures that the model can learn an accurate mapping from the LQ to the HQ distribution while maintaining the continuity and stability of inference at any step size (including a single step).

[0089] Based on the above, the following specific experiment is proposed:

[0090] Dataset

[0091] The model SCFlowFR of this invention was trained on the Flickr-Faces-HQ (FFHQ) dataset, which contains 70,000 high-resolution (HR) face images. All images were uniformly scaled to 512×512 resolution before training. During the evaluation phase, the model's performance was first benchmarked on the synthetic dataset CelebA-Test, which contains 3,000 pairs of HQ and LQ images. To further verify the model's generalization ability in real-world complex scenes, additional experiments were conducted on three real-world datasets: LFW-Tes, CelebChild-Test, and WebPhoto-Test. These datasets all contain face images affected by various real-world degradation factors.

[0092] Training details

[0093] In training and evaluation, this invention employs Codeformer's synthetic degradation process to generate low-quality input. The initial recovery module is implemented by SwinIR, while the core velocity field... The U-Net architecture is then adopted. The parameters of the SwinIR and VAE encoders are frozen during training.

[0094] This invention uses a batch size of 16 for 200k iterations of training, with Adam as the optimizer and a learning rate set to... To improve training stability, a self-consistent objective is calculated. At the same time, this invention introduces exponential moving average (EMA) model parameters. Furthermore, this invention also trains a lightweight variant, SCFlowFR-Tiny, which reduces the number of parameters by using a more compact VAE and U-Net architecture, and was trained for 150k iterations under the same settings.

[0095] Evaluation indicators

[0096] On the synthetic dataset CelebA-Test, this invention employs full-reference image quality evaluation metrics, including FID, PSNR, and LPIPS, as well as no-reference metrics MUSIQ and BRISQUE. For real-world datasets, due to the lack of real-world reference images, this invention uses only no-reference metrics NIQE and BRISQUE for evaluation. Model efficiency is measured by the total number of parameters and inference throughput (FPS).

[0097] Comparison with state-of-the-art methods

[0098] This invention provides a comprehensive comparison of the proposed method with the following representative baselines: DMDNet, RestoreFormer, StableSR, DiffBIR, OSEDiff, and FlowIE. Among them, DMDNet and RestoreFormer are based on Generative Adversarial Network (GAN) models, FlowIE is based on a flow matching model, and the remaining methods are all based on diffusion models.

[0099] 1. Results of the CelebA-Test dataset

[0100] Quantitative results are shown in Table 1. The SCFlowFR of this invention achieved the best performance with an FID score of 15.62, outperforming all comparative methods. Its lightweight variant, SCFlowFR-Tiny, ranked second, demonstrating its significant advantage in perceived quality. Furthermore, the method of this invention also demonstrated competitiveness in metrics such as PSNR, LPIPS, and BRISQUE.

[0101] Table 1 Quantitative results on the CelebA-Test dataset

[0102]

[0103] It is worth noting that the complete model of this invention requires far fewer parameters than models based on diffusion or flow matching; while the lightweight version has a parameter count comparable to GAN-based methods, but with superior performance. In contrast, some non-diffusion methods, although having advantages in model size and inference speed, show a significant decline in key metrics such as FID and PSNR.

[0104] The qualitative experimental results are shown in Figure 3. It can be observed that SCFlowFR can effectively preserve important structural information in the image, avoid over-generation or distortion caused by noise interference, and successfully restore fine-grained details in high-resolution images, ultimately achieving a restoration effect that is both realistic and high-fidelity.

[0105] Table 2 Comparison of no-reference metrics on three real-world scenario datasets

[0106]

[0107] ablation experiment

[0108] Training strategy analysis

[0109] To verify the necessity of the shortcut training strategy and explore the impact of different training strategies on performance, this invention designed an ablation experiment. First, this invention compared the performance of using only flow matching loss. The trained model (denoted as Model-w / o-) As shown in Table 3, the model's performance significantly degraded across all metrics. Secondly, this invention evaluated a training strategy based on Consistency-FM (denoted as Model-w / -). Since the model cannot achieve satisfactory results under single-step reasoning, this invention reports its five-step reasoning performance, which is still inferior to the method proposed in this invention.

[0110] LQ Image Conditioning Analysis

[0111] SCFlowFR not only uses the LQ image as the starting point for the ODE sampling trajectory but also inputs it as conditional information into the U-Net architecture to further improve the accuracy of path estimation. To evaluate the contribution of this design, this invention trained a simplified model without LQ conditional input (denoted as Model-w / o-conditioning). As shown in Table 3, in terms of FID, PSNR, and BRISQUE metrics, removing LQ conditionalization leads to a decrease in image restoration quality, confirming the necessity of introducing the LQ image as a condition.

[0112] Table 3 Ablation experimental results on the CelebA-Test dataset

[0113] In summary:

[0114] This invention proposes the SCFlowFR framework for single-step face reconstruction, the overall structure of which is as follows: Figure 2 As shown.

[0115] We constructed a coupled transport path (left side) between the LQ and HQ image distributions and utilized a lightweight recovery network. The generated preliminary restoration results serve as conditional information. Used to guide the velocity field The framework also introduces shortcut constraints, enabling the model to predict within time intervals. The average speed within.

[0116] Unlike traditional diffusion models or flow matching methods, SCFlowFR sets the LQ image distribution as the starting point, fully utilizing the coupling relationship between LQ and HQ images to reduce path crossings and improve trajectory linearity. Furthermore, by introducing shortcut constraints, the model can effectively predict the transport dynamics between arbitrary time points. Extensive experiments demonstrate that this method significantly improves inference efficiency while maintaining competitive recovery performance, achieving excellent results even with only single-step inference. Future work will extend this framework to general image restoration tasks and explore applications such as image inpainting to further enhance its robustness and generalization ability.

[0117] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A skip-step coupled flow matching method for face image restoration, characterized in that: At least the following steps are included: S1: Use lightweight network recovery The LQ input is initially recovered to generate a coarse result as conditional information, i.e., a lightweight recovery network is used. The generated preliminary restoration results serve as conditional information. ; S2: Perform coupled flow matching, construct coupled flow matching path, set the source distribution to low quality image distribution instead of Gaussian noise, set the target distribution to high quality image distribution, and utilize the inherent correlation between LQ and HQ images to reduce path intersection and reduce trajectory curvature; S3: Introducing shortcut constraints enables the flow matching model to predict the average velocity between any two time points, rather than the instantaneous velocity on the traditional ODE trajectory, thus forming a "shortcut" path. It also adopts a hybrid training mechanism, combining standard flow matching loss and shortcut constraint loss for training.

2. The skip-step coupled flow matching method for face image restoration according to claim 1, characterized in that: S1 includes at least the following steps: First, a lightweight pre-trained network is used. A preliminary recovery of the input yields a rough result; The output of this rough result is then fed into a pre-trained variational autoencoder to be transformed into conditional latent variables, i.e. ; Conditional latent variables It serves not only as the starting point of the transmission path but also as conditional information input into the velocity field prediction network.

3. The skip-step coupled flow matching method for face image restoration according to claim 2, characterized in that: The coupled flow matching path constructed in S2 includes at least the following steps: conditional latent variables As a starting point, a small amount of Gaussian noise is added to form the actual starting point of the transmission path. ; Simultaneously, the HQ image is encoded into a latent representation. As the end point of the path; For each pair of data Construct a conditional probability path in the form of a straight line, where a point on the path at any time t is defined as: The speed corresponding to this path can be determined: However, if only given There are multiple sets Data pairs can produce the same , making It becomes a random variable; A flow matching model is employed, which estimates the conditional expectation of the velocity by learning a neural network. That is to The velocity is averaged across all cases to approximate the true probability flow; Specifically, the velocity field Regression was performed on randomly sampled LQ-HQ data. The speed of gaining experience To perform optimization, the loss function is: During the inference phase, from the LQ latent code with added noise... Depart, according to The defined ODE is iteratively updated until it is obtained. ; Ultimately, Send to decoder This generates a high-quality restored face image.

4. The skip-step coupled flow matching method for face image restoration according to claim 3, characterized in that: The S3 includes at least the following steps: velocity field Used to predict from the current state To the ideal next state The average speed, i.e. the shortcut direction, is updated using the following formula: when At this point, the shortcut direction degenerates into the instantaneous velocity field in the standard flow matching; And when At the same time, the model learns to predict incremental movements with larger spans; To achieve this goal, the self-consistency principle of shortcut constraints is utilized: From state One of the departures A large step size should be equivalent to two consecutive steps. Small step size steps; Therefore, the model predicts large step speeds. Should be in line with target speed Consistent with this, the target velocity is defined by the average velocity of two consecutive small steps: The corresponding training objective—shortcut constraint loss Defined as: in, This indicates expectations for different data.

Images