Image editing model training method based on online reinforcement learning and GRPO
By using online reinforcement learning and GRPO technology, the problems of large data requirements, high computational resources, limited generalization ability, and difficulty in quality control in image editing model training were solved, achieving efficient and accurate image editing, reducing data annotation costs, and improving model performance.
Patent Information
- Application Number
- CN202511038215.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-28
- Publication Date
- 2025-10-31
AI Technical Summary
Existing image editing model training methods suffer from problems such as large data requirements, high computational resource consumption, limited generalization ability, weak instruction understanding ability, low editing success rate, difficulty in quality control, and lack of online optimization mechanisms.
We employ an image editing model training method based on online reinforcement learning and GRPO. Through ODE-to-SDE conversion technology, reward model scheme and policy gradient optimization of valueless network, combined with multimodal large language model and CLIP-based evaluation reward, we use LoRA module for parameter fine-tuning and adopt denoising and reduction strategies.
It significantly improves instruction compliance accuracy and training efficiency, reduces data annotation costs, expands the scale of training data, reduces the number of parameters, and enhances the model's generalization ability and editing quality.
Smart Images
Figure CN120876294A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a method for training an image editing model based on online reinforcement learning and GRPO. Background Technology
[0002] The following drawbacks exist in the training of image editing models in existing technologies:
[0003] (1) Limitations of training methods
[0004] Existing image editing models mainly employ supervised learning fine-tuning methods, such as InstructPix2Pix and MagicBrush. These methods have the following shortcomings: 1) Large data requirements: They require 450K to 10M high-quality edits for training; 2) High computational resource consumption: They typically require complete model parameter fine-tuning, with the number of parameters reaching billions; 3) Limited generalization ability: They perform poorly when faced with editing types not seen in the training data.
[0005] (2) The shortcomings of no training method
[0006] While training-free methods such as RF-inversion and RF-Solver Edit avoid large-scale data training, they have the following shortcomings: 1) Weak instruction comprehension: relying on manually designed prompt templates, the understanding of complex instructions is not accurate enough; 2) Low editing success rate: prone to failure in complex scenarios, lacking robustness; 3) Difficult quality control: unable to guarantee the consistency and high quality of editing results.
[0007] (3) Limitations of existing optimization strategies
[0008] The current field of image editing lacks effective online optimization mechanisms: 1) Static training mode: Once training is completed, the model cannot improve itself based on actual usage results; 2) Lack of quality feedback: There is no effective editing quality evaluation and feedback mechanism; 3) Single optimization objective: Only focusing on reconstruction loss, ignoring important indicators such as the degree of adherence to editing instructions.
[0009] Therefore, it is necessary to provide a training method for image editing models based on online reinforcement learning and GRPO, which can significantly improve the accuracy of instruction following and training efficiency while ensuring the quality of image editing. Summary of the Invention
[0010] The purpose of this invention is to provide an image editing model training method based on online reinforcement learning and GRPO, which significantly improves instruction compliance accuracy and training efficiency while ensuring image editing quality.
[0011] To address the problems existing in the prior art, this invention provides a method for training an image editing model based on online reinforcement learning and GRPO, comprising the following steps:
[0012] S1: The ODE-to-SDE conversion technique is used to transform the original deterministic ODE flow into an equivalent SDE flow, enabling the flow matching model to support the random sampling exploration required by reinforcement learning while maintaining the original edge distribution characteristics.
[0013] S2: Set up the reward model scheme.
[0014] S3: Based on the reward model scheme, the strategy gradient optimization of the valueless network is adopted to continuously optimize the parameters of the image editing model and obtain the overall objective function.
[0015] Optionally, in the image editing model training method based on online reinforcement learning and GRPO, the ODE-to-SDE conversion technique is as follows:
[0016] S11: The equivalent SDE of the original ODE stream is:
[0017]
[0018] Where t is the time step, x t Let v be the state vector at time step t. t Let σ be the velocity field function at time step t. t Let d be the noise scheduling parameter for time step t. t For the time derivative, d w For the differential of the Wiener process;
[0019] S12: Using the Euler-Maruyama discretization method, the actual update formula is obtained:
[0020]
[0021] Among them, X t+Δt For the state at the next time step, v θ (X t ,t) represents the velocity field predicted by the neural network; Δ t Let θ be the time step, θ be the neural network parameters, ò ~ N(0,I), and ò be standard Gaussian noise;
[0022] Adjusting noisy scheduling parameters using a time-dependent scheduling function:
[0023]
[0024] Where: a is a scalar hyperparameter controlling the noise level, t is the time step, and the value of t ranges from [0,1].
[0025] Optionally, in the image editing model training method based on online reinforcement learning and GRPO, policy gradient optimization using a valueless network is employed to continuously optimize the parameters of the image editing model, as follows:
[0026] S31: Calculation of the dominance function:
[0027] For multiple generated sample groups {x_0^j\}_{j=1}^G of the same c prompt word, the advantage function of the i-th sample is defined as:
[0028]
[0029] in: This represents the advantage function value of the i-th sample at time step t. Let represent the reward score for the i-th generated image, c represent the editing instruction conditions, and G represent the total number of generated samples in the same group. This represents the average reward within the group. This represents the standard deviation of the rewards within the group;
[0030] S32: Calculate the importance sampling ratio. The importance sampling ratio of the i-th sample at time step t is defined as:
[0031]
[0032] in: This represents the importance sampling ratio of the i-th sample. This represents the conditional probability distribution of the current image editing model. This represents the conditional probability distribution of the image editing model trained in the previous round. This represents the image editing model parameters from the previous iteration;
[0033] S33: Calculate the policy loss function for policy gradient optimization, using a clipped surrogate objective. The policy loss is calculated at each time step t, as shown in the following formula:
[0034]
[0035] in, This represents the policy loss at time step t. Let E represent the cutoff function, which is restricted to the range of 1-ε to 1+ε, where ε represents the cutoff parameter and E represents the expected value.
[0036] S34: Calculate the KL divergence loss. In the flow matching framework, the KL divergence between the current policy and the reference policy has a closed-form solution, as follows:
[0037]
[0038] Among them, D KL (π θ ||π ref ) represents the KL divergence between the current image editing model and the reference image editing model, v θ (x t ,t) represents the velocity field predicted by the current image editing model, v ref (x t ,t) represents the velocity field predicted by the reference image editing model; ||v θ (x t ,t)-v ref (x t ,t)|| 2 Represents the square of the Euclidean norm;
[0039] S35: Combining the complete training objective function with the policy loss and KL divergence constraint, we obtain the overall objective function as follows:
[0040] L total (θ)=L policy (θ)+β*E[D KL (π θ ||π ref )], where L total (θ) represents the total loss function, L policy (θ) represents the policy loss of the image editing model obtained by averaging the loss over all time steps, and β represents the KL penalty coefficient, which is used to balance the policy improvement and distribution preservation of the image editing model.
[0041] Optionally, in the image editing model training method based on online reinforcement learning and GRPO, two reward model schemes are adopted: one uses a multimodal large language model to evaluate rewards and the other uses CLIP-based instruction compliance to evaluate rewards.
[0042] Optionally, in the image editing model training method based on online reinforcement learning and GRPO, the reward is evaluated using a multimodal large language model as follows:
[0043] The GPT-4V multimodal large language model and the Qwen-VL multimodal large language model were used.
[0044] A scalar score of 0-100 is obtained using structured prompts;
[0045] The evaluation is based on three dimensions: instruction compliance, editing quality, and regional fidelity.
[0046] Optionally, in the image editing model training method based on online reinforcement learning and GRPO, the CLIP-based instruction-following evaluation reward is adopted as follows:
[0047] Calculate the CLIP embedding similarity between the edited image and the editing instructions;
[0048] Combine the visual similarity between the non-edited area and the original image;
[0049] A scalar reward function that forms a weighted combination.
[0050] Optionally, the image editing model training method based on online reinforcement learning and GRPO also includes the following step: S4: fine-tuning parameters by combining Low-Rank Adaptation.
[0051] Optionally, in the image editing model training method based on online reinforcement learning and GRPO, the parameter fine-tuning method is as follows:
[0052] S41: Add an editing-specific LoRA module to the basic flow-matching image editing model.
[0053] S42: Only update the low-dimensional adapter parameters, significantly reducing the number of training parameters.
[0054] S43: Supports fast convergence and lightweight deployment.
[0055] Optionally, the image editing model training method based on online reinforcement learning and GRPO also includes the following step: S5: adopting a noise reduction strategy.
[0056] Optionally, in the image editing model training method based on online reinforcement learning and GRPO, the noise reduction strategy is as follows:
[0057] S51: During training: Use denoising steps for fast sampling;
[0058] S52: During inference: Maintain the original number of denoising steps to ensure quality.
[0059] Compared with the prior art, the present invention has the following advantages:
[0060] (1) This invention aims to solve the problems of insufficient accuracy in instruction understanding, unstable editing quality, low training efficiency, and lack of online optimization mechanism in existing image editing models. The core task of this invention is to provide an image editing model training method based on online reinforcement learning and GRPO, which can significantly improve the accuracy of instruction following and training efficiency while ensuring the quality of image editing.
[0061] (2) Data annotation costs are significantly reduced. Traditional image editing methods require a large amount of manually annotated triple data (original image, editing instructions, standard editing results), typically requiring 450,000 to 12 million complete annotated samples. This invention uses the GRPO reinforcement learning framework, requiring only binary data (original image, editing instructions), completely eliminating the dependence on manually annotated editing results. Data acquisition costs are reduced from requiring professional annotators to annotate each editing result individually to only needing to collect the original image and editing instruction text, resulting in a significant reduction in annotation costs.
[0062] (3) The scale effect brought about by the significant expansion of training data size: Since there is no need to label standard editing results, the threshold for collecting training data is greatly reduced. Image-instruction pairs that were previously limited by labeling costs and could not be collected on a large scale can now be easily expanded to the millions. The closed-loop feedback learning mechanism of model autonomously generating candidate results and rewarding model evaluation makes large-scale weakly supervised training possible, thereby significantly improving the model's generalization ability and editing quality.
[0063] (4) Efficient parameter fine-tuning achieves high performance with few parameters. Combined with LoRA technology, only 1% of the model parameters need to be trained to achieve state-of-the-art performance. Compared with full parameter fine-tuning, training time is reduced by 70% and memory usage is reduced by 60%, while avoiding the additional overhead of the value network model in traditional PPO.
[0064] (5) Technological innovation breaks through the limitations of traditional frameworks and successfully applies GRPO technology to the field of image editing for the first time, providing a general theoretical basis and practical solution for reinforcement learning optimization of generative models. Attached Figure Description
[0065] Figure 1 This is a flowchart of a method provided in an embodiment of the present invention. Detailed Implementation
[0066] The specific embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. The advantages and features of the present invention will become clearer from the following description. It should be noted that the accompanying drawings are all in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the illustration of the embodiments of the present invention.
[0067] In the description of this application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.
[0068] In the following, if the methods described herein include a series of steps, the order of these steps presented herein is not necessarily the only order in which these steps can be performed, and some of the steps described may be omitted and / or some other steps not described herein may be added to the method.
[0069] The following drawbacks exist in the training of image editing models in the prior art: (1) limitations of training methods; (2) lack of training methods; and (3) limitations of existing optimization strategies.
[0070] To address the problems existing in the prior art, this invention provides a method for training an image editing model based on online reinforcement learning and GRPO, such as... Figure 1 As shown, the successful application of GRPO technology in the field of image editing mainly includes the following core technologies. GRPO, short for Group Relative Policy Optimization, is a novel reinforcement learning algorithm proposed by DeepSeek, aiming to improve the performance of large language models in inference tasks.
[0071] The method includes the following steps: S1:
[0072] To address the contradiction between the deterministic nature of flow matching models and the stochastic requirements of reinforcement learning, the ODE-to-SDE conversion technique is employed.
[0073] The ODE-to-SDE conversion technique is used to transform the original deterministic ODE flow into an equivalent SDE flow, enabling the flow matching model to support the random sampling exploration required by reinforcement learning while maintaining the original edge distribution characteristics.
[0074] Specifically, the ODE-to-SDE conversion technology is as follows:
[0075] S11: The equivalent SDE of the original ODE stream is:
[0076]
[0077] Where t is the time step, xt Let v be the state vector at time step t. t Let σ be the velocity field function at time step t. t Let d be the noise scheduling parameter for time step t. t For the time derivative, d w For the differential of the Wiener process;
[0078] S12: Using the Euler-Maruyama discretization method, the actual update formula is obtained:
[0079]
[0080] Among them, X t+Δt For the state at the next time step, v θ (X t ,t) represents the velocity field predicted by the neural network; Δ t Let θ be the time step, θ be the neural network parameters, ò ~ N(0,I), and ò be standard Gaussian noise;
[0081] Adjusting noisy scheduling parameters using a time-dependent scheduling function:
[0082]
[0083] Where: a is a scalar hyperparameter controlling the noise level, t is the time step, and the value of t ranges from [0,1].
[0084] This design ensures the sampling strategy of the image editing model is π. θ (x t-1 |x t c) It follows an isotropic Gaussian distribution, which facilitates the closed-form calculation of KL divergence.
[0085] Significance of ODE-to-SDE conversion: Formula X t+Δt The study shows that by introducing appropriate randomness, the originally deterministic ODE flow becomes an equivalent SDE flow, enabling the flow matching model to support the random sampling exploration required by reinforcement learning while maintaining the original marginal distribution characteristics.
[0086] Explanation of the "policy" concept: In this invention, the policy $\pi_\theta$ refers to the image editing model based on flow matching. Specifically: $\pi_\theta(x_{t-1}|x_t,c)$ represents the probability distribution of the image editing model outputting the next state $x_{t-1}$ given the current state $x_t$ and the editing instruction $c$; through GRPO training, we continuously optimize the parameters $\theta$ of the image editing model so that the generated editing results can obtain higher reward scores; "current policy" refers to the image editing model in the current training round, and "reference policy" refers to the image editing model used as the benchmark (usually the model trained in the initial state).
[0087] S2: Set up the reward model scheme.
[0088] Specifically, two reward model schemes are set up: one uses a multimodal large language model to evaluate rewards and the other uses CLIP-based instruction compliance to evaluate rewards.
[0089] The reward evaluation method using a multimodal large language model is as follows:
[0090] The GPT-4V multimodal large language model and the Qwen-VL multimodal large language model were used.
[0091] A scalar score of 0-100 is obtained using structured prompts;
[0092] The evaluation is based on three dimensions: instruction compliance, editing quality, and regional fidelity.
[0093] The CLIP-based instruction compliance evaluation reward method is as follows:
[0094] Calculate the CLIP embedding similarity between the edited image and the editing instructions;
[0095] Combine the visual similarity between the non-edited area and the original image;
[0096] A scalar reward function that forms a weighted combination.
[0097] S3: Based on the reward model scheme, a valueless network-based policy gradient optimization (GRPO) is used to continuously optimize the parameters of the image editing model to obtain the overall objective function. GRPO avoids the memory overhead of the value network in traditional PPO. GRPO achieves efficient policy optimization through within-group relative advantage estimation.
[0098] Specifically, the image editing model's parameters are continuously optimized using a strategy gradient optimization with a valueless network. The steps are as follows:
[0099] S31: Calculation of the dominance function:
[0100] For multiple generated sample groups {x_0^j\}_{j=1}^G of the same c prompt word, the advantage function of the i-th sample is defined as:
[0101]
[0102] in: This represents the advantage function value of the i-th sample at time step t. Let represent the reward score for the i-th generated image, c represent the editing instruction conditions, and G represent the total number of generated samples in the same group. This represents the average reward within the group. This represents the standard deviation of the rewards within the group;
[0103] S32: Calculate the importance sampling ratio. The importance sampling ratio of the i-th sample at time step t is defined as:
[0104]
[0105] in: This represents the importance sampling ratio of the i-th sample. This represents the conditional probability distribution of the current image editing model. This represents the conditional probability distribution of the image editing model trained in the previous round. This represents the image editing model parameters from the previous iteration;
[0106] S33: Calculate the policy loss function for Policy Gradient Optimization (GRPO) using the clipped surrogate objective. The policy loss is calculated at each time step t, as shown in the following formula:
[0107]
[0108] in, This represents the policy loss at time step t. Let E represent the cutoff function, which is restricted to the range of 1-ε to 1+ε, where ε represents the cutoff parameter and E represents the expected value.
[0109] S34: Calculate the KL divergence loss. In the flow matching framework, the KL divergence between the current policy and the reference policy has a closed-form solution, as follows:
[0110]
[0111] Among them, D KL (π θ ||π ref ) represents the KL divergence between the current image editing model and the reference image editing model, v θ (xt ,t) represents the velocity field predicted by the current image editing model, v ref (x t ,t) represents the velocity field predicted by the reference image editing model; ||v θ (x t ,t)-v ref (x t ,t)|| 2 Represents the square of the Euclidean norm;
[0112] S35: Combining the complete training objective function with the policy loss and KL divergence constraint, we obtain the overall objective function as follows:
[0113] L total (θ)=L policy (θ)+β*E[D KL (π θ ||π ref )], where L total (θ) represents the total loss function, L policy (θ) represents the policy loss of the image editing model obtained by averaging the loss over all time steps, and β represents the KL penalty coefficient, which is used to balance the policy improvement and distribution preservation of the image editing model.
[0114] In summary, the GRPO optimization process is as follows:
[0115] Group sampling: For each editing command, generate multiple candidate results and record the corresponding log probabilities;
[0116] Reward evaluation: Use a reward model to score the quality of all generated samples;
[0117] Dominance calculation: Group by prompt words and calculate the standardized dominance function value within each group;
[0118] Policy update: Update policy parameters using clipped surrogate objective, while applying KL constraints to prevent excessive deviation.
[0119] S4: Fine-tune the parameters using Low-Rank Adaptation. The parameter fine-tuning method is as follows:
[0120] S41: Add an editing-specific LoRA module to the basic flow-matching image editing model.
[0121] S42: Only update the low-dimensional adapter parameters, significantly reducing the number of training parameters.
[0122] S43: Supports fast convergence and lightweight deployment.
[0123] S5: A noise reduction strategy is adopted, and the noise reduction strategy is as follows:
[0124] S51: During training: Use denoising steps for fast sampling;
[0125] S52: During inference: Maintain the original number of denoising steps to ensure quality.
[0126] Results: Significantly accelerates the training process while maintaining inference quality.
[0127] In summary, compared with the prior art, the present invention has the following advantages:
[0128] (1) This invention aims to solve the problems of insufficient accuracy in instruction understanding, unstable editing quality, low training efficiency, and lack of online optimization mechanism in existing image editing models. The core task of this invention is to provide an image editing model training method based on online reinforcement learning and GRPO, which can significantly improve the accuracy of instruction following and training efficiency while ensuring the quality of image editing.
[0129] (2) Data annotation costs are significantly reduced. Traditional image editing methods require a large amount of manually annotated triple data (original image, editing instructions, standard editing results), typically requiring 450,000 to 12 million complete annotated samples. This invention uses the GRPO reinforcement learning framework, requiring only binary data (original image, editing instructions), completely eliminating the dependence on manually annotated editing results. Data acquisition costs are reduced from requiring professional annotators to annotate each editing result individually to only needing to collect the original image and editing instruction text, resulting in a significant reduction in annotation costs.
[0130] (3) The scale effect brought about by the significant expansion of training data size: Since there is no need to label standard editing results, the threshold for collecting training data is greatly reduced. Image-instruction pairs that were previously limited by labeling costs and could not be collected on a large scale can now be easily expanded to the millions. The closed-loop feedback learning mechanism of model autonomously generating candidate results and rewarding model evaluation makes large-scale weakly supervised training possible, thereby significantly improving the model's generalization ability and editing quality.
[0131] (4) Efficient parameter fine-tuning achieves high performance with few parameters. Combined with LoRA technology, only 1% of the model parameters need to be trained to achieve state-of-the-art performance. Compared with full parameter fine-tuning, training time is reduced by 70% and memory usage is reduced by 60%, while avoiding the additional overhead of the value network model in traditional PPO.
[0132] (5) Technological innovation breaks through the limitations of traditional frameworks and successfully applies GRPO technology to the field of image editing for the first time, providing a general theoretical basis and practical solution for reinforcement learning optimization of generative models.
[0133] The above are merely preferred embodiments of the present invention and do not constitute any limitation on the present invention. Any equivalent substitutions or modifications made by those skilled in the art to the technical solutions and content disclosed in the present invention without departing from the scope of the present invention shall be deemed to have remained within the protection scope of the present invention.
Claims
1. A method for training an image editing model based on online reinforcement learning and GRPO, characterized in that, Includes the following steps: S1: The ODE-to-SDE conversion technique is used to transform the original deterministic ODE flow into an equivalent SDE flow, enabling the flow matching model to support the random sampling exploration required by reinforcement learning while maintaining the original edge distribution characteristics. S2: Set up the reward model scheme. S3: Based on the reward model scheme, the strategy gradient optimization of the valueless network is adopted to continuously optimize the parameters of the image editing model and obtain the overall objective function.
2. The image editing model training method based on online reinforcement learning and GRPO as described in claim 1, characterized in that, The ODE-to-SDE conversion technology is as follows: S11: The equivalent SDE of the original ODE stream is: Where t is the time step, x t Let v be the state vector at time step t. t Let σ be the velocity field function at time step t. t Let d be the noise scheduling parameter for time step t. t For the time derivative, d w For the differential of the Wiener process; S12: Using the Euler-Maruyama discretization method, the actual update formula is obtained: Among them, X t+Δt For the state at the next time step, v θ (X t ,t) represents the velocity field predicted by the neural network; Δ t Let θ be the time step, θ be the neural network parameters, ò ~ N(0,I), and ò be standard Gaussian noise; Adjusting noisy scheduling parameters using a time-dependent scheduling function: Where: a is a scalar hyperparameter controlling the noise level, t is the time step, and the value of t ranges from [0,1].
3. The image editing model training method based on online reinforcement learning and GRPO as described in claim 2, characterized in that, The image editing model parameters are continuously optimized using a strategy gradient optimization with a valueless network. The steps are as follows: S31: Calculation of the dominance function: For multiple generated sample groups {x_0^j\}_{j=1}^G of the same c prompt word, the advantage function of the i-th sample is defined as: in: This represents the advantage function value of the i-th sample at time step t. Let represent the reward score for the i-th generated image, c represent the editing instruction conditions, and G represent the total number of generated samples in the same group. This represents the average reward within the group. This represents the standard deviation of the group's rewards; S32: Calculate the importance sampling ratio. The importance sampling ratio of the i-th sample at time step t is defined as: in: This represents the importance sampling ratio of the i-th sample. This represents the conditional probability distribution of the current image editing model. This represents the conditional probability distribution of the image editing model trained in the previous round. This represents the image editing model parameters from the previous iteration; S33: Calculate the policy loss function for policy gradient optimization, using a clipped surrogate objective. The policy loss is calculated at each time step t, as shown in the following formula: in, This represents the policy loss at time step t. Let E represent the cutoff function, which is restricted to the range of 1-ε to 1+ε, where ε represents the cutoff parameter and E represents the expected value. S34: Calculate the KL divergence loss. In the flow matching framework, the KL divergence between the current policy and the reference policy has a closed-form solution, as follows: Among them, D KL (π θ ||π ref ) represents the KL divergence between the current image editing model and the reference image editing model, v θ (x t ,t) represents the velocity field predicted by the current image editing model, v ref (x t ,t) represents the velocity field predicted by the reference image editing model; ||v θ (x t ,t)-v ref (x t ,t)|| 2 Represents the square of the Euclidean norm; S35: Combining the complete training objective function with the policy loss and KL divergence constraint, we obtain the overall objective function as follows: L total (θ)=L policy (θ)+β*E[D KL (π θ ||π ref )], where L total (θ) represents the total loss function, L policy (θ) represents the policy loss of the image editing model obtained by averaging the loss over all time steps, and β represents the KL penalty coefficient, which is used to balance the policy improvement and distribution preservation of the image editing model.
4. The image editing model training method based on online reinforcement learning and GRPO as described in claim 1, characterized in that, Two reward model schemes are adopted: one uses a multimodal large language model to evaluate rewards and the other uses CLIP-based instruction compliance to evaluate rewards.
5. The image editing model training method based on online reinforcement learning and GRPO as described in claim 4, characterized in that, The reward evaluation method using a multimodal large language model is as follows: The GPT-4V multimodal large language model and the Qwen-VL multimodal large language model were used. A scalar score of 0-100 is obtained using structured prompts; The evaluation is based on three dimensions: instruction compliance, editing quality, and regional fidelity.
6. The image editing model training method based on online reinforcement learning and GRPO as described in claim 4, characterized in that, The CLIP-based instruction compliance evaluation reward method is as follows: Calculate the CLIP embedding similarity between the edited image and the editing instructions; Combine the visual similarity between the non-edited area and the original image; A scalar reward function that forms a weighted combination.
7. The image editing model training method based on online reinforcement learning and GRPO as described in claim 1, characterized in that, It also includes the following steps: S4: Fine-tune parameters by combining Low-Rank Adaptation.
8. The image editing model training method based on online reinforcement learning and GRPO as described in claim 7, characterized in that, The parameter fine-tuning method is as follows: S41: Add an editing-specific LoRA module to the basic flow-matching image editing model. S42: Only updates low-dimensional adapter parameters, significantly reducing the number of training parameters. S43: Supports fast convergence and lightweight deployment.
9. The image editing model training method based on online reinforcement learning and GRPO as described in claim 7, characterized in that, It also includes the following steps: S5: Adopts noise reduction strategy.
10. The image editing model training method based on online reinforcement learning and GRPO as described in claim 9, wherein the noise reduction strategy is as follows: S51: During training: Use denoising steps for fast sampling; S52: During inference: Maintain the original number of denoising steps to ensure quality.