Automatic image editing method

Abstract

The invention relates to an automatic image editing method, which is characterized by comprising the following steps of S1, adopting a multi-technology fusion framework of a flow matching diffusion model, and taking an FLUX pre-training model as the basis of the multi-technology fusion framework; s2, realizing image editing through cooperative work of four core technology components, wherein the four core technology components are a target track starting point reinitialization mechanism, a dynamic velocity field scaling mechanism, a gradient intensity-based progressive contour generation and voting velocity field mask mechanism and a heterogeneous noise generation mechanism respectively. According to the method, training-free, inversion-free and source prompt word-free high-quality image editing can be realized.

Description

Technical Field

[0001] This invention relates to the field of computer vision and image editing technology, and in particular to an automated image editing method. Background Technology

[0002] Existing automated image editing methods have the following drawbacks:

[0003] (1) Training cost and computational efficiency issues: There are two main paradigms in existing image editing algorithms: 1) Pre-training methods, such as the InstructPix2Pix algorithm (Brooks et al., CVPR 2023), require the collection of a large amount of paired training data and training with a large-scale GPU cluster for several days to several weeks; 2) Online optimization methods, such as the Prompt-to-Prompt algorithm (Hertz et al., ICLR 2023), require iterative optimization during inference, resulting in a single image editing time of 1-2 minutes, which seriously affects practicality.

[0004] (2) Image Inversion Accuracy Dependence: Existing diffusion model-based image editing algorithms (such as DiffEdit, Couairon et al., ICLR 2023; SDEdit, Meng et al., ICLR 2022) generally adopt a strategy of inversion before editing. That is, the source image is first converted into Gaussian noise through DDIM inversion, and then the edited result is generated by resampling from the noise. The editing effect of this method is heavily dependent on the accuracy of the inversion step. Inaccurate inversion will lead to distortion of the final image or an insignificant editing effect.

[0005] (3) Source cue word dependency problem: Most existing algorithms (such as RF Inversion, Litu Rout et al., ICLR 2025) require users to provide both the text description of the source image (source cue words) and the target description. Obtaining source cue words requires manual annotation or additional image description models, which not only increases the system complexity and computational overhead, but also significantly affects the user experience.

[0006] (4) Insufficient precision in editing region control: Existing algorithms have obvious defects in editing region control: 1) The method without using a mask is prone to accidental modification of non-target regions; 2) The method that relies on the user to manually provide a mask is complicated to operate; 3) The method of automatically generating a mask (such as the method based on cross-attention) is too coarse and cannot adapt to editing objects of different scales, often resulting in unclear boundaries and inaccurate regions.

[0007] (5) Insufficient follow-up of editing instructions: Existing algorithms often have weak or completely ineffective editing effects in complex editing tasks, that is, the edited image is almost no different from the source image and cannot effectively respond to the user's editing instructions. This is mainly due to the lack of an effective editing intensity control mechanism.

[0008] Therefore, it is necessary to provide an automated image editing method to achieve high-quality image editing without training, inversion, or source prompts. Summary of the Invention

[0009] The purpose of this invention is to provide an automated image editing method that enables high-quality image editing without training, inversion, or source prompts.

[0010] To address the problems existing in the prior art, this invention provides an automated image editing method, comprising the following steps:

[0011] S1: A multi-technology fusion framework using a flow matching diffusion model, which is based on a FLUX pre-trained model;

[0012] S2: Image editing is achieved through the collaborative work of four core technology components, namely, the target trajectory starting point re-initialization mechanism, the dynamic velocity field scaling mechanism, the gradient intensity-based progressive contour generation and voting velocity field masking mechanism, and the heterogeneous noise generation mechanism.

[0013] Optionally, in the automated image editing method, the FLUX pre-trained model is a text-to-image generation model.

[0014] Optionally, in the automated image editing method, the technical principle of the target trajectory start-point re-initialization mechanism is as follows:

[0015] In the first time step of the editing process, the target trajectory is reinitialized by utilizing the velocity field information from the previous time step.

[0016] Optionally, in the automated image editing method, the technical principle of the dynamic velocity field scaling mechanism is as follows:

[0017] The scaling factor of the velocity field difference is dynamically adjusted by calculating the ratio of the actual forward distance to the theoretical forward distance of the source velocity field.

[0018] Optionally, in the automated image editing method, the technical principle of the gradient intensity-based progressive contour generation and voting velocity field masking mechanism is as follows:

[0019] By performing multi-threshold contour detection and voting fusion on the velocity field intensity map, a high-precision editing region mask is generated, enabling precise control over the editing region.

[0020] Optionally, in the automated image editing method, the threshold is obtained through progressive calculation.

[0021] Optionally, in the automated image editing method, the technical principle of the heterogeneous noise generation mechanism is as follows:

[0022] Different random noises are injected into the source and target trajectories at specified time steps to enhance the diversity of the editing process, while the same noise is used in time steps other than the specified time steps to ensure editing stability.

[0023] Optionally, in the automated image editing method, the overall processing flow of the automated image editing method is as follows:

[0024] Input preprocessing: The source image is encoded into a latent space representation using a variational autoencoder, and the target cue words are converted into semantic vectors using a text encoder;

[0025] Editing phase: From step n_start to step T_steps, the image editing trajectory is gradually modified through the collaborative work of four core technical components;

[0026] Output decoding: The edited latent representation is converted into the final image through the decoder of the autoencoder.

[0027] Optionally, in the automated image editing method, the autoencoder is a type of artificial neural network used in semi-supervised and unsupervised learning, whose function is to learn the representation of the input information by taking the input information as the learning target.

[0028] An autoencoder consists of two parts: an encoder and a decoder.

[0029] Compared with the prior art, the present invention has the following advantages:

[0030] (1) Efficiency improvement brought by training-free characteristics: realizes general image editing without training, and performs inference directly based on the pre-trained flow matching diffusion model. There is no need to collect data for specific tasks or retrain the model, reducing the training time required by traditional methods from several days to several weeks to zero, greatly reducing the consumption of computing resources and the threshold for use.

[0031] (2) Improved stability of the inversion-free architecture: The direct editing strategy completely eliminates the dependence on the image inversion step, avoids the impact of inversion accuracy on the final editing effect, and improves the robustness of the algorithm and the consistency of the editing results.

[0032] (3) User experience optimization without source prompts: Implement an editing mode that does not require source image description text. Users only need to provide the target description to complete the editing, which significantly simplifies the user interaction process and improves ease of use and practical application value.

[0033] (4) Improved editing quality through dynamic velocity field scaling: Through an innovative velocity field scaling mechanism, the editing intensity is adaptively adjusted, effectively filling the semantic gaps between different time steps, significantly improving the following of editing instructions, and reducing the generation of editing artifacts.

[0034] (5) An adaptive mask generation mechanism based on contour voting, combined with adaptive strategies of time and space dimensions, enables precise positioning and editing of objects at different scales, effectively protecting the invariance of non-editable areas. Attached Figure Description

[0035] Figure 1 This is a flowchart of a method provided in an embodiment of the present invention. Detailed Implementation

[0036] The specific embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. The advantages and features of the present invention will become clearer from the following description. It should be noted that the drawings are all in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the illustration of the embodiments of the present invention.

[0037] In the description of this application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0038] In the following, if the methods described herein include a series of steps, the order of these steps presented herein is not necessarily the only order in which these steps can be performed, and some of the steps described may be omitted and / or some other steps not described herein may be added to the method.

[0039] Existing automated image editing methods have the following drawbacks: training cost and computational efficiency issues, image inversion accuracy dependence issues, source cue word dependence issues, insufficient precision in editing region control, and insufficient follow-up of editing instructions.

[0040] To address the problems existing in the prior art, this invention provides an automated image editing method, such as... Figure 1 As shown, the method includes the following steps:

[0041] S1: A multi-technology fusion framework using a flow matching diffusion model, which is based on the FLUX pre-trained model; the FLUX pre-trained model is a text-to-image generation model.

[0042] S2: Image editing is achieved through the collaborative work of four core technology components, namely, the target trajectory starting point re-initialization mechanism, the dynamic velocity field scaling mechanism, the gradient intensity-based progressive contour generation and voting velocity field masking mechanism, and the heterogeneous noise generation mechanism.

[0043] Specifically, the technical principle of the target trajectory starting point re-initialization mechanism is as follows:

[0044] In the first time step of the editing process, the target trajectory is re-initialized by utilizing the velocity field information from the previous time step, thereby optimizing the starting point of the target trajectory and improving the editing quality and directionality.

[0045] Specific implementation steps:

[0046] Step 1: Check the starting conditions for editing

[0047] When the first edit time step is reached, the re-initialization mechanism is activated.

[0048] Step 2: Calculate the latent representation of the previous time step

[0049] Based on the scheduling parameter t_{i+1} of the previous time step, calculate the potential state of the previous step:

[0050] z_prev=(1-t_{i+1})·x_src+t_{i+1}·ε

[0051] Where x_src is the latent representation of the source image, and ε is standard Gaussian noise.

[0052] Step 3: Calculate the velocity field difference of the previous time step

[0053] Calculate the source velocity field and target velocity field at the previous time step respectively:

[0054] V_src^{prev}=F_θ(z_prev,"",t_{i+1})

[0055] V_tar^{prev}=F_θ(z_prev,P_tar,t_{i+1})

[0056] Where F_θ represents the stream matching model, "" represents an empty source prompt, and P_tar represents a target prompt.

[0057] Step 4: Reinitialize the target trajectory

[0058] The target trajectory is reinitialized based on the velocity field difference:

[0059] z_tar=z_src+β·(t_i-t_{i+1})·(V_tar^{prev}-V_src^{prev})

[0060] Where β is the reinitialization coefficient and (t_i-t_{i+1}) is the time step.

[0061] Technical advantages:

[0062] By leveraging the forward prediction capabilities of the flow matching model, better initialization of the target trajectory is provided at the beginning of the editing stage, avoiding deviations in the target trajectory that may be caused by random initialization, and significantly improving the accuracy and stability of editing.

[0063] 2. The technical principle of the dynamic velocity field scaling mechanism is as follows:

[0064] By calculating the ratio of the actual forward distance to the theoretical forward distance of the source velocity field, the scaling factor of the velocity field difference is dynamically adjusted to achieve adaptive control of the editing intensity.

[0065] Core calculation formula:

[0066] Step 1: Calculate the time step

[0067] Δt=t_{i-1}-t_i

[0068] Step 2: Calculate the distance the source velocity field has traveled.

[0069] S_src=||Δt·V_src||2

[0070] Where ||*||2 represents the quadratic norm.

[0071] Step 3: Calculate the theoretical distance traveled

[0072] S_noise=|Δt|·||x_src-ε||2

[0073] Step 4: Calculate the dynamic scaling factor

[0074] scale = S_noise / (S_src + ε0)

[0075] Where ε0 is a small constant to prevent division by zero.

[0076] Step 5: Apply scaling to velocity field difference

[0077] V_delta_scaled=scale·(V_tar-V_src)

[0078] Technical advantages:

[0079] This mechanism resolves the inconsistency between the predicted latent vector position at the current time step and the actual latent vector position at the next time step, bridging the semantic gaps between different time steps. Furthermore, its built-in dynamic scaling mechanism intelligently adjusts the editing intensity through real-time analysis of velocity field characteristics.

[0080] 3. The technical principle of the gradient intensity-based progressive contour generation and voting velocity field masking mechanism is as follows:

[0081] By performing multi-threshold contour detection and voting fusion on the velocity field intensity map, a high-precision editing region mask is generated, enabling precise control of the editing region. The threshold is obtained through progressive calculation.

[0082] Core algorithm flow:

[0083] Step 1: Generation of velocity field intensity map

[0084] Calculate the mean absolute value of the velocity field difference V_delta for each channel, and apply min-max normalization and Gaussian blur:

[0085] I_velocity=GaussianBlur(Normalize(mean(|V_delta_unpacked|, axis=channel)))

[0086] Step 2: Time-adaptive mask determination

[0087] Calculate the foreground / background ratio of the intensity map:

[0088] ratio_fg_bg=percentile(I_velocity,95) / percentile(I_velocity,5)

[0089] When ratio_fg_bg ≥ τ_ratio (τ_ratio is a preset threshold), mask generation is enabled; otherwise, a mask of all 1s is returned.

[0090] Step 3: Generation of progressive percentile thresholds

[0091] Multiple thresholds are generated using a gradient-guided adaptive step size adjustment algorithm:

[0092] (a) Initialization: p_cur = p_min

[0093] (b) Cyclic generation of thresholds:

[0094] - Calculate the threshold corresponding to the current percentile: thr = percentile(I_velocity, p_cur)

[0095] - Generate a binary mask: M_binary = (I_velocity > thr)

[0096] - Contour detection: contours = findContours(M_binary)

[0097] - Calculate the mean gradient of the maximum contour:

[0098] - Adaptive adjustment of the next percentile: A piecewise linear interpolation strategy is used based on the gradient mean. When the gradient mean is less than or equal to 0.25, the step size decreases from 20 to 12; when the gradient mean is between 0.25 and 0.7, the step size decreases from 12 to 2; when the gradient mean is greater than 0.7, the step size is fixed at 2.

[0099] - Update percentiles: p_cur = min(p_cur + step, p_max)

[0100] Step 4: Outline area filtering and filling

[0101] For each threshold, the corresponding binary mask is:

[0102] - Filter contours with an area greater than τ_area

[0103] - Fill the filtered outlines to generate a voting mask M_vote

[0104] Step 5: Multi-mask voting fusion

[0105] Add up all vote masks:

[0106] M_sum=ΣM_vote^(i)

[0107] Generate the final mask:

[0108] M_final = (M_sum ≥ τ_vote)

[0109] Where τ_vote is the voting threshold.

[0110] Step 6: Morphological optimization

[0111] Perform morphological operations on the final mask:

[0112] - Hole filling: Uses scipy.ndimage.binary_fill_holes

[0113] - Noise removal: Apply opening operation (erosion followed by dilation)

[0114] - Edge softening: Optional Gaussian blur processing

[0115] Technological innovation points:

[0116] (1) Gradient-guided adaptive step size: The threshold step size is dynamically adjusted according to the gradient characteristics of the contour boundary to ensure that target objects of different scales are captured.

[0117] (2) Contour voting mechanism: Improve the robustness and accuracy of the mask by using contour voting with multiple thresholds;

[0118] (3) Time-adaptive judgment: intelligently decide whether to enable the mask based on the foreground / background ratio to avoid unnecessary computational overhead and generating incorrect masks.

[0119] 4. The technical principle of heterogeneous noise generation mechanism is as follows:

[0120] Different random noises are injected into the source and target trajectories at specified time steps to enhance the diversity of the editing process, while the same noise is used in time steps other than the specified time steps to ensure editing stability.

[0121] Implementation strategy:

[0122] Step 1: Time Step Selection Strategy

[0123] Supports two heterogeneous noise application strategies:

[0124] - Specify a list of time steps: Apply heterogeneous noise at preset specific time steps (e.g., {4, 24, 25, 26, 27, 28}).

[0125] - Threshold cutoff strategy: Apply heterogeneous noise in steps where the time step number is less than or equal to a specified threshold.

[0126] Step 2: Heterogeneous noise generation

[0127] When in the heterogeneous noise time step:

[0128] -Generation source trajectory noise: ε1~N(0,I)

[0129] - Generate independent auxiliary noise: ε2~N(0,I)

[0130] - Mixed generation of target trajectory noise: ε2'=α·ε1+(1-α)·ε2

[0131] Here, α is the mixing coefficient, which maintains a certain correlation while introducing sufficient differences.

[0132] Step 2.1: Trajectory Construction

[0133] -Source trajectory: z_src=(1-t_i)·x_src+t_i·ε1

[0134] - Target trajectory: z_tar = z_edit + t_i·(ε2'-x_src)

[0135] Where z_edit represents the current editing status of the trajectory.

[0136] Step 3: Homogeneous noise processing

[0137] At the non-heterogeneous noise time step:

[0138] -Use the same noise: ε_common~N(0,I)

[0139] -Source trajectory: z_src=(1-t_i)·x_src+t_i·ε_common

[0140] - Target trajectory: z_tar = z_edit + z_src - x_src

[0141] Optionally, in the automated image editing method, the overall processing flow of the automated image editing method is as follows:

[0142] Input preprocessing: The source image is encoded into a latent space representation by a variational autoencoder (VAE), and the target prompt words are converted into semantic vectors by a text encoder. An autoencoder is a type of artificial neural network used in semi-supervised and unsupervised learning. Its function is to learn the representation of the input information by using the input information as the learning target. An autoencoder consists of two parts: an encoder and a decoder.

[0143] Editing phase: From step n_start to step T_steps, the image editing trajectory is gradually modified through the collaborative work of four core technical components;

[0144] Output decoding: The edited latent representation is converted into the final image by the decoder of the autoencoder (VAE).

[0145] Its technology integration strategy is as follows:

[0146] The four core technologies are organically integrated according to their time sequence and functional characteristics: the target trajectory start point re-initialization mechanism optimizes the target trajectory start point at the beginning of the editing stage; the dynamic velocity field scaling mechanism adjusts the editing intensity throughout the editing process and fills the semantic gaps between different time steps; the velocity field masking mechanism performs adaptive and precise region control in both spatial and temporal dimensions; and the heterogeneous noise generation mechanism enhances diversity exploration at specific time steps.

[0147] In summary, compared with the prior art, the present invention has the following advantages:

[0148] (1) Efficiency improvement brought by training-free characteristics: realizes general image editing without training, and performs inference directly based on the pre-trained flow matching diffusion model. There is no need to collect data for specific tasks or retrain the model, reducing the training time required by traditional methods from several days to several weeks to zero, greatly reducing the consumption of computing resources and the threshold for use.

[0149] (2) Improved stability of the inversion-free architecture: The direct editing strategy completely eliminates the dependence on the image inversion step, avoids the impact of inversion accuracy on the final editing effect, and improves the robustness of the algorithm and the consistency of the editing results.

[0150] (3) User experience optimization without source prompts: Implement an editing mode that does not require source image description text. Users only need to provide the target description to complete the editing, which significantly simplifies the user interaction process and improves ease of use and practical application value.

[0151] (4) Improved editing quality through dynamic velocity field scaling: Through an innovative velocity field scaling mechanism, the editing intensity is adaptively adjusted, effectively filling the semantic gaps between different time steps, significantly improving the following of editing instructions, and reducing the generation of editing artifacts.

[0152] (5) An adaptive mask generation mechanism based on contour voting, combined with adaptive strategies of time and space dimensions, enables precise positioning and editing of objects at different scales, effectively protecting the invariance of non-editable areas.

[0153] The above are merely preferred embodiments of the present invention and do not constitute any limitation on the present invention. Any equivalent substitutions or modifications made by those skilled in the art to the technical solutions and content disclosed in the present invention without departing from the scope of the present invention shall be deemed to have remained within the protection scope of the present invention.

Claims

1. An automated image editing method, characterized in that, Includes the following steps: S1: A multi-technology fusion framework using a flow matching diffusion model, which is based on a FLUX pre-trained model; S2: Image editing is achieved through the collaborative work of four core technology components, namely, the target trajectory starting point re-initialization mechanism, the dynamic velocity field scaling mechanism, the gradient intensity-based progressive contour generation and voting velocity field masking mechanism, and the heterogeneous noise generation mechanism.

2. The automated image editing method as described in claim 1, characterized in that, The FLUX pre-trained model is a text-to-image generation model.

3. The automated image editing method as described in claim 1, characterized in that, The technical principle of the target trajectory start-point re-initialization mechanism is as follows: In the first time step of the editing process, the target trajectory is reinitialized by utilizing the velocity field information from the previous time step.

4. The automated image editing method as described in claim 1, characterized in that, The technical principle of the dynamic velocity field scaling mechanism is as follows: The scaling factor of the velocity field difference is dynamically adjusted by calculating the ratio of the actual forward distance to the theoretical forward distance of the source velocity field.

5. The automated image editing method as described in claim 1, characterized in that, The technical principle of the gradient intensity-based progressive contour generation and voting velocity field masking mechanism is as follows: By performing multi-threshold contour detection and voting fusion on the velocity field intensity map, a high-precision editing region mask is generated, enabling precise control over the editing region.

6. The automated image editing method as described in claim 5, characterized in that, The threshold is obtained through incremental calculation.

7. The automated image editing method as described in claim 1, characterized in that, The technical principle of heterogeneous noise generation mechanism is as follows: Different random noises are injected into the source and target trajectories at specified time steps to enhance the diversity of the editing process, while the same noise is used in time steps other than the specified time steps to ensure editing stability.

8. The automated image editing method as described in claim 1, characterized in that, The overall processing flow of the automated image editing method is as follows: Input preprocessing: The source image is encoded into a latent space representation using a variational autoencoder, and the target cue words are converted into semantic vectors using a text encoder; Editing phase: From step n_start to step T_steps, the image editing trajectory is gradually modified through the collaborative work of four core technical components; Output decoding: The edited latent representation is converted into the final image through the decoder of the autoencoder.

9. The automated image editing method as described in claim 8, characterized in that, Autoencoders are a type of artificial neural network used in semi-supervised and unsupervised learning. Their function is to learn representations of input information by using the input information as the learning target. An autoencoder consists of two parts: an encoder and a decoder.

Images