Universal interactive image editing method and system based on anti-ambiguity point

Abstract

The invention provides a universal interactive image editing method and system based on anti-ambiguity points, and relates to the technical field of image editing.The method comprises the steps that firstly, an image editing task type is defined, a dense point set is constructed based on the image editing task type, and a dense point track in the dense point set is calculated; then carrying out adaptive motion supervision strategy processing on the input image to obtain dense point potential features; when the dense points move along the dense point tracks, the dense point potential features are optimized, and dense point potential optimization features are obtained; and finally, decoding the potential optimization features of the dense points to obtain an edited image. According to the method, ambiguity generated in editing intention is effectively reduced, the image editing difficulty is reduced, and the image editing quality is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image editing, and in particular to a general interactive image editing method and system based on anti-ambiguity points. Background Art

[0002] In the digital age, image editing is an important means of visual content creation and dissemination, and its efficiency and quality improvement have always been the focus of technological development. With the rapid development of computer vision and artificial intelligence technology, interactive image editing methods have emerged, which greatly improve the convenience and intuitiveness of editing by capturing the user's direct operations (such as dragging, clicking, etc.) to adjust the image content in real time.

[0003] At present, most interactive image editing methods are implemented by using an image editing model based on interaction points, but there are ambiguities in two aspects. On the one hand, in terms of the recognition of editing intentions, traditional image editing methods based on interaction points often face the problem of misunderstanding the user's editing intentions, that is, multiple possible editing results may correspond to the same interaction point trajectory (editing instructions). For example, the DragDiffusion algorithm pays too much attention to the position update of the control point during the optimization process, but ignores the specific editing tasks behind the user, resulting in the generated editing trajectory may deviate from the user's expectations, and even appear unreasonable or easy to deviate. Although the FreeDrag algorithm restricts the trajectory shape (such as a straight line) to The ambiguity problem can be alleviated, but this two-dimensional trajectory cannot fully represent three-dimensional operations such as out-of-plane rotations, which limits the diversity and flexibility of editing operations and affects the quality of image editing. On the other hand, in the processing of image content, most existing methods rely on the user-specified denoising time step to select the potential features of the control points for optimization. However, the information in the image is multi-level and multi-dimensional, and the features of different time steps or levels contain information of different scales and granularities. For complex editing tasks, since the potential features of a single time step often cannot fully reflect the true state of the image and the user's editing needs, it is impractical to generate satisfactory results using only the potential features of one time step. Therefore, how to reduce ambiguity in editing intent, reduce the difficulty of image editing and improve the quality of image editing is an extremely important technical problem to be solved. Summary of the invention

[0004] In order to solve the problem that the above-mentioned prior art generates ambiguity in editing intention, resulting in high difficulty in image editing and low quality in image editing, the present invention proposes a general interactive image editing method and system based on anti-ambiguity points, which effectively reduces ambiguity in editing intention, reduces the difficulty of image editing, and improves the quality of image editing.

[0005] In order to achieve the above technical effects, the technical solution of the present invention is as follows:

[0006] A general interactive image editing method based on anti-ambiguity points comprises the following steps:

[0007] S1. defining an image editing task type, constructing a dense point set based on the image editing task type, and calculating a dense point trajectory in the dense point set;

[0008] S2. Process the input image with an adaptive motion supervision strategy to obtain dense point potential features;

[0009] S3. When the dense points move along the dense point trajectory, the potential features of the dense points are optimized to obtain the potential optimized features of the dense points;

[0010] S4. Decode the potential optimization features of the dense points to obtain an edited image.

[0011] Preferably, the image editing task types include position movement tasks, rotation tasks and non-rigid transformation tasks.

[0012] Preferably, the constructing a dense point set based on the image editing task type and calculating the dense point trajectory in the dense point set includes:

[0013] In the position movement task, build a dense point set The calculation expression is as follows:

[0014]

[0015] in, Represents the edit object mask map;

[0016] Compute dense point trajectories in dense point sets The specific expression is as follows:

[0017]

[0018] in, p j Represents a point set is the initial position of the jth point in , Δ represents the average displacement of the control points, L is the number of steps to move each point to its target position, 0≤l≤L-1, l represents the number of moves, Indicates the new position of the point after moving l times;

[0019] In the rotation task, build a dense point set The calculation expression is as follows:

[0020]

[0021] Where ε represents the three-dimensional ellipsoid, Φ(ε) represents the projection point of the surface point of the three-dimensional ellipsoid ε on the two-dimensional image;

[0022] Compute dense point trajectories in dense point sets The specific expression is as follows:

[0023]

[0024] Among them, p ε It is used to construct the three-dimensional ellipsoid ε and satisfies the two-dimensional projection belonging to the point set The three-dimensional point set, p ε,j Indicates p ε The jth point in Φ(p ε,j ) indicates p ε,j The two-dimensional projection of the initial position of the point on the image, d j Indicates p ε,j The geometric displacement after rotation, Φ(d j ) represents the geometric displacement d j The two-dimensional projection on the image, Indicates the new position of the point on the image after being rotated l times;

[0025] In non-rigid transformation tasks, building dense point sets The calculation expression is as follows:

[0026]

[0027] in, represents the selected set of boundary points;

[0028] Compute dense point trajectories in dense point sets The specific expression is as follows:

[0029]

[0030] in, Represents a point set The initial position of the jth point in , Indicates the end position of the point.

[0031] Preferably, the performing adaptive motion supervision strategy processing on the input image comprises:

[0032] S21. Convert the input image into a series of latent feature maps {z t}, where 0≤t≤T-1, t represents any denoising time step;

[0033] S22. The potential feature map {z t} to update and obtain the potential feature update graph n represents any optimization step after a small range of movement on the dense point trajectory;

[0034] S23. Update the latent feature map Input pre-trained diffusion model, output dense point latent features

[0035] Preferably, the potential feature update map The calculation expression is as follows:

[0036]

[0037] Among them, ρ represents the parameter of the image editing task characteristics, represents the potential feature map after a small range of movement on the dense point trajectory, b n Represents potential feature bias for optimization.

[0038] Preferably, the diffusion model includes a first UNet network and a second UNet network in parallel.

[0039] Preferably, the diffusion model is fine-tuned in a low-rank adaptive manner, including:

[0040] S231. Update the potential feature map The first UNet network and the second UNet network are input respectively, and the first UNet network outputs the first feature And the second UNet network outputs the second feature

[0041] S232. Introducing optimizable weight graphs n , the first feature And the second feature Combine to get the potential features of dense points The calculation expression is as follows:

[0042]

[0043] Where ⊙ represents the Hadamard product, s n The spatial resolution and The spatial resolution is the same.

[0044] Preferably, the loss function is The potential features of the dense points are optimized, and the loss function The calculation expression is as follows:

[0045]

[0046] Among them, the first loss term It is used to measure the difference between the original feature and the feature after moving along the dense point trajectory. The second loss term Represents the corresponding feature difference calculated before and after each step of motion supervision, the third loss term represents the feature deviation of the latent feature map outside the target mask, β represents a task-specific parameter, and λ represents the same value for all tasks.

[0047] Preferably, the first loss term The calculation expression is as follows:

[0048]

[0049] in, represents the image features of the dense points located at the position l+1 in the dense trajectory extracted by linear interpolation on the feature map, The image features representing the dense points at the initial positions extracted in the same manner;

[0050] The second loss item The calculation expression is as follows:

[0051]

[0052] in, Represents the image features of the dense points located at l in the dense trajectory;

[0053] The third loss item The calculation expression is as follows:

[0054]

[0055] The present invention also proposes a general interactive image editing system based on anti-ambiguity points, comprising:

[0056] A calculation module, used for defining an image editing task type, constructing a dense point set based on the image editing task type, and calculating a dense point trajectory in the dense point set;

[0057] A processing module, used for performing adaptive motion supervision strategy processing on the input image to obtain dense point potential features;

[0058] An optimization module, used for optimizing the potential features of the dense points when the dense points move along the dense point trajectory, so as to obtain the potential optimized features of the dense points;

[0059] The output module is used to decode the potential optimization features of the dense points and output the edited image.

[0060] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0061] The present invention proposes a general interactive image editing method and system based on anti-ambiguous points. In order to solve the problem of ambiguity in editing intent, a framework of an interactive point-based image editing method for handling ambiguity by explicitly modeling editing tasks is provided. First, the image editing task type is defined, a dense point set is constructed according to the image editing task type, and the dense point trajectory in the dense point set is calculated, so as to provide more comprehensive and reasonable prior knowledge for image editing; then, an adaptive motion supervision strategy is performed on the input image, and task-aware fine-grained optimization parameters are introduced to optimize dense point potential features. The decoded image is output by decoding the optimized dense point potential features, thereby effectively reducing ambiguity in editing intent, reducing the difficulty of image editing, and improving the quality of image editing. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 A flowchart of a general interactive image editing method based on anti-ambiguity points proposed in an embodiment of the present invention is shown;

[0063] Figure 2 A working principle diagram of a general interactive image editing method based on anti-ambiguity points proposed in an embodiment of the present invention;

[0064] Figure 3 A diagram showing the image editing effects of the GDrag method and other model methods proposed in the embodiments of the present invention;

[0065] Figure 4 Another image editing effect diagram showing the GDrag method and other model methods proposed in the embodiment of the present invention;

[0066] Figure 5 A structural diagram of a general interactive image editing system based on anti-ambiguity points proposed in an embodiment of the present invention is shown; DETAILED DESCRIPTION

[0067] The accompanying drawings are only used for illustrative purposes and are not to be construed as limiting the present invention;

[0068] It is understandable to those skilled in the art that some well-known contents may be omitted in the drawings;

[0069] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0070] Example 1

[0071] like Figure 1As shown in the figure, the existing image editing method based on interactive points lacks clear modeling of editing intent, which inevitably leads to ambiguity. Such ambiguity not only increases the difficulty of editing, but also reduces the quality of edited images. Therefore, the present invention proposes a general interactive image editing method based on anti-ambiguity points, which includes the following steps:

[0072] S1. defining an image editing task type, constructing a dense point set based on the image editing task type, and calculating a dense point trajectory in the dense point set;

[0073] In S1, a set of image editing tasks with clear editing intentions are defined. These image editing tasks should be parameterized and guide subsequent processing to facilitate the completion of various complex editing tasks within a unified framework; the image editing task types include position movement tasks, rotation tasks, and non-rigid transformation tasks; position movement tasks involve translation of the editing object, rotation tasks include rotation of the editing object in or out of the image plane, and non-rigid transformation tasks include isotropic and anisotropic scaling, as well as removal or generation of the internal content of the editing object. In order to obtain satisfactory results, it is necessary to perform operations with different parameters for these three tasks and calculate the controlled dense point trajectories with different rules.

[0074] Given an image and user-specific controlled dense point trajectories, these dense point trajectories are set to belong to a certain image editing task, and the image editing task is specified. It should be noted that complex operations can be decomposed into multiple image editing tasks. Since the dense point trajectories controlled by the user are sparse and cannot fully represent the image information during the editing process, it is proposed to construct a dense point set based on the image editing task type and calculate the dense point trajectories in the dense point set, which is denoted as the anti-ambiguity dense trajectory calculation method (Anti-ambiguity Dense Trajectory, ADT) to identify neighborhood points that are semantically and geometrically related to the controlled dense points, thereby constructing a dense point set and calculating the trajectory of each point in the set according to the image editing task type. In previous point tracking methods, updating the position of the control point is achieved by searching a predefined neighborhood, while in this step, these dense points are only moved on the pre-calculated trajectory of each point. These more reasonable trajectories can reduce the ambiguity in editing intent.

[0075] Specifically, the goal of the ADT method involved in S1 is to obtain sufficient image information and reliable task perception trajectories. Specifically, see Figure 2 , given M user-specified control points and its corresponding end point First, the pre-trained image segmentation model SAM is used to obtain the semantic segmentation mask M of the edited object; then, a dense point set is constructed. And estimate the drag trajectory of dense points based on different tasks The specific steps are as follows:

[0076] In the position movement task, the points inside the edited object will move synchronously in similar directions and distances, thereby constructing a dense point set. The calculation expression is as follows:

[0077]

[0078] in, Represents the edit object mask map;

[0079] Compute dense point trajectories in dense point sets The specific expression is as follows:

[0080]

[0081] in, p j Represents a point set is the initial position of the jth point in , Δ represents the average displacement of the control points, L is the number of steps to move each point to its target position, 0≤l≤l-1, l represents the number of moves, Indicates the new position of the point after moving l times;

[0082] In the rotation task, since recovering 3D rotation from 2D image plane trajectory is an inappropriate problem, a 3D ellipsoid ε is used as an auxiliary tool to calculate the dense point trajectory. Specifically, ε is first placed near the center of the edited object, and the user can roughly cover the edited object by adjusting its axis length and angle. Let Φ(ε) represent the projection point of the surface point of ε on the 2D image, and approximate it by finding the nearest point among these projection points. (denoted as ), and then rotate around an axis of ε so that The projection is close to Construct a dense point set The calculation expression is as follows:

[0083]

[0084] Where ε represents the three-dimensional ellipsoid, Φ(ε) represents the projection point of the surface point of the three-dimensional ellipsoid ε on the two-dimensional image;

[0085] Compute dense point trajectories in dense point sets The specific expression is as follows:

[0086]

[0087] Among them, p ε It is used to construct the three-dimensional ellipsoid ε and satisfies the two-dimensional projection belonging to the point set The three-dimensional point set, p ε,j Indicates p ε The jth point in Φ(p ε,j ) indicates p ε,j The two-dimensional projection of the initial position of the point on the image, d j Indicates p ε,j The geometric displacement after rotation, Φ(d j ) represents the geometric displacement d j The two-dimensional projection on the image, Indicates the new position of the point on the image after being rotated l times;

[0088] Compared with the dense point trajectory directly calculated from the control points on the 2D plane, the dense point trajectory generated in this step can better represent the 3D rotation. In addition, the dense point trajectory generated in this step implicitly encodes geometric information, such as the different distances of the target surface points, which can guide the subsequent optimization process.

[0089] In the non-rigid transformation task, in order to complete the non-rigid transformation, we need to pay attention to the boundary points of the edited object because they determine the shape and posture of the target; in particular, A fixed width r is extracted from the boundary band; in order to reduce the distance The influence of distant boundary points and fine-grained local transformation are achieved, and only k points related to The closest boundary point.

[0090] Constructing a dense point set The calculation expression is as follows:

[0091]

[0092] in, represents the selected set of boundary points;

[0093] Compute dense point trajectories in dense point sets The specific expression is as follows:

[0094]

[0095] in, Represents a point set The initial position of the jth point in , Indicates the end position of the point. The starting and ending positions of a are set to the positions of its nearest control points.

[0096] S2. Process the input image with an adaptive motion supervision strategy to obtain dense point potential features;

[0097] S3. When the dense points move along the dense point trajectory, the potential features of the dense points are optimized to obtain the potential optimized features of the dense points;

[0098] S4. Decode the potential optimization features of the dense points to obtain an edited image.

[0099] In this embodiment, in order to solve the problem of ambiguity in editing intent, a framework of an image editing method based on interactive points is provided to handle ambiguity by explicitly modeling editing tasks. First, the image editing task type is defined, a dense point set is constructed according to the image editing task type, and the dense point trajectory in the dense point set is calculated, with the purpose of providing more comprehensive and reasonable prior knowledge for image editing; then, the input image is processed by an adaptive motion supervision strategy, and task-aware fine-grained optimization parameters are introduced to optimize the dense point potential features, and the decoded image is output by decoding the optimized dense point potential features, thereby effectively reducing the ambiguity in editing intent, reducing the difficulty of image editing, and improving the quality of image editing. It is also necessary to specifically state that the anti-ambiguous point-oriented general interactive image editing method proposed in this embodiment is a novel task-aware, training-free framework called GDrag (General-purpose Drag Editing). Specifically, complex operations can be decomposed into multiple basic tasks, and the intent of each task is clear and easy to execute, thereby reducing the difficulty of editing and reducing the ambiguity in editing intent. In addition, the framework introduces two strategies to alleviate content ambiguity, including the anti-ambiguity dense trajectory calculation method (Anti-ambiguity Dense Trajectory, ADT) mentioned in step S1 and the adaptive motion supervision method (Self-adaptive Motion Supervision, SMS) mentioned in step S2.

[0100] This embodiment also conducts extensive experiments on the DragBench dataset, proving that GDrag can produce accurate and user-satisfactory results in various editing tasks, and compared with other existing image editing methods based on interactive points, the anti-ambiguity point-based general interactive image editing method proposed in this embodiment can better balance the consistency of image content and the flexibility of editing. Figure 3 and Figure 4 In different task types, the effect obtained by the GDrag method in this embodiment is obviously better than other existing models.

[0101] Example 2

[0102] The motion supervision strategy described in S2 is a process of gradually optimizing the image features of the control points along their trajectories. In existing diffusion model methods, motion supervision requires a user-specified denoising time step to select latent features for optimization. However, recent studies have shown that diffusion models tend to generate high-level image content in earlier denoising steps and finer details in later stages. Therefore, optimizing latent features at a single time step is not sufficient to produce ideal content. To address this issue, S2 introduced the SMS method in GDrag.

[0103] See also Figure 2 , the step of performing adaptive motion supervision strategy processing on the input image comprises:

[0104] S21. Convert the input image into a series of latent feature maps {z t}, where 0≤t≤T-1, t represents any denoising time step;

[0105] In S21, the input image is converted into a series of potential feature maps {z t}, gradually changing a latent feature and removing noise from it to generate an edited image;

[0106] S22. The potential feature map {z t} to update and obtain the potential feature update graph n represents any optimization step after a small range of movement on the dense point trajectory;

[0107] In S22, the original motion supervision method assumes that the latent feature map of the desired edited image follows the same data prior as the input image. Therefore, it moves the control points step by step and minimizes the difference between the corresponding features before and after the control points are moved in a small range. Given a dense point trajectory consisting of L steps If we simply iterate over all T denoised time steps to exploit enough latent features, the computational complexity is O(TL), which is a significant burden. Therefore, this step performs N optimizations (N<<T) using randomly selected time steps after each move. Specifically, let 0≤n≤N-1 represent any optimization step after a small move on a dense trajectory. For simplicity, we ignore the index of the move step on the dense trajectory and represent the latent features after the move as Therefore, a denoising time step is randomly selected and the corresponding latent feature map is updated.

[0108] S23. Update the latent feature map Input pre-trained diffusion model, output dense point latent features

[0109] In S23, the potential feature update map The calculation expression is as follows:

[0110]

[0111] Among them, ρ represents the parameter of the image editing task characteristics, represents the potential feature map after a small range of movement on the dense point trajectory, b n Represents potential feature bias for optimization.

[0112] Consider encoding the latent features into b before moving, so that the latent features can be adaptively updated using the task-specific parameter ρ. To this end, the denoising formula in DDIM is used to calculate the approximate And initialize b:

[0113]

[0114] in, is the Gaussian variance that controls the transition probability of the Markov chain in DDIM, ∈ t is the noise predicted by the pre-trained diffusion model, b 0 is the value of initializing b. The value of ρ varies depending on the task type. For example, in the Relocation task, a larger ρ is more ideal because it usually better preserves the original latent features.

[0115] In addition, the diffusion model is fine-tuned through a low-rank adaptive method LoRA, including:

[0116] S231. Update the potential feature map The first UNet network and the second UNet network are input respectively, and the first UNet network outputs the first feature And the second UNet network outputs the second feature

[0117] S232. Introducing optimizable weight graphs n , the first feature And the second feature Combine to get the potential features of dense points The calculation expression is as follows:

[0118]

[0119] Where ⊙ represents the Hadamard product, s n The spatial resolution and The spatial resolution is the same.

[0120] In S232, the outputs of the two UNet networks in the diffusion model are linearly combined through a weight scalar, namely:

[0121]

[0122] However, such a global combination is not sufficient to model the transformation of fine-grained and heterogeneous features, so an optimizable weight map is introduced in SMS;

[0123] In the movement of the dense points along the dense point trajectory, in order to obtain the ideal potential features of the edited image, the loss function is used The potential features of the dense points are optimized b and s, and the loss function The calculation expression is as follows:

[0124]

[0125] Among them, the first loss term It is used to measure the difference between the original feature and the feature after moving along the dense point trajectory. The second loss term Represents the corresponding feature difference calculated before and after each step of motion supervision, which is used to encourage smooth feature transformation. The third loss term It represents the feature deviation of the potential feature map outside the target mask to ensure that the image information outside the mask remains unchanged; β represents a specific parameter related to the task, and λ represents the same value for all tasks.

[0126] The first loss term The calculation expression is as follows:

[0127]

[0128] in, represents the image features of the dense points located at the position l+1 in the dense trajectory extracted by linear interpolation on the feature map, The image features representing the dense points at the initial positions extracted in the same manner;

[0129] The second loss item The calculation expression is as follows:

[0130]

[0131] in, Represents the image features of the dense points located at l in the dense trajectory;

[0132] The third loss item The calculation expression is as follows:

[0133]

[0134] Among them, among them, represents the dense point set P obtained by interpolating the feature outputs of two UNet networks * potential characteristics.

[0135] Finally, the positions of the dense points are updated after each motion supervision. Since the pre-computed task-aware trajectories are reliable, for each point in them, its new position is determined as the position on its corresponding trajectory that minimizes the error, rather than searching in the neighborhood of the trajectory. This enables the control points to move along the dense point trajectory and avoid unnecessary drift.

[0136] Example 3

[0137] See also Figure 5 This embodiment proposes a general interactive image editing system based on anti-ambiguity points, including:

[0138] A calculation module, used for defining an image editing task type, constructing a dense point set based on the image editing task type, and calculating a dense point trajectory in the dense point set;

[0139] A processing module, used for performing adaptive motion supervision strategy processing on the input image to obtain dense point potential features;

[0140] An optimization module, used for optimizing the potential features of the dense points when the dense points move along the dense point trajectory, so as to obtain the potential optimized features of the dense points;

[0141] The output module is used to decode the potential optimization features of the dense points and output the edited image.

[0142] In this embodiment, in order to solve the problem of ambiguity in editing intent, a framework of an interactive point-based image editing method for handling ambiguous editing tasks by explicitly modeling editing tasks is provided. First, the image editing task type is defined, a dense point set is constructed according to the image editing task type, and the dense point trajectories in the dense point set are calculated, with the aim of providing more comprehensive and reasonable prior knowledge for image editing; then, an adaptive motion supervision strategy is performed on the input image, and task-aware fine-grained optimization parameters are introduced to optimize the dense point latent features; the decoded image is output by decoding the optimized dense point latent features, thereby effectively reducing ambiguity in editing intent, reducing the difficulty of image editing, and improving the quality of image editing.

[0143] Obviously, the above embodiments of the present invention are only examples for clearly illustrating the present invention, and are not intended to limit the implementation methods of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all implementation methods here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A general interactive image editing method based on anti-ambiguity points, characterized in that: The following steps are involved: S1. defining an image editing task type, constructing a dense point set based on the image editing task type, and calculating a dense point trajectory in the dense point set; S2. Process the input image with an adaptive motion supervision strategy to obtain dense point potential features; S3. When the dense points move along the dense point trajectory, the potential features of the dense points are optimized to obtain the potential optimized features of the dense points; S4. Decode the potential optimization features of the dense points to obtain an edited image.

2. The anti-ambiguity point-oriented general interactive image editing method according to claim 1, characterized in that: The image editing task types include position movement tasks, rotation tasks and non-rigid transformation tasks.

3. The anti-ambiguity point-oriented general interactive image editing method according to claim 2, characterized in that: The step of constructing a dense point set based on the image editing task type and calculating dense point trajectories in the dense point set includes: In the position movement task, build a dense point set The calculation expression is as follows: in, Represents the edit object mask map; Compute dense point trajectories in dense point sets The specific expression is as follows: in, p j Represents a point set is the initial position of the jth point in , Δ represents the average displacement of the control points, L is the number of steps to move each point to its target position, 0≤l≤L-1, l represents the number of moves, Indicates the new position of the point after moving l times; In the rotation task, build a dense point set The calculation expression is as follows: Where ε represents the three-dimensional ellipsoid, Φ(ε) represents the projection point of the surface point of the three-dimensional ellipsoid ε on the two-dimensional image; Compute dense point trajectories in dense point sets The specific expression is as follows: Among them, p ε It is used to construct the three-dimensional ellipsoid ε and satisfies the point set of the two-dimensional projection The three-dimensional point set, p ε,j Indicates p ε The jth point in Φ(p ε,j ) indicates p ε,j The two-dimensional projection of the initial position of the point on the image, d j Indicates p ε,j The geometric displacement after rotation, Φ(d j ) represents the geometric displacement d j The two-dimensional projection on the image, Indicates the new position of the point on the image after being rotated l times; In non-rigid transformation tasks, building dense point sets The calculation expression is as follows: in, represents the selected set of boundary points; Compute dense point trajectories in dense point sets The specific expression is as follows: in, Represents a point set The initial position of the jth point in , Indicates the end position of the point.

4. The anti-ambiguity point-oriented general interactive image editing method according to claim 3, characterized in that: The step of performing adaptive motion supervision strategy processing on the input image comprises: S21. Convert the input image into a series of latent feature maps {z t }, where 0≤t≤T-1, t represents any denoising time step; S22. The potential feature map {z t } to update and obtain the potential feature update graph n represents any optimization step after a small range of movement on the dense point trajectory; S23. Update the latent feature map Input pre-trained diffusion model, output dense point latent features 5. The anti-ambiguity point-oriented general interactive image editing method according to claim 4, characterized in that: The latent feature update graph The calculation expression is as follows: Among them, ρ represents the parameter of the image editing task characteristics, represents the potential feature map after a small range of movement on the dense point trajectory, b n Represents potential feature bias for optimization.

6. The anti-ambiguity point-oriented general interactive image editing method according to claim 5, characterized in that: The diffusion model includes a first UNet network and a second UNet network in parallel.

7. The anti-ambiguity point-oriented general interactive image editing method according to claim 6, characterized in that: The diffusion model is fine-tuned in a low-rank adaptive manner, including: S231. Update the potential feature map The first UNet network and the second UNet network are input respectively, and the first UNet network outputs the first feature And the second UNet network outputs the second feature S232. Introducing optimizable weight graphs n , the first feature And the second feature Combine to get the potential features of dense points The calculation expression is as follows: Where ⊙ represents the Hadamard product, s n The spatial resolution and The spatial resolution is the same.

8. The anti-ambiguity point-oriented general interactive image editing method according to claim 7, characterized in that: Using loss function The potential features of the dense points are optimized, and the loss function The calculation expression is as follows: Among them, the first loss term It is used to measure the difference between the original feature and the feature after moving along the dense point trajectory. The second loss term Represents the corresponding feature difference calculated before and after each step of motion supervision, the third loss term represents the feature deviation of the latent feature map outside the target mask, β represents a task-specific parameter, and λ represents the same value for all tasks.

9. The anti-ambiguity point-oriented general interactive image editing method according to claim 8, characterized in that: The first loss item The calculation expression is as follows: in, represents the image features of the dense points located at the position l+1 in the dense trajectory extracted by linear interpolation on the feature map, The image features representing the dense points at the initial positions extracted in the same manner; The second loss item The calculation expression is as follows: in, Represents the image features of the dense points located at l in the dense trajectory; The third loss item The calculation expression is as follows:

10. A general interactive image editing system based on anti-ambiguity points, characterized in that: include: A calculation module, used for defining an image editing task type, constructing a dense point set based on the image editing task type, and calculating a dense point trajectory in the dense point set; A processing module, used for performing adaptive motion supervision strategy processing on the input image to obtain dense point potential features; An optimization module, used for optimizing the potential features of the dense points when the dense points move along the dense point trajectory, so as to obtain the potential optimized features of the dense points; The output module is used to decode the potential optimization features of the dense points and output the edited image.

Images