A fast image editing method based on region

Abstract

The present invention discloses a region-based fast image editing method, comprising the following steps: step 1, a user selects a handle region H and a target region T on an image, wherein the handle region H represents a portion that the user wants to drag, and the target region T represents a desired position; step 2, a dense mapping is established between the handle region H and the target region T through a region-to-point mapping algorithm; step 3, a potential representation z in the handle region H is mapped to a region; and step 4, a potential representation z in the handle region H is mapped to a region. H Copy to the target area T and obtain the copied potential representation z T ; Step 4, the potential representation z after copying T Perform denoising operation; Step 5, the potential representation z after denoising new The decoder needs to convert the image back to complete the image editing. The region-based fast image editing method of the present invention allows the user to use the handle region H and the target region T to express the editing instruction through the settings of steps 1 to 5, thereby achieving more precise control and reducing ambiguity.

Description

Technical Field

[0001] The present invention relates to the field of computer image processing, and more particularly to a region-based rapid image editing method. Background Art

[0002] Image editing technology is the technology of using computers to modify and edit image information in detail. Most image editing technologies are based on image generation technology. Image generation technology is the technology of using computer programs to generate images. This technology can be used to generate various types of images, including digital art, animation, game scenes, product design, etc.

[0003] Most of the existing image editing technologies either cannot achieve precise control, such as using StyleGAN's intermediate latent vectors for direct interpolation processing, which can only change the style feature attributes of certain features, and cannot completely decouple the control, and the control is not accurate and reliable. Or it is DragGAN, which is based on the powerful discrimination ability of the intermediate features of the StyleGAN generator, and is highly associated with the generated image in both semantic and spatial dimensions. The w vector is optimized by the loss of the target point neighborhood and the handle point neighborhood of the intermediate features of the supervisory generator, and the final optimized w vector is passed through the StyleGAN generator to obtain the drag-edited image. Although DragGAN obtains accurate pixel-level drag editing effects, the optimization algorithm takes a long time in actual application scenarios, and the final result of the algorithm based on local optimization is not necessarily semantic, and cannot distinguish editing movements or editing deformations well. For example, large-scale dragging should lead to movement, but large deformations occur.

[0004] Therefore, in the prior art, there is an invention patent with publication number CN117541681A, entitled Image Editing Method, Device and Electronic Device Based on Deep Feature Generation, which discloses a more efficient, accurate and semantic interactive image editing based on "point dragging" by leveraging the powerful discrimination ability of the intermediate features of the generator of the StyleGAN pre-trained model and the powerful sequence modeling and prediction ability of the Transformer model structure. However, the interactive image editing method based on "point dragging" needs to infer the changes of the entire image from the movement of a single or a few points, which leads to a large computational overhead. At the same time, the point instructions are essentially ambiguous, and one drag operation may correspond to multiple possible editing effects, which leads to the problem of misalignment between user intentions and model outputs. Summary of the invention

[0005] In view of the shortcomings of the prior art, an object of the present invention is to provide a region-based fast image editing method, which is faster, more accurate and more in line with user intentions.

[0006] To achieve the above object, the present invention provides the following technical solution: a region-based fast image editing method, characterized in that it comprises the following steps:

[0007] Step 1: The user selects a handle area H and a target area T on the image, where the handle area H represents the part that the user wants to drag, and the target area T represents the desired location;

[0008] Step 2: Establish a dense mapping between the handle region H and the target region T through the region-to-point mapping algorithm;

[0009] Step 3: transform the potential representation z in the handle region H H Copy to the target area T and obtain the copied potential representation z T ;

[0010] Step 4: The copied potential representation z T Perform denoising operation;

[0011] Step 5: De-noised potential representation z new It needs to be converted back into an image through a decoder to complete image editing.

[0012] As a further improvement of the present invention, the specific steps of establishing dense mapping in step 2 are as follows:

[0013] Step 21: Sample point set P from handle region H H ;

[0014] Step 22: Sample point set P from target area T T ;

[0015] Step 2 and 3: Calculate the point set P H To P T The mapping relationship is established to complete the establishment of dense mapping.

[0016] As a further improvement of the present invention, in step 21, the sampling point set P is obtained from the handle area H. H The specific method is: Assuming H is an n×n rectangular area, the point set P can be uniformly sampled according to the grid structure H :

[0017] P H ={(x i ,y i )|x i =i·Δx,y i=i·Δy,i,j∈[0,n-1]};

[0018] Where Δx and Δy are the sampling intervals.

[0019] As a further improvement of the present invention, in step 22, a sampling point set P is obtained from the target area T. T The specific method is: Assuming that T is also an m×m rectangular area, the point set P can be uniformly sampled according to the grid structure T :

[0020] P T ={(x i ,y i )|x i =i·Δx,y i =i·Δy,i,j∈[0,m-1]};

[0021] Where Δx and Δy are the sampling intervals.

[0022] As a further improvement of the present invention, the point set P is calculated in steps 2 and 3. H To P T The specific mapping relationship is as follows: for each sampling point p in the handle area H H ∈P H , find the most similar point p T ∈P T Specifically, the handle region point P is calculated by the nearest neighbor method H and all points p in the target area T The Euclidean distance between them is calculated and the point with the smallest distance is selected as the mapping target:

[0023]

[0024] As a further improvement of the present invention, the potential representation z in step 3 H Calculated by the following formula:

[0025] z H =g encode (I H )

[0026] Among them, g encode is the image encoder, I H is the image block within the handle area H.

[0027] As a further improvement of the present invention, the potential representation z after copying in step 3 T Calculated by the following formula:

[0028]

[0029] Among them, z init is the potential representation of the original image.

[0030] As a further improvement of the present invention, the denoising operation in step 4 is implemented by the following formula:

[0031] z new =δ t (z orig -λ s ·φ θ (z orig , s, C))+η t ·φ θ (z orig , s, C)+σ s ·ω s

[0032] Among them, z orig represents the potential representation of the image, z new represents the potential representation of the newly generated image, that is, the potential representation after denoising, δ t and λ s is the scaling factor, φ θ is the improved denoising network, C is the conditional input, η t is the harmonic coefficient, σ s is the noise intensity, ω s Noise vector.

[0033] As a further improvement of the present invention, the specific steps of converting back to an image through a decoder in step 5 are as follows:

[0034] Step 5.1: Initialize the decoder. Assume the decoder is g decode ;

[0035] Step 52, through the decoder g decode The denoised potential representation z new Decoded into image I new .

[0036] The beneficial effects of the present invention are as follows: by introducing a region-based editing method, the present invention provides an image editing solution that is faster, more accurate and more in line with user intentions. This region selection method can express richer editing intentions and reduce the sparsity and ambiguity of instructions in the point-drag-based method. In addition, the present invention is also committed to generating images of different states in a data set through this technology, thereby improving the richness and diversity of the data set. DETAILED DESCRIPTION

[0037] The present invention will be further described in detail with reference to the given embodiments below.

[0038] The region-based fast image editing method of this embodiment allows the user to use the handle region and the target region to express the editing instruction, thereby achieving more precise control and reducing ambiguity, and includes the following steps:

[0039] 1. The user selects the handle area and the target area

[0040] The user selects a handle region (H) and a target region (T) on the image. The handle region H represents the part that the user wants to drag, and the target region T represents the desired location. This region selection method can express richer editing intent and reduce the sparsity and ambiguity of instructions in the point-based drag method.

[0041] 2. Create dense mapping

[0042] Through the region-to-point mapping algorithm, a dense mapping is established between the handle region H and the target region T.

[0043] The specific steps are as follows:

[0044] ① Sample point set P from handle region H H

[0045] First, sample the point set P from the handle region H H These points can be evenly distributed in the handle area to ensure that the features of the entire area are covered. Sampling methods can include random sampling or grid-based regular sampling. For example, assuming H is an n×n rectangular area, the point set P can be evenly sampled according to the grid structure. H :

[0046] P H ={(x i ,y i )|x i =i·Δx,y i =i·Δy,i,j∈[0,n-1]}

[0047] Where Δx and Δy are the sampling intervals.

[0048] ② Sample point set P from the target area T T

[0049] Similarly, the point set P is sampled from the target area T T These point sets can be evenly distributed in the target area to ensure that the features in the handle area can be accurately mapped. The sampling method is the same as that of the handle area, and can be random sampling or regular sampling. For example, assuming that T is also an m×m rectangular area, the point set P can be evenly sampled according to the grid structure. T :

[0050] P T ={(x i ,y i )|x i =i·Δx,y i =i·Δy,i,j∈[0,m-1]}

[0051] Where Δx and Δy are the sampling intervals.

[0052] ③Calculate the point set P H To P T The mapping relationship

[0053] Calculate the point set P H To P T Specifically, for each sampling point p in the handle area, H ∈P H , find the most similar point p T ∈P T .

[0054] The nearest neighbor method calculates the handle region point P H and all points p in the target area T The Euclidean distance between them is calculated and the point with the smallest distance is selected as the mapping target:

[0055]

[0056] Through the above steps, it is ensured that during the editing process, each pixel in the handle area can find a corresponding match in the target area.

[0057] 3. Copying latent representations

[0058] The potential representation z in the handle region H H Copy to the target area T. The potential representation refers to the representation of the image in the latent space. The specific calculation formula is as follows:

[0059] z H =g encode (I H )

[0060] Among them, g encode is the image encoder, I H is the image block within the handle area H.

[0061] The replication formula of the latent representation is as follows:

[0062]

[0063] Among them, z init is the potential representation of the original image.

[0064] 4. Denoising process

[0065] The following method is used in the denoising process to improve the stability and effect of editing. T Perform denoising operation:

[0066] z new =δ t (z orig -λ s ·φ θ (z orig , s, C))+η t ·φ θ (z orig , s, C)+σ s ·ω s

[0067] Among them, z orig represents the potential representation of the image, z new represents the potential representation of the newly generated image, that is, the potential representation after denoising, δ t and λ s is the scaling factor, φ θ is the improved denoising network, C is the conditional input (such as the handle area and the location information of the target), η t is the harmonic coefficient, σ s is the noise intensity, ω s Noise vector.

[0068] 5. Potential Representation Recovery

[0069] The latent representation z after denoising new It needs to be converted back to an image through a decoder. The specific steps are as follows:

[0070] ① Initialize the decoder

[0071] Assume the decoder is g decode , which is able to convert the latent representation back into an image.

[0072] ② Latent representation decoding

[0073] Through the decoder g decode The denoised potential representation z new Decoded into image I new .

[0074] I new =g decode (z new ).

[0075] The following examples are provided in this embodiment:

[0076] For example, the generation of switch state (on / off) images.

[0077] 1. Dataset preparation

[0078] First, an initial dataset containing switch state (on / off) images is selected. Assume that the number of images with the switch state "on" in the dataset is large, while the number of images with the state "off" is small.

[0079] 2. Use this method to edit images

[0080] To generate more images in the “off” state, we apply this method to edit the images in the “on” state in the initial dataset.

[0081] 3. Enhanced dataset

[0082] Through the above steps, the images in the initial data set with the state of "on" are edited to images with the state of "off", thereby enhancing the diversity and sample balance of the data set. The above example introduces how to use this method to edit the images in the data set to enhance the diversity and sample balance of the data set.

[0083] This is crucial for model training in machine learning and computer vision tasks, especially when there are few samples of certain states.

[0084] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A region-based fast image editing method, characterized in that: The steps include: Step 1: The user selects a handle area H and a target area T on the image, where the handle area H represents the part that the user wants to drag, and the target area T represents the desired location; Step 2: Establish a dense mapping between the handle region H and the target region T through the region-to-point mapping algorithm; Step 3: transform the potential representation z in the handle region H H Copy to the target area T and obtain the copied potential representation z T ; Step 4: The copied potential representation z T Perform denoising operation; Step 5: De-noised potential representation z new It needs to be converted back into an image through a decoder to complete image editing.

2. The region-based fast image editing method according to claim 1, characterized in that: The specific steps of establishing dense mapping in step 2 are as follows: Step 21: Sample point set P from handle region H H ; Step 22: Sample point set P from target area T T ; Step 2 and 3: Calculate the point set P H To P T The mapping relationship is established to complete the establishment of dense mapping.

3. The region-based fast image editing method according to claim 2, characterized in that: In step 21, a point set P is sampled from the handle region H. H The specific method is: Assuming H is an n×n rectangular area, the point set P can be uniformly sampled according to the grid structure H : P H ={(x i ,y i )|x i =i·Δx,y i =i·Δy,i,j∈[0,n-1]}; Where Δx and Δy are the sampling intervals.

4. The region-based fast image editing method according to claim 2 or 3, characterized in that: In step 22, a point set P is sampled from the target area T. T The specific method is: Assuming that T is also an m×m rectangular area, the point set P can be uniformly sampled according to the grid structure T : P T ={(x i ,y i )|x i =i·Δx,y i =i·Δy,i,j∈[0,m-1]}; Where Δx and Δy are the sampling intervals.

5. The region-based fast image editing method according to claim 2 or 3, characterized in that: In the steps 2 and 3, the point set P is calculated. H To P T The specific mapping relationship is as follows: for each sampling point p in the handle area H H ∈P H , find the most similar point p T ∈P T Specifically, the handle region point P is calculated by the nearest neighbor method H and all points p in the target area T The Euclidean distance between them is calculated and the point with the smallest distance is selected as the mapping target:

6. The region-based fast image editing method according to claim 1, 2 or 3, characterized in that: The potential representation z in step 3 H Calculated by the following formula: z H =g encode (I H ) Among them, g encode is the image encoder, I H is the image block within the handle area H.

7. The region-based fast image editing method according to claim 6, characterized in that: The potential representation z after copying in step 3 T Calculated by the following formula: Among them, z init is the potential representation of the original image.

8. The region-based fast image editing method according to claim 1, 2 or 3, characterized in that: The denoising operation in step 4 is implemented by the following formula: z new =d t (z orig -l s ·f θ (z orig ,s,C))+η t ·f θ (z orig ,s,C)+σ s ·oh s Among them, z orig represents the potential representation of the image, z new represents the potential representation of the newly generated image, that is, the potential representation after denoising, δ t and λ s is the scaling factor, φ θ is the improved denoising network, C is the conditional input, η t is the harmonic coefficient, σ s is the noise intensity, ω s Noise vector.

9. The region-based fast image editing method according to claim 1, 2 or 3, characterized in that: The specific steps of converting back to the image through the decoder in step 5 are as follows: Step 5.1: Initialize the decoder. Assume the decoder is g decode ; Step 52, through the decoder g decode The denoised potential representation z new Decoded into image I new .

Images