A method and system for editing and propagating based on deep similarity

Abstract

The present application discloses an edit propagation method and system based on deep similarity, the method comprising: performing Euclidean distance transformation on user interaction information to obtain a distance map, wherein the distance map is used to describe the distance relationship between each pixel point and the interaction point in the image; using the transformed distance map and the original image as inputs of the constructed neural network, training the network model, and obtaining the model parameters of the neural network model; inputting the distance map converted from the image to be processed and the user interaction information into the trained neural network model to perform feedforward neural network calculation to obtain a probability map; performing global energy optimization on the probability map for edit propagation. The present application solves at least one of the problems existing in the edit propagation method in the prior art, thereby achieving better results in grayscale image colorization and instance segmentation.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular, to an editing and propagation method and system based on depth similarity. Background Art

[0002] With the development of digital multimedia hardware and the rise of software technology, the demand for image color processing is growing, and fast and efficient image color processing on display devices has become particularly important. Edit propagation is a technology that propagates user interaction information to the entire image based on the sparse interactive pixels provided by the user in the image by utilizing the similarity of image features such as color, texture, and spatial coordinates. At present, this technology has been widely used in image chromaticity adjustment, image segmentation, grayscale image colorization, and color image recoloring. Edit propagation can be regarded as a multi-classification problem. According to the image feature vector X of image pixel i, i , estimate its probability vector y i ∈[0,1] n , that is, the possibility that pixel i belongs to n types of labels. There are many methods used to solve the edit propagation problem, such as global energy optimization, smooth function interpolation, manifold preservation and probability model, etc. These methods not only obtain high-quality image editing results, but also improve the efficiency of edit propagation.

[0003] With the development of deep learning technology, deep neural networks (DNNs) have been widely used in tasks such as image semantic classification, target object detection, and scene image semantic segmentation, and have achieved breakthrough results. This is because of DNN's supervised end-to-end layer-by-layer feature self-learning method and its powerful feature expression ability. In recent years, researchers have proposed many DNN-based image color editing processing methods. After combining with deep neural networks, the key to the example-based image color editing processing method is: first, according to the large-scale image database provided, the parameters of the network model are obtained by training the designed deep neural network, that is, building the network model. After the training is completed, the user only needs to use the image to be processed as the input of the deep neural network, and the image color can be edited quickly by performing the forward calculation of the network, and the color editing effect is significantly improved. The image color editing processing method based on user interaction uses the original image and the user interaction image as the input of the network and trains the network model; the trained network model can propagate the color information of the user interaction to the entire image, thereby obtaining the final color editing result.

[0004] Endo et al. (Endo Y, Iizuka S, Kanamori Y, et al. DeepProp: extracting deep features from a single image for edit propagation [J]. Computer Graphics Forum, 2016) used a neural network to learn and extract deep features from the input image for edit propagation, which can automatically adjust the feature weights according to the input image, avoiding the process of manually adjusting parameters in traditional methods. The network of this method consists of two sub-networks, the visual feature extraction layer and the spatial feature extraction layer, which respectively learn the deep visual features and spatial features of the input image. The input image is output through the network model to obtain a probability map, thereby finally obtaining an ideal colorization result.

[0005] Gui et al. (Yan Gui, Guang Zeng. Joint learning of visual and spatial features for edit propagation from a single image. Computer Graphics Forum, 2019) converted user interaction information into a distance map and concatenated it with the original image on the channel to generate a multi-channel image. Multi-channel image sub-blocks are randomly extracted according to the interacted pixels. The extracted image sub-blocks contain both appearance features and spatial feature information, and these sub-blocks are used for network training. The network learns to extract deep features from these sub-blocks. The learned deep neural network model is used as a classifier to estimate the probability value of each pixel in the image belonging to different categories. Finally, the probability map is further optimized after conditional random field (CRF) post-processing, and high-quality editing results are finally obtained.

[0006] Edit propagation methods based on single images can handle images with simple backgrounds well, but when dealing with images with complex scenes such as object occlusion and small objects, such methods usually have difficulty in accurately estimating the shape and position of the target object, which can easily lead to inaccurate editing results. In addition, since the similarity measurement between image pixels depends on low-level image features (such as color, texture, and pixel coordinates), traditional methods usually need to select appropriate image features and manually adjust the weights (importance) of the features, which takes a lot of computing time and occupies computer memory resources. This cannot provide users with a convenient and fast usage experience, hindering the practical application of edit propagation methods.

[0007] The method described in the literature (Endo Y, Iizuka S, Kanamori Y, et al. DeepProp: extracting deepfeatures from a single image for edit propagation [J]. Computer Graphics Forum, 2016) is not suitable for processing situations where edit propagation relies more on spatial features due to the different depths of the two sub-networks and the order of training. Although better editing results than traditional methods can be achieved without manually adjusting parameters, the number of user interactions provided has not decreased. In addition, the network must be retrained every time a new image is tested. Since the initial parameters of the network are random, the results of multiple training and predictions may be different, and the results of multiple predictions vary greatly, which may be better or worse.

[0008] The method described in the literature (Yan Gui, Guang Zeng. Joint learning of visual and spatial features for edit propagation from a single image. Computer Graphics Forum, 2019) only uses some multi-channel image sub-blocks for training. Although they contain spatial information and appearance information, these are local information and cannot make good use of local and global information. A large amount of user interaction is still required to obtain good editing results. On the other hand, this method has the problem of needing to retrain the network when testing new images. Similarly, since the network initialization is randomly generated, the model after training is uncertain, resulting in different image editing results under the same input.

[0009] The methods in the above literature have advantages and disadvantages, and a new solution needs to be proposed. Summary of the invention

[0010] The embodiments of the present application provide an edit propagation method and system based on depth similarity to at least solve at least one of the problems existing in the edit propagation method in the prior art.

[0011] According to one aspect of the present application, there is provided an edit propagation method based on deep similarity, comprising: performing a Euclidean distance transformation on user interaction information to obtain a distance map, wherein the distance map is used to describe the distance relationship between each pixel point and the interaction point in the image; using the transformed distance map and the original image as inputs of a constructed neural network, training the network model, and obtaining model parameters of the neural network model; inputting the distance map converted from the image to be processed and the user interaction information into the trained neural network model to perform a feedforward neural network calculation to obtain a probability map; and performing global energy optimization on the probability map for edit propagation.

[0012] Furthermore, before using the converted distance map and the original image as inputs of the constructed neural network and training the network model, it also includes: constructing the neural network model using a multi-scale feature extraction module, a target area positioning module and a multi-category classification model, wherein the neural network model is a deep neural network model.

[0013] Furthermore, the multi-scale feature extraction model is used to extract deep features of the image, and constructing the neural network model using the multi-scale feature extraction module, the target area positioning module and the multi-category classification model includes: inputting the image into the multi-scale feature extraction model to obtain the deep features output by the multi-scale feature extraction model; processing the deep features and the distance map through the target area positioning model to obtain a feature sequence; and using the multi-category classification model to dynamically classify the feature sequence.

[0014] Furthermore, the converted distance map and the original image are used as inputs of the constructed neural network, and the training network model includes: using images in the PascalVOC 2012 data set and the distance maps corresponding to these images as training data to train the neural network model.

[0015] Furthermore, training the constructed neural network model includes: using a stochastic gradient descent algorithm to perform back propagation to train the neural network model.

[0016] According to another aspect of the present application, there is also provided an editing propagation system based on deep similarity, including: a transformation module, used to perform Euclidean distance transformation on user interaction information to obtain a distance map, wherein the distance map is used to describe the distance relationship between each pixel point in the image and the interaction point; a training module, used to use the transformed distance map and the original image as inputs of a constructed neural network, train the network model, and obtain model parameters of the neural network model; a processing module, which inputs the distance map converted from the image to be processed and the user interaction information into the trained neural network model to perform feedforward neural network calculation to obtain a probability map; and an editing propagation module, used to perform global energy optimization on the probability map for editing propagation.

[0017] Furthermore, the training module is used to construct the neural network model using a multi-scale feature extraction module, a target area positioning module and a multi-category classification model, wherein the neural network model is a deep neural network model.

[0018] Furthermore, the constructed training module is used to: input the image into the multi-scale feature extraction model to obtain the depth features output by the multi-scale feature extraction model; process the depth features and the distance map through the target area positioning model to obtain a feature sequence; and use the multi-category classification model to dynamically classify the feature sequence.

[0019] Furthermore, the training module is used to train the neural network model using images in the PascalVOC 2012 dataset and distance maps corresponding to these images as training data.

[0020] Furthermore, the training module is constructed to: use a stochastic gradient descent algorithm to perform back propagation to train the neural network model.

[0021] In the embodiment of the present application, a distance map is obtained by performing Euclidean distance transformation on user interaction information, wherein the distance map is used to describe the distance relationship between each pixel point in the image and the interaction point; the transformed distance map and the original image are used as inputs of the constructed neural network to train the network model and obtain the model parameters of the neural network model; the distance map converted from the image to be processed and the user interaction information is input into the trained neural network model to perform feedforward neural network calculation to obtain a probability map; the probability map is globally energy optimized for edit propagation. The present application solves at least one of the problems existing in the edit propagation method in the prior art, thereby achieving better results in grayscale image colorization and instance segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0023] Figure 1 is a schematic diagram of a framework diagram according to an embodiment of the present application;

[0024] Figure 2 is a schematic diagram of interaction points according to an embodiment of the present application;

[0025] Figure 3 is a schematic diagram of a distance diagram according to an embodiment of the present application;

[0026] Figure 4 is a schematic diagram of a target area positioning module according to an embodiment of the present application;

[0027] Figure 5 is a schematic diagram of a multi-category and multi-classification module according to an embodiment of the present application;

[0028] Figure 6 is a schematic diagram of an image editing result according to an embodiment of the present application;

[0029] Figure 7 It is a flowchart of the editing propagation method based on deep similarity according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0031] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0032] In this embodiment, an edit propagation method based on depth similarity is provided. Figure 7 is a flowchart of an edit propagation method based on deep similarity according to an embodiment of the present application. Figure 7 As shown, the process includes the following steps:

[0033] Step S702, performing Euclidean distance transformation on the user interaction information to obtain a distance map, wherein the distance map is used to describe the distance relationship between each pixel point in the image and the interaction point;

[0034] Step S704, using the converted distance map and the original image as inputs of the constructed neural network, training the network model, and obtaining model parameters of the neural network model;

[0035] In this step, the neural network model can be constructed using a multi-scale feature extraction module, a target region positioning module and a multi-category classification model, wherein the neural network model is a deep neural network model. For example, the image can be input into the multi-scale feature extraction model to obtain the deep features output by the multi-scale feature extraction model; the deep features and the distance map are processed by the target region positioning model to obtain a feature sequence; and the feature sequence is dynamically classified using the multi-category classification model.

[0036] The neural network model can be trained using images in the PascalVOC 2012 dataset and the distance maps corresponding to these images as training data. During training, the neural network model can be trained by back propagation using a stochastic gradient descent algorithm.

[0037] Step S706 is to input the distance map converted from the image to be processed and the user interaction information into the trained neural network model to perform feedforward neural network calculation to obtain a probability map;

[0038] Step S708: performing global energy optimization on the probability map to perform edit propagation.

[0039] The above steps solve at least one of the problems existing in the editing and propagation methods in the prior art, thereby achieving better results in grayscale image colorization and instance segmentation.

[0040] An embodiment is described below, which provides an edit propagation method and system based on deep similarity. In this embodiment, a distance map is first generated using Euclidean distance transformation based on the provided user interaction information to describe the spatial distance relationship between each pixel and various sub-points in the image, thereby accurately describing the object of interest. Then, a deep neural network is constructed from the multi-scale feature extraction module, the target area positioning module, and the multi-category classification module, and the stochastic gradient descent algorithm (SGD) is used for back propagation to train the network model to obtain the model parameters. By using the learned network model for feedforward neural network calculations, the probability map corresponding to various tags can be output. Finally, the global energy optimization is solved for edit propagation, so that the color editing processing of the image is finally used.

[0041] Figure 1 is a schematic diagram of a framework diagram according to an embodiment of the present application, Figure 1 It is a principle framework of editing propagation method and system solution based on deep similarity, such as Figure 1As shown, in this framework, a multi-scale feature acquisition module, a target area positioning module and a multi-category classification module are included. In this framework, the PascalVOC 2012 data set, simulated user interaction and distance map generation are also included. Then, in this framework, an edit propagation network model based on deep similarity is trained, a probability map is estimated, and finally, color edit propagation is performed. Based on the above framework, the steps of edit propagation in this embodiment can be determined as follows:

[0042] Step 1: Generate distance map

[0043] In order to simulate user interaction, this embodiment randomly selects m interaction points for each target object area in the image, and the interaction points of different target objects are distinguished by different colors, such as Figure 2 Specifically, the user interaction information is represented as S = {S t |t=1,...,k}, where S t is the set of interaction points of the tth target object; k is the total number of interacted target objects, excluding the background. t Convert to Euclidean distance graph D t , the specific calculation is as follows:

[0044]

[0045] Then, the background distance map D is calculated based on these foreground distance maps bg , the specific calculation is as follows:

[0046] D bg =255-min(D 1 ,D 2 ,...,D k ) (2)

[0047] Among them, p ij Represents the coordinates of the pixel.

[0048] These generated distance maps are all two-dimensional matrices of the same size as the input image, such as Figure 3 As shown. In the generated distance map, if the surrounding area of ​​a pixel is darker, it means that the pixel is closer to the interactive pixel; conversely, the surrounding area of ​​the pixel farther from the interactive pixel is brighter. The value range of the minimum spatial distance constrained by this embodiment is between 0 and 255. In addition, this embodiment will calculate the distance map corresponding to each type of interactive point set, and these distance maps will be used to describe different target objects marked in the image.

[0049] Step 2: Construct a multi-scale feature extraction module

[0050] This embodiment of the patent uses a high-resolution network (HRNet) to extract multi-scale features from a single input image. The network contains four parallel branches of different resolutions, each of which consists of residual blocks. Specifically, an image I of size H×W is used as the input of the module, and four feature maps of size H / 4×W / 4×48, H / 8×W / 8×96, H / 16×W / 16×192 and H / 32×W / 32×384 are output. This embodiment of the patent upsamples the output features of the other three branches to the resolution size of the output features of the first branch, and cascades these upsampled feature maps with the feature maps of the first branch, and finally outputs a deep feature of size H / 4×W / 4×720, denoted as f. The reason for this approach is that if the output features of the first branch are used alone, the network will lose the deep visual features of larger target objects. Therefore, cascading the output features of the other three low-resolution branches can retain useful spatial information and semantic information.

[0051] Step 3: Build the target area positioning module

[0052] This module locates the target area according to the distance map calculated in step one. The module consists of two branches. The first branch is composed of an embedding head consisting of 5 convolutional layers and a weight-sharing dynamic fusion head consisting of 3 convolutional layers. The size of the 1st, 2nd and 4th convolutional filters in the embedding head is 3×3, and the size of the convolutional filters in the other two layers is 1×1, with a step size of 1 pixel. The size of the two convolutional filters in the dynamic fusion head is 3×3, and the size of the convolutional filter in the last layer is 1×1, with a step size of 1 pixel. The second branch consists of two 3×3 convolutional filters with a step size of 1 and one 1×1 convolutional filter. The structure of this module is as follows: Figure 4 shown.

[0053] Therefore, the deep feature f extracted by the multi-scale feature extraction module is used as input, and the embedding head of the first branch outputs an embedded feature map e of H / 4×W / 4×128. The output embedded features are cascaded with k+1 distance maps. Since the calculated distance map is the same size as the original image, it is necessary to first downsample the distance map to the size of the embedded feature, and then cascade it with the embedded feature to obtain k+1 feature sequences of size H / 4×W / 4×129. After passing through the weight-sharing dynamic fusion head in sequence, the branch outputs k+1 deep feature sequences of size H / 4×W / 4×128, which integrate the appearance and spatial position information of the target object to be edited. In addition, the deep feature f is also used as the input of the second branch, and the feature map d of size H / 4×W / 4×64 is output.

[0054] By concatenating the output features of the first branch with the output features of the second branch, we finally get a feature sequence {si} i＝1,...,k+1 , obtain better semantic and boundary contour information, and supplement the missing information of the dynamic fusion head output features.

[0055] Step 4: Build a multi-category classification module

[0056] This module can dynamically classify according to the output features of the target region positioning module. It consists of a convolutional long short-term memory unit (Convolutional Long Short-term Memory Unit, ConvLSTM) and a convolution layer with a convolution filter size of 1×1. The convolutional long short-term memory unit has a memory function and can produce outputs of variable length. For the target region i, this module takes the feature s i As input, it outputs a probability estimation map of size H / 4×W / 4×(k+1), such as Figure 5 Specifically, given feature s i and the previous feature s i-1 The hidden state h i-1 , the memory unit outputs a new hidden state h i :

[0057] h i =ConvLSTM(s i |h i-1 ) (3)

[0058] When i is 1, h0 is a zero vector matrix. i It can encode the appearance and location information of previously marked targets and provide sufficient target prior information.

[0059] A=Softmax([conv(h1),...,conv(h k+1 )]) (4)

[0060] These hidden states After the same 1×1 convolutional layer is used for dimensionality reduction, a single-channel logical feature map is obtained. These single-channel feature maps are stacked and processed by a softmax layer to obtain a probability estimation map A corresponding to each category label (user interaction).

[0061] Step 5: Network training

[0062] This example trains the network on the PascalVOC 2012 dataset and optimizes the parameters of the entire network by minimizing the following loss function:

[0063] L=L embedding +λL cls (5)

[0064] Among them, L embedding represents the binary cross entropy loss based on the similarity of embedded features, L cls is the multi-classification cross entropy loss.

[0065] Where L embedding The loss function is defined as follows:

[0066]

[0067]

[0068] Among them, S is a set of k pixels randomly selected in each instance object area in the image, and y p Represents the instance label of the pixel. σ(p,q) calculates the similarity between two pixels; in the embedded feature space, pixels belonging to the same instance have high similarity, and σ(p,q) is 1. Pixels between different instances have low similarity, and σ(p,q) is 0.

[0069] Finally, the stochastic gradient descent (SGD) algorithm is used for error back propagation, and the loss function L is minimized through supervised training. The network training is initially set to 0, and the embedding space features are learned first. After learning for 10 epochs, λ is set to 1 and the step 4 module is added for training. The whole process uses the Adam optimizer, the learning rate is set to 1e-3, and the learning rate is reduced by one tenth every 50 epochs, and training is performed for 200 epochs.

[0070] Step 5: Color Editing and Spreading

[0071] The trained network model is used as a classifier. Given any image to be edited and user interaction information, the network model outputs a probability map corresponding to each color label category. Each element in the probability graph It describes the possibility that the pixel belongs to the current color label category. In this embodiment, the color editing propagation problem based on user interaction is regarded as a global energy optimization problem: E(C) = ∑ i∈v D(c i )+λ·∑ (i,j)∈ε V(c i ,c j ), where λ is the balance data item D(l i ) and the smoothing term V(l i ,l j ) between the weight coefficients; C = {c1,...,c k} is the set of color label categories. Finally, the multi-label graph cutting is used to minimize the energy function to assign the correct color label category to each pixel, thus finally obtaining the color editing processing result.

[0072] Different from previous methods, this embodiment introduces user interaction information, namely the distance map, in the middle. Users only interact with foreground information, and background information is calculated based on the distance map converted from the interactive foreground information. Secondly, the target area positioning module can roughly locate the foreground based on the distance map, and then obtain the multi-classification result through the memory function of the convolutional long short-term memory unit (ConvLSTM) in the multi-classification module. This can handle a variable number of user interactions and respond to user interaction graphs. Importantly, the network proposed in this embodiment does not need to be retrained when testing new images after supervised training on the data set, and the network model is directly used for image color editing.

[0073] In this embodiment, an electronic device is provided, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to execute the method in the above embodiment.

[0074] The above program can be run in the processor, or it can be stored in the memory (or computer-readable medium), which includes permanent and non-permanent, removable and non-removable media. Information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0075] These computer programs can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps of the functions specified in one or more blocks can be implemented by different modules corresponding to different steps.

[0076] Such a device or system is provided in this embodiment. The system is called an editing propagation system based on deep similarity, including: a transformation module, which is used to perform Euclidean distance transformation on user interaction information to obtain a distance map, wherein the distance map is used to describe the distance relationship between each pixel point and the interaction point in the image; a training module, which is used to use the transformed distance map and the original image as the input of the constructed neural network, train the network model, and obtain the model parameters of the neural network model; a processing module, which has the function of inputting the distance map converted from the image to be processed and the user interaction information into the trained neural network model to perform feedforward neural network calculation to obtain a probability map; an editing propagation module, which is used to perform global energy optimization on the probability map for editing propagation.

[0077] The system or device is used to implement the functions of the method in the above-mentioned embodiment. Each module in the system or device corresponds to each step in the method, which has been explained in the method and will not be repeated here.

[0078] For example, the construction training module is used to: construct the neural network model using a multi-scale feature extraction module, a target area positioning module and a multi-category classification model, wherein the neural network model is a deep neural network model. Optionally, the construction training module is used to: input the image into the multi-scale feature extraction model to obtain the deep features output by the multi-scale feature extraction model; process the deep features and the distance map through the target area positioning model to obtain a feature sequence; and dynamically classify the feature sequence using the multi-category classification model.

[0079] For another example, the training module is used to train the neural network model using images in the PascalVOC 2012 dataset and distance maps corresponding to these images as training data. Optionally, the training module is used to train the neural network model using a stochastic gradient descent algorithm for back propagation.

[0080] Compared with the prior art, the technical solution proposed in this embodiment uses a deep network to extract image depth features, generates distance maps of corresponding color categories through a small amount of foreground interaction information, and calculates the distance map of the background based on these distance maps. According to the trained network model, the color probability distribution of each pixel in the image can be accurately estimated, and by combining the global energy optimization solution based on graph cutting, high-quality and fast image color editing processing can be finally achieved. Importantly, the network after training in this embodiment does not need to be retrained during testing, and good results can also be achieved in the direction of grayscale image colorization and instance segmentation.

[0081] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A deep similarity-based edit propagation method, characterized in that: include: The user interaction information is subjected to Euclidean distance transformation to obtain a distance map, wherein the distance map is used to describe the distance relationship between each pixel point and the interaction point in the image; wherein a plurality of interaction points are randomly selected for each target object region in the image, and the interaction points of different target objects are distinguished by different colors, wherein the user interaction information is represented as S={S t |t=1,...,k}, where S t is the set of interaction points of the tth target object; k is the total number of interacted target objects, excluding the background; the user interaction S of the target object t Convert to Euclidean distance graph D t , calculated as follows: The background distance map D is calculated based on these foreground distance maps bg , calculated as follows: D bg =255-min(D 1 ,D 2 ,...,D k ); Among them, p ij Indicates the coordinates of the pixel point; p mn ∈S t is a subset of interaction points; these generated distance maps are two-dimensional matrices of the same size as the input image; Using the converted distance map and the original image as inputs of the constructed neural network, training the network model, and obtaining model parameters of the neural network model; Inputting the distance map converted from the image to be processed and the user interaction information into the trained neural network model to perform feedforward neural network calculation to obtain a probability map; performing global energy optimization on the probability map for edit propagation; Wherein, before using the converted distance map and the original image as the input of the constructed neural network and training the network model, it also includes: using a multi-scale feature extraction module, a target area positioning module and a multi-category classification module to construct the neural network model, wherein the neural network model is a deep neural network model; the multi-scale feature extraction module is used to extract the deep features of the image, and using the multi-scale feature extraction module, the target area positioning module and the multi-category classification module to construct the neural network model includes: inputting the image into the multi-scale feature extraction module, obtaining the deep features output by the multi-scale feature extraction module The invention relates to a method for extracting multi-scale features from a single input image by using a high-resolution network, wherein the network comprises four parallel branches of different resolutions, each of which is composed of a residual block, wherein an image I of size H×W is used as an input of the module, and an output size of H / 4×W / 4×48, H / 8×W / 8×96, H / 16×W / 16×192 and H / 32×W / 32× 384 four feature maps; upsample the output features of the other three branches to the resolution size of the output features of the first branch, and cascade these upsampled feature maps with the feature map of the first branch, and output a depth feature of size H / 4×W / 4×720, denoted as f; the target area positioning module uses the calculated distance map to locate the target area, wherein the target area positioning module is composed of two branches, the first branch is composed of an embedding head composed of 5 convolutional layers and a weight-sharing dynamic fusion head composed of 3 convolutional layers, the size of the 1st, 2nd and 4th convolutional filters in the embedding head is 3×3, and the The convolution filters of the other two layers are 1×1 in size, and the step size is 1 pixel width; the two convolution filters in the dynamic fusion head are 3×3 in size, and the convolution filter size of the last layer is 1×1, and the step size is 1 pixel width; the second branch is composed of two 3×3 convolution filters with a step size of 1 and a 1×1 convolution filter; the multi-category classification module performs dynamic classification according to the output features of the target area positioning module, and the multi-category classification module is composed of a convolutional long short-term memory unit and a convolution layer with a convolution filter size of 1×1. For the target area i, the multi-category classification module uses the feature s i As input, it outputs a probability estimation map of size H / 4×W / 4×(k+1).

2. The method according to claim 1, characterized in that Using the converted distance map and the original image as the input of the constructed neural network, training the network model includes: The neural network model is trained using images in the PascalVOC 2012 dataset and the distance maps corresponding to these images as training data.

3. The method according to claim 1, characterized in that Training the constructed neural network model includes: The neural network model is trained by back propagation using a stochastic gradient descent algorithm.

Images