Interactive image segmentation method combining global seeds and sparse local linear reconstruction

Abstract

The application provides an interactive image segmentation method combining global seeds and sparse local linear reconstruction, comprising introducing a global seed information term, constructing a global seed information flow; designing a sparse local linear reconstruction regularization term, using L1 norm to realize the segmentation result piecewise smoothness, and combining the global seed information term to obtain a target function; using the ADMM mode to iteratively solve the target function, obtaining a more complete segmentation object while achieving a more sparse regional boundary segmentation effect; modifying the L1 norm to L p norm, converting the L p norm to the L1 norm by an iterative reweighting method to solve, obtaining the solution of the minimized target function; obtaining the final segmentation result by a threshold method from the solution of the minimized target function. The model constructed by the application is composed of the global seed information term and the sparse local linear reconstruction regularization term, which can not only retain complete semantic information, but also show sparse smoothness when processing complex images, and can obtain clearer segmentation boundaries.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of interactive image segmentation, in particular to an interactive image segmentation method combining global seeds and sparse local linear reconstruction. BACKGROUND

[0002] Image segmentation algorithm plays a vital role in the field of computer vision and pattern recognition, especially in the application of medical image segmentation, video processing, target tracking, etc., which promotes the rapid development of image segmentation technology and derives many different kinds of segmentation algorithms.

[0003] The existing interactive image segmentation algorithm can be divided into three categories: graph cut theory based algorithm, random walk based algorithm and deep learning based algorithm. Among them, the graph cut theory based algorithm is to transform the image segmentation task into the maximum flow / minimum cut problem in graph theory, and the final segmentation result is obtained by minimizing the graph energy function. After the user labels part of the pixels in the image as foreground and background, the random walk algorithm calculates the random walk probability of the labeled pixels to all unlabeled pixels, and the class of the labeled pixel with the maximum arrival probability is the class of the unlabeled pixel. The image segmentation algorithm based on deep learning can well express and depict the structure information of complex data through multi-layer network to realize the semantic segmentation of the target.

[0004] The main purpose of interactive image segmentation is to obtain complete and clear boundary objects with less user interaction prior information. However, due to the objective differences between users, the prior information obtained by the algorithm is often quite different. How to achieve better segmentation performance with less interaction information is still a problem to be solved. Most of the existing algorithms are based on the principle of local linear reconstruction, which believes that each pixel point can be linearly represented by the neighborhood pixel points. The higher the similarity between the neighborhood pixel points, the greater the possibility of label consistency, that is, the pixel label is only determined by the similarity between the neighborhood pixels. However, there are differences between the texture information or color information of the same target and different targets on the image. When measuring the similarity between the pixel pairs far apart on the same target and the pixel pairs similar to different targets, these differences may cause ambiguity in the positioning of the boundary between targets. Please refer to Figure 1 When the seed information is sparse, label propagation only relying on local seed information usually cannot achieve accurate division of complete segmentation objects. In many segmentation methods, unknown pixels obtain label information from adjacent pixels without using information outside the neighborhood pixels, that is, most algorithms only consider the similarity of eight-neighborhood pixels in the image, ignoring the similarity of pixels outside the eight-neighborhood pixels, which is one of the main reasons why image labels cannot be completely propagated. For example, Figure 1(a) as shown, three pixel points A to C are non-neighbor pixel points, the similarity of A point and C point is high, and the similarity of A point and B point is low. If pixel point A has been marked as a foreground point, and points B and C are not marked, due to the large pixel difference between A and B, point B may hinder the information propagation of point A, so that point C cannot obtain label information from point A, and the propagation ability of local seed information is weak.

[0005] In interactive image segmentation, since the object boundary only accounts for a small part of the entire image, in the local connection graph, only a small number of pixel pairs come from different object regions. The embedding satisfying the above properties should make a subset of pixel pairs with a distance large enough to be preserved, and the distance between any remaining pixel pairs to be pushed to zero. Most existing interactive segmentation algorithms use L2 norm to measure the distance between pixel points, which often leads to a smooth change of the region boundary, resulting in boundary positioning errors, and L2 norm does not produce sparse solutions, which means that in high-dimensional data, the solution is less interpretable because most feature weights are non-zero. SUMMARY

[0006] In view of the technical problems that the existing interactive image segmentation algorithm has weak local seed information propagation ability, and the distance between pixel points is measured by L2 norm, which does not produce sparse solutions, resulting in boundary positioning errors, the present application provides an interactive image segmentation method combining global seed and sparse local linear reconstruction, which can not only segment more complete objects with less seed interaction, but also has excellent boundary positioning ability compared with existing methods. Compared with existing models, the global seed information flow and sparse regular model GSSR p (0 < p < 1) is composed of a global seed information term and a sparse local linear reconstruction regularization term, so that not only the complete semantic information can be preserved, but also the sparse smoothing effect can be achieved when processing simple and complex non-texture images, and clearer segmentation boundaries can be obtained.

[0007] In order to solve the above technical problems, the present application adopts the following technical scheme:

[0008] The interactive image segmentation method combining global seed and sparse local linear reconstruction comprises the following steps:

[0009] S1, a global seed information term is introduced to construct a global seed information flow, so that a more complete segmented object can be obtained under sparse seeds, and the specific global seed information term is as follows:

[0010]

[0011] Wherein, F represents foreground, B represents background, V represents the vertex set of the graph, xi Represents an unknown pixel, w i* Represents the weight of the unknown pixel to the foreground model or background model, x * Indicates whether the pixel belongs to the foreground or background;

[0012] S2. Based on the idea of ​​local linear reconstruction, a sparse local linear reconstruction regularization term is designed, and the L1 norm is used to achieve the smoothness of the segmentation results. At the same time, the final objective function constrained by the L1 norm is obtained by combining the global seed information term in step S1. The specific sparse local linear reconstruction regularization term is shown in the following formula:

[0013]

[0014] in, represents the degree of pixel i, N(i) represents the local set of neighboring pixels, and w ij represents the edge weight, x j is a pixel in N(i);

[0015] The objective function is shown as follows:

[0016]

[0017] Where X=(x1,x2,…,x n ) represents the final solution of the objective function, n is the total number of pixels; λ is a balance parameter used to balance the sparse local linear reconstruction regularization term and the global seed information term;

[0018] S3. The obtained objective function is iteratively solved by the alternating direction multiplier method, so as to obtain a more complete segmentation object and achieve a more sparse regional boundary segmentation effect;

[0019] S4. Modify the L1 norm to L p norm, where 0<p<1, L is p The norm is converted into the L1 norm, and the non-convex optimization problem is converted into a convex optimization problem to obtain the solution that minimizes the objective function;

[0020] S5. According to the obtained solution of the minimized objective function, the final segmentation result is obtained through the threshold method.

[0021] Furthermore, the step S1 includes the following steps:

[0022] S11. The user interaction information is modeled using a Gaussian mixture model. The foreground model and background model are obtained by solving the EM algorithm. The calculation formula is shown in the following formula:

[0023]

[0024] where p v denotes the Gaussian mixture probability distribution of the background model or the foreground model; z i denotes the ith observation data I i the label belonging to the kth Gaussian component; v∈{F,B}; Θ v =(π v1 ,…,π vk ,θ v1 ,…,θ vk ) denotes the parameter set; N vk (I i |θ vk ), N vl (I i |θ vl ) respectively denote the Gaussian distribution determined by the parameters θ vk , θ vl ; π vk , π vl respectively denote the weight of the kth and the lth Gaussian component in the model v; K denotes the number of Gaussian components;

[0025] S12, calculate the probability of the picture pixel point to the foreground model and the background model as the distance of the unlabeled pixel point to the foreground and the background, construct the foreground matrix and the background matrix, so that the seed information can be globally propagated; wherein the distance calculation formula of the unlabeled pixel point to the foreground is as follows:

[0026]

[0027] The distance calculation formula of the unlabeled pixel point to the background is as follows:

[0028]

[0029] where P F denotes the foreground model, P B denotes the background model, k denotes the kth Gaussian component, denotes the best model parameter of the foreground model, denotes the best model parameter of the background model.

[0030] Further, the step S2 comprises the following steps:

[0031] S21, calculate the graph weight model according to the input image:

[0032] Let the input image be I, in order to construct the sparse local linear reconstruction regularization term, define a weight graph G=(V,E,W E ), where V denotes the vertex set of the graph, its elements are the pixel points of the image, E is the edge set of all connected pixel points in the eight neighborhood, W Eis the edge weight set; the local neighbor pixel set N(i) = {j: (i, j) e E} represents all pixel sets in the eight-neighborhood of the current pixel i, and the edge weight is defined as follows:

[0033]

[0034] where exp(·) represents the exponential function, and σ = max (i,j)∈E ||I i -I j || ∞ To ensure the stability of propagation, this weight calculation method ensures that the edge weight value is always a positive number and is symmetric, i.e., w ij = w ji ; I i and I j are color vectors, and the pixel intensity is measured by RGB or Lab color space, and the weight matrix model of the final graph is as follows:

[0035]

[0036] S22, the propagation of local seed information is constrained by L1 norm, and the final target function is obtained by combining the foreground matrix and the background matrix obtained in step S1.

[0037] Further, the step S3 includes the following steps:

[0038] S31, the labeled pixel points and the unlabeled pixel points are separated, and the energy function is reconstructed:

[0039] To ensure that the pixel points solved by the foreground and background regions of the target function are consistent with the labeled pixel points, and and are used as constraint terms of the target function, and the matrix form of the target function can be written as follows:

[0040]

[0041] where L = D-W, L is the Laplace matrix; D is a diagonal matrix, satisfying D ii = d i , 1 is a full one vector; W is the adjacency weight matrix; D B and D F represent the constructed background weight matrix and foreground weight matrix, respectively, and the elements satisfy i.e.

[0042] In order to minimize the objective function, and avoid the memory consumption and time overhead brought by the inverse operation of the extremely sparse matrix, the marked pixel points and the unmarked pixel points are separated, and the L matrix is reorganized, which is simple and effective and does not affect the logical structure of the L matrix; U represents the set of unknown pixel points, and S is the set of seed pixel points, N B 、N F 、N U respectively represent the number of background seeds, the number of foreground seeds and the number of unmarked pixels, so that the objective function can be decomposed as shown in the following formula:

[0043]

[0044] Wherein, X U represents the set of unmarked pixel points, The dimension of the matrix D B and D F is N U ×N U , L S is determined by the row and column of the marked pixel points in the L matrix, and L U is determined by the row and column of the unmarked pixel points, and R matrix is an adjacent weight matrix determined by the row of seed pixel point index and the column of unmarked pixel point index, so that the objective function is further expanded as:

[0045]

[0046] S32, the reconstructed objective function is solved by using the alternating direction multiplier method:

[0047] In order to effectively solve the above formula (17), L S X S -RX U =d0, L U X U -R T X S =d1, by introducing Lagrange multiplier, the augmented Lagrangian function is constructed, and the objective function is expanded as the following formula:

[0048]

[0049] Wherein, b0 and b1 are the introduced Lagrange multipliers, and ρ is the Lagrange penalty coefficient, and ρ>0; for solving the last objective function formula (19), the objective function only contains L1 norm, L2 norm and does not contain constraint, and direct derivation minimization can be carried out.

[0050] Further, the step S4 comprises the following steps:

[0051] S41, the sparse local linear reconstruction regular term in the reconstructed objective function is constrained by L p norm, and the objective function after the constraint is redefined as shown in the following formula:

[0052]

[0053] S42, the L p norm solution is converted into L1 norm problem solution, and each L p norm term is converted into L1 norm for constraint. Specifically, formula (26) is modified as shown in the following formula:

[0054]

[0055] Where, N F represents the number of foreground seeds, represents an element in vector L s X s -RX U , and α i represents N B represents the number of background seeds, and ψ i represents an element in vector-R T X s +L U X U , and β i represents |ψ i | p-1 ;

[0056] converted into a diagonal matrix form, then ‖diag(β)ψ‖1, then L p norm corresponding objective function is converted into the following formula:

[0057]

[0058] Where, ψ=-R T X s +L U X U , α is a vector composed of α i , and β is a vector composed of β i , and thus L p regular embedding has been converted into L1 regular embedding; in order to ensure the stability of the calculation, the weight term is designed in the following manner at the next iteration weight update after each iteration ends:

[0059]

[0060] Wherein, k represents the current iteration round, epsilon is a very small positive number to prevent division by zero, at this time the objective function is solved into the same form as formula (17), which will not be described here.

[0061] Further, the step S5 of obtaining the final segmentation result by threshold method comprises: i ∈{x B ,x F}, i is in V, and the threshold segmentation adopts the following rules:

[0062]

[0063] Wherein, x F represents the foreground pixel value, x B represents the background pixel value, and x i represents the value of the marked pixel point obtained after minimizing the objective function.

[0064] Compared with the prior art, the interactive image segmentation method combining global seeds and sparse local linear reconstruction has the following advantages:

[0065] 1. The application provides a new interactive segmentation model, which sufficiently mines seed information, introduces a global seed information item, constructs a global seed information flow, establishes the connection between the seed and the pixel point outside the neighborhood, increases the propagation ability of the seed, and still can obtain a relatively complete segmentation object under the condition of less user interaction.

[0066] 2. The application not only considers the global propagation of seed information, but also uses a sparse local linear reconstruction regular term, so that the local structure information of the image is retained, and weak structure information is better retained.

[0067] 3. The application adopts L p norm constraint output gradient, obtains a more sparse smooth result, helps the label propagation to generate a clear boundary, when p is equal to 1, an alternating direction multiplier method (ADMM) is used to perform iterative minimization calculation on the objective function, when 0 < p < 1, an iterative reweighting method is used, a preconditioned conjugate gradient (PCG) is used to accelerate matrix operation, so that the calculation efficiency is improved, and the effectiveness is verified through experiments. BRIEF DESCRIPTION OF DRAWINGS

[0068] Figure 1 is a segmentation result graph of some classical algorithms related to the application, and different regions on the same object cause segmentation difficulties due to pixel differences.

[0069] Figure 2 is a global seed information model graph constructed by the application.

[0070] Figure 3 is the boundary localization capability function display diagram of the present application.

[0071] Figure 4 is the model of the present application using L1 regularization and L p (p=0.7) and L p (p=0.8) segmentation result comparison diagram.

[0072] Figure 5 is the visualization comparison diagram of the segmentation result of the model of the present application and several classical algorithms.

[0073] Figure 6 is the PR curve, F-score and AUC value of the present application on the BSD dataset.

[0074] Figure 7 is the PR curve, F-score and AUC value of the present application on the MSRC dataset.

[0075] Figure 8 is the PR curve, F-score and AUC value of the present application on the MSRC dataset using S1 seed.

[0076] Figure 9 is the PR curve, F-score and AUC value of the present application on the MSRC dataset using S2 seed.

[0077] Figure 10 is the multi-class segmentation result diagram obtained by the present application.

[0078] Figure 11 is the robustness analysis result diagram of the present application on different seed placement positions.

[0079] Figure 12 is the robustness analysis result diagram of the present application under Gaussian noise. DETAILED DESCRIPTION

[0080] In order to make the technical means, creative features, purposes and effects realized by the present application easy to understand, the present application is further described below in combination with specific diagrams.

[0081] The present application provides an interactive image segmentation method combining global seed and sparse local linear reconstruction, comprising the following steps:

[0082] S1, the present application designs a new interactive segmentation model, by introducing a global seed information term, a global seed information flow is constructed, so that a relatively complete segmentation object can be obtained under the condition of sparse seed, and the specific global seed information term is as follows:

[0083]

[0084] Wherein, F represents a foreground model, B represents a background model, V represents a vertex set of a graph, x i represents an unknown pixel point, w i* represents a weight of the unknown pixel point to the foreground model or the background model, x * represents whether the pixel point belongs to the foreground or the background.

[0085] As a specific embodiment, the step S1 comprises the following steps:

[0086] S11, the interaction information of the user is modeled by using a Gaussian mixture model, and a foreground model and a background model are respectively obtained by using an EM algorithm:

[0087] In the spatial domain of a picture, an image is represented by a three-channel matrix (r, g, b) composed of n pixel points. Given a seed pixel set of the background B and the foreground F. The seed diffusion of the present application is no longer limited to the local area of the seed, but diffuses to the whole picture, as shown in the following figure: Figure 2 The present application uses a Gaussian mixture model (Gaussian Mixture Model) to construct a color data model, considers different color information in the image, in order to make each unknown pixel point find the most similar pixel point in the labeled data as possible, the present application respectively performs Gaussian modeling for the foreground and the background, and sets the Gaussian components of each to K, and the expression of the Gaussian mixture model probability model of the background B and the foreground F is as follows:

[0088]

[0089] Wherein, v∈{F,B};I i represents observation data; Θ v =(π v1 ,…,π vk ,θ v1 ,…,θ vk ) represents a parameter set; K represents the number of Gaussian components; π vk represents the weight of the kth Gaussian component in the model v; N vk (I i |θ vk ) represents a Gaussian distribution determined by the parameter θ vk , two parameters of N vk (I i |θ vk ) are mean μ vk and covariance ∑ vk , and the expression of N vk (I i |θ vk ) is as follows:

[0090]

[0091] Solve the background and background Gaussian mixture model using EM algorithm to obtain the optimal parameters And combined with Bayes formula to calculate the i-th observation data I i The probability of belonging to the k-th Gaussian, the calculation formula is as follows:

[0092]

[0093] Where p v Indicates the Gaussian mixture probability distribution of the background model or the foreground model; z i Indicates the i-th observation data I i The label of the k-th Gaussian component; π vl Indicates the weight of the l-th Gaussian component in the model v; N vl (I i |θ vl ) represents the Gaussian distribution determined by the parameter θ vl .

[0094] S12, calculate the probability of the picture pixel point to the foreground model and the background model, as the distance of the unlabeled pixel point to the foreground and the background, construct the foreground matrix and the background matrix, so that the seed information can be globally propagated:

[0095] Each pixel point x i Can be brought into the foreground and background model to calculate the probability, and the weight w iF And w iB are defined as the probability of the unlabeled pixel point to the foreground model and the background model respectively.

[0096] Wherein, the distance calculation formula of the unlabeled pixel point to the foreground is as follows:

[0097]

[0098] The distance calculation formula of the unlabeled pixel point to the background is as follows:

[0099]

[0100] Where P F Indicates the foreground model, P B Indicates the background model, k indicates the k-th Gaussian component, Indicates the best model parameter of the foreground model, Indicates the best model parameter of the background model. Based on w iF And w iB , the foreground matrix W And the background matrix W Can be constructed, wherein the elements satisfy That is

[0101] Given background and foreground labels x B and x F Based on the above discussion, the following global seed information item can be constructed:

[0102]

[0103] S2, based on the idea of local linear reconstruction, a sparse local linear reconstruction regularization term is designed, and L1 norm is used to realize the piecewise smoothness of the segmentation result, and the final objective function of L1 norm constraint is obtained by combining the global seed information item in step S1, and the specific sparse local linear reconstruction regularization term is as follows:

[0104]

[0105] Wherein, N(i) represents the local neighbor pixel point set, w ij represents the edge weight, and x j is a pixel point in N(i);

[0106] The objective function is as follows:

[0107]

[0108] Wherein, X=(x1,x2,…,x n ) represents the solution obtained by the objective function, n is the total number of pixel points; λ is a balance parameter, which is used to balance the sparse local linear reconstruction regularization term and the global seed information item.

[0109] As a specific embodiment, the step S2 comprises the following steps:

[0110] S21, calculate the graph weight model according to the input image:

[0111] Let the input image be I, in order to construct the sparse local linear reconstruction regularization term, the present application defines a weight graph G=(V,E,W E ), wherein V represents the vertex set of the graph, the elements of which are pixel points of the image, E is the edge set of all connected pixel points in the eight neighborhood, and W E is the edge weight set; the local neighbor pixel point set N(i)={j:(i,j)∈E} represents the set of all pixel points in the eight neighborhood of the current pixel point i, N(i) represents the degree of pixel point i.

[0112] There are many methods for calculating the edge weight of the graph, such as pixel intensity, image gradient, etc. The edge weight of the present application is defined as follows:

[0113]

[0114] Where exp(·) represents the exponential function, σ=max (i,j)∈E ||I i -I j || ∞ In order to ensure the stability of propagation, this weight calculation method ensures that the edge weight value is always a positive number and is symmetrical, that is, w ij =w ji ;I i and I j They are all color vectors. The pixel intensity is measured using RGB or Lab color space. The weight matrix model of the final image is as follows:

[0115]

[0116] S22. Use the L1 norm to constrain the propagation of local seed information, and combine the foreground matrix and background matrix obtained in step S1 to obtain the final objective function:

[0117] Based on the principle of local linear reconstruction, the value of a pixel is represented by its eight neighboring pixels. Finally, the sparse local linear reconstruction regularization term designed by the present invention is shown as follows:

[0118]

[0119] Combined with the above step S1, the model proposed by the present invention finally consists of two parts: a global seed information term and a sparse local linear reconstruction regularization term. The objective function expression of the model is as follows:

[0120] E(X)=λE G (X)+E S (X) Formula (13)

[0121] Where X=(x1,x2,…,x n ) represents the final solution of the objective function, n is the total number of pixels; λ is a balance parameter used to balance the sparse local linear reconstruction regularization term and the global seed information term. The larger λ is, the greater the influence of the global seed information term is. E G (X) is the global seed information item, E S (X) is the sparse local linear reconstruction regularization term. Therefore, the image segmentation task is finally transformed into minimizing the objective function E(X), that is,

[0122] S3. The obtained objective function is iteratively solved using the alternating direction multiplier method (ADMM) to obtain a more complete segmentation object while achieving a more sparse region boundary segmentation effect.

[0123] As a specific embodiment, the step S3 comprises the following steps:

[0124] S31, the marked pixel points and the unmarked pixel points are separated, and an energy function is reconstructed:

[0125] In order to effectively solve the global seed information flow and the sparse local linear reconstruction regular algorithm proposed in the application, the application adopts an iterative method of an alternating direction multiplier method to solve.

[0126] When ρ is set to 1, the objective function of the L1 norm constraint can be obtained as follows:

[0127]

[0128] In order to ensure that the pixel points solved by the foreground and background regions of the objective function are consistent with the marked pixel points, the following formula is used: and As a constraint term of the objective function, the matrix form of the objective function can be written as follows:

[0129]

[0130] Wherein, L=D-W, L is a Laplace matrix; D is a diagonal matrix, and satisfies D ii =d i , 1 is a full 1 vector; W is an adjacent weight matrix; D B and D F respectively represent the constructed background weight matrix and the foreground weight matrix, wherein the elements satisfy That is

[0131] In order to minimize the objective function, and avoid the memory consumption and time overhead caused by the inverse operation of the extremely sparse matrix, the marked pixel points and the unmarked pixel points are separated, and the L matrix can be simply and effectively reorganized without affecting the logical structure of the L matrix; U represents a set of unknown pixel points, and S is a set of seed pixel points, N B , N F , N U respectively represent the number of background seeds, the number of foreground seeds and the number of unmarked pixels, so that the objective function can be decomposed as follows:

[0132]

[0133] Wherein, X U represents a set of unmarked pixel points, At this time, the matrix D B and D FThe dimension of L is N U ×N U L S is determined by the row and column of the marked pixel points in the L matrix, L U is determined by the row and column of the unmarked pixel points, and R matrix is a neighbor weight matrix determined by the row of the seed pixel point index and the column of the unmarked pixel point index, so the objective function is further expanded as:

[0134]

[0135] S32, the alternating direction multiplier method is used to solve the reconstructed objective function:

[0136] For the specific L1 regularization constraint term of the application, that is, to effectively solve the above formula (17), let L S X S -RX U =d0, L U X U -R T X S =d1, then the problem is equivalent to minimizing the following formula:

[0137]

[0138] By introducing the Lagrange multiplier, the augmented Lagrange function is constructed, and the objective function is expanded as follows:

[0139]

[0140] Where b0, b1 are the introduced Lagrange multipliers, and ρ is the Lagrange penalty coefficient, and ρ>0; for solving the final objective function formula (19), the objective function only contains L1 norm, L2 norm and does not contain constraints, and direct derivation minimization can be carried out.

[0141] Specifically, formula (19) can be solved by iterative minimization (ADDM), wherein the steps of solving the minimum sub-problem of each parameter in the k+1 iteration problem are as follows:

[0142] First, fix d0, d1, b0, b1, and solve X U :

[0143]

[0144] Then the final solution of can be transformed into the solution of the following linear equation group:

[0145]

[0146] Where, Variables are all the intermediate variables introduced in the ADDM algorithm.

[0147] Second, fix X U , b0, b1, respectively solve d0, d1:

[0148]

[0149] Here, the shrink operation can be defined as max(·) represents the maximum value function, y represents the input vector, and γ represents the penalty factor.

[0150] Finally, fix X U , d0, d1, respectively solve b0, b1 d1:

[0151]

[0152] In summary, the present application summarizes all the steps of minimizing the energy function (formula 19) and obtaining the solution of the unknown pixel point in Algorithm 1, wherein the linear equation group of the sub-problem (solving X U ) is defined in lines 3-5, and there are three parameters λ, ρ, and η in the algorithm, wherein η is used to control the end of the loop, and the functions of λ and ρ are the same as described above. The specific algorithm 1 is as follows:

[0153] Input: input image g, seed information X S , λ, ρ, η

[0154] Output: unmarked pixel set X U

[0155] 1: initialization:

[0156] 2:

[0157] 3: construct A matrix as shown in formula (21)

[0158] 4: construct v vector as shown in formula (21)

[0159] 5: update by solving

[0160] 6: calculate formula (22) to obtain

[0161] 7: calculate formula (23) to obtain

[0162] 8: calculate formula (24) to obtain

[0163] 9: Calculate formula (25) to obtain

[0164] 10: k = k + 1

[0165] 11: end while

[0166] 12:

[0167] S4. Modify the L1 norm to L p norm, where 0<p<1, L is p The norm is converted into the L1 norm, and the non-convex optimization problem is converted into a convex optimization problem to obtain the solution that minimizes the objective function.

[0168] In this embodiment, step S4 includes the following steps:

[0169] S41, the sparse local linear reconstruction regularization term in the reconstructed objective function is replaced by L p Norm constraint:

[0170] In order to improve the robustness of the algorithm and further improve the sparsity of the solution, the present invention adopts L p (0<p<1) regular embedding is used to obtain a sparser solution. After the constraint, the objective function is redefined as follows:

[0171]

[0172] S42, using iterative reweighting method to p Norm solution is transformed into L1 norm problem solution. For each L p The norm terms are converted into L1 norms for constraints:

[0173] When 0<p<1, Equation (26) is a non-convex optimization problem. When faced with a complex objective function optimization task, a wise strategy is to try to convert it into a simpler optimization problem. At present, a class of optimization methods based on iterative reweighting has been developed. These methods iteratively optimize smooth or non-smooth convex objective functions, and the weight of each round is obtained by dynamically weighting the coefficients evaluated in the previous iteration. According to the above method and in order to reflect the iterative reweighting process, the present invention optimizes and derives L on the basis of L1 p Solution: Modify Equation (26) as follows:

[0174]

[0175] Among them, N F Indicates the number of foreground seeds, Represents vector Ls X s -RX U α i β N B ψ i -R T X s +L U X U β i |ψ i | p-1 ;

[0176] Transformed into a diagonal matrix form then have ‖diag(β)ψ‖1, then L p norm corresponding objective function is transformed into the following formula shown:

[0177]

[0178] Wherein, ψ=-R T X s +L U X U , alpha is by alpha i Vector, beta is by beta i Vector, thus L p Regular embedding has been transformed into L1 regular embedding;In order to ensure the stability of calculation, the weight term designed by the application is updated as follows in the next round of iteration after each round of iteration ends:

[0179]

[0180] Wherein, k represents the current iteration round, epsilon is a very small positive number to prevent division by zero, at this time the objective function solving is transformed into the same form as formula (17), next only needs to use the similar solving method of formula (17) can, here no longer tedious.

[0181] S5, according to the solution of the minimum objective function obtained, the final segmentation result is obtained by threshold method:

[0182] Once the minimum result X vector of formula (20) is obtained, in combination with the known X S Vector above, some simple and convenient classification algorithms can be used to help get the final segmentation result, such as k-means clustering algorithm or Otsu threshold segmentation method, binary x vector into y i ∈{x B ,x F}, i∈V, the simplest threshold segmentation can adopt the following rules:

[0183]

[0184] Among them, x F represents the foreground pixel value, x B Represents the background pixel value, x i Indicates that the value of the marked pixel is obtained after minimizing the objective function.

[0185] Figure 4 Shows the use of L1 regularization and L p (p=0.7) and L p (p=0.8) Segmentation results on the same mark. It can be seen from the figure that L p (p = 0.8) Regularization can better distinguish the object boundaries and obtain more accurate segmentation results, which is better than L1 and L p (p=0.7) norm-constrained GSSR model. Therefore, in order to achieve more accurate results, for L p (0<p<1) Constrained GSSR Model The present invention will use p=0.8 for subsequent experimental comparisons.

[0186] This paper proposes a new algorithm framework based on global seed information term and sparse local linear reconstruction regularization term. This algorithm can not only segment more complete objects with less seed interaction, but also has excellent boundary localization ability compared with existing methods (see Figure 3 ). In order to verify the necessity and rationality of adopting sparse regularization constraint, the present invention adopts L1 regularization and L p (p = 0.8) regularization is used for constraint.

[0187] The present invention combines GSSR1 and GSSR 0.8Compared with several methods for interactive image segmentation, which are Grow Cuts (GRO), Random Walk (RW), Normalized Random Walk (NRW), Normalized Lazy Random Walk (NLRW), One Cut (ONE), Laplacian Coordinates (LC), Multi-layer RandomForest (MRF). The application realizes all the above comparison algorithms in the MATLAB2020b programming environment, and the above methods are open source, and all use the default parameters provided by the above algorithms. In order to quantitatively evaluate the segmentation results of the algorithm, the application adopts some widely used evaluation indexes: Land Index (RI), Information Difference Index (VoI), Boundary Drift Error (BDE), Intersection Over Union (IOU). The application verifies the feasibility of the GSSR framework on two very famous and GT-provided data sets: the BSD data set and the MSRC data set. Both of the two data sets are open source and free. The MSRC data set contains 50 natural pictures, and the BSD data set contains 500 pictures, most of which have interference that makes segmentation more difficult, such as texture clutter and different types of lighting conditions. In order to reduce the index error caused by the inconsistency of the seed information of the human subjective annotation, further use the annotation provided by Andrade and Carrera, two different sparse seed point sets S1 and S2 on the MSRC data set, further show the advantages of the algorithm of the application. In order to more intuitively show the segmentation ability of each method mentioned above compared with the algorithm of the application, Figure 5 The specific segmentation results of each algorithm on 9 pictures are presented. As can be seen from the figure, although most of the algorithms can output similar results, there are still some subtle differences that need to be emphasized. In the starfish picture, no background seed is set in the upper background, and it can be seen that most of the algorithms cannot handle this situation, only ONE and GSSR 0.8 achieve better segmentation results. In the church and boat pictures, MRF and LC algorithms produce almost the same segmentation results, but also slightly worse than GSSR 0.8 When in the squirrel picture, GSSR 0.8 algorithm performs best. When comparing the remaining pictures, it can be seen that GSSR1 and GSSR 0.8 algorithm all get more accurate edges and more complete targets.

[0188] To show the segmentation details more comprehensively, the median, mean and variance of the four indexes of RI, VoI, BDE and IoU are used to measure. At the same time, the PR curve, F-score and AUC index are used to measure the boundary fitting of the segmentation result more comprehensively. Table 1 below shows the IoU, VoI, BDE and RI indexes of each segmentation algorithm on the BSD dataset. It is noted that the GSSR1 method and the GSSR 0.8 The method almost exceeds the above-mentioned comparative method in all indexes.

[0189] Table 1 shows the IoU, VoI, BDE and RI indexes of each method on the BSD dataset

[0190]

[0191] To further evaluate the boundary fitting ability of the segmentation algorithm, Figure 6 (a) shows the PR curve of each algorithm. Here, the higher the recall rate and the precision rate, the better the segmentation ability of the algorithm. At the same time, in order to comprehensively consider the precision rate and the recall rate, we show the F-score and the AUC index of each algorithm in (b) to evaluate the model. Figure 6 (b) can be seen that the boundary fitting degree of GSSR1 and GSSR 0.8 is very similar, but slightly worse than GSSR 0.8 Both of them show better edge fitting ability than the rest of the comparative methods and the highest F-score and AUC value in all methods.

[0192] The experimental results on the MSRC dataset are shown in Table 2. From the data shown in the table, the model added with global seed information, whether using L1 norm or L p (p=0.8) norm, almost exceeds other segmentation methods in all evaluation indexes. It shows that the method proposed in the application is better in overall segmentation quality and more stable in segmentation. Similar to the BSD dataset, the PR curve, F-score and AUC index are shown in Figure 7 It can be seen that GSSR 0.8 achieves the highest score.

[0193] Table 2 shows the IoU, VoI, BDE and RI indexes of each method on the MSRC dataset

[0194]

[0195] To reduce the index error caused by the inconsistency of the seed information manually labeled by subjective, the application adopts a sparse seed set S1 labeled on the MSRC dataset to further evaluate the above algorithm in depth and complexity. Table 3 below summarizes the index of each algorithm when using the sparse seed S1. By analyzing the data in the above table, the following conclusions can be drawn: when using a small amount of labeled seeds, the algorithm using global seed information flow is almost in the best position in all indicators, and each indicator has a relatively significant improvement. Although GSSR1 and GSSR 0.8 Algorithm has a slight gap, but all indicators are still in the second place of all the above segmentation algorithms. In order to show the boundary fitting index, the application gives the PR curve and F-score, AUC chart in Figure 8

[0196] Table 3 quantitative analysis of IoU, VoI, BDE, RI indexes of each method on MSRC dataset using S1 seeds

[0197]

[0198]

[0199] Table 4 below shows another set of comparisons on the MSRC dataset, which uses another set of sparse seed sets S2 provided by Andrade and Carrera. As can be seen from Table 4, the method proposed in the application completely exceeds the above-mentioned comparison methods in terms of mean and variance, and is in the first and second place, which shows that GSSR1 and GSSR 0.8 The overall segmentation quality is higher and more stable. Moreover, the algorithm of the application is in the second and third place of all methods in the median, only falling behind the ONE method. Figure 9 The PR curve, F-score and AUC index of each model on the MSRC dataset using S2 seeds are shown.

[0200] Table 4 quantitative analysis of IoU, VoI, BDE, RI indexes of each method on MSRC dataset using S2 seeds

[0201]

[0202] GSSR p Can be easily extended to a multi-classification problem, here L1 norm is used as an example. For an N-class problem, only N-1 times solving formula (19) as solving the binary classification problem is needed.

[0203]

[0204] ​The final segmentation result can be obtained by solving the above formula. The present application shows some multi-region segmentation results, as shown in Figure 10 The seed label set S is defined, and different kinds of labels are represented as where k={1,…,N}, then for the jth solving satisfies the following formula:

[0205]

[0206] Figure 11 The random seed position robustness of the GSSR p is shown. Different segmentation results can be generated by marking different objects in the image. From the three input seeds (left, middle and right columns) in Figure 11 , the seeds can be accurately segmented into multiple targets from the GSSR p model. In order to highlight the robustness of the algorithm in the anti-noise, the present application shows several image segmentation results under the influence of Gaussian noise in Figure 12 , and it can be seen that under the influence of Gaussian noise with a mean of 0.2 and a variance of 0.01, the algorithm can still accurately segment the target object.

[0207] Compared with the prior art, the interactive image segmentation method combining global seeds and sparse local linear reconstruction provided by the present application has the following advantages:

[0208] 1. The present application proposes a new interactive segmentation model, which fully excavates the seed information, introduces a global seed information term, constructs a global seed information flow, establishes the connection between the seed and the pixel point outside the neighborhood, increases the propagation ability of the seed, and still can obtain a relatively complete segmentation object under the condition of less user interaction.

[0209] 2. The present application not only considers the global propagation of seed information, but also uses a sparse local linear reconstruction regularization term, which preserves the local structure information of the image and better preserves the weak structure information.

[0210] 3. The present application uses L p norm constraint output gradient to obtain a more sparse smooth result, which helps the label propagation to produce clear boundaries; when p=1, the alternating direction multiplier method (ADMM) is used to iteratively minimize the calculation of the objective function; when 0

[0211] Finally, it is to be explained that the above embodiments are only used to illustrate the technical solutions of the present application but not to limit the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalently replaced without departing from the purpose and scope of the present application, and all of them should be covered in the scope of the claims of the present application.

Claims

1. An interactive image segmentation method combining global seeding and sparse local linear reconstruction, characterized in that: Including steps: S1. Introduce the global seed information item and construct the global seed information flow to achieve a relatively complete segmentation object in the case of sparse seeds. The specific global seed information item is shown in the following formula: Among them, F represents the foreground, B represents the background, V represents the vertex set of the graph, and x i Represents an unknown pixel, w i* Represents the weight of the unknown pixel to the foreground model or background model, x * Indicates whether the pixel belongs to the foreground or background; S2. Based on the idea of ​​local linear reconstruction, a sparse local linear reconstruction regularization term is designed, and the L1 norm is used to achieve the smoothness of the segmentation results. At the same time, the final objective function constrained by the L1 norm is obtained by combining the global seed information term in step S1. The specific sparse local linear reconstruction regularization term is shown in the following formula: in, represents the degree of pixel i, N(i) represents the local set of neighboring pixels, and w ij represents the edge weight, x j is a pixel in N(i); The objective function is shown as follows: Where X=(x1,x2,…,x n ) represents the final solution of the objective function, n is the total number of pixels; λ is a balance parameter used to balance the sparse local linear reconstruction regularization term and the global seed information term; S3. The obtained objective function is iteratively solved by the alternating direction multiplier method, so as to obtain a more complete segmentation object and achieve a more sparse regional boundary segmentation effect; S4. Modify the L1 norm to L p norm, where 0<p<1, L is p The norm is converted into the L1 norm, and the non-convex optimization problem is converted into a convex optimization problem to obtain the solution that minimizes the objective function; S5. According to the obtained solution of the minimized objective function, the final segmentation result is obtained through the threshold method.

2. The interactive image segmentation method combining global seed and sparse local linear reconstruction according to claim 1, characterized in that: The step S1 comprises the following steps: S11. The user interaction information is modeled using a Gaussian mixture model. The foreground model and background model are obtained by solving the EM algorithm. The calculation formula is shown in the following formula: Among them, p v represents the Gaussian mixture probability distribution of the background model or foreground model; z i Represents the i-th observation data I i The label belonging to the kth Gaussian component; v∈{F,B}; Θ v =(v v1 ,…,π vk ,θ v1 ,…,θ vk ) represents a parameter set; N vk (I i |θ vk ), N vl (I i |θ vl ) represent the parameters θ vk ,θ vl Determined Gaussian distribution; π vk , π vl Respectively represent the weights of the kth and lth Gaussian components in the model v; K represents the number of Gaussian components; S12. Calculate the probability of the image pixel to the foreground model and the background model as the distance from the unlabeled pixel to the foreground and background, and construct the foreground matrix and the background matrix so that the seed information can be globally propagated. The distance calculation formula from the unlabeled pixel to the foreground is as follows: The distance calculation formula from unmarked pixels to the background is as follows: Among them, P F represents the foreground model, P B represents the background model, k represents the kth Gaussian component, represents the optimal model parameters of the foreground model, Represents the optimal model parameters of the background model.

3. The interactive image segmentation method combining global seed and sparse local linear reconstruction according to claim 2, characterized in that: The step S2 comprises the following steps: S21. Calculate the graph weight model based on the input image: Let the input image be I. In order to construct the sparse local linear reconstruction regularization term, define a weight graph G = (V, E, W E ), where V represents the vertex set of the graph, whose elements are the pixels of the image, E is the edge set of all connected pixels in the eight-neighborhood, and W E is the set of edge weights; the local neighbor pixel set N(i) = {j:(i,j)∈E} represents the set of all pixels in the eight-neighborhood of the current pixel i. The edge weight is specifically defined as follows: Where exp(·) represents the exponential function, σ=max (i,j)∈E ||I i -I j || ∞ In order to ensure the stability of propagation, this weight calculation method ensures that the edge weight value is always a positive number and is symmetrical, that is, w ij =w ji ;I i and I j They are all color vectors. The pixel intensity is measured using RGB or Lab color space. The weight matrix model of the final image is as follows: S22. Use the L1 norm to constrain the propagation of local seed information, and combine the foreground matrix and background matrix obtained in step S1 to obtain the final objective function.

4. The interactive image segmentation method combining global seed and sparse local linear reconstruction according to claim 3, characterized in that: The step S3 comprises the following steps: S31. Separate the marked pixels and the unmarked pixels and reconstruct the energy function: In order to ensure that the pixels obtained by the objective function in the foreground and background areas are consistent with the marked pixels, x i =x B , and x i =x F , As the constraint term of the objective function, the matrix form of the objective function can be written as follows: Where L = DW, L is the Laplace matrix; D is a diagonal matrix that satisfies D ii =d i , 1 is a full 1 vector; W is the adjacency weight matrix; D B and D F Represent the constructed background weight matrix and foreground weight matrix respectively, where the elements satisfy Right now In order to minimize the objective function and avoid the memory consumption and time overhead caused by the inversion of the extremely large sparse matrix, the marked pixels and the unmarked pixels are separated. By reorganizing the L matrix, it is simple and effective without affecting the logical structure of the L matrix; U represents the set of unknown pixels, S is the set of seed pixels, N B 、N F 、N U They represent the number of background seeds, the number of foreground seeds, and the number of unlabeled pixels, respectively. The objective function can be decomposed into the following formula: Among them, X U represents the set of unlabeled pixels, The matrix D at this time B and D F The dimension is N U ×N U , L S Determined by the rows and columns of the marked pixels in the L matrix, L U It is determined by the rows and columns of unlabeled pixels, and the R matrix is ​​an adjacency weight matrix whose rows are determined by the seed pixel index and whose columns are determined by the unlabeled pixel index. The objective function is further expanded as follows: S32. Use the alternating direction multiplier method to solve the reconstructed objective function: In order to effectively solve the above formula (17), let L S X S -RX U =d0,L U X U -R T X S =d1, by introducing the Lagrange multiplier and constructing the augmented Lagrangian function, the objective function is expanded to the following formula: Among them, b0 and b1 are both introduced Lagrange multipliers, ρ is the Lagrange penalty coefficient, and ρ>0; for solving the final objective function formula (19), the objective function only contains the L1 norm and L2 norm and has no constraints, and can be directly minimized by derivative.

5. The interactive image segmentation method combining global seed and sparse local linear reconstruction according to claim 4, characterized in that: The step S4 comprises the following steps: S41, the sparse local linear reconstruction regularization term in the reconstructed objective function is replaced by L p The norm is constrained, and the objective function is redefined as follows: S42, using iterative reweighting method to p Norm solution is transformed into L1 norm problem solution. For each L p The norm terms are all converted into L1 norms for constraint. Specifically, Equation (26) is modified as shown below: Among them, N F Indicates the number of foreground seeds, Represents vector L s X s -RX U An element in i express N B represents the number of background seeds, ψ i Representation vector - R T X s +L U X U An element in i Indicates |ψ i | p-1 ; Converted into diagonal matrix form, we have ‖diag(β)ψ‖1, then L p The objective function corresponding to the norm is transformed into the following formula: in, ψ=-R T X s +L U X U , α is given by α i The vector composed of β is i The vector composed of L p Regularized embedding has been converted to L1 regularized embedding. To ensure the stability of the calculation, the weight items are updated in the following way after each iteration: Where k represents the current iteration round, and ε is a small positive number to prevent it from being divided by zero. At this time, the objective function solution is transformed into the same form as Equation (17), which will not be repeated here.

6. The interactive image segmentation method combining global seed and sparse local linear reconstruction according to claim 5, characterized in that: The final segmentation result obtained by the threshold method in step S5 includes: converting the x vector into y i ∈{x B ,x F }, i∈V, the threshold segmentation adopts the following rules: Among them, x F represents the foreground pixel value, x B Represents the background pixel value, x i Indicates that the value of the marked pixel is obtained after minimizing the objective function.

Images