Interactive image segmentation method combining global seed and sparse local linear reconstruction

Abstract

The invention provides an interactive image segmentation method combining global seed and sparse local linear reconstruction, which comprises the following steps: introducing global seed information items, and constructing a global seed information stream; designing a sparse local linear reconstruction regular term, adopting an L1 norm to realize fragment smoothness of a segmentation result, and combining a global seed information item to obtain a target function; the objective function is iteratively solved in an ADMM mode, and a more sparse region boundary segmentation effect is achieved while a more complete segmentation object is obtained; the L1 norm is modified into an Lp norm, the Lp norm is converted into the L1 norm to be solved in an iterative reweighting mode, and a solution of a minimized objective function is obtained; and obtaining a final segmentation result from the solution of the minimized objective function through a threshold value method. The model constructed by the invention is composed of a global seed information item and a sparse local linear reconstruction regular item, so that complete semantic information can be reserved, a sparse smooth effect is shown when a complex image is processed, and a clearer segmentation boundary can be obtained.

Description

Technical Field

[0001] The present invention relates to the technical field of interactive image segmentation, and in particular to an interactive image segmentation method combining global seed and sparse local linear reconstruction. Background Art

[0002] Image segmentation algorithms play a vital role in fields such as computer vision and pattern recognition. Especially in the fields of medical image segmentation, video processing, and target tracking, they have promoted the rapid development of image segmentation technology and spawned many different types of segmentation algorithms.

[0003] Existing interactive image segmentation algorithms can be divided into three categories: algorithms based on graph cut theory, algorithms based on random walks, and algorithms based on deep learning. Among them, the algorithm based on graph cut theory converts the image segmentation task into the maximum flow / minimum cut problem in graph theory, and obtains the final segmentation result by minimizing the graph energy function. After the user marks some pixels in the image as foreground and background, the random walk algorithm calculates the random walk probability from the labeled pixels to all unlabeled pixels, and the labeled pixel category with the highest probability is the category to which the unlabeled pixels belong. The image segmentation algorithm based on deep learning explores the hierarchical feature representation of data through a multi-layer network, which can well express and characterize the structural information of complex data and achieve semantic segmentation of the target.

[0004] The main purpose of interactive image segmentation is to obtain complete objects with clear boundaries with less prior information of user interaction. However, due to the objective differences between users, the prior information obtained by the algorithm is often far apart. How to achieve better segmentation performance with less interactive information is still an urgent problem to be solved. Most of the existing algorithms are based on the principle of local linear reconstruction, which believes that each pixel can be linearly represented by neighboring pixels. The higher the similarity with the neighboring pixels, the greater the possibility of label consistency, that is, the pixel label is only determined by the similarity with the neighboring pixels. However, there are differences in texture information or color information between the same target and different targets in the image. When measuring the similarity between distant pixel pairs on the same target and close pixel pairs between different targets, these differences may cause the model to be ambiguous about the boundary positioning between targets. Please refer to Figure 1 As shown in , when the seed information is relatively sparse, it is usually difficult to achieve accurate segmentation of the complete segmentation object by relying solely on local seed information for label propagation. In many segmentation methods, unknown pixels obtain label information through adjacent pixels without using information outside the neighborhood pixels. That is, most algorithms only consider the similarity of the eight neighborhoods of pixels in the image, while ignoring the similarity between pixels and pixels outside the eight neighborhoods. This is one of the main reasons why image labels cannot be fully propagated. Figure 1As shown in (a), the three pixels A to C are non-neighborhood pixels, and the similarity between point A and point C is higher, and the similarity with point B is lower. If pixel point A has been marked as a foreground point, and points B and C are not marked, then due to the large pixel gap between A and B, point B may hinder the information propagation of point A, resulting in point C being unable to obtain label information from point A, and the propagation ability of local seed information is weak.

[0005] In interactive image segmentation, since object boundaries only occupy a small number of pixels in the entire image, there are only a small number of pixel pairs from different object regions in the local connection map. An embedding that satisfies the above properties should retain a subset of pixel pairs with sufficiently large distances, while the distances between any remaining pixel pairs are pushed to zero. Most existing interactive segmentation algorithms use the L2 norm to measure the distance between pixels, which often results in obvious smooth changes in region boundaries, leading to boundary positioning errors. In addition, the L2 norm does not produce sparse solutions, which means that in high-dimensional data, the solution is less interpretable because most feature weights are non-zero. Summary of the invention

[0006] Aiming at the technical problems that the existing interactive image segmentation algorithms have weak local seed information propagation ability and the L2 norm used to measure the distance between pixels does not produce a sparse solution, resulting in boundary positioning errors, the present invention provides an interactive image segmentation method combining global seed and sparse local linear reconstruction. This method can not only segment more complete objects with less seed interaction, but also has excellent boundary positioning ability compared with the existing methods. At the same time, compared with the existing models, the GSSR model constructed by the present invention that combines global seed information flow and sparse regularization is p (0<p≤1) consists of two parts: a global seed information term and a sparse local linear reconstruction regularization term. It can not only retain the complete semantic information, but also show a sparse smoothing effect when processing simple and complex non-texture images, and can obtain clearer segmentation boundaries.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0008] The interactive image segmentation method combining global seed and sparse local linear reconstruction includes the following steps:

[0009] S1. Introduce global seed information items and construct global seed information flow to obtain relatively complete segmentation objects in the case of sparse seeds. The specific global seed information items are shown in the following formula:

[0010]

[0011] Among them, F represents the foreground, B represents the background, V represents the vertex set of the graph, and xi Represents unknown pixel, w i* Represents the weight of the unknown pixel to the foreground model or background model, x * Indicates whether the pixel belongs to the foreground or background;

[0012] S2. Based on the idea of ​​local linear reconstruction, a sparse local linear reconstruction regularization term is designed, and the L1 norm is used to achieve the smoothness of the segmentation result. At the same time, the final objective function constrained by the L1 norm is obtained by combining the global seed information term in step S1. The specific sparse local linear reconstruction regularization term is shown in the following formula:

[0013]

[0014] in, represents the degree of pixel i, N(i) represents the local set of neighboring pixels, and w ij represents the edge weight, x j is a pixel in N(i);

[0015] The objective function is as follows:

[0016]

[0017] Where X=(x1,x2,…,x n ) represents the final solution of the objective function, n is the total number of pixels; λ is a balance parameter used to balance the sparse local linear reconstruction regularization term and the global seed information term;

[0018] S3, the obtained objective function is iteratively solved by the alternating direction multiplier method, so as to obtain a more complete segmentation object and achieve a more sparse regional boundary segmentation effect;

[0019] S4. Modify the L1 norm to L p norm, where 0<p<1, L is iterated by reweighting p The norm is converted into the L1 norm, and the non-convex optimization problem is converted into a convex optimization problem to obtain the solution that minimizes the objective function;

[0020] S5. According to the obtained solution of minimizing the objective function, the final segmentation result is obtained through a threshold method.

[0021] Further, the step S1 comprises the following steps:

[0022] S11. The user's interactive information is modeled using a Gaussian mixture model, and the foreground model and the background model are obtained by solving the EM algorithm. The calculation formula is as follows:

[0023]

[0024] Among them, p v represents the Gaussian mixture probability distribution of the background model or foreground model; z i Represents the i-th observation data I i The label belonging to the kth Gaussian component; v∈{F,B}; Θ v =(π v1 ,…,π vk ,θ v1 ,…,θ vk ) represents a parameter set; N vk (I i |θ vk ), N vl (I i |θ vl ) represent the parameters θ vk ,θ vl Determined Gaussian distribution; π vk , π vl They represent the weights of the kth and lth Gaussian components in the model v respectively; K represents the number of Gaussian components;

[0025] S12, calculate the probability of the image pixel to the foreground model and the background model, as the distance from the unlabeled pixel to the foreground and the background, construct the foreground matrix and the background matrix, so that the seed information can be globally propagated; wherein, the distance calculation formula from the unlabeled pixel to the foreground is as follows:

[0026]

[0027] The distance calculation formula from unmarked pixels to the background is as follows:

[0028]

[0029] Among them, P F represents the foreground model, P B represents the background model, k represents the kth Gaussian component, represents the optimal model parameters of the foreground model, Represents the optimal model parameters of the background model.

[0030] Further, the step S2 comprises the following steps:

[0031] S21. Calculate the graph weight model based on the input image:

[0032] Let the input image be I. In order to construct the sparse local linear reconstruction regularization term, a weight graph G = (V, E, W E ), where V represents the vertex set of the graph, whose elements are the pixels of the image, E is the edge set of all connected pixels in the eight-neighborhood, and W Eis the weight set of the edge; the local neighbor pixel set N(i) = {j:(i,j)∈E} represents the set of all pixels in the eight neighborhoods of the current pixel i. The edge weight is specifically defined as follows:

[0033]

[0034] Where exp(·) represents the exponential function, σ=max (i,j)∈E ||I i -I j || ∞ In order to ensure the stability of propagation, this weight calculation method ensures that the edge weight value is always a positive number and is symmetrical, that is, w ij =w ji ;I i and I j They are all color vectors. The pixel intensity is measured by RGB or Lab color space. The weight matrix model of the final image is as follows:

[0035]

[0036] S22, using the L1 norm to constrain the propagation of local seed information, combining the foreground matrix and the background matrix obtained in step S1, to obtain the final objective function.

[0037] Further, the step S3 comprises the following steps:

[0038] S31, separate the marked pixels and the unmarked pixels, and reconstruct the energy function:

[0039] In order to ensure that the pixels obtained by the objective function in the foreground and background areas are consistent with the marked pixels, as well as As the constraint term of the objective function, the matrix form of the objective function can be written as shown below:

[0040]

[0041] Where L = DW, L is the Laplace matrix; D is a diagonal matrix that satisfies D ii =d i , 1 is a vector of all 1s; W is the adjacency weight matrix; D B and D F Represent the constructed background weight matrix and foreground weight matrix respectively, where the elements satisfy Right now

[0042] In order to minimize the objective function and avoid the memory consumption and time overhead caused by the inversion of the extremely large sparse matrix, the marked pixels and the unmarked pixels are separated. By reorganizing the L matrix, it is simple and effective without affecting the logical structure of the L matrix; U represents the set of unknown pixels, S is the set of seed pixels, N B 、N F 、N U They represent the number of background seeds, the number of foreground seeds and the number of unlabeled pixels respectively. Therefore, the objective function can be decomposed into the following formula:

[0043]

[0044] Among them, X U represents the set of unlabeled pixels, At this time, the matrix D B and D F The dimension is N U ×N U , L S Determined by the rows and columns of the marked pixels in the L matrix, L U The R matrix is ​​determined by the rows and columns of unlabeled pixels, and the rows are determined by the seed pixel index, and the columns are determined by the unlabeled pixel index, which is an adjacency weight matrix. The objective function is further expanded as follows:

[0045]

[0046] S32. Use the alternating direction multiplier method to solve the reconstructed objective function:

[0047] In order to effectively solve the above formula (17), let L S X S -RX U =d0,L U X U -R T X S =d1, by introducing the Lagrange multiplier and constructing the augmented Lagrangian function, the objective function is expanded to the following formula:

[0048]

[0049] Among them, b0 and b1 are introduced Lagrange multipliers, ρ is the Lagrange penalty coefficient, and ρ>0; for solving the final objective function formula (19), the objective function only contains the L1 norm and L2 norm and has no constraints, and the derivative can be directly minimized.

[0050] Further, the step S4 comprises the following steps:

[0051] S41, the sparse local linear reconstruction regularization term in the reconstructed objective function is replaced by L p The norm is constrained, and the objective function is redefined as follows:

[0052]

[0053] S42, using iterative reweighting method to p The norm solution is transformed into the L1 norm problem solution. For each L p The norm terms are all converted into L1 norms for constraint. Specifically, formula (26) is modified as shown below:

[0054]

[0055] Among them, N F Indicates the number of foreground seeds, Represents the vector L s X s -RX U An element in i express N B represents the number of background seeds, ψ i Representation vector - R T X s +L U X U An element in i Indicates |ψ i | p-1 ;

[0056] Transformed into a diagonal matrix, we have ‖diag(β)ψ‖1, then L p The objective function corresponding to the norm is transformed into the following formula:

[0057]

[0058] in, ψ=-R T X s +L U X U , α is given by α i The vector composed of β is i The vector composed of L p The regular embedding has been converted to L1 regular embedding. In order to ensure the stability of the calculation, the weight item is designed to be updated in the next round of iteration after each round of iteration as follows:

[0059]

[0060] Where k represents the current iteration round, and ε is a very small positive number to prevent it from being divided by zero. At this time, the objective function solution is transformed into the same form as equation (17), which will not be repeated here.

[0061] Furthermore, the final segmentation result obtained by the threshold method in step S5 includes: converting the x vector into y i ∈{x B ,x F}, i∈V, the threshold segmentation adopts the following rules:

[0062]

[0063] Among them, x F represents the foreground pixel value, x B Represents the background pixel value, x i It means the value of the marked pixel is obtained after minimizing the objective function.

[0064] Compared with the prior art, the interactive image segmentation method combining global seed and sparse local linear reconstruction provided by the present invention has the following advantages:

[0065] 1. The present invention proposes a new interactive segmentation model, which fully mines seed information, introduces global seed information items, constructs global seed information flows, establishes connections between seeds and pixels outside the neighborhood, and increases the seed propagation capability. In this way, a relatively complete segmentation object can still be obtained with less user interaction.

[0066] 2. The present invention not only considers the global propagation of seed information, but also uses a sparse local linear reconstruction regularization term to retain the local structural information of the image while better retaining the weak structural information.

[0067] 3. The present invention adopts L p The norm constrains the output gradient to obtain a sparser smooth result, which helps to produce clear boundaries for label propagation. When p=1, the alternating direction method of multipliers (ADMM) is used to iteratively minimize the objective function. When 0<p<1, iterative reweighting is used to accelerate matrix operations using preconditioned conjugate gradient (PCG), thereby improving computational efficiency, and its effectiveness is verified through experiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 This is a segmentation result diagram of some classic algorithms involved in the present invention, which leads to segmentation difficulties due to pixel differences between different regions on the same object.

[0069] Figure 2 It is a global seed information model diagram constructed by the present invention.

[0070] Figure 3 It is a functional display diagram of the boundary positioning capability of the present invention.

[0071] Figure 4 The model of the present invention adopts L1 regularization and L p (p = 0.7) and L p (p=0.8) segmentation results comparison diagram.

[0072] Figure 5 This is a visual comparison diagram of the segmentation results of the model of the present invention and several classic algorithms.

[0073] Figure 6 PR curve, F-score and AUC value of the present invention on the BSD data set.

[0074] Figure 7 It is the PR curve, F-score and AUC value of the present invention on the MSRC dataset.

[0075] Figure 8 PR curve, F-score and AUC value of the present invention using S1 seed on the MSRC dataset.

[0076] Fig. 9 PR curve, F-score and AUC value of the present invention using S2 seeds on the MSRC dataset.

[0077] Fig.10 This is a multi-classification segmentation result diagram obtained by the present invention.

[0078] Fig.11 It is a diagram of the robustness analysis results of the present invention at different seed placement positions.

[0079] Fig.12 This is a diagram showing the robustness analysis results of the present invention under Gaussian noise. DETAILED DESCRIPTION

[0080] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below with reference to specific diagrams.

[0081] The present invention provides an interactive image segmentation method combining global seed and sparse local linear reconstruction, comprising the steps of:

[0082] S1. The present invention designs a new interactive segmentation model. By introducing a global seed information item and constructing a global seed information flow, a relatively complete segmentation object can be obtained in the case of sparse seeds. The specific global seed information item is shown in the following formula:

[0083]

[0084] Among them, F represents the foreground model, B represents the background model, V represents the vertex set of the graph, and x i Represents unknown pixel, w i* Represents the weight of the unknown pixel to the foreground model or background model, x * Indicates whether the pixel belongs to the foreground or background.

[0085] As a specific embodiment, the step S1 includes the following steps:

[0086] S11. Gaussian mixture model is used to model the user's interactive information, and the foreground model and background model are obtained by solving the EM algorithm:

[0087] In the spatial domain of a picture, the image is represented by a three-channel matrix (r, g, b) consisting of n pixels. Given a set of seed pixels of background B and foreground F, the seed diffusion of the present invention is no longer limited to the local area of ​​the seed, but spreads to the entire image, such as Figure 2 The present invention adopts Gaussian mixture model to construct the color data model. Considering the different color information in the image, in order to make each unknown pixel point find the pixel point that is most similar to itself in the labeled data as much as possible, the present invention performs mixed Gaussian modeling for the foreground and background respectively, and sets each Gaussian component to K. Then the Gaussian mixture model probability model of the background B and the foreground F is expressed as follows:

[0088]

[0089] Where v∈{F,B}; I i represents the observed data; Θ v =(π v1 ,…,π vk ,θ v1 ,…,θ vk ) represents the parameter set; K represents the number of Gaussian components; π vk represents the weight of the kth Gaussian component in the model v; N vk (I i |θ vk ) represents the parameter θ vk Determined Gaussian distribution, N vk (I i |θ vk ) are the two parameters of the mean μ vk and covariance∑ vk , N vk (I i |θ vk ) is:

[0090]

[0091] Use the EM algorithm to solve the background and background Gaussian mixture model to obtain the optimal parameters And combined with the Bayesian formula to calculate the i-th observation data I i The probability of belonging to the kth Gaussian is calculated as follows:

[0092]

[0093] Among them, p v represents the Gaussian mixture probability distribution of the background model or foreground model; z i Represents the i-th observation data I i The label belonging to the kth Gaussian component; π vl represents the weight of the lth Gaussian component in model v; N vl (I i |θ vl ) represents the parameter θ vl Determine the Gaussian distribution.

[0094] S12. Calculate the probability of the image pixel to the foreground model and the background model as the distance from the unlabeled pixel to the foreground and the background, and construct the foreground matrix and the background matrix so that the seed information can be globally propagated:

[0095] Through the Gaussian mixture model, each pixel x i Bring in the foreground and background models to calculate the probability and set the weight w iF and w iB are defined as the probability of the unlabeled pixel to the foreground model and the background model respectively.

[0096] Among them, the distance calculation formula from the unmarked pixel to the foreground is as follows:

[0097]

[0098] The distance calculation formula from unmarked pixels to the background is as follows:

[0099]

[0100] Among them, P F represents the foreground model, P B represents the background model, k represents the kth Gaussian component, denotes the optimal model parameters for the foreground model, represents the optimal model parameters of the background model. iF and w iB A prospect matrix can be constructed and background matrix The elements satisfy Right now

[0101] Given background and foreground labels x respectively B and x F , based on the above discussion, the following global seed information item can be constructed:

[0102]

[0103] S2. Based on the idea of ​​local linear reconstruction, a sparse local linear reconstruction regularization term is designed, and the L1 norm is used to achieve the smoothness of the segmentation result. At the same time, the final objective function constrained by the L1 norm is obtained by combining the global seed information term in step S1. The specific sparse local linear reconstruction regularization term is shown in the following formula:

[0104]

[0105] in, represents the degree of pixel i, N(i) represents the local set of neighboring pixels, and w ij represents the edge weight, x j is a pixel in N(i);

[0106] The objective function is as follows:

[0107]

[0108] Where X=(x1,x2,…,x n ) represents the final solution of the objective function, n is the total number of pixels; λ is a balance parameter used to balance the sparse local linear reconstruction regularization term and the global seed information term.

[0109] As a specific embodiment, step S2 includes the following steps:

[0110] S21. Calculate the graph weight model based on the input image:

[0111] Let the input image be I. In order to construct the sparse local linear reconstruction regularization term, the present invention defines a weight graph G = (V, E, W E ), where V represents the vertex set of the graph, whose elements are the pixels of the image, E is the edge set of all connected pixels in the eight-neighborhood, and W E is the weight set of the edge; the local neighbor pixel set N(i) = {j:(i,j)∈E} represents the set of all pixels in the eight neighborhoods of the current pixel i. Represents the degree of pixel i.

[0112] There are many methods for calculating the edge weight of a graph, and strategies such as pixel intensity and image gradient can be used. The edge weight of the present invention is specifically defined as follows:

[0113]

[0114] Where exp(·) represents the exponential function, σ=max (i,j)∈E ||I i -I j || ∞ In order to ensure the stability of propagation, this weight calculation method ensures that the edge weight value is always a positive number and is symmetrical, that is, w ij =w ji ;I i and I j They are all color vectors. The pixel intensity is measured by RGB or Lab color space. The weight matrix model of the final image is as follows:

[0115]

[0116] S22, the propagation of local seed information is constrained by L1 norm, and the final objective function is obtained by combining the foreground matrix and background matrix obtained in step S1:

[0117] Based on the principle of local linear reconstruction, the value of a pixel is represented by its eight neighboring pixels. Finally, the sparse local linear reconstruction regularization term designed by the present invention is as follows:

[0118]

[0119] Combined with the above step S1, the model proposed by the present invention finally consists of two parts, namely the global seed information term and the sparse local linear reconstruction regularization term. The objective function expression of the model is as follows:

[0120] E(X)=λE G (X)+E S (X) Formula (13)

[0121] Where X=(x1,x2,…,x n ) represents the final solution of the objective function, n is the total number of pixels; λ is a balance parameter used to balance the sparse local linear reconstruction regularization term and the global seed information term. The larger the λ, the greater the influence of the global seed information term. E G (X) is the global seed information item, E S (X) is the sparse local linear reconstruction regularization term. Therefore, the image segmentation task is finally transformed into minimizing the objective function E(X), that is,

[0122] S3. The obtained objective function is iteratively solved by the alternating direction multiplier method (ADMM) to obtain a more complete segmentation object while achieving a more sparse regional boundary segmentation effect.

[0123] As a specific embodiment, step S3 includes the following steps:

[0124] S31, separate the marked pixels and the unmarked pixels, and reconstruct the energy function:

[0125] In order to effectively solve the global seed information flow and sparse local linear reconstruction regularization algorithm proposed in the present invention, the present invention will adopt the iterative method of alternating direction multiplier method to solve.

[0126] By setting ρ to 1, the objective function of the L1 norm constraint can be obtained as follows:

[0127]

[0128] In order to ensure that the pixels obtained by the objective function in the foreground and background areas are consistent with the marked pixels, as well as As the constraint term of the objective function, the matrix form of the objective function can be written as shown below:

[0129]

[0130] Where L = DW, L is the Laplace matrix; D is a diagonal matrix that satisfies D ii =d i , 1 is a vector of all 1s; W is the adjacency weight matrix; D B and D F Represent the constructed background weight matrix and foreground weight matrix respectively, where the elements satisfy Right now

[0131] In order to minimize the objective function and avoid the memory consumption and time overhead caused by the inversion of the extremely large sparse matrix, the marked pixels and the unmarked pixels are separated. By reorganizing the L matrix, it is simple and effective without affecting the logical structure of the L matrix; U represents the set of unknown pixels, S is the set of seed pixels, N B 、N F 、N U They represent the number of background seeds, the number of foreground seeds and the number of unlabeled pixels respectively. Therefore, the objective function can be decomposed into the following formula:

[0132]

[0133] Among them, X U represents the set of unlabeled pixels, At this time, the matrix D B and D FThe dimension is N U ×N U , L S Determined by the rows and columns of the marked pixels in the L matrix, L U The R matrix is ​​determined by the rows and columns of unlabeled pixels, and the rows are determined by the seed pixel index, and the columns are determined by the unlabeled pixel index, which is an adjacency weight matrix. The objective function is further expanded as follows:

[0134]

[0135] S32. Use the alternating direction multiplier method to solve the reconstructed objective function:

[0136] For the specific L1 regularization constraint term of the present invention, in order to effectively solve the above equation (17), let L S X S -RX U =d0,L U X U -R T X S =d1, then the problem is equivalent to minimizing the following formula:

[0137]

[0138] By introducing the Lagrange multiplier and constructing the augmented Lagrangian function, the objective function is expanded to the following formula:

[0139]

[0140] Among them, b0 and b1 are introduced Lagrange multipliers, ρ is the Lagrange penalty coefficient, and ρ>0; for solving the final objective function formula (19), the objective function only contains the L1 norm and L2 norm and has no constraints, and the derivative can be directly minimized.

[0141] Specifically, equation (19) can be solved by iterative minimization (ADDM), where the steps for solving the minimization subproblem of each parameter in the k+1th iteration are as follows:

[0142] First, fix d0, d1, b0, b1 and solve for X U :

[0143]

[0144] Then finally for The solution of can be transformed into the solution of the following linear equations:

[0145]

[0146] in, variable These are all intermediate variables introduced in the ADDM algorithm.

[0147] Next, fix X U , b0, b1, solve for d0, d1 respectively:

[0148]

[0149] The shrink operation here can be defined as max(·) represents the maximum value function, y represents the input vector, and γ represents the penalty factor.

[0150] Finally, fix X U , d0, d1, solve b0, b1 d1 respectively:

[0151]

[0152] In summary, the present invention summarizes all the steps of minimizing the energy function (Equation 19) and obtaining the unknown pixel solution in Algorithm 1, where the subproblem (solving X U ) is defined in lines 3-5. There are three parameters λ, ρ, and η in this algorithm, where η is used to control the end of the loop, and the functions of λ and ρ are the same as described above. The specific algorithm 1 is as follows:

[0153] Input: input image g, seed information X S ,λ,ρ,η

[0154] Output: unlabeled pixel set X U

[0155] 1: Initialization:

[0156] 2:

[0157] 3: Construct the A matrix as shown in formula (21)

[0158] 4: Construct the v vector as shown in formula (21)

[0159] 5: By solving renew

[0160] 6: Calculate formula (22) to obtain

[0161] 7: Calculate formula (23) to obtain

[0162] 8: Calculate formula (24) to obtain

[0163] 9: Calculate formula (25) to obtain

[0164] 10: k = k + 1

[0165] 11: end while

[0166] 12:

[0167] S4. Modify the L1 norm to L p norm, where 0<p<1, L is iterated by reweighting p The norm is converted into the L1 norm, and the non-convex optimization problem is converted into a convex optimization problem to obtain the solution that minimizes the objective function.

[0168] As a specific embodiment, step S4 includes the following steps:

[0169] S41, the sparse local linear reconstruction regularization term in the reconstructed objective function is replaced by L p The norm is constrained:

[0170] In order to improve the robustness of the algorithm and further improve the sparsity of the solution, the present invention adopts L p (0<p<1) regular embedding is used to obtain a sparser solution. After the constraint, the objective function is redefined as follows:

[0171]

[0172] S42, using iterative reweighting method to p The norm solution is transformed into the L1 norm problem solution. For each L p The norm terms are all converted to L1 norms for constraints:

[0173] When 0<p<1, equation (26) is a non-convex optimization problem. When faced with a complex objective function optimization task, a wise strategy is to try to convert it into a more simplified optimization problem. At present, a class of optimization methods based on iterative reweighting has been developed. These methods iteratively optimize smooth or non-smooth convex objective functions, and the weight of each round is obtained by dynamically weighting the coefficients evaluated in the previous iteration. According to the above method and in order to reflect the iterative reweighting process, the present invention optimizes and derives L based on L1 p Solution method: Specifically, modify equation (26) as shown below:

[0174]

[0175] Among them, N F Indicates the number of foreground seeds, Represents the vector Ls X s -RX U An element in i express N B represents the number of background seeds, ψ i Representation vector - R T X s +L U X U An element in i Indicates |ψ i | p-1 ;

[0176] Transformed into a diagonal matrix, we have ‖diag(β)ψ‖1, then L p The objective function corresponding to the norm is transformed into the following formula:

[0177]

[0178] in, ψ=-R T X s +L U X U , α is given by α i The vector composed of β is i The vector composed of L p The regular embedding has been converted to L1 regular embedding; in order to ensure the stability of the calculation, the weight item designed by the present invention is updated in the following way after each round of iteration:

[0179]

[0180] Among them, k represents the current iteration round, and ε is a very small positive number to prevent it from being divided by zero. At this time, the objective function solution is transformed into the same form as formula (17). Next, we only need to use a solution method similar to formula (17), which will not be repeated here.

[0181] S5. According to the obtained solution of minimizing the objective function, the final segmentation result is obtained by the threshold method:

[0182] Once the value of the minimization result X vector of the above formula (20) is obtained, combined with the above known X S Vector, you can use some simple and convenient classification algorithms to help get the final segmentation result, such as clustering algorithms such as k-means or threshold segmentation methods such as Otsu, to convert the x vector into y i ∈{x B ,x F}, i∈V, the simplest threshold segmentation can adopt the following rules:

[0183]

[0184] Among them, x F represents the foreground pixel value, x B Represents the background pixel value, x i It means the value of the marked pixel is obtained after minimizing the objective function.

[0185] Figure 4 Demonstrates the use of L1 regularization and L p (p = 0.7) and L p (p = 0.8) segmentation results on the same mark. It can be seen from the figure that L p (p = 0.8) Regularization can better distinguish the object boundaries and obtain more accurate segmentation results, which is better than L1 and L p (p=0.7) norm-constrained GSSR model. Therefore, in order to achieve more accurate results, for L p (0<p<1) Constrained GSSR Model The present invention will use p=0.8 for subsequent experimental comparison.

[0186] This paper proposes a new algorithm framework of global seed information term and sparse local linear reconstruction regularization term. This algorithm can not only segment more complete objects with less seed interaction, but also has excellent boundary localization ability compared with existing methods (see Figure 3 ). In order to verify the necessity and rationality of using sparse regularization constraints, the present invention uses L1 regularization and L p (p = 0.8) regularization is used for constraint.

[0187] The present invention combines GSSR1 and GSSR 0.8It is compared with several methods for interactive image segmentation, namely: Grow Cuts (GRO), Random Walk (RW), Normalized Random Walk (NRW), Normalized LazyRandom Walk (NLRW), One Cut (ONE), Laplacian Coordinates (LC), Multi-layer RandomForest (MRF). The present invention implements all the above-mentioned comparison algorithms in the MATLAB2020b programming environment. The above-mentioned methods are all open source, and all use the default parameters provided by the above-mentioned algorithms. In order to quantitatively evaluate the segmentation results of the algorithm, the present invention adopts some widely used evaluation indicators: Rand Index (RI), Information Difference Index (VoI), Boundary Drift Error (BDE), and Intersection Over Union (IOU). The feasibility of the GSSR framework is verified on two very famous datasets that provide GT: the BSD dataset and the MSRC dataset. Both datasets are open source and free. The MSRC dataset contains 50 natural images, and the BSD dataset contains 500 images, most of which have interference that makes segmentation more difficult, such as cluttered textures and different types of lighting conditions. In order to reduce the indicator error caused by the inconsistency of the seed information of subjective annotations, the annotations provided by Andrade and Carrera are further adopted. Two different sparse seed point sets S1 and S2 on the MSRC dataset are used to further demonstrate the advantages of the algorithm of the present invention. In order to more intuitively show the segmentation capabilities of the above-mentioned methods compared with the algorithm of the present invention, Figure 5 The specific segmentation results of each algorithm on 9 images are presented. It can be seen from the figure that although most algorithms can output similar results, there are still some subtle differences that need to be emphasized. In the starfish image, there is no background seed set at the upper background. It can be seen that most algorithms cannot handle this situation, only ONE and GSSR can handle it. 0.8 In the church and boat images, MRF and LC algorithms both produced similar segmentation results but also worse than GSSR. 0.8 Slightly worse. When it comes to squirrel images, GSSR is slower when facing a more complex background. 0.8 The algorithm performs best. When comparing the rest of the images, we can see that GSSR1 and GSSR 0.8 The algorithms all obtain more precise edges and more complete targets.

[0188] In order to more comprehensively display the segmentation details, the present invention uses median, mean and variance to measure the four indicators of RI, VoI, BDE and IoU. At the same time, PR curve, F-score and AUC indicators are used to more comprehensively measure the boundary fitting of the segmentation results. Table 1 below shows the IoU, VoI, BDE and RI indicators of each segmentation algorithm on the BSD dataset. Note that the GSSR1 method and GSSR 0.8 The proposed method surpasses the above mentioned comparison methods in almost all indicators.

[0189] Table 1 Quantitative results of IoU, VoI, BDE, and RI indicators of various methods on the BSD dataset

[0190]

[0191] In order to further evaluate the boundary fitting ability of the segmentation algorithm, Figure 6 (a) shows the PR curve of each algorithm. The higher the recall rate and precision rate, the better the segmentation ability of the algorithm. At the same time, in order to comprehensively consider both precision rate and recall rate, we Figure 6 (b) shows an overview of the F-score and AUC indicators to evaluate the model. 0.8 The boundary fit is very similar, but better than GSSR 0.8 Slightly worse. Both methods show better marginal fitting ability than other comparison methods and the highest F-score and AUC values ​​among all methods.

[0192] The experimental results on the MSRC dataset are shown in Table 2. From the data shown in the table, the model with global seed information has good performance, whether using L1 norm or L p (p=0.8) norm, and the performance of almost all evaluation indicators is better than other segmentation methods. This shows that the method proposed in this invention is better in overall segmentation quality and the segmentation is more stable. Similar to the BSD dataset, the PR curve, F-score and AUC indicators are as follows Figure 7 As shown, you can see GSSR 0.8 Achieved the highest score.

[0193] Table 2 Quantitative analysis of IoU, VoI, BDE, and RI indicators of each method on the MSRC dataset

[0194]

[0195] In order to reduce the index error caused by the inconsistency of seed information labeled by manual subjective methods. The present invention uses a sparse seed set S1 labeled on the MSRC dataset to further conduct a more in-depth and complex evaluation of the above algorithm. The following Table 3 summarizes the indicators of each algorithm when using sparse seed S1. By analyzing the data in the above table, the following conclusions can be drawn: When a small number of labeled seeds are used, the algorithm using the global seed information flow is in the best position in almost all indicators, and each indicator has a more significant improvement. Although GSSR1 and GSSR 0.8 The algorithm is slightly different, but all indicators are still in the second place among all the above segmentation algorithms. Figure 8 The PR curve, F-score and AUC charts are given in the figure.

[0196] Table 3 Quantitative analysis of IoU, VoI, BDE, and RI indicators of various methods on the MSRC dataset using S1 seeds

[0197]

[0198]

[0199] Table 4 below shows another set of comparisons on the MSRC dataset, which uses another set of sparse seed set S2 provided by Andrade and Carrera. From Table 4, it can be seen that the performance of the method proposed in the present invention in terms of mean and variance completely exceeds the above-mentioned comparison methods, both ranking first and second, indicating that GSSR1 and GSSR 0.8 The overall segmentation quality is higher and more stable. Moreover, the algorithm of the present invention is ranked second or third among all the methods in terms of median value, only behind the ONE method. Fig. 9 The specific situation of PR curve, F-score and AUC indicators of each model on the MSRC dataset using S2 is shown.

[0200] Table 4 Quantitative analysis of IoU, VoI, BDE, and RI indicators of various methods on the MSRC dataset using S2 seeds

[0201]

[0202] GSSR p It can be easily expanded to a multi-classification problem. Here, the L1 norm is used as an example. For an N-class problem, it is only necessary to solve equation (19) N-1 times like solving a binary classification problem.

[0203]

[0204] Solving the above equation can get the final segmentation result. The present invention shows some multi-region segmentation results, such as Fig.10 As shown. Define the seed tag set S, and different types of tags are represented as Where k = {1,…,N}, then for the jth solution Satisfies the following formula:

[0205]

[0206] Fig.11 Demonstrated GSSR p The robustness of the random seed position of , can generate different segmentation results by simply marking different objects in the image. Fig.11 From the three input seeds (left, middle, and right columns), we can see that only from GSSR p By sowing seeds into the model, multiple targets can be accurately segmented. In order to highlight the robustness of the algorithm in noise resistance, Fig.12 In the figure, the present invention shows several image segmentation results under the influence of Gaussian noise. It can be seen that under the influence of Gaussian noise with a mean of 0.2 and a variance of 0.01, the algorithm of the present invention can still segment the target object more accurately.

[0207] Compared with the prior art, the interactive image segmentation method combining global seed and sparse local linear reconstruction provided by the present invention has the following advantages:

[0208] 1. The present invention proposes a new interactive segmentation model, which fully mines seed information, introduces global seed information items, constructs global seed information flows, establishes connections between seeds and pixels outside the neighborhood, and increases the seed propagation capability. This allows a relatively complete segmentation object to be obtained even with less user interaction.

[0209] 2. The present invention not only considers the global propagation of seed information, but also uses a sparse local linear reconstruction regularization term to retain the local structural information of the image while better retaining the weak structural information.

[0210] 3. The present invention adopts L p The norm constrains the output gradient to obtain a sparser smooth result, which helps to produce clear boundaries for label propagation. When p=1, the alternating direction method of multipliers (ADMM) is used to iteratively minimize the objective function. When 0<p<1, iterative reweighting is used to accelerate matrix operations using preconditioned conjugate gradient (PCG), thereby improving computational efficiency, and its effectiveness is verified through experiments.

[0211] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. An interactive image segmentation method combining global seeding and sparse local linear reconstruction, characterized in that: Includes steps: S1. Introduce global seed information items and construct global seed information flow to obtain relatively complete segmentation objects in the case of sparse seeds. The specific global seed information items are shown in the following formula: Among them, F represents the foreground, B represents the background, V represents the vertex set of the graph, and x i Represents unknown pixel, w i* Represents the weight of the unknown pixel to the foreground model or background model, x * Indicates whether the pixel belongs to the foreground or background; S2. Based on the idea of ​​local linear reconstruction, a sparse local linear reconstruction regularization term is designed, and the L1 norm is used to achieve the smoothness of the segmentation result. At the same time, the final objective function constrained by the L1 norm is obtained by combining the global seed information term in step S1. The specific sparse local linear reconstruction regularization term is shown in the following formula: in, represents the degree of pixel i, N(i) represents the local set of neighboring pixels, and w ij represents the edge weight, x j is a pixel in N(i); The objective function is as follows: Where X=(x1,x2,…,x n ) represents the final solution of the objective function, n is the total number of pixels; λ is a balance parameter used to balance the sparse local linear reconstruction regularization term and the global seed information term; S3, the obtained objective function is iteratively solved by the alternating direction multiplier method, so as to obtain a more complete segmentation object and achieve a more sparse regional boundary segmentation effect; S4. Modify the L1 norm to L p norm, where 0<p<1, L is iterated by reweighting p The norm is converted into the L1 norm, and the non-convex optimization problem is converted into a convex optimization problem to obtain the solution that minimizes the objective function; S5. According to the obtained solution of minimizing the objective function, the final segmentation result is obtained through a threshold method.

2. The interactive image segmentation method combining global seed and sparse local linear reconstruction according to claim 1, characterized in that: The step S1 comprises the following steps: S11. The user's interactive information is modeled using a Gaussian mixture model, and the foreground model and the background model are obtained by solving the EM algorithm. The calculation formula is as follows: Among them, p v represents the Gaussian mixture probability distribution of the background model or foreground model; z i Represents the i-th observation data I i The label belonging to the kth Gaussian component; v∈{F,B}; Θ v =(v v1 ,…,π vk ,θ v1 ,…,θ vk ) represents a parameter set; N vk (I i |θ vk ), N vl (I i |θ vl ) represent the parameters θ vk ,θ vl Determined Gaussian distribution; π vk , π vl They represent the weights of the kth and lth Gaussian components in the model v respectively; K represents the number of Gaussian components; S12, calculate the probability of the image pixel to the foreground model and the background model, as the distance from the unlabeled pixel to the foreground and the background, construct the foreground matrix and the background matrix, so that the seed information can be globally propagated; wherein, the distance calculation formula from the unlabeled pixel to the foreground is as follows: The distance calculation formula from unmarked pixels to the background is as follows: Among them, P F represents the foreground model, P B represents the background model, k represents the kth Gaussian component, represents the optimal model parameters of the foreground model, Represents the optimal model parameters of the background model.

3. The interactive image segmentation method combining global seed and sparse local linear reconstruction according to claim 2, characterized in that: The step S2 comprises the following steps: S21. Calculate the graph weight model based on the input image: Let the input image be I. In order to construct the sparse local linear reconstruction regularization term, a weight graph G = (V, E, W E ), where V represents the vertex set of the graph, whose elements are the pixels of the image, E is the edge set of all connected pixels in the eight-neighborhood, and W E is the weight set of the edge; the local neighbor pixel set N(i) = {j:(i,j)∈E} represents the set of all pixels in the eight neighborhoods of the current pixel i. The edge weight is specifically defined as follows: Where exp(·) represents the exponential function, σ=max (i,j)∈E ||I i -I j || ∞ In order to ensure the stability of propagation, this weight calculation method ensures that the edge weight value is always a positive number and is symmetrical, that is, w ij =w ji ;I i and I j They are all color vectors. The pixel intensity is measured by RGB or Lab color space. The weight matrix model of the final image is as follows: S22, using the L1 norm to constrain the propagation of local seed information, combining the foreground matrix and the background matrix obtained in step S1, to obtain the final objective function.

4. The interactive image segmentation method combining global seed and sparse local linear reconstruction according to claim 3, characterized in that: The step S3 comprises the following steps: S31, separate the marked pixels and the unmarked pixels, and reconstruct the energy function: In order to ensure that the pixels obtained by the objective function in the foreground and background areas are consistent with the marked pixels, x i =x B , and x i =x F , As the constraint term of the objective function, the matrix form of the objective function can be written as shown below: Where L = DW, L is the Laplace matrix; D is a diagonal matrix that satisfies D ii =d i , 1 is a vector of all 1s; W is the adjacency weight matrix; D B and D F Represent the constructed background weight matrix and foreground weight matrix respectively, where the elements satisfy Right now In order to minimize the objective function and avoid the memory consumption and time overhead caused by the inversion of the extremely large sparse matrix, the marked pixels and the unmarked pixels are separated. By reorganizing the L matrix, it is simple and effective without affecting the logical structure of the L matrix; U represents the set of unknown pixels, S is the set of seed pixels, N B 、N F 、N U They represent the number of background seeds, the number of foreground seeds and the number of unlabeled pixels respectively. Therefore, the objective function can be decomposed into the following formula: Among them, X U represents the set of unlabeled pixels, At this time, the matrix D B and D F The dimension is N U ×N U , L S Determined by the rows and columns of the marked pixels in the L matrix, L U The R matrix is ​​determined by the rows and columns of unlabeled pixels, and the rows are determined by the seed pixel index, and the columns are determined by the unlabeled pixel index, which is an adjacency weight matrix. The objective function is further expanded as follows: S32. Use the alternating direction multiplier method to solve the reconstructed objective function: In order to effectively solve the above formula (17), let L S X S -RX U =d0,L U X U -R T X S =d1, by introducing the Lagrange multiplier and constructing the augmented Lagrangian function, the objective function is expanded to the following formula: Among them, b0 and b1 are introduced Lagrange multipliers, ρ is the Lagrange penalty coefficient, and ρ>0; for solving the final objective function formula (19), the objective function only contains the L1 norm and L2 norm and has no constraints, and the derivative can be directly minimized.

5. The interactive image segmentation method combining global seed and sparse local linear reconstruction according to claim 4, characterized in that: The step S4 comprises the following steps: S41, the sparse local linear reconstruction regularization term in the reconstructed objective function is replaced by L p The norm is constrained, and the objective function is redefined as follows: S42, using iterative reweighting method to p The norm solution is transformed into the L1 norm problem solution. For each L p The norm terms are all converted into L1 norms for constraint. Specifically, formula (26) is modified as shown below: Among them, N F Indicates the number of foreground seeds, Represents the vector L s X s -RX U An element in i express N B represents the number of background seeds, ψ i Representation vector - R T X s +L U X U An element in i Indicates |ψ i | p-1 ; Transformed into a diagonal matrix, we have ‖diag(β)ψ‖1, then L p The objective function corresponding to the norm is transformed into the following formula: in, ψ=-R T X s +L U X U , α is given by α i The vector composed of β is i The vector composed of L p The regular embedding has been converted to L1 regular embedding. In order to ensure the stability of the calculation, the weight item is designed to be updated in the next round of iteration after each round of iteration as follows: Where k represents the current iteration round, and ε is a very small positive number to prevent it from being divided by zero. At this time, the objective function solution is transformed into the same form as equation (17), which will not be repeated here.

6. The interactive image segmentation method combining global seed and sparse local linear reconstruction according to claim 5, characterized in that: The final segmentation result obtained by the threshold method in step S5 includes: converting the x vector into y i ∈{x B ,x F }, i∈V, the threshold segmentation adopts the following rules: Among them, x F represents the foreground pixel value, x B Represents the background pixel value, x i It means the value of the marked pixel is obtained after minimizing the objective function.

Images