Image segmentation method based on sparse optimization algorithm heuristic neural network

Abstract

The invention discloses an image segmentation method based on a sparse optimization algorithm heuristic neural network. The method comprises the following steps: firstly, constructing a mathematical optimization model of a sparse feature image segmentation problem; secondly, solving the mathematical optimization model by using a Proximal Point Algorithm (PPA), and constructing a neural network module, namely a PPA module, according to the algorithm; then, a PPA module is used for building a U-shaped neural network for solving the image segmentation problem through sparse features; dividing a training set, designing a loss function, and completing neural network training; and finally, inputting an image, and completing image segmentation by using the trained neural network. According to the method, the neural network is designed under the inspiration of the approximate point algorithm, the interpretability in mathematical meaning is given to the neural network structure, and in the face of an image segmentation task with sparse features, the segmentation effect can be improved while the parameter quantity of a network model is reduced, and more accurate segmentation is realized.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, relates to the field of image segmentation technology, and in particular to an image segmentation method based on a sparse optimization algorithm-inspired neural network. Background Art

[0002] With the development of deep learning, neural networks have been widely used in the field of image segmentation. The process of using neural networks to segment the target area from the complete image is fully automatically learned by computers based on data, eliminating the manual dependence in traditional segmentation methods. Based on this background, many complex neural networks such as U-net have emerged to widely solve image segmentation problems. However, such neural networks face certain challenges when dealing with problems in specific scenarios, such as segmenting images with sparse features, that is, images where the targets to be segmented are sparsely distributed everywhere. The general neural network framework cannot utilize the sparsity characteristics unique to the problem, which is not only a waste of known information, but also leads to poor segmentation results.

[0003] Many scholars have tried to use optimization methods to assist neural networks in completing specific tasks. They have greatly improved the application effect of neural networks by establishing a mathematical model that describes the characteristics of the problem and designing the neural network structure inspired by the optimization algorithm for solving the model. Initially, this type of method had a significant impact on the field of compressed sensing. In this field, scholars have conducted a step-by-step exploration of neural networks inspired by the iterative soft threshold algorithm (ISTA). These methods complement each other, using the sparse characteristics of signals in compressed sensing for mathematical modeling, and using neural networks that rely on big data learning instead of fixed iterative algorithms for solving, fully integrating the advantages of both.

[0004] The poor segmentation effect of general neural networks on sparse feature images is often due to insufficient segmentation accuracy in image details. Therefore, using ideas inspired by optimization algorithms and introducing sparsity constraints into the mathematical modeling and network design process can improve the neural network's processing of sparse image details, enhance the neural network's ability to extract sparse features, and effectively improve the segmentation accuracy of sparse targets. Summary of the invention

[0005] The technical problem to be solved by the present invention is to address the defects involved in the background technology and provide an image segmentation method based on a sparse optimization algorithm inspired by a neural network. The Proximal Point Algorithm (PPA) for solving sparse optimization problems is combined with the neural network U-net, which can not only reduce the number of parameters of the U-net model, but also show better results when segmenting images with sparse features.

[0006] The present invention adopts the following technical solutions to solve the above technical problems:

[0007] An image segmentation method based on a sparse optimization algorithm inspired neural network includes the following steps:

[0008] Step 1), modeling the image segmentation problem with sparse features and constructing a mathematical optimization model for minimizing the l1 norm;

[0009] Step 2), using PPA to solve the mathematical optimization model, and building N neural network modules under the guidance of the PPA iteration format, that is, N PPA modules, N is an odd number;

[0010] Step 3), using the PPA module to build a U-shaped neural network that uses sparse features to solve the image segmentation problem;

[0011] Step 4), randomly divide the training set, design the loss function and train the U-shaped neural network to obtain a trained U-shaped neural network;

[0012] Step 5), input the image to be segmented into the trained U-shaped neural network for segmentation to obtain its segmentation probability map, and then perform image binarization processing on the segmentation probability map to obtain the final segmentation result.

[0013] As a further optimization scheme of the image segmentation method based on the sparse optimization algorithm inspired neural network of the present invention, the detailed steps of step 1) are as follows:

[0014] The segmentation problem of the input sparse image x is modeled and the following mathematical optimization model is obtained:

[0015]

[0016] Here, F(x) represents a learnable nonlinear transformation F(·) applied to the image x; ||F(x)||1 represents the l1 norm of F(x), requiring the transformed image F(x) to be sparse.

[0017] As a further optimization scheme of the image segmentation method based on the sparse optimization algorithm inspired neural network of the present invention, the detailed steps of step 2) are as follows:

[0018] Step 2.1), for the mathematical optimization model, use PPA to iteratively solve it;

[0019] Step 2.1.1), construct an iterative formula to solve the problem through the sub-gradient descent algorithm. The k-th iteration process is implicitly expressed as the solution process of the following optimization problem:

[0020]

[0021] Among them, x k represents the input of the k-th iteration, x k+1 represents the output of the k-th iteration; y is the independent variable in the optimization problem, and λ is the preset weight parameter;

[0022] Step 2.1.2), further equivalent transformation is performed based on data statistical analysis to obtain the implicit iterative expression containing the minimization problem, as follows:

[0023]

[0024] Among them, θ is the preset weight parameter;

[0025] Step 2.1.3), directly use PPA to solve the minimization problem in the implicit iterative expression, and obtain the following iterative expression:

[0026] F(x k+1 )=prox(F(x k ))=sgn(F(x k ))·max(|F(x k )|-θ,0)

[0027] Among them, prox(·) represents the approximate point operator; sgn(·) represents the sign function, |·| is the absolute value operation, and max(·,·) is the maximum value operation;

[0028] Step 2.1.4), for the result F(x k+1 ) Take the inverse and get in, It is the inverse operation of F(·);

[0029] Step 2.2), according to the formula Construct N neural network modules, that is, N PPA modules. The construction steps of a single PPA module are as follows:

[0030] Step 2.2.1), arrange the 3*3 convolutional layer, batch normalization, rectified linear unit, and 3*3 convolutional layer in sequence to form F(·);

[0031] Step 2.2.2), construct the inverse operation Its structure is the same as F(·);

[0032] Step 2.2.3), strictly follow the mathematical expression prox(F(x k ))=sgn(F(x k ))·max(|F(x k )|-θ,0) constructs the approximate point operator prox(·), F(x k ) is the input to the operation;

[0033] Step 2.2.4), F(·), prox(·), Arrange them in order to form a PPA module. The order is consistent with the formula The priority of each operation is consistent; the priority of each operation is from x k Start by decreasing from near to far.

[0034] As a further optimization scheme of the image segmentation method based on the sparse optimization algorithm inspired neural network of the present invention, the detailed steps of step 3) are as follows:

[0035] Step 3.1), before each PPA module, add a 3*3 convolution operation with batch normalization and rectified linear unit, denoted as f(·);

[0036] Step 3.2), using the N PPA modules after adding the f(·) operation, arrange them into a neural network according to the U-shaped architecture of U-net;

[0037] Step 3.2.1), split the N PPA modules in half, and take the two groups of PPA modules before and after the median to construct the contraction path and expansion path in the U-shaped network; in the contraction path, the image needs to be max-pooled each time it passes through the PPA module; in the expansion path, the image needs to be up-sampled each time it passes through the PPA module;

[0038] Step 3.2.2), the PPA module located at the median is arranged between the contraction path and the expansion path, the PPA module takes the output result of the contraction path as input, and the calculation result is output to the expansion path after upsampling;

[0039] Step 3.2.3), introduce a skip connection between each pair of PPA modules that are symmetric about the median;

[0040] Step 3.3), organize the output results into probability graph representation;

[0041] At the end of the neural network expansion path, a convolution layer with a convolution kernel size of 1*1 is added; this convolution layer organizes the multi-channel feature map output by the expansion path into two single-channel feature maps, and then further converts them into two probability maps using a softmax layer. The value of each pixel in the probability map is between [0,1], indicating the probability that the pixel belongs to the segmentation target or the image background.

[0042] As a further optimization scheme of the image segmentation method based on the sparse optimization algorithm inspired neural network of the present invention, the detailed steps of step 4) are as follows:

[0043] Step 4.1), divide the dataset containing the image and its corresponding mask into a training set and a test set. The training set will be used for subsequent training of the neural network;

[0044] Step 4.2), use the following loss function Loss total Training a U-shaped neural network:

[0045] Loss total =Loss fin +ηLoss sym

[0046] Among them, Loss fin is the binary cross entropy function, Loss sym This is to ensure that the operation F(·) in each PPA module is consistent with Inverse loss function; Loss sym The operation process is: in each PPA module, skip The prox(·) operation in The output of the two operations is subtracted from the input, and the sum of the differences at all pixels is calculated; the differences output by all PPA modules are accumulated to obtain the final Loss sym ;

[0047] After determining the loss function, use the back propagation algorithm to minimize the loss total The neural network parameters are updated in the process.

[0048] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0049] 1. This method takes into account the sparsity of the target in the image when modeling, and expands the optimization algorithm for dealing with the sparsity problem into a neural network. The network structure has mathematical significance and strong interpretability;

[0050] 2. Compared with the classic U-net, the neural network provided by the present invention requires significantly fewer parameters to learn. Moreover, when faced with the problem of image segmentation with sparse features, the segmentation effect of this method is better than that of U-net. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a flow chart of the image segmentation method provided by the present invention.

[0052] Figure 2 It is a schematic diagram of the structure of the neural network provided by the present invention.

[0053] Figure 3 (a) Figure 3 (b) Figure 3 (c) are schematic diagrams showing the comparison of the original image, mask, and segmentation result in the present invention. DETAILED DESCRIPTION

[0054] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings:

[0055] The present invention can be implemented in many different forms and should not be considered to be limited to the embodiments described herein. On the contrary, these embodiments are provided to make this disclosure thorough and complete, and will fully express the scope of the present invention to those skilled in the art. In the accompanying drawings, components are enlarged for clarity.

[0056] In order to verify the effectiveness of the sparse optimization algorithm-based neural network method in the image segmentation task proposed by the present invention, the following will be described in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following implementation is only used as a reference to help those skilled in the art better understand the present invention. After reading the present invention, the modifications of various equivalent forms of the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0057] like Figure 1 As shown, the present invention provides an image segmentation method based on a sparse optimization algorithm inspired neural network, comprising the following steps:

[0058] Step 1), modeling the image segmentation problem with sparse features and constructing a mathematical optimization model for minimizing the l1 norm;

[0059] The segmentation problem of the input sparse image x is modeled and the following mathematical optimization model is obtained:

[0060]

[0061] Here, F(x) represents a learnable nonlinear transformation F(·) applied to the image x; ||F(x)||1 represents the l1 norm of F(x), requiring the transformed image F(x) to be sparse.

[0062] Step 2), using PPA to solve the mathematical optimization model, and building N neural network modules under the guidance of the PPA iteration format, that is, N PPA modules, N is an odd number, and in this embodiment, N is set to 5;

[0063] Step 2.1), for the mathematical optimization model, use PPA to iteratively solve it;

[0064] Step 2.1.1), construct an iterative formula to solve the problem through the sub-gradient descent algorithm. The k-th iteration process is implicitly expressed as the solution process of the following optimization problem:

[0065]

[0066] Among them, x k represents the input of the k-th iteration, x k+1 represents the output of the k-th iteration; y is the independent variable in the optimization problem, and λ is the preset weight parameter;

[0067] Step 2.1.2), further equivalent transformation is performed based on data statistical analysis to obtain the implicit iterative expression containing the minimization problem, as follows:

[0068]

[0069] Wherein, θ is a preset weight parameter, and in this embodiment, θ is set to 0.1;

[0070] Step 2.1.3), directly use PPA to solve the minimization problem in the implicit iterative expression, and obtain the following iterative expression:

[0071] F(x k+1 )=prox(F(x k ))=sgn(F(x k ))·max(|F(x k )|-θ,0)

[0072] Among them, prox(·) represents the approximate point operator; sgn(·) represents the sign function, |·| is the absolute value operation, and max(·,·) is the maximum value operation;

[0073] Step 2.1.4), for the result F(x k+1 ) Take the inverse and we get in, It is the inverse operation of F(·);

[0074] Step 2.2), according to the formula Construct 5 neural network modules, that is, 5 PPA modules. The construction steps of a single PPA module are as follows:

[0075] Step 2.2.1), arrange the 3*3 convolutional layer, batch normalization, rectified linear unit, and 3*3 convolutional layer in sequence to form F(·);

[0076] Step 2.2.2), construct the inverse operation Its structure is the same as F(·);

[0077] Step 2.2.3), strictly follow the mathematical expression prox(F(x k ))=sgn(F(x k ))·max(|F(x k )|-θ,0) constructs the approximate point operator prox(·), F(x k ) is the input to the operation;

[0078] Step 2.2.4), F(·), prox(·), Arrange them in order to form a PPA module. The order is consistent with the formula The priority of each operation is consistent; the priority of each operation is from x k Start by decreasing from near to far.

[0079] Step 3), using the PPA module to build a U-shaped neural network that uses sparse features to solve the image segmentation problem;

[0080] Step 3.1), before each PPA module, add a 3*3 convolution operation with batch normalization and rectified linear unit, denoted as f(·); the operation f(·) mainly changes the number of feature channels of the image; the five f(·) operations in this embodiment adjust the number of feature channels of the input feature map to 64, 128, 256, 128 and 64 respectively before outputting them to the PPA module;

[0081] Step 3.2), using the 5 PPA modules after adding the f(·) operation, arrange them into a neural network according to the U-shaped architecture of U-net;

[0082] Step 3.2.1), split the five PPA modules in half, take the first and second PPA modules to construct the contraction path in the U-shaped network, and take the fourth and fifth PPA modules to construct the expansion path; in the contraction path, the image needs to be max-pooled each time it passes through a PPA module; in the expansion path, the image needs to be up-sampled each time it passes through a PPA module;

[0083] Step 3.2.2), arranging the third PPA module between the contraction path and the expansion path, the PPA module takes the output result of the contraction path as input, and outputs the calculation result to the expansion path after upsampling;

[0084] Step 3.2.3), introduce skip connections between the 1st and 5th PPA modules and between the 2nd and 4th PPA modules respectively;

[0085] Step 3.3), organize the output results into probability graph representation;

[0086] At the end of the neural network expansion path, a convolution layer with a convolution kernel size of 1*1 is added; this convolution layer organizes the multi-channel feature map output by the expansion path into two single-channel feature maps, and then further converts them into two probability maps using a softmax layer. The value of each pixel in the probability map is between [0,1], indicating the probability that the pixel belongs to the segmentation target or the image background.

[0087] Step 4), randomly divide the training set, design the loss function and train the U-shaped neural network to obtain a trained U-shaped neural network;

[0088] Step 4.1), divide the data set containing the image and its corresponding mask into a training set and a test set; in this embodiment, the extended DRIVE (Digital Retinal Images for Vessel Extraction) is used as the data set. The images in the data set are all fundus retinal images, and the goal of the embodiment is to segment the fundus blood vessels from them. The extended DRIVE data set contains a total of 100 images with masks, of which 90 groups of images are used as training sets for network training, and 10 groups of images are used as test sets for network testing.

[0089] Step 4.2), use the following loss function Loss total Training a U-shaped neural network:

[0090] Loss total =Loss fin +ηLoss sym

[0091] Among them, Loss fin is the binary cross entropy function, Loss sym This is to ensure that the operation F(·) in each PPA module is consistent with Inverse loss function; Loss sym The operation process is: in each PPA module, skip The prox(·) operation in The output of the two operations is subtracted from the input, and the sum of the differences at all pixels is calculated; the differences output by all PPA modules are accumulated to obtain the final Loss sym ;

[0092] After determining the loss function, use the back propagation algorithm to minimize the loss total The neural network parameters are updated in the process.

[0093] Step 5), input the image to be segmented into the trained U-shaped neural network for segmentation to obtain its segmentation probability map, and then perform image binarization processing on the segmentation probability map to obtain the final segmentation result.

[0094] In this embodiment, the segmentation of 10 images in the above test set is completed and one group of images is selected to display the segmentation visual effect. Figure 3 (a) Figure 3 (b) is the display of the image and its mask. Figure 3 (c) is the corresponding segmentation result. Obviously, the neural network provided by the present invention can achieve relatively accurate segmentation of both the main trunk of the retina blood vessels and the small branches of the blood vessels.

[0095] On this basis, this embodiment compares the provided neural network with the classic U-net. After completing the segmentation of the 10 images in the above test set, the average pixel accuracy of the segmentation results, the average Dice coefficient and the parameter amount of the neural network are selected as indicators to evaluate the performance of the neural network image segmentation. In terms of parameter amount, U-net contains 13,394,242 learnable parameters, while the neural network provided by the present invention contains 6,678,850 learnable parameters, which is about half of the parameter amount of U-net. On the other hand, the average pixel accuracy of U-net segmentation is 0.952, and the average Dice coefficient is 0.768; while the average pixel accuracy of the neural network provided by the present invention is 0.954, and the average Dice coefficient is 0.783. It is not difficult to find that the neural network provided by the present invention has a lower parameter amount while achieving higher segmentation accuracy under other test indicators, and the segmentation performance is significantly better than U-net.

[0096] In summary, the present invention proposes a neural network that can be explained by a mathematical optimization algorithm. When segmenting a target image with sparse features, the network successfully utilizes the characteristics of the mathematical model and exhibits a better segmentation effect with fewer parameters.

[0097] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0098] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An image segmentation method based on a sparse optimization algorithm inspired neural network, characterized in that: The following steps are included: Step 1), modeling the image segmentation problem with sparse features and constructing a mathematical optimization model for minimizing the l1 norm; Step 2), using PPA to solve the mathematical optimization model, and building N neural network modules under the guidance of the PPA iteration format, that is, N PPA modules, N is an odd number; Step 3), using the PPA module to build a U-shaped neural network that uses sparse features to solve the image segmentation problem; Step 4), randomly divide the training set, design the loss function and train the U-shaped neural network to obtain a trained U-shaped neural network; Step 5), input the image to be segmented into the trained U-shaped neural network for segmentation to obtain its segmentation probability map, and then perform image binarization processing on the segmentation probability map to obtain the final segmentation result.

2. The image segmentation method based on sparse optimization algorithm inspired neural network according to claim 1, characterized in that: The detailed steps of step 1) are as follows: The segmentation problem of the input sparse image x is modeled and the following mathematical optimization model is obtained: Here, F(x) represents a learnable nonlinear transformation F(·) applied to the image x; ||F(x)||1 represents the l1 norm of F(x), requiring the transformed image F(x) to be sparse.

3. The image segmentation method based on sparse optimization algorithm inspired neural network according to claim 2, characterized in that: The detailed steps of step 2) are as follows: Step 2.1), for the mathematical optimization model, use PPA to iteratively solve it; Step 2.1.1), construct an iterative formula to solve the problem through the sub-gradient descent algorithm. The k-th iteration process is implicitly expressed as the solution process of the following optimization problem: Among them, x k represents the input of the k-th iteration, x k+1 represents the output of the k-th iteration; y is the independent variable in the optimization problem, and λ is the preset weight parameter; Step 2.1.2), further equivalent transformation is performed based on data statistical analysis to obtain the implicit iterative expression containing the minimization problem, as follows: Among them, θ is the preset weight parameter; Step 2.1.3), directly use PPA to solve the minimization problem in the implicit iterative expression, and obtain the following iterative expression: F(x k+1 )=prox(F(x k ))=sgn(F(x k ))·max(|F(x k )|-θ,0) Among them, prox(·) represents the approximate point operator; sgn(·) represents the sign function, |·| is the absolute value operation, and max(·,·) is the maximum value operation; Step 2.1.4), for the result F(x k+1 ) Take the inverse and get in, It is the inverse operation of F(·); Step 2.2), according to the formula Construct N neural network modules, that is, N PPA modules. The construction steps of a single PPA module are as follows: Step 2.2.1), arrange the 3*3 convolutional layer, batch normalization, rectified linear unit, and 3*3 convolutional layer in sequence to form F(·); Step 2.2.2), construct the inverse operation Its structure is the same as F(·); Step 2.2.3), strictly follow the mathematical expression prox(F(x k ))=sgn(F(x k ))·max(|F(x k )|-θ,0) constructs the approximate point operator prox(·), F(x k ) is the input to the operation; Step 2.2.4), F(·), prox(·), Arrange them in order to form a PPA module. The order is consistent with the formula The priority of each operation is consistent; the priority of each operation is from x k Start by decreasing from near to far.

4. The image segmentation method based on sparse optimization algorithm inspired neural network according to claim 3, characterized in that: The detailed steps of step 3) are as follows: Step 3.1), before each PPA module, add a 3*3 convolution operation with batch normalization and rectified linear unit, denoted as f(·); Step 3.2), using the N PPA modules after adding the f(·) operation, arrange them into a neural network according to the U-shaped architecture of U-net; Step 3.2.1), split the N PPA modules in half, and take the two groups of PPA modules before and after the median to construct the contraction path and expansion path in the U-shaped network; In the contraction path, the image needs to be max-pooled each time it passes through the PPA module; in the expansion path, the image needs to be up-sampled each time it passes through the PPA module; Step 3.2.2), the PPA module located at the median is arranged between the contraction path and the expansion path, the PPA module takes the output result of the contraction path as input, and the calculation result is output to the expansion path after upsampling; Step 3.2.3), introduce a skip connection between each pair of PPA modules that are symmetric about the median; Step 3.3), organize the output results into probability graph representation; At the end of the neural network expansion path, a convolution layer with a convolution kernel size of 1*1 is added; this convolution layer organizes the multi-channel feature map output by the expansion path into two single-channel feature maps, and then further converts them into two probability maps using a softmax layer. The value of each pixel in the probability map is between [0,1], indicating the probability that the pixel belongs to the segmentation target or the image background.

5. The image segmentation method based on sparse optimization algorithm inspired neural network according to claim 4, characterized in that: The detailed steps of step 4) are as follows: Step 4.1), divide the dataset containing the image and its corresponding mask into a training set and a test set. The training set will be used for subsequent training of the neural network; Step 4.2), use the following loss function Loss total Training a U-shaped neural network: Loss total =Loss fin +ηLoss sym Among them, Loss fin is the binary cross entropy function, Loss sym This is to ensure that the operation F(·) in each PPA module is consistent with Inverse loss function; Loss sym The operation process is: in each PPA module, skip The prox(·) operation in The output of the two operations is subtracted from the input, and the sum of the differences at all pixels is calculated; the differences output by all PPA modules are accumulated to obtain the final Loss sym ; After determining the loss function, use the back propagation algorithm to minimize the loss total The neural network parameters are updated in the process.

Images