An Image Segmentation Method Based on a Neural Network Inspired by a Sparse Optimization Algorithm

Abstract

The present invention discloses an image segmentation method based on a neural network inspired by a sparse optimization algorithm. First, a mathematical optimization model for the sparse feature image segmentation problem is constructed; secondly, the Proximal Point Algorithm (PPA) is used to solve the mathematical optimization model, and a neural network module, namely the PPA module, is constructed according to the algorithm; then the PPA module is used to build a U-shaped neural network for solving the image segmentation problem using sparse features; then the training set is divided, the loss function is designed, and the neural network training is completed; finally, the input image is used, and the trained neural network is used to complete the image segmentation. The neural network designed under the inspiration of the proximal point algorithm in the present invention not only endows the neural network structure with interpretability in the mathematical sense, but also can improve the segmentation effect while reducing the number of network model parameters when facing the image segmentation task with sparse features, realizing more accurate segmentation.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence, relates to the technical field of image segmentation, and particularly relates to an image segmentation method based on a neural network inspired by a sparse optimization algorithm. Background Art

[0002] With the development of deep learning, neural networks have been widely applied to the field of image segmentation. The process of using a neural network to segment the target area from a complete image is fully automatic and data-oriented learning by the computer, eliminating the artificial dependence in traditional segmentation methods. Based on this background, many complex neural networks such as U-net have emerged and widely solved the problem of image segmentation. However, such neural networks face certain challenges when dealing with problems in specific scenarios, such as segmenting images with sparse features, that is, images where the targets to be segmented are sparsely distributed everywhere. The general neural network framework cannot utilize the sparse feature specific to the problem, which is not only a waste of known information but also results in poor segmentation effects.

[0003] Many scholars have tried to assist neural networks to complete specific tasks with the help of optimization methods. They design neural network structures inspired by the optimization algorithms for solving the mathematical models that describe the problem characteristics, greatly improving the application effects of neural networks. Initially, such methods had a significant impact on the field of compressive sensing. In this field, scholars have gradually explored neural networks inspired by the Iterative Shrinkage Thresholding Algorithm (ISTA). These methods complement each other's advantages, using both the sparse feature of the signal in compressive sensing for mathematical modeling and the neural network relying on big data learning to replace the fixed iterative algorithm for solution, fully integrating the advantages of both.

[0004] The poor segmentation effect of general neural networks on sparse feature images is often due to insufficient segmentation accuracy at the image details. Therefore, adopting the idea inspired by optimization algorithms and introducing sparse constraints into the mathematical modeling and network design process can improve the processing of neural networks for sparse image details, enhance the ability of neural networks to extract sparse features, and effectively improve the segmentation accuracy of sparse targets. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide an image segmentation method based on a neural network inspired by a sparse optimization algorithm to address the deficiencies in the background art. By combining the Proximal Point Algorithm (PPA) for solving sparse optimization problems with the neural network U-net, not only can the number of parameters of the U-net model be reduced, but also better performance can be achieved when segmenting images with sparse features.

[0006] The present invention adopts the following technical solutions to solve the above technical problems:

[0007] An image segmentation method based on a neural network inspired by a sparse optimization algorithm, comprising the following steps:

[0008] Step 1), model the image segmentation problem for an image with sparse features, and construct a mathematical optimization model for minimizing the l1 norm.

[0009] Step 2), use PPA to solve the mathematical optimization model, and build N neural network modules under the guidance of the PPA iteration format, that is, N PPA modules, where N is an odd number.

[0010] Step 3), use the PPA modules to build a U-shaped neural network for solving the image segmentation problem using sparse features.

[0011] Step 4), randomly divide the training set, design a loss function and train the U-shaped neural network to obtain a trained U-shaped neural network.

[0012] Step 5), input the image to be segmented into the trained U-shaped neural network for segmentation to obtain its segmentation probability map, and then perform image binarization on the segmentation probability map to obtain the final segmentation result.

[0013] As a further optimized solution of the image segmentation method based on a neural network inspired by a sparse optimization algorithm of the present invention, the detailed steps of the step 1) are as follows:

[0014] Model the segmentation problem of the input sparse image x to obtain the following mathematical optimization model:

[0015]

[0016] Among them, F(x) represents applying a learnable non-linear transformation F(·) to the image x; ||F(x)||1 represents the l1 norm of F(x), and it is required that the transformed image F(x) has sparsity.

[0017] As a further optimized solution of the image segmentation method based on a neural network inspired by a sparse optimization algorithm of the present invention, the detailed steps of the step 2) are as follows:

[0018] Step 2.1), for the mathematical optimization model, use PPA to iteratively solve it;

[0019] Step 2.1.1), construct an iterative formula to solve through the subgradient descent algorithm, and the process of the k-th iteration is implicitly expressed as the solution process of the following optimization problem:

[0020]

[0021] where, x k represents the input of the k-th iteration, and x k+1 represents the output of the k-th iteration; y is the independent variable in the optimization problem, and λ is the preset weight parameter;

[0022] Step 2.1.2), further equivalently transform according to data statistical analysis to obtain an implicit iterative expression containing a minimization problem, as follows:

[0023]

[0024] where, θ is the preset weight parameter;

[0025] Step 2.1.3), directly use PPA to solve the minimization problem in the implicit iterative expression to obtain the following iterative expression:

[0026] F(x k+1 ) = prox(F(x k )) = sgn(F(x k ))·max(|F(x k )| - θ, 0)

[0027] where, prox(·) represents the proximal operator; sgn(·) represents the sign function, |·| is the absolute value operation, and max(·, ·) is the maximum operation;

[0028] Step 2.1.4), invert the result F(x k+1 ) to obtain where, is the inverse operation of F(·);

[0029] Step 2.2), construct N neural network modules according to the formula , that is, N PPA modules. The construction steps of a single PPA module are as follows:

[0030] Step 2.2.1), arrange a 3*3 convolutional layer, batch normalization, rectified linear unit, and 3*3 convolutional layer in sequence to form F(·);

[0031] Step 2.2.2), construct the inverse operation Its structure is the same as F(·);

[0032] Step 2.2.3), strictly construct the proximal operator prox(·) according to the mathematical expression prox(F(x k )) = sgn(F(x k ))·max(|F(x k )| - θ, 0), where F(x k ) is the input of this operation;

[0033] Step 2.2.4), arrange F(·), prox(·), in sequence to form a PPA module, and this sequence is consistent with the operation priorities in the formula ; the operation priorities decrease successively from near to far starting from x k .

[0034] As a further optimization scheme of an image segmentation method based on a sparse optimization algorithm inspired neural network of the present invention, the detailed steps of step 3) are as follows:

[0035] Step 3.1), before each PPA module, add a 3*3 convolutional operation with batch normalization and rectified linear unit, denoted as f(·);

[0036] Step 3.2), use N PPA modules after adding the f(·) operation and arrange them into a neural network according to the U-shaped architecture in U-net;

[0037] Step 3.2.1), divide the N PPA modules into two halves, and respectively take the two groups of PPA modules before and after the median to construct the contracting path and the expanding path in the U-shaped network; in the contracting path, max pooling is required every time the image passes through a PPA module; in the expanding path, upsampling is required every time the image passes through a PPA module;

[0038] Step 3.2.2), arrange the PPA module at the median between the contracting path and the expanding path, and this PPA module takes the output result of the contracting path as the input, and the operation result is output to the expanding path after upsampling;

[0039] Step 3.2.3), introduce skip connections between each pair of PPA modules symmetric about the median;

[0040] Step 3.3), organize the output result into a probability map representation;

[0041] At the end of the neural network dilation path, a convolutional layer with a kernel size of 1*1 is added; this convolutional layer reorganizes the multi-channel feature map output by the dilation path into two single-channel feature maps, and then uses a softmax layer to further transform them into two probability maps. The value at each pixel point in the probability map is between [0,1], indicating the probability that the pixel point belongs to the segmentation target or the image background.

[0042] As a further optimization scheme of the image segmentation method based on the sparse optimization algorithm inspired neural network of the present invention, the detailed steps of step 4) are as follows:

[0043] Step 4.1), divide the dataset containing the image and its corresponding mask into two parts: a training set and a test set. The training set will be used for the subsequent training of the neural network;

[0044] Step 4.2), use the following loss function Loss total Train the U-shaped neural network:

[0045] Loss total = Loss fin + ηLoss sym

[0046] where, Loss fin is the binary cross-entropy function, Loss sym is the loss function that ensures the operation F(·) in each PPA module is inverse to ; The operation process of Loss sym is: within each PPA module, skip the prox(·) operation in , directly subtract the output and input of the two operations passed through F(·) and , and calculate the sum of the differences at all pixel points; accumulate the differences output by all PPA modules to obtain the final Loss sym ;

[0047] After determining the loss function, use the backpropagation algorithm to update the neural network parameters during the process of minimizing Loss total .

[0048] Compared with the prior art, the present invention adopts the above technical solutions and has the following technical effects:

[0049] 1. This method considers the sparsity of the target in the image during modeling, and unfolds the optimization algorithm for dealing with the sparse problem into a neural network. The network structure has mathematical significance and has strong interpretability;

[0050] 2. Compared with the classical U-net, the number of parameters to be learned by the neural network provided by the present invention is significantly reduced. Moreover, when facing the image segmentation problem with sparse features, the segmentation effect of this method is better than that of U-net. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flowchart of the image segmentation method provided by the present invention.

[0052] Figure 2 is a schematic structural diagram of the neural network provided by the present invention.

[0053] Figure 3 (a), Figure 3 (b), Figure 3 (c) are respectively schematic comparison diagrams of the original image, mask, and segmentation result in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0054] The technical solution of the present invention will be further described in detail below with reference to the drawings:

[0055] The present invention can be implemented in many different forms and should not be considered limited to the embodiments described herein. On the contrary, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present invention to those skilled in the art. In the drawings, components are enlarged for clarity.

[0056] In order to verify the effectiveness of the neural network method inspired by the sparse optimization algorithm proposed by the present invention in the image segmentation task, it will be described below in combination with the drawings and specific embodiments. It should be understood that the following embodiments are only for reference to help those skilled in the art better understand the present invention. After reading the present invention, various equivalent modifications of the present invention by those skilled in the art fall within the scope defined by the appended claims of this application.

[0057] As Figure 1 shown, the present invention provides an image segmentation method based on a neural network inspired by a sparse optimization algorithm, including the following steps:

[0058] Step 1), model the image segmentation problem with sparse features and construct a mathematical optimization model for minimizing the l1 norm;

[0059] Model the segmentation problem of the input sparse image x to obtain the following mathematical optimization model:

[0060]

[0061] Among them, F(x) represents applying a learnable non-linear transformation F(·) to the image x; ||F(x)||1 represents the l1 norm of F(x), and it is required that the transformed image F(x) has sparsity.

[0062] Step 2), use PPA to solve the mathematical optimization model, and build N neural network modules under the guidance of the PPA iteration format, that is, N PPA modules. N is an odd number. In this embodiment, N is set to 5;

[0063] Step 2.1), for the mathematical optimization model, use PPA to iteratively solve it;

[0064] Step 2.1.1), construct an iterative formula to solve through the subgradient descent algorithm. The process of the k-th iteration is implicitly expressed as the solution process of the following optimization problem:

[0065]

[0066] where, x k represents the input of the k-th iteration, and x k+1 represents the output of the k-th iteration; y is the independent variable in the optimization problem, and λ is the preset weight parameter;

[0067] Step 2.1.2), further equivalently transform according to data statistical analysis to obtain an implicit iterative expression containing a minimization problem, as follows:

[0068]

[0069] where, θ is the preset weight parameter. In this embodiment, θ takes 0.1;

[0070] Step 2.1.3), directly use PPA to solve the minimization problem in the implicit iterative expression to obtain the following iterative expression:

[0071] F(x k+1 ) = prox(F(x k )) = sgn(F(x k ))·max(|F(x k )| - θ, 0)

[0072] where, prox(·) represents the proximal operator; sgn(·) represents the sign function, |·| is the absolute value operation, and max(·, ·) is the maximum operation;

[0073] Step 2.1.4), take the inverse of the result F(x k+1 ) to obtain where, is the inverse operation of F(·);

[0074] Step 2.2), construct 5 neural network modules, that is, 5 PPA modules, according to the formula . The construction steps of a single PPA module are as follows:

[0075] Step 2.2.1), arrange a 3×3 convolutional layer, batch normalization, rectified linear unit, and 3×3 convolutional layer in sequence to form F(·);

[0076] Step 2.2.2), construct an inverse operation whose structure is the same as F(·);

[0077] Step 2.2.3), strictly construct the proximal operator prox(·) according to the mathematical expression prox(F(x k )) = sgn(F(x k ))·max(|F(x k )| - θ, 0), where F(x k ) is the input of this operation;

[0078] Step 2.2.4), arrange F(·), prox(·), in sequence to form a PPA module, and this sequence is consistent with the precedence levels of each operation in the formula ; the precedence levels of each operation start from x k and decrease successively from near to far.

[0079] Step 3), use the PPA module to build a U-shaped neural network for solving the image segmentation problem using sparse features;

[0080] Step 3.1), before each PPA module, add a 3×3 convolutional operation with batch normalization and rectified linear unit, denoted as f(·); the main function of the operation f(·) is to change the number of feature channels of the image; in this embodiment, the 5 f(·) operations respectively adjust the number of feature channels of the input feature map to 64, 128, 256, 128, and 64 and then output to the PPA module;

[0081] Step 3.2), use the 5 PPA modules after adding the f(·) operation and arrange them into a neural network according to the U-shaped architecture in U-net;

[0082] Step 3.2.1), divide the 5 PPA modules into two halves, respectively take the 1st and 2nd PPA modules to construct the contracting path in the U-shaped network, and take the 4th and 5th PPA modules to construct the expanding path; in the contracting path, the image needs to perform max pooling every time it passes through a PPA module; in the expanding path, the image needs to perform upsampling every time it passes through a PPA module;

[0083] Step 3.2.2), arrange the 3rd PPA module between the contracting path and the expanding path, and this PPA module takes the output result of the contracting path as the input, and the operation result is output to the expanding path after upsampling;

[0084] In step 3.2.3), introduce skip connections between the 1st and the 5th PPA modules and between the 2nd and the 4th PPA modules respectively.

[0085] In step 3.3), organize the output result into a probability map representation.

[0086] At the end of the neural network dilation path, add a convolutional layer with a kernel size of 1*1; this convolutional layer organizes the multi-channel feature map output by the dilation path into two single-channel feature maps, and then further transforms them into two probability maps using a softmax layer. The value at each pixel point in the probability map is between [0, 1], indicating the probability that the pixel point belongs to the segmentation target or the image background.

[0087] In step 4), randomly divide the training set, design a loss function and train the U-shaped neural network to obtain a trained U-shaped neural network.

[0088] In step 4.1), divide the dataset containing images and their corresponding masks into two parts: a training set and a test set; in this embodiment, the extended DRIVE (Digital Retinal Images for Vessel Extraction) is used as the dataset. The images in the dataset are all fundus retinal images, and the goal of the embodiment is to segment the fundus blood vessels from them. The extended DRIVE dataset has a total of 100 pictures with masks, among which 90 groups of pictures are used as the training set for network training, and 10 groups of pictures are used as the test set for network testing.

[0089] In step 4.2), use the following loss function Loss total to train the U-shaped neural network:

[0090] Loss total = Loss fin + ηLoss sym

[0091] where Loss fin is the binary cross-entropy function, and Loss sym is a loss function that ensures the operations F(·) in each PPA module are inverse to each other; the operation process of Loss sym is as follows: within each PPA module, skip the prox(·) operation in, and directly calculate the difference between the output and the input of the two operations F(·) and , and calculate the sum of the differences at all pixel points; accumulate the differences output by all PPA modules to obtain the final Loss sym ;

[0092] After determining the loss function, the backpropagation algorithm is used to update the neural network parameters while minimizing the Loss. total In the process.

[0093] Step 5): Input the image to be segmented into the trained U-shaped neural network for segmentation to obtain its segmentation probability map, and then perform image binarization on the segmentation probability map to obtain the final segmentation result.

[0094] In this embodiment, the segmentation of 10 images in the above test set is completed respectively, and one group of images is selected for the display of the segmentation visual effect. Among them, Figure 3 (a), Figure 3 (b) is the display of the image and its mask, Figure 3 (c) is its corresponding segmentation result. Obviously, the neural network provided by the present invention can achieve relatively accurate segmentation for both the main trunk of blood vessels in the retina and the thin branches of blood vessels.

[0095] On this basis, the neural network provided in this embodiment is compared with the classical U-net. After the segmentation of 10 images in the above test set is completed respectively, the average pixel accuracy, average Dice coefficient of the segmentation results, and the number of parameters of the neural network are selected as indicators to evaluate the pros and cons of the neural network image segmentation performance. In terms of the number of parameters, U-net contains 13,394,242 learnable parameters, while the neural network provided by the present invention contains 6,678,850 learnable parameters, which is about half of the number of parameters of U-net. On the other hand, the average pixel accuracy of the U-net segmentation is ......

[0096] To sum up, the present invention proposes a neural network that can be explained by a mathematical optimization algorithm. When segmenting target images with sparse features, the network successfully utilizes the characteristics of the mathematical model and exhibits better segmentation effects with fewer parameters.

[0097] Those skilled in the art of the present technology can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used here have the same meaning as the general understanding of those of ordinary skill in the art in the field to which the present invention belongs. It should also be understood that terms defined in general dictionaries should be understood to have a meaning consistent with the meaning in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as such here.

[0098] The specific embodiments described above further elaborate on the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only for the specific embodiments of the present invention and is not used to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. An image segmentation method based on a neural network inspired by a sparse optimization algorithm, characterized in that, It includes the following steps: Step 1), model the image x segmentation problem with sparse features, and construct a mathematical optimization model that minimizes the l1 norm: Among them, F(x) represents applying a learnable non-linear transformation F(·) to the image x; ||F(x)||1 represents the l1 norm of F(x), and it is required that the transformed image F(x) has sparsity; Step 2), use the PPA to solve the mathematical optimization model, and build N neural network modules under the guidance of the PPA iteration format, that is, N PPA modules, where N is an odd number; Step 2.1), for the mathematical optimization model, use the PPA to iteratively solve it; Step 2.1.1), construct an iterative formula to solve it through the subgradient descent algorithm, and the process of the k-th iteration is implicitly expressed as the solution process of the following optimization problem: where x k represents the input of the k-th iteration, and x k+1 represents the output of the k-th iteration; y is the independent variable in the optimization problem, and λ is the preset weight parameter; Step 2.1.2), further equivalently transform according to data statistical analysis to obtain an implicit iterative expression containing a minimization problem, as follows: Among them, θ is a preset weight parameter; Step 2.1.3), directly use the PPA to solve the minimization problem in the implicit iterative expression to obtain the following iterative expression: F(x k+1 ) = prox(F(x k )) = sgn(F(x k ))·max(|F(x k )| - θ, 0) Among them, prox(·) represents the proximal operator; sgn(·) represents the sign function, |·| is the absolute value operation, and max(·,·) is the maximum value operation; Step 2.1.4), invert the result F(x k+1 ), to obtain where is the inverse operation of F(·); Step 2.2), according to the formula Construct N neural network modules, that is, N PPA modules. The construction steps of a single PPA module are as follows: Step 2.2.1), arrange a 3*3 convolutional layer, batch normalization processing, rectified linear unit, and 3*3 convolutional layer in sequence to form F(·); Step 2.2.2), construct the inverse operation Its structure is the same as F(·); Step 2.2.3), strictly construct the proximal operator prox(·) according to the mathematical expression prox(F(x k )) = sgn(F(x k ))·max(|F(x k )| - θ, 0), where F(x k ) is the input of this operation; Step 2.2.4), arrange F(·), prox(·), in sequence according to the order, to form a PPA module, and this order is consistent with the priority levels of each operation in the formula; the priority levels of each operation start from x k and decrease in turn from near to far; Step 3), use the PPA module to build a U-shaped neural network that uses sparse features to solve the image segmentation problem; Step 4), randomly divide the training set, design a loss function and train the U-shaped neural network to obtain a trained U-shaped neural network; Step 5), input the image to be segmented into the trained U-shaped neural network for segmentation to obtain its segmentation probability map, and then perform image binarization processing on the segmentation probability map to obtain the final segmentation result.

2. The image segmentation method based on a neural network inspired by a sparse optimization algorithm according to claim 1, characterized in that The detailed steps of step 3) are as follows: Step 3.1), before each PPA module, add a 3*3 convolutional operation with batch normalization processing and rectified linear unit, denoted as f(·); Step 3.2), use the N PPA modules after adding the f(·) operation and arrange them into a neural network according to the U-shaped architecture in U-net; Step 3.2.1), divide the N PPA modules into two halves, and respectively take the two groups of PPA modules before and after the median to construct the contraction path and the expansion path in the U-shaped network; In the contraction path, the image needs to perform max pooling every time it passes through a PPA module; in the expansion path, the image needs to perform upsampling every time it passes through a PPA module; Step 3.2.2), arrange the PPA module at the median between the contraction path and the expansion path. This PPA module takes the output result of the contraction path as the input, and the operation result is output to the expansion path after upsampling; Step 3.2.3), introduce skip connections between each pair of PPA modules symmetric about the median; Step 3.3), organize the output result into a probability map representation; At the end of the neural network dilation path, a convolutional layer with a kernel size of 1*1 is added; this convolutional layer reorganizes the multi-channel feature map output by the dilation path into two single-channel feature maps, and then further converts them into two probability maps using a softmax layer. The value at each pixel point in the probability map is between [0,1], indicating the probability that the pixel point belongs to the segmentation target or the image background.

3. The image segmentation method based on a neural network inspired by a sparse optimization algorithm according to claim 2, wherein The detailed steps of step 4) are as follows: Step 4.1), divide the dataset containing the image and its corresponding mask into two parts: a training set and a test set. The training set will be used for subsequent training of the neural network; Step 4.2), using the following loss function Loss total Train the U-shaped neural network: Loss total = Loss fin + ηLoss sym Among them, Loss fin is the binary cross-entropy function, and Loss sym is the loss function that ensures the operation F(·) in each PPA module is inverse to sym ; The operation process of Loss is as follows: within each PPA module, skip the prox(·) operation in , directly subtract the output and input of the two operations of F(·) and sym and calculate the sum of the differences at all pixel points; accumulate the differences output by all PPA modules to obtain the final Loss After determining the loss function, use the backpropagation algorithm to update the neural network parameters during the process of minimizing Loss total ​

Images