A neural network search method for medical image segmentation
By introducing multi-scale feature aggregation module and gate function search method in medical image segmentation, the problem of slow update and iteration of neural network structure design and lack of scalability in the prior art is solved, and more efficient neural network structure search and Performance improvement.
Patent Information
- Application Number
- CN202010973006.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-09-16
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2040-09-16
AI Technical Summary
The prior art lacks optimization for multi-scale feature aggregation in medical image segmentation, resulting in slow update and iteration of neural network structure design, inability to effectively improve performance, and lack scalability.
A neural network search method is proposed, including building an initial neural network and training to convergence, searching for the best neural network structure through multiple iterative sampling training, and using searchable multi-scale feature aggregation module and gate function to determine the feature aggregation strategy, enlargement Search for space and optimize feature fusion.
It realizes more efficient neural network structure search in medical image segmentation, significantly improves segmentation accuracy, is suitable for different medical image tasks, and improves the scalability and performance of network design.
Smart Images

Figure CN114202669B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the medical field, in particular to the field of using neural networks to process medical image data, and more particularly to a neural network search method for medical image segmentation. Background Art
[0002] With the development of neural network technology, based on its outstanding performance, it is increasingly widely used in various fields, most notably in the field of computer vision. The research on neural network structure search originated in the field of computer vision, and many neural network structures searched out have excellent performance. However, in the field of computer vision, the application of neural networks is based on a single task with a large number of annotations, the so-called "small tasks, big data". In 3D / 2D medical scenarios, data is often high-dimensional and high-density (such as CT, MRI, lack of sufficient annotations, and different medical image analysis tasks vary greatly (for example, the segmentation process for brain tumors and liver tumors is very different). Therefore, in 3D medicine, there are large individual differences between models and lack of scalability. Compared with the network design of computer vision, the backbone network of the neural network in 3D medicine is often relatively simple, and the aggregation of multi-scale features becomes more important. At present, image segmentation based on neural networks has become the mainstream in 3D medical segmentation. However, it is very expensive and time-consuming to artificially design a neural network structure for each specific application scenario. These huge losses lead to slow updates and iterations of neural network structure design, which greatly restricts the development of industry-university-research cooperation. Although the current mainstream neural network search algorithms can be used in medical images, these algorithms have not been specifically optimized or designed for medical image tasks, and thus the performance obtained is limited.
[0003] As we all know, convolutional neural networks extract features by imitating the neural structure of the human brain mechanism, and have outstanding performance in the segmentation and detection of medical image processing. In the field of medical image processing, in order to improve the performance of segmentation algorithms, researchers have designed a variety of neural network architectures such as Unet, ResUnet, Hybrid-Dense-Unet, etc. However, in the medical field, the differences between different medical image analysis tasks are very large, resulting in large individual differences in the artificially designed neural network architectures and lack of scalability. In order to reduce the repeated manual design consumption when facing new tasks, more and more researchers have conducted in-depth research on the implementation of neural network structure search. Researchers have found that many neural structure search algorithms have excellent performance in the original field, but can only obtain limited performance improvement when applied to the field of medical image processing. The main reason is that the search space does not consider the key to the application of neural networks in the field of medical image processing: the aggregation of multi-scale features. The recognition of targets (tumors, etc.) in medical images depends on multiple scales to judge. Therefore, the latest search algorithms can only obtain limited improvement when applied to the field of medical image processing. Summary of the invention
[0004] Therefore, the purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a new search method to search for neural networks for medical image segmentation.
[0005] According to a first aspect of the present invention, there is provided a neural network search method for medical image segmentation, wherein the neural network comprises an encoder, a searchable multi-scale feature aggregation module, a decoder, and a classifier, and the method comprises: S1, constructing an initial neural network and training it to convergence, wherein the encoder of the initial neural network comprises one or more first searchable backbone constituent modules, each of which comprises a normal layer and a compression layer, and the decoder of the initial neural network comprises one or more second searchable backbone constituent modules, each of which comprises a normal layer and an extended layer, wherein each normal layer is randomly selected from a candidate set of its corresponding normal layer, each compression layer is randomly selected from a candidate set of its corresponding compression layer, and each extension layer is randomly selected from a candidate set of its corresponding compression layer. The corresponding extended layer is randomly selected from the candidate set; S2, the initial neural network trained in step S1 is subjected to multiple rounds of iterative sampling training using the training set corresponding to the medical image to obtain multiple neural networks, and the best neural network with the highest segmentation accuracy is selected from the multiple neural networks; wherein each round of iterative training is carried out in accordance with a preset sampling method, for each layer in the neural network encoder and decoder, a new candidate layer is independently sampled from the candidate layer corresponding to each layer to replace the current layer, and the new neural network composed of the sampled encoder, decoder, searchable multi-scale feature aggregation module and classifier is trained with the training set, and at the same time, the best feature aggregation strategy corresponding to the searchable multi-scale feature aggregation module in the current neural network is searched; S3, the neural network selected in step S2 is retrained until convergence.
[0006] Preferably, the candidate set of common layers includes the following candidate layers: 3x3x3 convolution layer, 3x3x1 convolution layer, 5x5x5 convolution layer, pseudo 3d convolution layer, two stacked 3x3x3 convolution layers, 3x3x3 dilated convolution with a stride of 2, 5x5x5 dilated convolution layer with a stride of 2; the candidate set of compression layers includes the following candidate layers: maximum pooling layer, average pooling layer, 3x3x3 convolution layer with a stride of 2; the candidate set of extension layers includes the following candidate layers: transposed convolution layer, trilinear interpolation layer.
[0007] In some embodiments of the present invention, the candidate layers in the candidate set corresponding to each layer in the encoder or decoder in the neural network have independent control weights, and the sum of the control weights of all candidate layers corresponding to the same layer in the encoder or decoder is 1; wherein before iterative sampling training, the initialization control weights of all candidate layers corresponding to the same layer are equal values.
[0008] In some embodiments of the present invention, in step S2, the ProxylessNAS method is used to sample from the candidate set of each layer in each round of iterative training, and the training set corresponding to the medical image is input into all the sampled new candidate layers and the new neural network composed of the searchable multi-scale feature aggregation module and the classifier for prediction, and the prediction result is compared with the manual label corresponding to the training set to calculate the loss value, and the standard gradient backpropagation is performed based on the loss value to update the neural network parameters and the control weights of the candidate layers in the candidate set corresponding to each layer in the encoder and the decoder.
[0009] In some embodiments of the present invention, a searchable connection between a certain node in the searchable multi-scale feature aggregation module and an input feature has a learnable weight; in each round of iterative training of the neural network, a gate function is used to map the learnable weight on the searchable connection between a certain node in the searchable multi-scale feature aggregation module and the input feature to [0, 1] to measure the importance of the connection propagation feature, retain the feature connection whose importance is greater than or equal to the preset threshold, delete the feature connection whose importance is less than the preset threshold, and finally retain the feature connection to form a feature aggregation strategy; wherein the gate function is:
[0010]
[0011] Among them, σ represents the gate function, β is the learnable weight, and e is the base of the natural logarithm function. During initialization, each learnable weight is set to 0. In each iterative training, standard gradient backpropagation is performed based on the loss value to update the learnable weights of all searchable connections in the special aggregation module.
[0012] In some embodiments of the present invention, a neural network is selected by arranging the weights of the candidate layers in the candidate set corresponding to each layer in the encoder and decoder of the neural network after iterative sampling training in descending order, selecting the candidate layer with the largest weight in each layer and forming the best neural network with the searchable multi-scale feature aggregation module and the classifier. Preferably, in step S2, the initial neural network is subjected to 60,000 times of iterative sampling training, and the neural network is verified on the verification set corresponding to the medical image every 500 times of iterative training, and the neural network with the highest segmentation accuracy on the verification set is selected.
[0013] According to a second aspect of the present invention, there is provided a neural network search method for medical image assignment, comprising:
[0014] T1. Construct an initial neural network set and train each initial neural network to convergence, wherein each initial neural network in the initial neural network set includes an encoder, a searchable multi-scale feature aggregation module, a decoder, and a classifier, the encoder includes one or more first searchable backbone constituent modules, each of which includes a normal layer and a compression layer, and the decoder of the initial neural network includes one or more second searchable backbone constituent modules, each of which includes a normal layer and an extended layer, wherein each normal layer is selected from a candidate set of its corresponding normal layer, each compression layer is selected from a candidate set of its corresponding compression layer, and each extension layer is selected from a candidate set of its corresponding extension layer;
[0015] T2, using the training set corresponding to the medical image to perform multiple rounds of training on all the initial neural networks trained in step T1 until convergence, and determine the best feature aggregation strategy for searching the multi-scale feature aggregation module in each neural network;
[0016] T3, verifying the segmentation accuracy of the initial neural network trained in step T2 on the verification set corresponding to the medical image, and selecting the neural network with the highest segmentation accuracy as the search result;
[0017] T4. Retrain the neural network selected in step T3 until convergence.
[0018] Preferably, the candidate set of common layers includes the following candidate layers: 3x3x3 convolution layer, 3x3x1 convolution layer, 5x5x5 convolution layer, pseudo 3d convolution layer, two stacked 3x3x3 convolution layers, 3x3x3 dilated convolution with a stride of 2, 5x5x5 dilated convolution layer with a stride of 2; the candidate set of compression layers includes the following candidate layers: maximum pooling layer, average pooling layer, 3x3x3 convolution layer with a stride of 2; the candidate set of extension layers includes the following candidate layers: transposed convolution layer, trilinear interpolation layer.
[0019] In some embodiments of the present invention, a searchable connection between a certain node in the searchable multi-scale feature aggregation module and the input feature has a learnable weight, and in step T2, the optimal feature aggregation strategy of the searchable multi-scale feature aggregation module in the neural network is determined in the following manner: a gate function is used to map the learnable weight on the searchable connection between a certain node in the searchable multi-scale feature aggregation module and the input feature to [0, 1] to measure the importance of the connection propagation feature, retaining feature connections whose importance is greater than or equal to a preset threshold, deleting feature connections whose importance is less than the preset threshold, and finally retaining feature connections to form a feature aggregation strategy, wherein the gate function is:
[0020]
[0021] Among them, σ represents the gate function, β is the learnable weight, and e is the base of the natural logarithm function. During initialization, each learnable weight is set to 0. In each training, standard gradient backpropagation is performed based on the loss value to update the learnable weights of all searchable connections in the special aggregation module.
[0022] According to a third aspect of the present invention, there is provided a neural network for medical image segmentation searched by the method described in the first and second aspects of the present invention, the neural network comprising: an encoder for extracting features of input data, each layer in the encoder being a candidate layer with the highest weight selected from a candidate set corresponding to the layer; a searchable multi-scale feature aggregation module for aggregating features of various scales extracted by the encoder, the feature connections in the searchable multi-scale feature aggregation module being feature connections whose importance is greater than or equal to a preset threshold; a decoder for determining a segmentation target based on the features aggregated by the feature aggregation module, each layer in the decoder being a candidate layer with the highest weight selected from a candidate set corresponding to the layer; and a classifier for outputting a classification result of the input data based on a target determination result of the decoder.
[0023] Compared with the prior art, the advantages of the present invention are:
[0024] The existing methods usually search the backbone module, that is, only the encoder and decoder are searched. Although this approach can effectively extract features at various scales, it ignores how to further fuse these features for the final prediction. Feature fusion has always been a very important link in the design of 3D medical neural networks. Ignoring this link can only achieve limited performance improvement. In the present invention, we have added the feature fusion aspect, thereby increasing the overall search space, and we have also used an effective search strategy to jointly couple the two parts of the search space (backbone construction module, multi-scale feature aggregation), so that the best structure can be searched for analysis tasks with different data sources and different task definitions. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:
[0026] Figure 1 A schematic diagram of a neural network framework structure searched in a neural network search method for medical image segmentation according to an embodiment of the present invention;
[0027] Figure 2 A schematic diagram of a searchable backbone component framework in an encoder and a decoder according to an embodiment of the present invention;
[0028] Figure 3This is a schematic diagram of an example of randomly initializing a first searchable backbone component module when constructing an initial neural network according to an embodiment of the present invention;
[0029] Figure 4 A schematic diagram of an example of training a first searchable backbone component module according to an embodiment of the present invention;
[0030] Figure 5 A schematic diagram of neural network feature connections of a neural network search method for medical image segmentation according to an embodiment of the present invention;
[0031] Figure 6 This is a schematic diagram of an example of feature aggregation according to an embodiment of the present invention;
[0032] Figure 7 is a schematic diagram of a gating function characteristic curve according to an embodiment of the present invention;
[0033] Figure 8 Schematic diagram of search feature connection according to an embodiment of the present invention. DETAILED DESCRIPTION
[0034] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0035] As mentioned in the background technology, under the existing technology, the neural network in the field of medical image segmentation has poor scalability and high requirements for multi-scale feature aggregation. Therefore, a neural network search method that can quickly search for a neural network suitable for specific medical image segmentation is needed to accelerate the development of a high-performance network structure. In order to solve this problem, the inventor analyzed the existing medical segmentation method and abstracted the current neural network structure into two parts. One part is the constituent module of the backbone network, which is responsible for the extraction of multi-scale features, and the other part is the multi-scale feature aggregation module, which is responsible for further aggregating features of different scales, understanding features of different scales, and selectively aggregating multi-scales. Based on these two parts, the inventor proposes a method for neural network search starting from the two parts to quickly and automatically search for the best network structure in a specific medical occasion. In short, the neural network search method proposed in the present invention includes the following main stages: in the first stage, an initial neural network is constructed and trained to convergence; in the second stage, a network search is performed on the basis of the trained initial neural network to select the best neural network; in the third stage, the searched best neural network is retrained to convergence. Among them, the initialized neural network includes an encoder, a decoder, a searchable multi-scale feature aggregation module, and a decoder. The encoder and the decoder respectively include one or more searchable backbone constituent modules. During the search and training process, according to the performance of the backbone constituent modules in the neural network and the searchable multi-scale feature aggregation module on the data set, the network structure is automatically adjusted, including the type of functional layer in the backbone constituent module, the way of feature connection in the searchable multi-scale feature aggregation module, etc., and then the best performing neural network is selected as the search result.
[0036] According to one embodiment of the present invention, the present invention provides a neural network search method for medical image segmentation, comprising the following steps:
[0037] Step 1: Preparation stage:
[0038] Obtain the target data set required for the search neural network, that is, the data set corresponding to the field to which the neural network is to be applied. For example, if it is applied to the image segmentation of brain tumors, then the corresponding target data set is the medical image data set of brain tumors. If it is applied to the image segmentation of liver tumors, then the corresponding target data set is the medical image data set of liver tumors. Divide the target data set into a training set and a validation set in a ratio of 4:1.
[0039] Step 2: Construct the initial neural network stage:
[0040] like Figure 1As shown in , the initial neural network includes an encoder, a decoder, a searchable multi-scale feature aggregation module, and a classifier. The encoder is used to analyze and extract the features of the input data, and the searchable multi-scale feature aggregation module is used to better aggregate the various scale features obtained by the encoder. The aggregated features are used as the input of the decoder and then used for the prediction of the classifier. Figure 1 As shown, the encoder includes a plurality of first searchable trunk constituent modules (for example, 4), and the decoder includes a plurality of second searchable trunk constituent modules (for example, 4). According to one embodiment of the present invention, Figure 2 As described above, each first searchable backbone module includes two functional layers, a common layer and a compressed layer, and each second searchable backbone module includes two functional layers, a common layer and an extended layer, and each functional layer has multiple candidate layers. Figure 1 As shown, the searchable feature connection in the searchable multi-scale feature aggregation module has multiple configurations, so that the first searchable backbone component module, the second searchable backbone component module, the searchable multi-scale feature aggregation module, and the classifier together constitute multiple neural network combinations. When constructing the initial neural network, each functional layer randomly selects a candidate layer from its corresponding candidate set, and the searchable multi-scale feature aggregation module randomly configures the feature aggregation strategy, and uses the training set to train the constructed initial neural network until convergence, wherein the training convergence is determined when the classification accuracy of the initial neural network on the validation set after a certain training is compared with that before training and does not change by more than 0.2%.
[0041] According to one embodiment of the present invention, the candidate set of the common layer includes the following candidate layers: 3x3x3 convolution layer, 3x3x1 convolution layer, 5x5x5 convolution layer, pseudo 3d convolution layer, two stacked 3x3x3 convolution layers, 3x3x3 dilated convolution with a stride of 2, 5x5x5 dilated convolution layer with a stride of 2; the candidate set of the compression layer includes the following candidate layers: maximum pooling layer, average pooling layer, 3x3x3 convolution layer with a stride of 2; the candidate set of the expansion layer includes the following candidate layers: transposed convolution layer, trilinear interpolation layer. Figure 3 As shown, taking the first searchable backbone module as an example, when constructing the initial neural network, each common layer randomly selects a candidate layer from the candidate set of the common layer, and each compressed layer also randomly selects a candidate layer from the candidate set of the compressed layer; Figure 4 As shown, the first searchable backbone module consists of a 3x3x3 convolutional layer as the common layer and a maximum pooling layer as the compression layer.
[0042] Step 3: Neural network search phase:
[0043] like Figure 5As shown, the encoder of the neural network includes four first searchable backbone constituent modules, and the decoder includes four second searchable backbone constituent modules. All first searchable backbone constituent modules have fixed connections with all second searchable backbone constituent modules. Each first searchable backbone constituent module is used to extract features of different scales of input data, such as Figure 5 As shown in {N 0,0 、N 0,1 、N 0,2 、N 0,3 、N 0,4}, the searchable multi-scale feature aggregation module is in the shape of an inverted pyramid and is divided into multiple aggregation stages. Each stage contains multiple aggregation nodes, and the number of aggregation nodes is reduced by one in each stage, such as Figure 5 There are four stages in 1,0 ,N 1,1 ,N 1,2 ,N 1,3},{N 2,0 ,N 2,1 ,N 2,2},{N 3,0 ,N 3,1},{N 4,0}, the final feature N 4,0 The optimal neural network structure is obtained by searching for the best first searchable backbone component module, the second backbone component module, and the optimal feature aggregation strategy of the searchable multi-scale feature aggregation module.
[0044] For any searchable backbone component module in the first searchable backbone component module and the second searchable backbone component module, each functional layer (i.e., normal layer, compression layer, and expansion layer) of each searchable backbone component module has a set of control weights, that is, each candidate layer in the candidate set of each functional layer has a learnable control weight. Taking the first searchable backbone component module as an example, in the first searchable backbone component module, when the normal layer and the compression layer are initially set, each candidate layer of the normal layer and the compression layer will be additionally set with a learnable control weight, and this weight meets the following two conditions: one is that the value range is [0-1], and the other is that the sum of the learnable control weights of the same type of search layers is one. Therefore, at the time of initialization, each candidate layer will be assigned an equal value of learnable weight, and the sum of the learnable control weights of candidate layers of the same type is one. Specifically, since there are a total of 7 candidate layers in the candidate set of the common layer, the learnable weight value of a candidate layer in the candidate set of the common layer is initialized to 1 / 7 = 0.14285, and each candidate layer in the candidate set of the compression layer is initialized to 1 / 3 = 0.333. It should be noted that each functional layer of the searchable backbone constituent module is independently set with a learnable control weight, that is, each candidate set of the functional layer will have an independent set of candidate layers with control weights. In the present invention, for each functional layer of the independent searchable backbone constituent module, the control weight represents the importance of its candidate layer. In the initial stage of the neural network, the control weight values of each candidate layer are equivalent, indicating that these candidate layers are equally important. As the network search training proceeds, the control weight values of each candidate layer begin to change, and the importance of the corresponding candidate layer also changes. When searching for the best structure of the searchable backbone module, for each functional layer, a new candidate layer is sampled from its corresponding candidate set to replace the current layer, the sampled new neural network structure is trained, and then the learnable control weights of the candidate layers in the candidate set corresponding to each functional layer are updated. With the continuous iteration of training, the learnable control weights of the candidate layers in the candidate set corresponding to each functional layer change and become different from each other. The learnable control weights of the same type of candidate layers in different functional layers will become different due to the segmentation accuracy of the neural network on the training set. Take the first iteration of the neural network on the training set as an example. Figure 4 The training process of the first searchable backbone building block in the encoder is shown. As mentioned before, in the initial stage, the control weights of all candidate layers in the candidate set of the common layer are equal, which is 1 / 7. The control weights of all candidate layers in the candidate set of the compression layer are also equal, which is 1 / 3. Here we use the ProxylessNas sampling method to perform sampling search on the given candidate layer, where the one with a higher control weight is more likely to be selected, such as Figure 4As shown, since the control weights of the candidate layers of each functional layer are the same at this time, based on equal probability, the first first searchable backbone constituent module randomly selects a 3x3x3 convolution layer from the candidate set of the common layer, and the compression layer selects the maximum pooling layer. In the next step, the input of the first searchable backbone constituent module passes through the selected common layer (i.e., 3x3x3 convolution layer) and compression layer (maximum pooling layer) in turn to obtain the output of the module. It should be noted that since each searchable backbone constituent module independently sets the control weights of the candidate layers in the corresponding candidate set of its functional layer, the sampling results are also inconsistent, that is, in another first searchable backbone constituent module (such as the second first searchable backbone constituent module), the common layer may select a 5x5x5 convolution layer, and the compression layer selects an average pooling layer. Finally, as Figure 5 The encoder (including four first searchable backbone constituent modules) and the decoder (including four second searchable backbone constituent modules) in the neural network shown in the figure each use the obtained candidate layer and the searchable multi-scale feature aggregation module to form a network combination, output predictions, and compare them with the corresponding manual labels, calculate the loss value to perform standard gradient backpropagation updates, and update the network parameters and each set of independent control weights. The independent control weights of the functional layers of different searchable backbone constituent modules are updated independently. For example, in the first first searchable backbone constituent module, the control weight of the 3x3x3 convolution layer changes from 1 / 7 to 1 / 4, and in the second first searchable backbone constituent module, the control weight of the 3x3x3 convolution layer changes from 1 / 7 to 1 / 8. The weight of each functional layer corresponding to the candidate layer in the candidate set reflects the excellent performance of the candidate layer. The better the performance of the candidate layer, the higher its weight will be updated during the training update process, and the corresponding probability of being selected will also be higher.
[0045] In addition, while the searchable backbone constituent module is searching, the searchable multi-scale feature aggregation module is synchronously searching for the optimal multi-scale feature aggregation strategy. The two influence each other, and are jointly driven by the accuracy of the network output obtained by their joint action on the training set, and synchronously search for the overall optimal network structure. As mentioned earlier, the searchable multi-scale feature aggregation module is used to better aggregate the various scale features obtained by the encoder, and gradually use the aggregated features as the input of the decoder for classifier prediction. Therefore, the searchable multi-scale feature aggregation module and the searchable backbone constituent module work together on the network output, and are also jointly driven by the segmentation accuracy to collaboratively search for the structure. Regarding the searchable multi-scale feature aggregation module, it is still based on Figure 5 For example, in the first aggregation stage, the aggregation node uses the multi-scale features generated in the encoder as input to perform weighted aggregation. Specifically, the aggregation node N 1,0 For example, the encoder output {N0,0 、N 0,1 、N 0,2 、N 0,3 、N 0,4} Enter the aggregation node N via the searchable connection 1,0 , each searchable connection is additionally equipped with a learnable weight and gate function. Figure 6 As shown, Figure 6 Shown in Figure 5 {N 2,0 ,N 2,1 ,N 2,2} to N 3,0 An aggregation process of searchable connections, with N 2,0 To N 3,0 For example, N 2,0 First, a function F (a regular convolutional layer) is used to convert the feature size to N 3,0 The features are aligned and then with a weight σ(β 2,0→3,0 ) are multiplied, where β 2,0→3,0 Representative as N 2,0 Node and feature N 3,0 The learnable weights on the searchable connections are represented by σ, where σ represents the sigmoid gate function:
[0046]
[0047] like Figure 7 As shown in , the input of the gate function will be mapped to the range [0,1]. At the time of initialization, each learnable weight β is set to 0, and the value of σ(β) is 0.5. The value of σ(β) represents the importance of each searchable connection. The higher the value of σ(β), the more important the corresponding searchable connection is and should be retained, and vice versa. When the value of σ(β) is 1, the corresponding searchable path is fully open. When σ(β) is 0, the searchable path is completely closed (multiplying the input by 0 has no effect), as shown in Figure 8 As shown in a, b, and c, by changing the value of σ(β), the neural network connects / disconnects the corresponding scale feature propagation path. The aggregated features obtained in each aggregation stage will be used as the input of the next aggregation stage. Iteratively, the last aggregated feature of each aggregation stage will be used as the input of the decoder. The aggregated features of the final stage {N 4,0}For the prediction of the classifier, we can see that the aggregate feature N used for neural network prediction 4,0 They are jointly influenced by the searchable backbone construction module and the searchable multi-scale feature aggregation module, so these two types of modules are jointly driven by the segmentation accuracy and collaboratively search for structures.
[0048] As mentioned above, the searchable multi-scale feature aggregation module and the searchable component backbone module work together to search for the overall optimal structure, such as Figure 5 The searchable multi-scale feature aggregation module shown in the figure is completed in four stages. As mentioned above, the multi-scale features output by the encoder are used as the input of the feature aggregation in the first aggregation stage. The aggregated features are used as the input of the next aggregation stage, and iteratively, the last aggregated features of each aggregation stage are used as the input of the decoder. In this process, the newly aggregated features can be expressed as:
[0049]
[0050] Among them, N i,j It is represented as the i-th stage and j-th aggregation node of the searchable multi-scale feature aggregation module, and β represents N i,j The learnable weights on the searchable connections between nodes and input features use the σ function (i.e., sigmoid gate function) to map β to [0,1] to measure the importance of its connection propagation features. The larger the score, the more important and useful the corresponding searchable connection is. When the score is 1, it means that the connection is very important and the path is completely open. When the score is 0, it means that the connection is not important and the path is completely closed. In fact, during the training phase, all searchable connections will participate in the calculation. During the search process, some rules such as the value of the gate function are directly used to determine whether the corresponding connection should be retained. L represents the number of output features in the previous stage, and T represents the alignment of these feature maps of different scales through a series of transformations, such as Figure 6 As shown in Figure 1, we use the addition operator and the weights of the σ gate function to weight all the input features transformed by T for aggregation, that is, summation. For an input data on a given training set, N 4,0The searchable multi-scale feature aggregation module and the searchable backbone module work together to obtain the prediction results obtained by the classifier and compare them with the manual labels corresponding to the training set, calculate the loss value to perform standard gradient backpropagation, and update the learnable weights on the searchable connection. Note that, like the searchable backbone module, the searchable connections in the searchable multi-scale feature aggregation module jointly determine the searched network structure, but update their learnable weights independently. When the prediction result is not accurate in a certain iteration, the learnable weight of a searchable connection decreases from 0.5 to 0.3, and the influence of the connection decreases, while the learnable weight of another searchable connection increases from 0.5 to 0.7, and the influence of the connection increases. After continuous iterative training, the learnable weight of each feature connection path is constantly updated and changed. Finally, when selecting the best neural network structure, the feature path whose weight meets the preset threshold requirement is retained and opened. The preset threshold is determined according to the segmentation accuracy requirement of the specific application scenario. For example, it can be determined to be 0.9. At this time, all feature connections with learnable weights greater than or equal to 0.9 are retained and opened, forming the optimal feature aggregation strategy of the searchable feature aggregation module.
[0051] According to one embodiment of the present invention, a total of 60,000 training iterations are performed on the neural network on the training set. During the training process, the neural network is verified on the verification set every 500 iterations. The network structure with the best performance on the verification set is taken as the optimal neural network. The optimal candidate layer is selected for each functional layer in the searchable backbone component module, and the control weights of all candidate layers of each functional layer are sorted in descending order, and the candidate layer with the largest control weight is retained as the optimal structure of the functional layer. Due to the independence of the control weights of each searchable backbone component module, the optimal candidate layers obtained by searching different searchable backbone component modules may not be consistent. For example, the ordinary layer in the first searchable backbone component module selects a 3x3x3 convolutional layer as the optimal candidate layer, while the ordinary layer in the second first searchable backbone component module selects a 5x5x5 convolutional layer. Although the optimal layers of each functional layer may not be consistent, the overall network structure obtained by their search results and the search results of the searchable backbone component module is regarded as the optimal network structure on a specific data set.
[0052] Step 4: Retraining phase:
[0053] The optimal neural network selected after iterative sampling training is retrained with the training set until convergence, wherein the training convergence is determined when the classification accuracy of the initial neural network on the validation set does not change by more than 0.2% after a certain training compared with that before training.
[0054] In order to better illustrate the effect of the present invention, the segmentation accuracy of the neural network searched by the present invention is compared with other existing mainstream neural networks on the brain segmentation dataset. The comparison results are shown in Table 1. Table 1 shows the comparison of the segmentation performance of the present invention and other mainstream methods on the brain segmentation dataset. The results use the intersection-over-union ratio indicator to measure the segmentation results. Among them, this series of methods can be divided into three groups. One is the manually designed network, such as Unet, NNUnet, Nunet++, Un-ResNet, which are all network structures manually designed by researchers. The second is SCNAS and RONASMIS, both of which apply neural network search to medical images. However, their search space is too small, that is, the set of selectable sub-networks is too small, because it often does not include the optimal network structure. Finally, the present invention UXNet considers and designs more search spaces that are more in line with the characteristics of medical images, can search for better network structures and has better performance.
[0055] Table 1
[0056]
[0057] It can be seen from the above embodiments that the technical solution of the present invention can be summarized as follows: T1, constructing an initial neural network set and training each initial neural network to convergence, wherein each initial neural network in the initial neural network set includes an encoder, a searchable multi-scale feature aggregation module, a decoder, and a classifier, the encoder includes at least four first searchable backbone constituent modules, each of which includes a normal layer and a compression layer, and the decoder of the initial neural network includes at least four second searchable backbone constituent modules, each of which includes a normal layer and an expansion layer, wherein each normal layer is from its Each compression layer is selected from the candidate set of the corresponding compression layer, and each expansion layer is selected from the candidate set of the corresponding expansion layer; T2, use the training set corresponding to the medical image to perform multiple rounds of training on all the initial neural networks trained in step T1 until convergence, and determine the optimal feature aggregation strategy for the searchable multi-scale feature aggregation module in each neural network; T3, verify the segmentation accuracy of the initial neural network trained in step T2 on the verification set corresponding to the medical image, and select the neural network with the highest segmentation accuracy as the search result; T4, retrain the neural network selected in step T3 until convergence.
[0058] Among them, the candidate set of common layers includes the following candidate layers: 3x3x3 convolution layer, 3x3x1 convolution layer, 5x5x5 convolution layer, pseudo 3d convolution layer, two stacked 3x3x3 convolution layers, 3x3x3 dilated convolution with a stride of 2, and 5x5x5 dilated convolution layer with a stride of 2; the candidate set of compression layers includes the following candidate layers: maximum pooling layer, average pooling layer, 3x3x3 convolution layer with a stride of 2; the candidate set of extension layers includes the following candidate layers: transposed convolution layer and trilinear interpolation layer.
[0059] In addition, each node in the searchable multi-scale feature aggregation module has a learnable weight on the searchable connection between the input feature. In step T2, the optimal feature aggregation strategy of the searchable multi-scale feature aggregation module in the neural network is determined in the following manner: a gate function is used to map the learnable weight on the searchable connection between a node in the searchable multi-scale feature aggregation module and the input feature to [0, 1] to measure the importance of the connection propagation feature, and feature connections whose importance is greater than or equal to a preset threshold are retained, and feature connections whose importance is less than the preset threshold are deleted. Finally, the retained feature connections form a feature aggregation strategy, wherein,
[0060] The gate function is:
[0061]
[0062] Among them, σ represents the gate function, β is the learnable weight, and e is the base of the natural logarithm function. During initialization, each learnable weight is set to 0. In each training, standard gradient backpropagation is performed based on the loss value to update the learnable weights of all searchable connections in the special aggregation module.
[0063] The aforementioned embodiments are based on the idea of the technical solution and can more quickly search for the optimal neural network structure through iterative sampling training.
[0064] The existing methods usually search the backbone module, that is, only the encoder and decoder are searched. Although this approach can effectively extract features at various scales, it ignores how to further fuse these features for the final prediction. Feature fusion has always been a very important link in the design of 3D medical neural networks. Ignoring this link can only achieve limited performance improvement. In the present invention, we have added the feature fusion aspect, thereby increasing the overall search space, and we have also used an effective search strategy to jointly couple the two parts of the search space (backbone construction module, multi-scale feature aggregation), so that the best structure can be searched for analysis tasks with different data sources and different task definitions.
[0065] It should be noted that although the above describes the various steps in a specific order, it does not mean that the various steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.
[0066] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.
[0067] Computer readable storage medium can be a tangible device that holds and stores instructions used by an instruction execution device. Computer readable storage medium can include, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a protruding structure in a groove on which instructions are stored, and any suitable combination thereof.
[0068] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.
Claims
1. A neural network search method for medical image segmentation, the neural network comprising an encoder, a searchable multi-scale feature aggregation module, a decoder, and a classifier, characterized in that: The method comprises: S1. Construct an initial neural network and train it to convergence, wherein the encoder of the initial neural network includes one or more first searchable backbone constituent modules, each of which includes a normal layer and a compression layer, and the searchable multi-scale feature aggregation module of the initial neural network is used to aggregate features of various scales extracted by the encoder, and the feature connections in the searchable multi-scale feature aggregation module are all feature connections whose importance is greater than or equal to a preset threshold; the decoder of the initial neural network includes one or more second searchable backbone constituent modules, each of which includes a normal layer and an extended layer, wherein each normal layer is randomly selected from a candidate set of its corresponding normal layer, each compression layer is randomly selected from a candidate set of its corresponding compression layer, and each extension layer is randomly selected from a candidate set of its corresponding extension layer; S2. Perform multiple rounds of iterative sampling training on the initial neural network trained in step S1 using the training set corresponding to the medical image to obtain multiple neural networks, and select the best neural network with the highest segmentation accuracy from the multiple neural networks; wherein each round of iterative training is carried out in accordance with a preset sampling method, and for each layer in the neural network encoder and decoder, a new candidate layer is independently sampled from the candidate layers corresponding to each layer to replace the current layer, and the new neural network composed of the sampled encoder, decoder, searchable multi-scale feature aggregation module and classifier is trained using the training set, and at the same time, the best feature aggregation strategy corresponding to the searchable multi-scale feature aggregation module in the current neural network is searched; S3. Retrain the neural network selected in step S2 until convergence.
2. A neural network search method for medical image segmentation according to claim 1, characterized in that: The candidate set of common layers includes the following candidate layers: 3x3x3 convolution layer, 3x3x1 convolution layer, 5x5x5 convolution layer, pseudo 3d convolution layer, two stacked 3x3x3 convolution layers, 3x3x3 dilated convolution with a step size of 2, and 5x5x5 dilated convolution layer with a step size of 2; The candidate set of the compression layer includes the following candidate layers: a maximum pooling layer, an average pooling layer, and a 3x3x3 convolutional layer with a stride of 2; The candidate set of the extended layer includes the following candidate layers: a transposed convolution layer and a trilinear interpolation layer.
3. A neural network search method for medical image segmentation according to claim 2, characterized in that: The candidate layers in the candidate set corresponding to each layer in the encoder or decoder in the neural network have independent control weights, and the sum of the control weights of all candidate layers corresponding to the same layer in the encoder or decoder is 1; wherein before iterative sampling training, the initialization control weights of all candidate layers corresponding to the same layer are equal values.
4. A neural network search method for medical image segmentation according to claim 3, characterized in that: In step S2, the ProxylessNAS method is used in each round of iterative training to sample from the candidate set of each layer, and the training set corresponding to the medical image is input into all the sampled new candidate layers and the new neural network composed of the searchable multi-scale feature aggregation module and the classifier for prediction. The prediction result is compared with the manual label corresponding to the training set to calculate the loss value, and the standard gradient is returned based on the loss value to update the neural network parameters and the control weights of the candidate layers in the candidate set corresponding to each layer in the encoder and the decoder.
5. A neural network search method for medical image segmentation according to claim 4, characterized in that: A searchable connection between a node in a searchable multi-scale feature aggregation module and an input feature has a learnable weight; In each round of iterative training of the neural network, a gate function is used to map the learnable weight of the searchable connection between a node in the searchable multi-scale feature aggregation module and the input feature to [0, 1] to measure the importance of the connection propagation feature. Feature connections whose importance is greater than or equal to the preset threshold are retained, and feature connections whose importance is less than the preset threshold are deleted. The feature connections that are retained at last form a feature aggregation strategy. Wherein, the gate function is: Among them, σ represents the gate function, β is the learnable weight, and e is the base of the natural logarithm function. During initialization, each learnable weight is set to 0. In each iterative training, standard gradient backpropagation is performed based on the loss value to update the learnable weights of all searchable connections in the special aggregation module.
6. A neural network search method for medical image segmentation according to claim 5, characterized in that: The neural network is selected as follows: The weights of the candidate layers in the candidate set corresponding to each layer in the encoder and decoder of the neural network after iterative sampling training are arranged in descending order, and the candidate layer with the largest weight in each layer is selected to form the best neural network with the searchable multi-scale feature aggregation module and the classifier.
7. A neural network search method for medical image segmentation according to claim 6, characterized in that: In step S2, the initial neural network is trained by 60,000 iterations of sampling, and the neural network is verified on the verification set corresponding to the medical image every 500 iterations of training, and the neural network with the highest segmentation accuracy on the verification set is selected.
8. A neural network search method for medical image segmentation, characterized in that: include: T1. Construct an initial neural network set and train each initial neural network to convergence, wherein each initial neural network in the initial neural network set includes an encoder, a searchable multi-scale feature aggregation module, a decoder, and a classifier. The encoder includes one or more first searchable backbone constituent modules, each of which includes a normal layer and a compression layer. The searchable multi-scale feature aggregation module is used to aggregate features of various scales extracted by the encoder, and the feature connections in the searchable multi-scale feature aggregation module are all feature connections whose importance is greater than or equal to a preset threshold. The decoder of the initial neural network includes one or more second searchable backbone constituent modules, each of which includes a normal layer and an extended layer, wherein each normal layer is selected from a candidate set of its corresponding normal layer, each compression layer is selected from a candidate set of its corresponding compression layer, and each extension layer is selected from a candidate set of its corresponding extension layer; T2, using the training set corresponding to the medical image to perform multiple rounds of training on all the initial neural networks trained in step T1 until convergence, and determine the best feature aggregation strategy for searching the multi-scale feature aggregation module in each neural network; T3, verifying the segmentation accuracy of the initial neural network trained in step T2 on the verification set corresponding to the medical image, and selecting the neural network with the highest segmentation accuracy as the search result; T4. Retrain the neural network selected in step T3 until convergence.
9. A neural network search method for medical image segmentation according to claim 8, characterized in that: The candidate set of common layers includes the following candidate layers: 3x3x3 convolution layer, 3x3x1 convolution layer, 5x5x5 convolution layer, pseudo 3d convolution layer, two stacked 3x3x3 convolution layers, 3x3x3 dilated convolution with a step size of 2, and 5x5x5 dilated convolution layer with a step size of 2; The candidate set of the compression layer includes the following candidate layers: a maximum pooling layer, an average pooling layer, and a 3x3x3 convolutional layer with a stride of 2; The candidate set of the extended layer includes the following candidate layers: a transposed convolution layer and a trilinear interpolation layer.
10. A neural network search method for medical image segmentation according to claim 9, characterized in that: The searchable connection between a node in the searchable multi-scale feature aggregation module and the input feature has a learnable weight, and the optimal feature aggregation strategy of the searchable multi-scale feature aggregation module in the neural network is determined in the step T2 in the following manner: The gate function is used to map the learnable weight of the searchable connection between a node in the searchable multi-scale feature aggregation module and the input feature to [0, 1] to measure the importance of the connection propagation feature. The feature connections whose importance is greater than or equal to the preset threshold are retained, and the feature connections whose importance is less than the preset threshold are deleted. The feature connections retained at last form a feature aggregation strategy, where The gate function is: Among them, σ represents the gate function, β is the learnable weight, and e is the base of the natural logarithm function. During initialization, each learnable weight is set to 0. In each training, standard gradient backpropagation is performed based on the loss value to update the learnable weights of all searchable connections in the special aggregation module.
11. A neural network for medical image segmentation searched by the method according to any one of claims 1 to 10, characterized in that: The neural network comprises: An encoder, used to extract features of input data, wherein each layer in the encoder is a candidate layer with the highest weight selected from the candidate set corresponding to the layer; A searchable multi-scale feature aggregation module is used to aggregate features of various scales extracted by the encoder, wherein the feature connections in the searchable multi-scale feature aggregation module are all feature connections whose importance is greater than or equal to a preset threshold; A decoder, used to determine the segmentation target according to the features aggregated by the feature aggregation module, wherein each layer in the decoder is the candidate layer with the highest weight selected from the candidate set corresponding to the layer; The classifier is used to output the classification result of the input data according to the target determination result of the decoder.
12. A computer-readable storage medium, characterized in that: A computer program is included thereon, and the computer program can be executed by a processor to implement the steps of any one of the methods of claims 1 to 10.
13. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the steps of the method as claimed in any one of claims 1 to 10.
Citation Information
Patent Citations
Method for searching convolutional neural network based on number of channels
CN110991621A
Method and system for semantic segmentation involving multi-task convolutional neural network
US10467500B1
Neural architecture search for dense image prediction tasks
US20190370648A1
Convolutional neural network framework using reverse connections and objectness priors for object detection
WO2019028725A1
