Pancreatic cancer image segmentation method based on self-attention mechanism and neural network architecture search

Abstract

The invention discloses a pancreatic cancer image segmentation method based on a self-attention mechanism and neural network architecture search. The method comprises the following steps: firstly, extracting local feature information of an image through a CNN, acquiring global features by using a cascaded Transform structure, secondly, carrying out up-sampling operation on coding features of previous two-stage networks through a decoder, and combining the coding features with a high-resolution CNN feature map to realize accurate positioning. And finally, in the NAS architecture, adopting different primitive operation sets for searching, in the down-sampling operation, constructing two search spaces and respectively adopting different search strategies to retrieve spatial information, performing feature fusion on the obtained structure through up-sampling and a convolutional layer, and outputting a more accurate segmentation result. Through experiments, the thought is feasible in the segmentation problem, the method also has good performance in the segmentation field of migrating to other image data sets in the same way, and a network form with a better structure can be searched out through an NAS in an attempt, so that the image segmentation capability is further improved.

Description

Technical Field

[0001] The present invention belongs to the field of image segmentation and is a pancreatic cancer image segmentation method based on self-attention mechanism and neural network architecture search. Background Art

[0002] Cancer is the most common fatal disease. Pancreatic cancer is a malignant tumor that originates from pancreatic tissue. It is usually difficult to detect in the early stages and develops rapidly. The pancreas is an important organ located deep in the abdominal cavity, responsible for producing digestive enzymes and regulating blood sugar levels. The early symptoms of pancreatic cancer are often non-specific, including indigestion, abdominal pain, weight loss, etc., so it is often discovered in the late stages. The causes of pancreatic cancer are complex and may be related to genetic factors, smoking, eating habits, pancreatitis and other factors. The diagnosis of pancreatic cancer usually requires comprehensive judgment through imaging examinations (such as CT, MRI, ultrasound, etc.), blood tests and tissue biopsies. It is highly malignant, prone to metastasis, and has strong tolerance to chemotherapy and radiotherapy. Therefore, early diagnosis and precise treatment of pancreatic cancer are particularly important. Medical imaging plays a key role in the diagnosis and treatment of pancreatic cancer, and pancreatic cancer image segmentation technology can help doctors accurately identify tumor areas, guide the formulation of subsequent treatment plans, and improve patients' survival rate and quality of life.

[0003] With the continuous maturity of computer vision and image processing in the medical field, the use of computer technology with deep learning methods to analyze medical images has become the mainstream trend in today's medical community. For fully convolutional neural networks, U-Net has become the most classic algorithm. It is a symmetric fully convolutional neural network based on the encoder-decoder structure. The added jump connection splices low-level spatial detail features and high-level abstract features to segment cells, achieving good results. It has been widely used in the field of medical image segmentation. Based on the U-Net model, new network structures such as H-DenseUNet and Unet++ have been developed, showing strong advantages in a variety of medical image segmentation tasks, but due to the limitations of convolution operations, it performs poorly in constructing explicit long-range relationships in images. In particular, it shows great differences between different patients in terms of texture, shape and size. In order to overcome this limitation, the Transformer model based on the attention mechanism has gradually demonstrated its potential. In recent years, the combination of attention mechanism and convolutional neural network has been applied to the field of medical image segmentation, and great breakthroughs have been made. Chen et al. used Transformer to pay attention to global spatial information and combined it with a convolutional neural network to propose the TransUNet model. This was the first attempt to apply Transformer to medical image segmentation, and it has the advantages of both Transformer and U-Net. Thanks to the development of artificial intelligence, automated neural architecture search (NAS) has gradually become a research hotspot. Compared with manually designed neural network architectures, it can automatically generate suitable network structures based on corresponding data sets, greatly reducing the number of manual trial and error. However, NAS is often used in the field of image classification, and there is still room for improvement in image segmentation.

[0004] This method adds the structure of CNN-Transformers to the neural architecture search and automatically searches for U-shaped networks. First, the local feature information of the image is extracted through CNN, and the global features are obtained using the cascaded Transformer structure. Secondly, the encoded features of the first two levels of the network are upsampled through the decoder, and the encoded features are combined with the high-resolution CNN feature map to achieve accurate positioning. Finally, in the NAS architecture, different primitive operation sets are used for search. In the downsampling operation, two search spaces are constructed and different search strategies are used to retrieve spatial information. The obtained structure is feature fused through upsampling and convolution layers to output more accurate segmentation results. Summary of the invention

[0005] In the field of medical image segmentation, technicians often need to conduct a large number of experiments on existing network architectures in order to select the neural network architecture model with the best performance. The present invention aims to provide a technical support for the automatic identification and diagnosis of pancreatic cancer, and proposes a pancreatic cancer image segmentation method based on self-attention mechanism and neural network architecture search with high detection accuracy. The structure of CNN-Transformers is added to the neural architecture search, which can automatically search for network structures with global and local feature information, so that the feature information can be accurately located in the image, and the automatic neural network model can be built, which greatly reduces the workload of technicians.

[0006] A pancreatic cancer image segmentation method based on self-attention mechanism and neural network architecture search, characterized in that the specific steps are as follows:

[0007] S1: Randomly preprocess the pancreatic cancer dataset and use the processed dataset as the input dataset for model training;

[0008] S2: Construct a U-shaped network structure of encoding and decoding of multiple CNN combinations, construct the NAS stage 1 search space, and input the image training data set and tags ;

[0009] S3: Deploy the weight parameters of the CNN encoding and decoding U-type network structure , add the Transformer encoding structure of different Transformer layers to form the search space of NAS stage 2, and use the search data of stage 1 as the input sequence;

[0010] S4: Use the reinforcement learning search strategy to train the overall hypernetwork according to the architecture parameters , select the Transformer encoding network structure with better performance and save its network weight parameters ,according to and Deploy the CNN-Transformer network structure and input test data , complete the segmentation of medical images.

[0011] The specific steps of preprocessing the pancreatic cancer data set in step S1 are as follows:

[0012] S1.1: Download and standardize the original pancreatic cancer dataset images, resize all photos to the same size, and clean the photos by filtering, noise reduction, etc.

[0013] S1.2: Convert the preprocessed photos into a digital sequence array using numpy array to obtain the training data set train, the test data set test, and the validation data set validate;

[0014] The step S2 constructs the stage 1 search space module of the encoder, and the specific steps are as follows:

[0015] S2.1: Construct a NAS search space composed of only CNNs. The same number of CNN encoding downsampling modules and CNN decoding upsampling modules are combined to form a set of candidate modules for the search space. The number of convolutional layers, the size of the convolution kernel, and the channel level of the downsampling and upsampling (CNN encoder and CNN decoder) are all variable. Therefore, a CNN search space composed of multiple candidate modules can be constructed. All candidate modules form a super network.

[0016] S2.2: Each module can be To indicate that is the input data, It is The weights of the modules, each of which has an associated architectural parameter , these parameters are learned from a set of , through the softmax function, which means that for each module, there is a Parameters, which together determine the importance of different modules in the overall network output. Program a set of probability distributions after normalization by the softmax function ,in Indicates The probability or importance of modules being selected, all The sum of the addition is 1. The output z of the entire hypernetwork is the weighted sum of all candidate modules, and the mathematical expression is:

[0017]

[0018] S2.3: Finally, the constructed super network is trained through a continuous gradient search algorithm, and the weight of each module and architecture parameters are learned during training, and the architecture parameters It directly determines the role of each module in the final network architecture. is the input sequence The output result of the hypernetwork. According to the symbols defined in the previous article, the cross-entropy loss function can be written as:

[0019]

[0020] in About The function of It is The target of the module's expected output is and For each module and So each stage can be differentiated, so the parameters can be updated by back propagation After the training is completed, according to Parameters select the most important modules to determine the optimal CNN network architecture. The module with the largest value will be selected.

[0021] The step S3 will deploy the weight parameters , adding different Transformer encoding structures, the specific steps to form the search space of NAS stage 2 are as follows:

[0022] S3.1: Selecting parameters After the appropriate CNN sampling model is selected, the weight value of the CNN sampling model is deployed, and the image information extracted by CNN features is directly passed into the Transformer network structure in the encoder stage;

[0023] S3.2: Setting the network structure with different Transofrmer layers constitutes the search space of stage 2. Here, it is assumed that For the input sequence The output result of the CNN-Transformer hypernetwork. Represents the number of candidate networks in the search space composed of CNN-Transformer, and Represents the weights of each module of the Transformer network and architecture parameters A collection of;

[0024] The specific steps of using the search strategy of reinforcement learning to train the overall super network in step S4 are as follows:

[0025] S4.1: In stage 2, an agent, i.e., an agent network, is constructed to execute the defined search space by using reinforcement learning. First, the state space of the agent is defined by the architecture or parameter settings of the neural network, and then operations such as adding or deleting specific layers and modifying network structure parameters are added to form the action space, and a reward function is designed to evaluate the quality of the agent taking specific actions in a specific state. Finally, the environment simulator is used to train the agent. The agent learns how to choose actions to maximize long-term rewards by interacting with the environment;

[0026] S4.2: After training is completed, the structure of each layer of transformers is evaluated through this dynamic generation, so that a better Transformer network architecture can be selected and its network weight parameters can be saved. ;

[0027] S4.3: Combining the local features and global feature information obtained by training the CNN module and the Transformers module, a cascade upsampler (CUP) is introduced to decode the hidden features of the final segmentation output through multi-level upsampling operations;

[0028] S4.4: Based on and Deploy the CNN-Transformer network structure and input test data , complete the segmentation of medical images. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 The present invention discloses a pancreatic cancer image segmentation method based on self-attention mechanism and neural network architecture search.

[0030] Figure 2 The figure is a flowchart of the pancreatic cancer dataset preprocessing according to the present invention.

[0031] Figure 3 This is a schematic diagram of the CNN search space in stage one of the present invention.

[0032] Figure 4 Schematic diagram of the search space information transmission between the first stage and the second stage described in the present invention.

[0033] Figure 5 Schematic diagram of the neural network structure of the present invention. DETAILED DESCRIPTION

[0034] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings:

[0035] like Figure 1As shown, a pancreatic cancer image segmentation method based on self-attention mechanism and neural network architecture search from step 101 to step 104:

[0036] Step 101: performing random preprocessing on the pancreatic cancer dataset, and using the processed dataset as an input dataset for model training;

[0037] Step 102: Construct a U-shaped network structure of multiple CNN combinations, construct the NAS stage 1 search space, and input the image training data set and tags ;

[0038] Step 103: Deploy the weight parameters of the CNN encoding and decoding U-type network structure , add the Transformer encoding structure of different Transformer layers to form the search space of NAS stage 2, and use the search data of stage 1 as the input sequence;

[0039] Step 104: Use the reinforcement learning search strategy to train the entire hypernetwork according to the architecture parameters , select the Transformer encoding network structure with better performance and save its network weight parameters ,according to and Deploy the CNN-Transformer network structure and input test data , complete the segmentation of medical images.

[0040] like Figure 2 As shown: the preprocessing process step 101 of the pancreatic cancer data set from step 201 to step 204:

[0041] Step 201: Download the pancreatic cancer dataset from the website;

[0042] Step 202: Clean the photo by filtering, noise reduction and other methods;

[0043] Step 203: pre-process the image randomly and crop the center to 224×224 pixels;

[0044] Step 204: Obtain the final pancreatic cancer dataset.

[0045] like Figure 3 As shown: Design stage 1 CNN encoder search space schematic diagram Step 102 from step 301 to step 304:

[0046] Step 301: randomly extract candidate operations in the search space, including pooling layer, fully connected layer, 3x3 convolution layer, 5x5 convolution layer, and skip connection layer;

[0047] Step 302: training the randomly extracted candidate operations for feature extraction through a neural network;

[0048] Step 303: Perform test set verification on the trained neural network and record the test results;

[0049] Step 304: Select a set of experimental data with the highest accuracy as the final network model structure;

[0050] like Figure 4 As shown: Design the connection layer to transfer the search space information between stage one and stage two, step 103, step 401 to step 404:

[0051] Step 401: define a connection layer in the search space;

[0052] Step 402: Use x.view(x.size(0), -1) to flatten the features into a two-dimensional tensor with shape (batch_size, channelsheightwidth);

[0053] Step 403: The flattened feature tensor passes through the linear transformation layer nn.Linear(in_channels, out_channels) to adjust its dimension from in_channels to out_channels;

[0054] Step 404: The feature map is adjusted to a shape and dimension compatible with the Transformer layer input.

[0055] like Figure 5 Shown: This is a schematic diagram of the overall network structure:

[0056] First, we construct a NAS search space composed of only CNNs. We combine the same number of CNN encoding downsampling modules with CNN decoding upsampling modules to form a set of candidate modules for the search space (the candidate modules are designed based on our own experience). The number of convolutional layers, the size of the convolution kernel, and the channel level of the downsampling and upsampling (CNN encoder and CNN decoder) are all variable. Therefore, we can construct a CNN search space composed of multiple candidate modules. All candidate modules form a super network. Secondly, we assume that each module can be To indicate that is the input data, It is The weights of the modules, each of which has an associated architectural parameter , these parameters are learned from a set of , through the softmax function, which means that for each module, there is a Parameters, which together determine the importance of different modules in the overall network output. Program a set of probability distributions after normalization by the softmax function ,in Indicates The probability or importance of modules being selected, all The sum of the addition is 1. The output z of the entire hypernetwork is the weighted sum of all candidate modules, and the mathematical expression is:

[0057]

[0058] In the above formula, n represents the number of modules. Finally, the constructed hypernetwork is trained through a continuous gradient search algorithm, and the weight of each module is and architecture parameters are learned during training, and the architecture parameters It directly determines the role of each module in the final network architecture. is the input sequence The output result of the hypernetwork. According to the symbols defined in the previous article, the cross-entropy loss function can be written as:

[0059]

[0060] From the above formula we can see that About The function of It is The target of the module's expected output is and For each module and So each stage can be differentiated, so the parameters can be updated by back propagation ,therefore can be calculated as:

[0061]

[0062] After the training is completed, according to Parameters select the most important modules to determine the optimal CNN network architecture. The module with the largest value will be selected.

[0063] Phase 2: Selecting parameters After the appropriate CNN sampling model is obtained, the weight value of the CNN sampling model is deployed. In the encoder stage, the image information extracted by CNN features is directly passed into the Transformer network structure, and then the image segmentation result is obtained through the decoder. Among them, the network structure with different Transformer layers constitutes the search space of the second stage. Here, it is assumed that For the input sequence The output result of the CNN-Transformer hypernetwork. Represents the number of candidate networks in the search space composed of CNN-Transformer, and Represents the weights of each module of the Transformer network and architecture parameters The search steps of the network are consistent with those in stage 1, but the difference is that the search strategy adopts reinforcement learning. By building an intelligent agent (i.e., agent network) to execute the defined search space, the agent can obtain better long-term rewards by learning the action strategy of interacting with the environment. After training, the structure of each layer of transformers is evaluated through this dynamic generation, so that a better Transformer network architecture can be selected to improve image segmentation performance.

Claims

1. A pancreatic cancer image segmentation method based on self-attention mechanism and neural network architecture search, characterized in that: The specific steps are as follows: S1: The pancreatic cancer dataset is randomly preprocessed and the processed dataset is used as the input dataset for model training; S2: Construct a U-shaped network structure of encoding and decoding of multiple CNN combinations, construct the NAS stage 1 search space, and input the image training data set and tags ; S3: Deploy the weight parameters of the CNN encoding and decoding U-type network structure , add the Transformer encoding structure of different Transformer layers to form the search space of NAS stage 2, and use the search data of stage 1 as the input sequence; S4: Use the reinforcement learning search strategy to train the overall hypernetwork according to the architecture parameters , select the Transformer encoding network structure with better performance and save its network weight parameters ,according to and Deploy the CNN-Transformer network structure and input test data , complete the segmentation of medical images.

2. A pancreatic cancer image segmentation method based on self-attention mechanism and neural network architecture search according to claim 1, characterized in that: The specific steps of preprocessing the pancreatic cancer data set in step S1 are as follows: S1.1: Download and standardize the original pancreatic cancer dataset images, resize all photos to the same size, and clean the photos by filtering, noise reduction, etc. S1.2 converts the preprocessed photos into arrays in the form of digital sequences through numpy arrays to obtain the training data set train, the test data set test, and the validation data set validate.

3. A pancreatic cancer image segmentation method based on self-attention mechanism and neural network architecture search according to claim 1, characterized in that: The step S2 constructs the stage 1 search space module of the encoder, and the specific steps are as follows: S2.1: Construct a NAS search space composed of only CNNs. The same number of CNN encoding downsampling modules and CNN decoding upsampling modules are combined to form a set of candidate modules for the search space. The number of convolutional layers, the size of the convolution kernel, and the channel level of the downsampling and upsampling (CNN encoder and CNN decoder) are all variable. Therefore, a CNN search space composed of multiple candidate modules can be constructed. All candidate modules form a super network. S2.2: Each module can be To indicate that is the input data, It is The weights of the modules, each of which has an associated architectural parameter , these parameters are learned from a set of , through the softmax function, which means that for each module, there is a Parameters, which together determine the importance of different modules in the overall network output. Program a set of probability distributions after normalization by the softmax function ,in Indicates The probability or importance of modules being selected, all The sum of the addition is 1; the output z of the entire hypernetwork is the weighted sum of all candidate modules, and the mathematical expression is: S2.3: Finally, the constructed super network is trained through a continuous gradient search algorithm, and the weight of each module and architecture parameters are learned during training, and the architecture parameters It directly determines the role of each module in the final network architecture. is the input sequence The output of the hypernetwork; according to the symbols defined above, the cross-entropy loss function can be written as: in About The function of It is The target of the module's expected output is and For each module and So each stage can be differentiated, so the parameters can be updated by back propagation After the training is completed, according to Parameters select the most important modules to determine the optimal CNN network architecture. The module with the largest value will be selected.

4. A pancreatic cancer image segmentation method based on self-attention mechanism and neural network architecture search according to claim 1, characterized in that: The step S3 will deploy the weight parameters , adding different Transformer encoding structures, the specific steps to form the search space of NAS stage 2 are as follows: S3.1: Selecting parameters After the appropriate CNN sampling model is selected, the weight value of the CNN sampling model is deployed, and the image information extracted by CNN features is directly passed into the Transformer network structure in the encoder stage; S3.2: Setting the network structure with different Transofrmer layers constitutes the search space of stage 2. Here, it is assumed that For the input sequence The output result of the CNN-Transformer hypernetwork; Represents the number of candidate networks in the search space composed of CNN-Transformer, and Represents the weights of each module of the Transformer network and architecture parameters A collection of .

5. A pancreatic cancer image segmentation method based on self-attention mechanism and neural network architecture search according to claim 1, characterized in that: The specific steps of using the search strategy of reinforcement learning to train the overall super network in step S4 are as follows: S4.1: In stage 2, an agent, i.e., an agent network, is constructed to execute the defined search space by using reinforcement learning. First, the state space of the agent is defined by the architecture or parameter settings of the neural network. Then, operations such as adding or deleting specific layers and modifying network structure parameters are added to form the action space. A reward function is designed to evaluate the performance of the agent in taking specific actions in a specific state. Finally, an environment simulator is used to train the agent. The agent learns how to choose actions to maximize long-term rewards by interacting with the environment. S4.2: After training is completed, the structure of each layer of transformers is evaluated through this dynamic generation, so that a better Transformer network architecture can be selected and its network weight parameters can be saved. ; S4.3: Combining the local features and global feature information obtained by training the CNN module and the Transformers module, a cascade upsampler (CUP) is introduced to decode the hidden features of the final segmentation output through multi-level upsampling operations; S4.4: Based on and Deploy the CNN-Transformer network structure and input test data , complete the segmentation of medical images.

Images