A Multi-scale Fusion Fully Convolutional Network Lymph Node Detection Method Based on CT Images

Abstract

The present invention discloses a method for detecting abdominal lymph nodes based on a multi-scale fusion convolutional network of CT images. The characteristics of this method include: performing data preprocessing on CT images to obtain patch images; sending the patch images into the MiniUNet network to extract features and obtain lymph node candidate regions; extracting lymph node candidate ROI regions of three scales, large, medium, and small; sending the ROI region images into the MSENet network to extract features and perform classification, using the SPPLayer to unify the output feature size and perform feature fusion, performing binary classification from multiple scales, and outputting the binary classification results, etc. Compared with the prior art, the present invention can accurately predict whether it is a true lymph node, can automatically and quickly detect the shape and position of abdominal lymph nodes, provide reference for doctors, and provide ROI regions for subsequent lymph node metastasis prediction, with strong effectiveness, time-saving, labor-saving, and high efficiency, and has good application prospects.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision and deep learning, and in particular to a two-stage multi-scale fusion convolutional network abdominal lymph node detection method based on CT images. Background Art

[0002] Lymph nodes are important immune organs of the human body. Lymph nodes can effectively fight against bacteria, viruses or other harmful substances invading the body. However, at the same time, lymphocytes and histiocytes in the lymph nodes reactively proliferate, causing the lymph nodes to swell, which is called reactive hyperplasia of lymph nodes. Abdominal lymph nodes are located in the abdominal cavity, and there are lymph nodes around the mesentery, intestinal tract, and organs. Inflammation or tumor metastasis in the abdomen can cause lymph node swelling. Local lymph node swelling can reflect lesions in the area where the lymph nodes receive lymph. If the local lymph nodes cannot intercept and remove these bacteria or toxins, the lesions can spread and diffuse further along the efferent ducts of the local lymph nodes. Therefore, accurately detecting the location of lymph nodes, the range of lymph received, and the direction of lymphatic drainage has important clinical significance.

[0003] In clinical practice, due to the small size of lymph nodes themselves, and the fact that abdominal medical images have numerous organs, dense tissues, blood vessels, and fat, and large temporal variations, the identification of lymph nodes highly depends on the clinical experience of doctors. Detecting lymph node metastasis based on CT images is one of the most important methods in clinical practice. It requires doctors to view each image layer by layer to identify the shape and location of lymph nodes. However, this method consumes a large amount of time and is subjective.

[0004] The identification of lymph nodes in the prior art requires doctors to view each CT image layer by layer to identify the shape and location of lymph nodes, highly depending on the clinical experience of doctors. It is not only time-consuming, laborious, and inefficient, but also subjective and has poor effectiveness. Summary of the Invention

[0005] The purpose of the present invention is to propose a multi-scale fusion convolutional network abdominal lymph node detection method based on CT images in view of the deficiencies of the prior art. The constructed MiniUNet network is used to extract features from the preprocessed patch images, and the network is trained using a pixel-level loss function. The three-scale images obtained are sent to the MSENet network for feature extraction and classification. The SPPLayer is used to unify the output feature size and perform feature fusion. Binary classification is performed from multiple scales and the classification results are output. This method can automatically and quickly detect the shape and location of abdominal lymph nodes, provide reference for doctors, and provide ROI regions for subsequent lymph node metastasis prediction. It has strong effectiveness, saves time and effort, is efficient, and has good application prospects.

[0006] The specific technical solution for achieving the object of the present invention is: a multi-scale fusion convolutional network abdominal lymph node detection method based on CT images. The characteristic of this method is that after data processing on the obtained CT image data, smaller-scale training patch images are cropped. After preprocessing each patch image, it is sent into a newly constructed MiniUNet network to obtain image features at the patch size scale and generate lymph node candidate regions. During the training process, a pixel-level loss function is used to accelerate network convergence and improve the network's ability to detect lymph nodes. Then, CT images of large, medium, and small scales are extracted from the lymph node candidate regions, and the images of the three scales are sent into a classification network based on MSENet. During the training process, the SPPLayer is used to unify the feature sizes, binary classification is performed on the outputs and fused features of the three scales, and the soft voting algorithm is used to obtain the final classification result. Those exceeding the preset threshold are recorded as true lymph nodes. The detection of abdominal lymph nodes specifically includes the following steps:

[0007] Step 1: One-stage data preprocessing

[0008] After processing the CT image data, the ROI regions containing lymph nodes and background regions without lymph nodes are cropped. Rotation and random center cropping are combined, etc., to perform data augmentation and expansion on the dataset to obtain images of smaller scale sizes.

[0009] Step 2: One-stage network design module

[0010] Design the MiniUNet network module to extract image features from smaller-scale patch inputs and use the skip-attention module to fuse the image features.

[0011] Step 3: One-stage network training module

[0012] Use a loss function based on pixel selection to select and update partial gradients to optimize the network and improve the accuracy of generating lymph node candidate regions.

[0013] Step 4: Two-stage data processing

[0014] For the lymph node candidate regions obtained in the one-stage, extract the lymph node candidate ROI regions at large, medium, and small scales, and use rotation and contrast enhancement to perform data augmentation on the dataset to obtain images at the three scales.

[0015] Step 5: Two-stage network design module

[0016] Design the MSENet with multi-scale fusion, extract features from image inputs at three scales, use SPPLer to unify the output feature size, perform binary classification on the outputs at three scales and the fused features, and use the soft voting algorithm to obtain the final classification result. Those exceeding the preset threshold are recorded as true lymph nodes.

[0017] The soft voting algorithm is to perform weighted averaging on the probabilities of the outputs at three scales and the fused output to obtain the predicted value. Those exceeding the preset threshold are recorded as true lymph nodes.

[0018] Compared with the prior art, the present invention can accurately predict whether it is a true lymph node, can automatically and quickly detect the shape and position of abdominal lymph nodes, provide reference for doctors, and provide the ROI area for subsequent lymph node metastasis prediction. It has strong effectiveness, saves time and effort, is efficient, and has good application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is the flow chart of the present invention;

[0020] Figure 2 is the architecture diagram of the one-stage convolutional neural network of the present invention;

[0021] Figure 3 is the structure diagram of the one-stage skip-attention of the present invention;

[0022] Figure 4 is the architecture diagram of the two-stage convolutional neural network of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0023] Refer to Figure 1 , the present invention uses a two-stage multi-scale fusion convolutional network for feature extraction, performs binary classification from multiple scales, and obtains the shape and position of abdominal lymph nodes, which specifically includes the following steps:

[0024] Step 1: Perform data preprocessing and cropping on the CT image to obtain a smaller-scale patch image required for training;

[0025] Step 2: Send the patch image into the MiniUNet network to extract features;

[0026] Step 3: Use a loss function based on pixel selection to train the network to obtain lymph node candidate regions;

[0027] Step 4: Process the lymph node candidate regions and extract large, medium, and small-scale lymph node candidate ROI regions;

[0028] Step 5: Send the lymph node candidate ROI region image into the MSENet network to extract features and perform classification. Use the SPPLayer to unify the feature size and perform feature fusion, perform binary classification from multiple scales, and output the binary classification result.

[0029] Combined with the following specific embodiments and accompanying drawings, the present invention will be further described in detail. The processes, conditions, experimental methods, etc. for implementing the present invention, except for the specifically mentioned content below, are all common knowledge and well-known common sense in the art, and the present invention has no particularly limited content.

[0030] Embodiment 1

[0031] The present invention uses CT images, crops to obtain the ROI region containing lymph nodes and the background region without lymph nodes, and combines the use of rotation and random central cropping, etc. to perform data augmentation and expansion on the data set. After preprocessing each patch image, it is sent into the newly constructed MiniUNet network to obtain the image features at the patch size scale and generate lymph node candidate regions. During the training process, a pixel-level loss function is used to accelerate network convergence and improve the detection ability of lymph nodes. After obtaining the lymph node candidate regions, images of three scales, large, medium, and small, are extracted from the candidate regions, and the images of the three scales are sent into the classification network based on MSENet. During the training process, the SPPLayer is used to unify the feature size and perform feature fusion, perform binary classification on the outputs of the three scales and the fused features, and record as true lymph nodes when exceeding the preset threshold, and finally use the voting algorithm to obtain the final classification result. The specific operations are carried out according to the following steps:

[0032] (1) Data preprocessing in the first stage

[0033] For CT images, cropping is required, and the region of interest containing lymph nodes and a randomly selected background region without lymph nodes are retained. Then, 4 rotation angles are randomly selected (the rotation angles are between 0 - 10°), combined with 5 random central cropping methods, that is, the combination of rotation and central cropping with the upper left, lower left, upper right, lower right offset amounts of the centroid coordinates of the ROI and the centroid coordinates themselves, to perform 20-fold data augmentation on the original data set. Finally, each image is used as the input.

[0034] (2) Design of the network model in the first stage

[0035] The one-stage network has an encoder-decoder structure. The Encoder feature extractor is composed of Resblocks and downsamples the image. The Decoder module upsamples the features extracted by the Encoder to the size of the corresponding layer, and uses the skip-attention module to fuse the Encoder and Decoder feature maps at different downsampling multiples to obtain a feature map of patch size for generating lymph node candidate regions. In the inference stage, the entire image is used as the input and inference is performed in the form of patches, and then fused and restored to the original image size.

[0036] See Figure 2 , the MiniUNet network consists of an Encoder-Decoder. On the left side of the dotted line is the Encoder module for feature extraction, which is composed of four groups of convolutions, namely E1-E4 in the figure, and performs three downsamplings. In the Decoder module, a structure symmetric to the Encoder part is adopted, namely D4-D1. The difference is that the features of the Encoder are upsampled three times and then restored to the input size.

[0037] The working process of the MiniUNet network is as follows:

[0038] The patch image is fed into the MiniUNet network. After being processed by E1, the feature EF-1 is obtained and continues to be input into E2 to obtain the feature EF-2, and so on, obtaining the features EF-3 and EF-4 respectively for backup. In the Decode module, first, the EF-4 is input into D4 to obtain the DF-4 feature, which is then upsampled to obtain the feature DF-3. The upsampled feature is connected with the feature of the Encoder of the same size through skip-attention. Taking the first upsampling as an example, the feature EF-3 is subjected to skip-attention operation and then fused with DF-3 for feature fusion, and so on and finally output.

[0039] See Figure 3 , the skip-attention network structure consists of three branches, which generate query, key, and value respectively. The working process of the skip-attention network is as follows:

[0040] The first step is to calculate the similarity between the query and each key to obtain weights;

[0041] The second step uses a softmax function to normalize these weights;

[0042] The third step is to perform weighted summation of the weights and the corresponding key-value values, and use residual connection to input features to obtain the final output features.

[0043] (3) Training of the one-stage network

[0044] The network is trained using a loss function based on pixel selection, selectively updating partial gradients to constrain the network to better learn for lymph nodes and suspected regions. The loss function is expressed by the following equations (a) - (b):

[0045] l p = -y∑log(y p ) (a);

[0046] L pixel = Top k ∑Sort(l p ) (b).

[0047] Among them, l p is the basic loss; y and y p are the GroundTruth and the predicted value respectively; L pixel is the loss based on pixel selection; Top k is to take the loss of the top K% of the pixel losses after sorting each pixel loss for gradient update, and the value range of K is 50% - 90%.

[0048] First, calculate the basic loss between the prediction result and the GroundTruth. In this embodiment, the cross-entropy loss function is adopted, and each pixel loss is retained to obtain a loss matrix consistent with the image size. Then, sort the pixel losses belonging to the foreground and background in this matrix respectively, and take the top K% of the pixels for gradient update respectively, where the value range of K is 50% - 90%.

[0049] (4) Two-stage data processing and model design

[0050] For the lymph node candidate regions obtained in the one-stage, extract the lymph node candidate ROI regions at three scales of large, medium, and small, and use rotation and contrast enhancement to perform data augmentation on the dataset. Obtain images at three scales.

[0051] Refer to Figure 4 , the MSENet network structure and working process are as follows:

[0052] The two-stage MSENet network model consists of a feature extractor and a classifier. The feature extractor selects the ResNet network, and the classifier consists of two fully connected layers. First, features are extracted from the image input at three scales. The features of different scales output by each feature extractor are unified by using SPPLayer to output feature sizes and then feature fusion is performed. The feature fusion method uses the weighted average fusion method. The outputs of the three scales and the fused features are classified into two categories, and the final classification result is obtained using the soft voting algorithm. The specific method of the soft voting algorithm is to weighted average the probabilities of the three scales and the fused output to obtain the predicted value. Those exceeding the preset threshold are recorded as true lymph nodes, which greatly enhances the accuracy of the algorithm.

[0053] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the inventive concept, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the attached claims.

Claims

1. A method for detecting abdominal lymph nodes based on a multi-scale fusion convolutional network of CT images, characterized in that This method uses MiniUNet and MSENet as a two-stage multi-scale fusion convolutional network for feature extraction to obtain the shape and location of abdominal lymph nodes. The specific detection includes the following steps: Step 1: Data preprocessing in the first stage For CT images of the ROI region with lymph nodes and the background region without lymph nodes, data augmentation and expansion are performed using rotation and random center cropping to obtain patch images with a scale of 128*128; Step 2: Network design in the first stage Design the MiniUNet network to extract image features from small-scale patch image inputs and use the skip-attention module to fuse the image features; Step 3: Feature extraction in the first stage Input the patch images into the constructed MiniUNet network, use a loss function based on pixel selection for network training, extract features and obtain lymph node candidate regions; Step 4: Data preprocessing in the second stage For the obtained lymph node candidate regions, extract lymph node candidate ROI regions at three scales of large, medium, and small as the dataset, and use rotation and contrast enhancement to perform data augmentation on the dataset to obtain ROI region images at the three scales; Step 5: Network design in the second stage Design the MSENet network with multi-scale fusion composed of three ResNet-based feature extractors; Step 6: Feature extraction in the second stage Input the ROI region images at the three scales into the constructed MSENet network, use the SPPLayer to unify the output feature size and perform feature fusion, perform binary classification from multiple scales, and output the classification results to obtain the shape and location of abdominal lymph nodes.

2. The abdominal lymph node detection method based on a multi-scale fusion convolutional network of CT images according to claim 1, wherein, The data preprocessing in the first stage specifically includes: 1-1: Use a sliding window to find the region where the lymph nodes are located, crop to obtain the ROI region at a scale of 128*128, and randomly select the background region of the same scale without lymph nodes to form the CT image dataset; 1-2: Combine the image processing methods of rotation and random center cropping to perform 20-fold data augmentation on the dataset.

3. The abdominal lymph node detection method based on a multi-scale fusion convolutional network for CT images according to claim 1, characterized in that, The MiniUNet network consists of an encoder and a decoder. The feature extraction of the encoder is based on Resblock to downsample the image; the decoder upsamples the features extracted by the encoder to restore the size of the corresponding layer, and uses the skip-attention module to fuse the feature maps extracted by the encoder and decoder at different downsampling multiples to obtain a feature map of the patch size for the generation of lymph node candidate regions.

4. The abdominal lymph node detection method based on a multi-scale fusion convolutional network of CT images according to claim 1, characterized in that, The network training in the first stage uses a loss function based on pixel selection to selectively update the gradient to constrain the network to better generate lymph node candidate regions, reduce omissions, and improve the lymph node detection ability of the network.

5. The abdominal lymph node detection method based on a multi-scale fusion convolutional network for CT images according to claim 1, wherein The feature extraction in the second stage uses three feature extractors to extract features from image inputs at three scales, and uses an SPP Layer to unify the output feature sizes and then perform feature fusion. Binary classification is performed on the outputs at the three scales and the fused features, and the soft voting algorithm is used to obtain the final classification result. Those exceeding the preset threshold are recorded as true lymph nodes.

6. The abdominal lymph node detection method based on a multi-scale fusion convolutional network of CT images according to claim 5, wherein The soft voting algorithm is to perform weighted averaging on the probabilities of the outputs at the three scales and the fused output to obtain the predicted value, and those exceeding the preset threshold are recorded as true lymph nodes.

Images