Urinary stone detection and classification method based on improved Faster RCNN algorithm

Abstract

The present invention provides a urinary stone detection and classification method based on an improved Faster RCNN algorithm. The method comprises the following steps: ① inputting CT images of patients with urinary stones; ② extracting features through a ResNet network; ③ fusing the features extracted from different stages of the ResNet network; ④ using a convolutional attention mechanism to weight the fused features to form a new feature map; ⑤ the new feature map generates a candidate frame through an RPN network; ⑥ the new feature map and the candidate frame obtain a fixed-size feature map through an ROI network; ⑦ classifying and locating stones through a prediction module. The present invention integrates the shallow features and deep features of the image, reduces the missed diagnosis rate of stones, improves the detection performance of the urinary stone detection and classification model based on the Faster RCNN algorithm, realizes the prediction of stone composition before surgery, assists doctors in selecting appropriate treatment plans, reduces the pain and burden of patients, and has clinical application value.

Description

Technical Field

[0001] The invention relates to the technical field of medical image processing and disease diagnosis, and in particular to a urinary stone detection and classification method based on an improved FasterRCNN algorithm. Background Art

[0002] Urinary tract stones are one of the most common urological diseases, with a high incidence rate that is increasing year by year. According to whether their chemical composition contains inorganic salts, they can be divided into: calcium oxalate stones, calcium phosphate stones, ammonium magnesium phosphate stones, cystine stones, and uric acid stones.

[0003] Different types of stones require different treatments. Uric acid and cystine stones require alkalinization of urine first; calcium stones are hard and poorly treated with extracorporeal shock wave lithotripsy (ESWL), so percutaneous nephrolithotomy (PCNL) is often used; magnesium ammonium phosphate stones require anti-infection treatment, and ESWL is more effective.

[0004] Therefore, analyzing the composition of stones before treatment can not only guide the selection of treatment options, but also provide a basis for preventing stone recurrence, which has important clinical value. Traditional methods for analyzing stone composition include chemical qualitative or quantitative analysis, spectral analysis, X-ray diffraction analysis, etc. These methods are complicated, time-consuming and expensive. With the rise of artificial intelligence technology, a method for classifying urinary stones through machine learning has emerged, but this method is limited to in vitro. So far, there is no reliable detection method in clinical practice that can accurately determine the composition of stones in the body before a specific treatment plan is implemented. Summary of the invention

[0005] The purpose of the present invention is to provide a urinary stone detection and classification method based on the improved Faster RCNN algorithm. The method can detect and classify stones according to the CT images of patients with urinary stones before surgery, and can accurately measure the chemical composition of stones, so as to assist doctors in finding the cause of the disease and selecting appropriate treatment plans based on the cause of the disease.

[0006] The present invention is implemented as follows: a urinary stone detection and classification method based on an improved Faster RCNN algorithm comprises the following steps:

[0007] S1. Input CT images of patients with urinary stones;

[0008] S2, feature extraction through ResNet network;

[0009] S3, multi-scale feature fusion of the features extracted from the first, second, third and fourth stages of the ResNet network;

[0010] S4, using the convolutional attention mechanism to weight the fused features obtained in step S3 to form a new feature map;

[0011] S5, passing the new feature map through the RPN network to generate a candidate box;

[0012] S6, passing the new feature map and the candidate box through the ROI network to obtain a feature map of a preset size;

[0013] S7. Classify and locate stones through the prediction module.

[0014] Preferably, in step S2, the ResNet network is ResNet101. The present invention uses ResNet101 to extract stone features, thereby extracting deeper features of the stone CT image and increasing the accuracy of judgment.

[0015] Preferably, in step S3, the output of the fourth stage is used as the basic feature map, and the outputs of the first stage, the second stage, and the third stage are transformed into a feature map of the same size as the basic feature map after 1×1 convolution and downsampling, and then the feature maps of the four stages are channel-superimposed to achieve multi-scale feature fusion.

[0016] Preferably, step S3 fuses the shallow features and deep features of the stone. The fusion method is different from that of the FPN network. Instead, the feature maps extracted from each stage of resnet101 are unified into the same size, and then the cancat operation is used to perform splicing and fusion in the channel dimension. Specifically,

[0017] ① The output of the fourth stage is used as the basic feature map, whose size is 1024*32*32;

[0018] ②The output size of the first stage is 64*128*128, which is downsampled twice to 64*32*32 through a 1×1 convolutional layer;

[0019] ③The output size of the second stage is 256*128*128, which is downsampled twice to 256*32*32 through a 1×1 convolutional layer;

[0020] ④The output size of the third stage is 512*64*64, which is downsampled to 512*32*32 through a 1×1 convolutional layer;

[0021] ⑤ The output of the fourth stage is channel-superimposed with the outputs of the first, second and third stages after transformation to fuse multi-scale features. The size of the fused feature map is 1856*32*32;

[0022] ⑥ Make the feature map fused in step ⑤ pass through a 1×1 convolution layer for channel compression, and finally the size of the fused feature map is 1024*32*32.

[0023] Preferably, the convolutional attention mechanism in step S4 includes a channel attention mechanism and a spatial attention mechanism; the features fused in step S3 are not directly input into the RPN network, but weighted by the CBAM attention mechanism, so that important features in the feature map have higher weights.

[0024] Preferably, step S5 is specifically as follows: first, the new feature map generated in step S4 is used as the input of the RPN network, and then passed through a sliding window, i.e., a 3*3 convolution layer, 9 rectangular windows are set on the original image corresponding to each pixel point in the sliding window to obtain an anchor frame, and the anchor frame is input into two 1*1 convolution layers for classification and regression, and finally a candidate frame is obtained through post-processing NMS.

[0025] Preferably, in step S6, the ROI network is Roi Align. Step S6 specifically includes: mapping the candidate box output by the RPN network to the new feature map formed in step S4 to obtain the region of interest, and then pooling the region of interest into a feature map of preset size and length for output.

[0026] Preferably, in step S7, the prediction module includes two branch networks, one of which is a classification network, which performs classification prediction and outputs the category of the stone; the other branch network is a boundary regression network, which performs regression prediction and outputs the location of the stone. Step S7 is specifically: the prediction module first passes the feature map output by the ROI network through the fifth stage of ResNet101, then performs a full connection operation, and then uses the Softmax function for classification, and completes the bounding box regression operation for positioning.

[0027] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0028] 1. The present invention can accurately detect and classify stones on CT images of patients with urinary stones before surgery without removing the stones, and predict their chemical composition, which helps assist doctors in finding the cause of the disease and selecting appropriate treatment plans based on the cause, as well as preventing recurrence and alleviating the pain of secondary surgery for patients.

[0029] 2. The present invention improves the Faster RCNN algorithm. The feature extraction network of Faster RCNN adopts ResNet101. This feature extraction network adopts a skip connection method, which solves the gradient vanishing and explosion problems faced by deep networks, realizes the extraction of deeper features of the image, and improves the network classification accuracy.

[0030] 3. The Faster RCNN of the present invention combines the shallow and deep features of the image, and uses the convolutional attention mechanism to weight the fused features to form a new feature map as the input of the RPN network, which solves the problem of insufficient position information of deep features and inaccurate positioning, thereby improving the network positioning accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of the network framework structure of the method of the present invention.

[0032] Figure 2 It is a schematic diagram of the framework structure of the RseNet101 network in the present invention.

[0033] Figure 3 It is a schematic diagram of the framework structure of the CBAM attention mechanism in the present invention.

[0034] Figure 4 It is the original CT image of a patient with urinary stones in an embodiment of the present invention.

[0035] Figure 5 The method of the present invention is Figure 4 The result of detection and classification of the original CT images of patients with urinary stones. DETAILED DESCRIPTION

[0036] The classic Faster RCNN algorithm uses VGG16 as the feature extraction network, and only uses the output of the last layer as the feature map for detection and classification. In the present invention, the Faster RCNN algorithm is improved, and ResNet101 is used as the feature extraction network to extract deeper features, and shallow features and deep features are fused at the same time, and the attention mechanism is used to assign weights, thereby improving the detection performance.

[0037] The network framework of the present invention is as follows Figure 1 As shown, it involves ResNet network, feature fusion module, CBAM attention mechanism module, RPN network, ROI network and prediction module, where the functions of each module are as follows:

[0038] The ResNet network uses a skip connection network to solve the gradient vanishing and explosion problems faced by deep networks and achieve the extraction of deeper features of the image.

[0039] The ResNet network in the present invention adopts ResNet101, and its structure is as follows Figure 2As shown. ResNet101 can be divided into five stages, namely the first stage (Stage0), the second stage (Stage1), the third stage (Stage2), the fourth stage (Stage3), and the fifth stage (Stage4). The first stage is mainly composed of a 7*7 convolution layer and a maximum pooling layer, and the other four stages are mainly composed of a stack of residual blocks, and the residual block is composed of a 1*1 convolution layer, a 3*3 convolution layer and a jump connection. In the method of the present invention, the features extracted from the first stage, the second stage, the third stage and the fourth stage are fused, and the fifth stage is used in the prediction module to further extract features from the output of the ROI network.

[0040] The feature fusion module first performs 1*1 convolution and downsampling on the feature maps output by the first, second and third stages respectively, transforms the size of the feature maps to the same size as the feature maps of the fourth stage, and superimposes and fuses them in the channel dimension. It then compresses the number of channels through a 1*1 convolution and outputs them as the feature maps after preliminary fusion.

[0041] like Figure 3 As shown in Figure 1, the CBAM attention mechanism module includes a channel attention mechanism module and a spatial attention mechanism module. The CBAM attention mechanism module takes the feature map output by the feature fusion module as input, and performs weight distribution on the feature map after preliminary fusion to form a new feature map after final fusion, thereby perfectly integrating the features extracted in the first four stages.

[0042] The RPN network is used to generate candidate frames. First, the final fused new feature map output by the CBAM attention mechanism module is used as the input of the RPN network, and then through the sliding window (which is essentially a 3*3 convolution layer), 9 rectangular windows are set on the original image corresponding to each pixel in the sliding window to obtain the anchor frame, and the anchor frame is input into two 1*1 convolution layers for classification and regression to determine whether the anchor frame contains stones, and finally the candidate frame is obtained through post-processing NMS.

[0043] The ROI network uses the RoiAlign method, which mainly maps the candidate box output by the RPN network to the feature map to obtain the region of interest, and then pools the region of interest into a feature map output with a fixed size and length.

[0044] The prediction module first passes the feature map output by the ROI network through the fifth stage of ResNet101, then performs a full connection operation, and then uses the Softmax function for classification, and completes the bounding box regression operation for accurate positioning.

[0045] The structure of the entire network of the present invention is as follows Figure 1 As shown, it mainly includes the following steps:

[0046] Step 1: Input the CT image of a patient with urinary stones.

[0047] Step 2: Perform feature extraction through the ResNet network; specifically, the ResNet network is ResNet101.

[0048] Step 3: Perform multi-scale feature fusion on the features extracted from different stages of the ResNet network.

[0049] Combination Figure 1 , the output of the fourth stage (Stage 3) of the ResNet network is used as the basic feature map, whose size is 1024*32*32, and the output size of the first, second and third stages is changed to 32*32 pixels.

[0050] The output size of the first stage (Stage0) is 64*128*128, so the output of the first stage passes through a 1×1 convolution layer with a step size of 2 and is downsampled twice to 64*32*32.

[0051] The output size of the second stage (Stage 1) is 256*128*128, so that the output of the second stage passes through a 1×1 convolution layer with a step size of 2 and is downsampled twice to 256*32*32.

[0052] The output size of the third stage (Stage 2) is 512*64*64, so the output of the third stage passes through a 1×1 convolution layer with a step size of 2 and is downsampled to 512*32*32.

[0053] The output of the fourth stage (Stage 3) and the output of the first, second, and third stages after downsampling are channel-superimposed (i.e., concat operation) to fuse multi-scale features. The size of the fused feature map is 1856*32*32.

[0054] Finally, channel compression is performed through a 1×1 convolutional layer, so that the size of the feature map after channel compression becomes 1024*32*32.

[0055] Step 4: Use the convolutional attention mechanism to assign weights to the fused features obtained in the previous step to form a new feature map.

[0056] The convolutional attention mechanism (CBAM) in this step includes the channel attention mechanism and the spatial attention mechanism.

[0057] Step 5: Pass the new feature map through the RPN network to generate candidate boxes.

[0058] The process of using the RPN network to generate candidate frames in this step is as follows: first, the new feature map generated in step 4 is used as the input of the RPN network, and then it passes through the sliding window, that is, the 3*3 convolution layer, and 9 rectangular windows are set on the original image corresponding to each pixel point in the sliding window to obtain the anchor frame. The anchor frame is input into two 1*1 convolution layers for classification and regression, and finally the candidate frame is obtained after post-processing NMS.

[0059] Step 6: Pass the new feature map and the candidate box through the ROI network to obtain a feature map of fixed size. Preferably, the ROI network is RoiAlign.

[0060] Step 7: Classify and locate stones through the prediction module.

[0061] The prediction module in this step includes two branch networks, one of which is a classification network that performs classification prediction and outputs the specific category of the stone; the other branch network is a boundary regression network that performs regression prediction and outputs the precise location of the stone.

[0062] In this step, the feature map output by the ROI network is first passed through the fifth stage of ResNet101, and then fully connected, and then classified using the Softmax function, and the bounding box regression operation is completed for accurate positioning.

[0063] The following experiments and evaluation analyses were performed on the urinary stone detection and classification method based on the improved Faster RCNN algorithm in the present invention.

[0064] First, a data set was established to collect CT images of patients with urinary stones and the corresponding test reports of each patient. Professional urologists annotated the stone locations on the CT images and classified and annotated them according to the test reports. This experiment mainly included two types of stones, calcium oxalate stones and uric acid stones. The annotated data were then randomly divided into training set, validation set, and test set in a ratio of 8:1:1. The parameters required for network training were then set, such as training batch size, number of epochs, initialization learning rate, optimization method used, etc. The training set and validation set were then used to train the stone detection classification model, and the test set was used to test the performance of the model.

[0065] The experiment uses recall rate (Recall), average precision (AP), and average precision mean (MAP) to measure the experimental results:

[0066]

[0067]

[0068]

[0069]

[0070] N: number of categories;

[0071] M: the number of positive examples in all samples;

[0072] P: Maximum Precision under different Recall;

[0073] True negative (TN): It is actually the number of samples predicted as negative samples;

[0074] False positive (FP): It is actually the number of negative samples predicted as positive samples;

[0075] False Negative (FN): It is actually the number of samples predicted as negative samples from positive samples;

[0076] True positive (TP): It is actually the number of samples predicted as positive samples;

[0077] Table 1

[0078]

[0079] It can be seen from the experimental results that the method of the present invention performs well in distinguishing calcium oxalate stones from uric acid stones and can meet the requirements for distinguishing calcium oxalate stones from uric acid stones.

[0080] like Figure 4 and Figure 5 As shown, Figure 4 This is the original CT image of a patient with urinary stones. Figure 5 This is the result of detecting and classifying the original CT image of a patient with urinary calculi using the method of the present invention. As can be seen from the figure, the present invention accurately detects the location of the calculus and the composition of the calculus (calcium-containing calculus).

[0081] The present invention is not limited to the above-described embodiments. The above description of the specific embodiments is intended to describe and illustrate the technical solution of the present invention. The above-mentioned specific embodiments are merely illustrative and not restrictive. Without departing from the scope of the present invention and the scope of protection of the claims, a person of ordinary skill in the art can also make many forms of specific changes under the guidance of the present invention, which all fall within the scope of protection of the present invention.

Claims

1. A urinary stone detection and classification method based on an improved Faster RCNN algorithm, characterized in that: The steps include: S1. Input CT images of patients with urinary stones; S2, feature extraction through ResNet network; S3, multi-scale feature fusion of the features extracted from the first, second, third and fourth stages of the ResNet network; S4, using the convolutional attention mechanism to weight the fused features obtained in step S3 to form a new feature map; S5, passing the new feature map through the RPN network to generate a candidate box; S6, passing the new feature map and the candidate box through the ROI network to obtain a feature map of a preset size; S7, stone classification and location through prediction module; In step S2, the ResNet network is ResNet101; ResNet101 is divided into five stages, namely the first stage, the second stage, the third stage, the fourth stage, and the fifth stage; the first stage includes a 7×7 convolution layer and a maximum pooling layer, and the other four stages include stacked residual blocks, which are composed of a 1×1 convolution layer, a 3×3 convolution layer, and a skip connection; In step S3, the output of the fourth stage is used as the basic feature map, and the outputs of the first stage, the second stage, and the third stage are transformed into a feature map of the same size as the basic feature map after 1×1 convolution and downsampling, and then the feature maps of the four stages are channel-superimposed to achieve multi-scale feature fusion; Step S3 specifically includes: ① The output of the fourth stage is used as the basic feature map, whose size is 1024 32 32; ②The output size of the first stage is 64 128 128, so that it is downsampled twice through a 1×1 convolutional layer to 64 32 32; ③The output size of the second stage is 256 128 128, so that it is downsampled twice to 256 through a 1×1 convolution layer 32 32; ④The output size of the third stage is 512 64 64, so that it is downsampled to 512 through a 1×1 convolution layer 32 32; ⑤ The output of the fourth stage is channel-superimposed with the output of the first, second, and third stages after transformation to fuse multi-scale features. The size of the fused feature map is 1856 32 32; ⑥ Make the fused feature map in step ⑤ pass through a 1×1 convolution layer for channel compression, and finally obtain a fused feature map size of 1024 32 32.

2. The urinary stone detection and classification method based on the improved Faster RCNN algorithm according to claim 1 is characterized in that: The convolutional attention mechanism in step S4 includes a channel attention mechanism and a spatial attention mechanism.

3. The urinary stone detection and classification method based on the improved Faster RCNN algorithm according to claim 1 is characterized in that: Step S5 is as follows: first, the new feature map generated in step S4 is used as the input of the RPN network, and then the new feature map is passed through the sliding window, i.e., 3 3 convolutional layer, set 9 rectangular windows on the original image corresponding to each pixel point in the sliding window to obtain anchor frames, and input the anchor frames into two 1 Classification and regression are performed in the convolutional layer of 1, and finally the candidate boxes are obtained through post-processing NMS.

4. The urinary stone detection and classification method based on the improved Faster RCNN algorithm according to claim 1 is characterized in that: In step S6, the ROI network is Roi Align.

5. The urinary stone detection and classification method based on the improved Faster RCNN algorithm according to claim 4 is characterized in that: Step S6 specifically includes: mapping the candidate box output by the RPN network to the new feature map formed in step S4 to obtain the region of interest, and then pooling the region of interest into a feature map output of a preset size and length.

6. The urinary stone detection and classification method based on the improved Faster RCNN algorithm according to claim 1 is characterized in that: In step S7, the prediction module includes two branch networks, one of which is a classification network that performs classification prediction and outputs the category of the stone; the other branch network is a boundary regression network that performs regression prediction and outputs the location of the stone.

7. The urinary stone detection and classification method based on the improved Faster RCNN algorithm according to claim 6 is characterized in that: Step S7 is specifically as follows: the prediction module first passes the feature map output by the ROI network through the fifth stage of ResNet101, then performs a full connection operation, and then uses the Softmax function for classification, and completes the bounding box regression operation for positioning.

Images