Pathological image classification method and system based on improved initial residual network

Abstract

The present invention proposes a pathological image classification method and system based on an improved initial residual network, which relates to the technical field of image classification. First, original pathological image samples are collected, and preprocessing operations are performed on the original pathological image samples to expand and enhance sample data. After the initial preparation of the data is completed, the initial network and the residual network are combined and improved to build an improved initial residual network model. The input module in the initial residual network is improved, and a parallel asymmetric input module is introduced as a substitute to increase the number of times the original image features are used, thereby realizing feature reuse. Then, the conventional training and testing stages of the model are entered, and finally the classification results are evaluated. This scheme can improve the classification performance of the model for pathological image classification and reduce network parameters and calculation amount.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and more specifically, to a pathological image classification method and system based on an improved initial residual network. Background Art

[0002] In the classification of pathological images, the existing pathological image classification methods generally classify pathological images according to their quality, that is, they are classified into malignant or benign, but there are few technologies for the specific classification of malignant or benign. In medicine, the classification of diseases should be specific. For example, in the pathological classification of breast cancer, the specific name of the tumor should be specifically classified. The purpose of doing so is to be able to provide better auxiliary diagnosis for doctors or medical experts.

[0003] The feature analysis and classification recognition of tissue pathology images are the focus of tissue pathology image analysis research. Thanks to the development of computer vision technology, many pathology image diagnosis technologies based on computer vision technology have emerged. Compared with manual pathology image classification, pathology image classification based on computer vision technology has made great progress, especially in feature extraction. However, the characteristics of each pathology are different, mainly concentrated on local features such as the size, shape, structure and edge of the cell nucleus. This requires personnel with professional knowledge to do this work, but this process is very long and costly for the training of medical professionals. It is unrealistic to arrange different medical personnel for feature extraction for each pathology.

[0004] Thanks to the rise of deep learning and the excellent adaptability of convolutional neural networks in feature extraction and feature expression, a number of methods for pathological image classification using initial neural networks have emerged. For example, a method for histopathological image classification based on convolutional neural networks is disclosed in the prior art. The main process includes: creating a data set, fine-tuning the protocol, pre-trained convolutional neural network (CNN) as a feature extractor and fine-tuned CNN as a classifier. First, images of different body parts with different texture patterns are manually selected from the scans, patch selection and accuracy calculation are performed, and then a specific single-layer fully connected layer is selected for network training. The pre-trained CNN is used as a feature extractor, and the linear support vector machine for classification is trained using deep features. Finally, the fine-tuned CNN is used as a classifier, and the obtained network is used to classify the test patch. The data set of the present invention is used to design and train a deep network, with richer details and more complete data, so that the accuracy of retrieval and classification is significantly improved, and the classification of histopathological images is effectively realized. However, this method has some shortcomings: first, a large amount of pathological image data is required to train deep learning models, but the existing pathological image data is very small, and there is not much effective information for sample training. In addition, in terms of the selected model, there is still the problem of insufficient image feature extraction. These shortcomings make the training of convolutional neural network models more time-consuming and costly, so further improvement is urgently needed. Summary of the invention

[0005] In order to solve the problem that the current method of pathological image classification using deep learning technology has high requirements on the amount of image data when training the model, the present invention proposes a pathological image classification method and system based on an improved initial residual network, which combines and improves the initial network and the residual network, has low requirements on the amount of original pathological image data, and can make full use of the characteristics of the original image data, reduce the amount of calculation, and improve the performance of the network for pathological image classification.

[0006] In order to achieve the above technical effects, the technical solution of the present invention is as follows:

[0007] The present invention proposes a pathological image classification method based on an improved initial residual network, comprising the following steps:

[0008] S1. Collecting original pathological image samples, performing preprocessing operations on the original pathological image samples, and randomly dividing the preprocessed pathological image samples into a training sample set and a test sample set;

[0009] S2. Building an improved initial residual network model, the improved initial residual network model includes a parallel asymmetric input module, a first initial module, a first reduction module, a second initial module, a second reduction module, a third initial module and an output module connected in sequence;

[0010] S3. Using the pathological image samples of the training sample set to train the improved initial residual network model, to obtain a trained improved initial residual network model;

[0011] S4. Input the pathological image samples of the test sample set into the trained improved initial residual network model for testing, and output the classification results;

[0012] S5. Evaluate the classification results.

[0013] In this technical solution, the original pathological image samples are first collected. Considering that the current pathological image data is small, the original pathological image samples are preprocessed to expand and enhance the sample data. After the initial data preparation is completed, the initial network and the residual network are combined and improved to build an improved initial residual network model. The input module in the initial residual network is improved, and a parallel asymmetric input module is introduced as a substitute to increase the number of times the original image features are used, realizing feature reuse, and then entering the regular training and testing phase of the model. Finally, the classification results are evaluated. This solution combines the improved model with the data, which can improve the classification performance of the model for pathological image classification and reduce network parameters and computational complexity.

[0014] Preferably, the preprocessing operations performed on the original pathological image samples in step S1 include: scaling, flipping, shearing and rotating, and performing data enhancement processing on the original pathological image samples to increase the number of samples.

[0015] Preferably, the parallel asymmetric input module described in step S2 includes: a 3*(3×3) convolution layer, a maximum pooling layer branch structure, an asymmetric convolution kernel structure and a 2*(3×3) convolution layer, the output end of the 3*(3×3) convolution layer is respectively connected to the input end of the first maximum pooling layer branch structure and the input end of the first asymmetric convolution kernel structure, the output end of the first maximum pooling layer branch structure and the output end of the first asymmetric convolution kernel structure are fused and connected to the 2*(3×3) convolution layer; the maximum pooling layer branch structure is composed of a maximum pooling layer, and the asymmetric convolution kernel structure includes a 1×1 convolution layer, a 7×1 convolution layer, and a 1×7 convolution layer connected in sequence; the pathological image samples in the training sample set or the pathological image samples in the test sample set are input into the parallel asymmetric input module, and feature reuse is realized in the parallel asymmetric input module based on the maximum pooling layer branch structure and the asymmetric convolution kernel structure. Here, the data processed by the parallel asymmetric input module is a feature fusion after being processed by two channels at the same time, and the information contained is more comprehensive, so that the subsequent data can be more close to the original data sample.

[0016] Preferably, the first initial module includes a first ReLU layer, a parallel four-channel layer, a first 1×1 convolutional layer, and a second ReLU layer, wherein the parallel four-channel layer includes a parallel downsampling layer, a second 1×1 convolutional layer, a (1×1+3×3) convolutional layer, and a [1×1+2*(3×3)] convolutional layer, and the output end of the parallel asymmetric input module is connected to the input end of the first ReLU layer, and the output end of the first ReLU layer is respectively connected to the downsampling layer, the second 1×1 convolutional layer, the (1×1+3×3) convolutional layer, and the [1×1+ 2*(3×3)] convolution layer input and the second ReLU layer input, the output of the parallel four-channel layer is connected to the input of the first 1×1 convolution layer, the output of the first 1×1 convolution layer is fused with the output of the first ReLU layer and then connected to the second ReLU layer; the pathological image samples output from the parallel asymmetric input module are extracted with shallow features by the first ReLU layer and simultaneously input to the parallel four-channel layer, and then convolutionally extracted with the first 1×1 convolution layer. The deep features are fused with the shallow features and then input to the second ReLU layer. Here, a method of downsampling layer downsampling processing is introduced to reduce the size of the feature matrix of the data processing process. The module simultaneously inputs the data output by the parallel asymmetric input module into the four channels including downsampling, and then adds the data processed by the four channels to the shallow data and inputs it into the first reduction module.

[0017] Preferably, the first reduction module includes: a downsampling layer, a (3×3) convolution layer, a [2*(3×3)+1×1] convolution layer and a maximum pooling layer, the downsampling layer, a (3×3) convolution layer, a [2*(3×3)+1×1] convolution layer and a maximum pooling layer are in parallel, the output end of the first initial module is respectively connected to the input end of the downsampling layer, the input end of the (3×3) convolution layer, the input end of the [2*(3×3)+1×1] convolution layer and the input end of the maximum pooling layer, the output end of the downsampling layer, the output end of the (3×3) convolution layer, the output end of the [2*(3×3)+1×1] convolution layer and the output end of the maximum pooling layer are fused and connected to the input end of the second initial module.

[0018] A downsampling layer is also introduced in the first reduction module to apply the downsampling technology. The data output by the first initial module enters the four channels including downsampling for calculation, and then the data processed by all channels are fused and directly input into the second initial module.

[0019] Preferably, the second initial module includes: a splicing layer, a ReLU layer, a parallel three-channel layer and a 1×1 convolution layer, wherein the parallel three-channel layer includes a parallel 1×1 convolution layer, a parallel (1×1+1×7+7×1) convolution layer and a first parallel downsampling layer, the output end of the first reduction module is connected to the input end of the splicing layer, the output end of the splicing layer is connected to the input end of the ReLU layer, the output end of the ReLU layer is connected to the input end of the parallel 1×1 convolution layer in the parallel three-channel layer, the input end of the parallel (1×1+1×7+7×1) convolution layer and the input end of the first parallel downsampling layer, the output end of the parallel 1×1 convolution layer, the output end of the parallel (1×1+1×7+7×1) convolution layer and the output end of the first parallel downsampling layer are connected to the input end of the 1×1 convolution layer, and the output end of the 1×1 convolution layer is fused with the output end of the ReLU layer and connected to the second reduction module;

[0020] The second reduction module includes: a ReLU layer and a parallel five-channel layer, the parallel five-channel layer includes a second parallel downsampling layer, a parallel maximum pooling layer, a parallel first (1×1+3×3) convolution layer, a parallel second (1×1+3×3) convolution layer and a [1×1+2*(3×3)] convolution layer, the output end of the second initial module is connected to the input end of the ReLU layer, the output end of the ReLU layer is respectively connected to the input end of the second parallel downsampling layer, the input end of the parallel maximum pooling layer, the input end of the parallel first (1×1+3×3) convolution layer, the input end of the parallel second (1×1+3×3) convolution layer and the input end of the [1×1+2*(3×3)] convolution layer, the output end of the second parallel downsampling layer, the output end of the parallel maximum pooling layer, the output end of the parallel first (1×1+3×3) convolution layer, the output end of the parallel second (1×1+3×3) convolution layer and the output end of the [1×1+2*(3×3)] convolution layer are fused and connected to the input end of the third initial module.

[0021] Here, downsampling layers are introduced in the second initial module and the second reduction module to apply downsampling processing technology. The data output from the first reduction module is input into the second initial module of block downsampling, and the three channels are added for calculation. Then, the processed data of each channel and the unprocessed shallow feature data are directly added to complete the fusion of multiple features. Finally, the processed data are input into the second reduction module together, including the five channels of downsampling for calculation, and then the processed data of each channel are added to complete the fusion of multiple features, and the fused data is passed to the third initial module.

[0022] Preferably, the third initial module includes: a splicing layer, a ReLU layer, a parallel three-channel layer, and a 1×1 convolution layer, the parallel three-channel layer includes a parallel (1×1+1×3+3×1) convolution layer, a parallel 1×1 convolution layer and a parallel downsampling layer, the output end of the second reduction module is connected to the input end of the splicing layer, the output end of the splicing layer is connected to the input end of the ReLU layer, the output end of the ReLU layer is respectively connected to the input end of the parallel (1×1+1×3+3×1) convolution layer, the input end of the parallel 1×1 convolution layer and the input end of the parallel downsampling layer, the output end of the parallel (1×1+1×3+3×1) convolution layer, the output end of the parallel 1×1 convolution layer and the output end of the parallel downsampling layer are fused and connected to the input end of the 1×1 convolution layer, and the output end of the 1×1 convolution layer is fused with the output end of the ReLU layer and connected to the output module; the output module includes: an output ReLU layer, an average pooling layer, a regularization layer and a softmax layer, and the output ReLU layer, the average pooling layer, the regularization layer and the softmax layer are connected in sequence.

[0023] Here, a splicing layer is also introduced in the third initial module to process data using downsampling technology. The third initial module inputs the data output from the second initial module into the three downsampling channels for calculation, and then fuses the calculated data to complete the fusion of multiple features. The fused data is then passed through a convolution layer to fix the dimension of the data matrix. Finally, the processed data is added to the shallow data and input into the output module.

[0024] Preferably, the process of training the improved initial residual network model using the pathological image samples in the training sample set in step S3 is:

[0025] S31. Setting the parameters of the improved initial residual network model, including the learning rate, the number of training rounds M, and setting the image input format and data batch size when the training sample set is pre-input into the improved initial residual network model;

[0026] S32. Assume that the loss function of the improved initial residual network model is L, input the training sample set into the improved initial residual network model, use L as the training guide, and use back propagation and gradient descent as the training method. Each time the improved initial residual network model is trained, the pathological image classification result predicted by the model is trained, and then the loss function L is updated;

[0027] S33. Modifying the parameters of the improved initial residual network model according to the result of the loss function L;

[0028] S34. Repeat S32 to S33 until the loss function no longer decreases in M ​​training rounds, and the training of the improved initial residual network model is completed, and the parameters of the improved initial residual network model are retained as the parameters of the trained improved initial residual network model.

[0029] Preferably, when evaluating the classification result in step S5, the evaluation criteria adopted include: accuracy, recall rate and F1 value.

[0030] The present invention also proposes a pathological image classification system based on an improved initial residual network, comprising:

[0031] An image sample acquisition and processing module is used to acquire original pathological image samples, perform preprocessing operations on the original pathological image samples, and randomly divide the preprocessed pathological image samples into a training sample set and a test sample set;

[0032] A model building module, used to build an improved initial residual network model, the improved initial residual network model includes a parallel asymmetric input module, a first initial module, a first reduction module, a second initial module, a second reduction module, a third initial module and an output module connected in sequence;

[0033] A training module, using the pathological image samples of the training sample set to train the improved initial residual network model, to obtain a trained improved initial residual network model;

[0034] The testing module inputs the pathological image samples of the test sample set into the trained improved initial residual network model for testing and outputs the classification results;

[0035] Evaluation module evaluates the classification results.

[0036] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0037] The present invention proposes a pathological image classification method and system based on an improved initial residual network. First, the original pathological image samples are collected. Considering that the current pathological image data is small, the original pathological image samples are preprocessed to expand and enhance the sample data. After the preliminary preparation of the data is completed, the initial network and the residual network are combined and improved to build an improved initial residual network model. The input module in the initial residual network is improved, and a parallel asymmetric input module is introduced as a substitute to increase the number of times the original image features are used, thereby realizing feature reuse. Then the regular training and testing stages of the model are entered, and finally the classification results are evaluated. This scheme can improve the classification performance of the model for pathological image classification and reduce network parameters and computational complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram showing a method flow of a pathological image classification method based on an improved initial residual network proposed in Embodiment 1 of the present invention;

[0039] Figure 2A diagram showing the experimental process of pathological image classification based on an improved initial residual network proposed in Example 1 of the present invention;

[0040] Figure 3 A model structure diagram showing the improved initial residual network proposed in Embodiment 2 of the present invention;

[0041] Figure 4 A schematic diagram showing the structure of a pathological image classification system based on an improved initial residual network proposed in Example 3 of the present invention. DETAILED DESCRIPTION

[0042] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0043] In order to better illustrate the present embodiment, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the actual size;

[0044] It is understandable to those skilled in the art that descriptions of certain well-known contents in the drawings may be omitted.

[0045] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0046] Example 1

[0047] Considering that the computer vision technology currently used for pathological image classification involves the necessary training process of the experimental model, a large amount of original pathological image data is required. However, in the field of pathology, it is very difficult to obtain a large amount of original pathological image data. Therefore, this embodiment focuses on data enhancement of a small amount of original pathological image data to expand the amount of data, and improves the input module of the initial residual network to fully utilize the characteristics of the image data.

[0048] Specifically, a pathological image classification method based on an improved initial residual network is proposed. The flowchart of this method is shown in Figure 1 , including the following steps:

[0049] S1. Collecting original pathological image samples, performing preprocessing operations on the original pathological image samples, and randomly dividing the preprocessed pathological image samples into a training sample set and a test sample set;

[0050] S2. Building an improved initial residual network model, the improved initial residual network model includes a parallel asymmetric input module, a first initial module, a first reduction module, a second initial module, a second reduction module, a third initial module and an output module connected in sequence;

[0051] S3. Using the pathological image samples of the training sample set to train the improved initial residual network model, to obtain a trained improved initial residual network model;

[0052] S4. Input the pathological image samples of the test sample set into the trained improved initial residual network model for testing, and output the classification results;

[0053] S5. Evaluate the classification results.

[0054] On the whole, the original pathology image samples are first collected. Considering that the current pathology image data is small, the original pathology image samples are preprocessed to expand and enhance the sample data. After the initial data preparation is completed, the initial network and the residual network are combined and improved to build an improved initial residual network model. The input module in the initial residual network is improved, and a parallel asymmetric input module is introduced as a substitute to increase the number of times the original image features are used, realizing feature reuse, and then entering the regular training and testing phase of the model. Finally, the classification results are evaluated. This scheme combines the improved model with the data, which can improve the classification performance of the model for pathology image classification and reduce network parameters and computational complexity.

[0055] In this embodiment, the preprocessing operations performed on the original pathological image samples include: scaling, flipping, shearing and rotating. The original pathological image samples are subjected to data enhancement processing to increase the number of samples. After the pathological images are subjected to data enhancement, the processed image data are used to train the improved initial residual network. The preprocessed pathological image samples are randomly divided into training sample sets and test sample sets. 70% of the data samples are randomly selected as the training sample sets and sent to the improved initial residual network for training. During each training process, the output result of the output module is the probability that the input pathological image belongs to a certain category. After the training is completed, the remaining 30% of the data samples are tested as the test sample set. The overall experimental process is shown in the figure. Figure 2 As shown, during the preprocessing of the training set samples, the original training set pathology image samples are scaled to a fixed size of 299x299, and then the fixed-size images are enhanced, that is, the images are flipped, sheared, rotated, etc., to increase the number of data samples, and then the improved initial residual network is trained;

[0056] The process of training the improved initial residual network model using the pathological image samples of the training sample set is:

[0057] S31. Setting the parameters of the improved initial residual network model, including the learning rate, the number of training rounds M, and setting the image input format and data batch size when the training sample set is pre-input into the improved initial residual network model;

[0058] S32. Assume that the loss function of the improved initial residual network model is L, input the training sample set into the improved initial residual network model, use L as the training guide, and use back propagation and gradient descent as the training method. Each time the improved initial residual network model is trained, the pathological image classification result predicted by the model is trained, and then the loss function L is updated;

[0059] S33. Modifying the parameters of the improved initial residual network model according to the result of the loss function L;

[0060] S34. Repeat S32 to S33 until the loss function no longer decreases in M ​​training rounds, and the training of the improved initial residual network model is completed, and the parameters of the improved initial residual network model are retained as the parameters of the trained improved initial residual network model.

[0061] Finally, when evaluating the classification results as described in S5, the evaluation criteria used include: precision, recall, and F1 value;

[0062] Among them, the accuracy rate refers to the proportion of true positive classes among all the results predicted as positive classes, and the formula is:

[0063]

[0064] The recall rate is also called the recall rate, which refers to the proportion of all positive classes found by the classifier. The formula is:

[0065]

[0066] The F1 value is the harmonic mean of precision and recall, and the formula is:

[0067]

[0068] Among them, TP (True Positive) represents the number of true positives, that is, the number of samples classified as positive and actually positive; FP (False Positive) represents the number of false positives, that is, the number of samples classified as positive but actually negative; FN (False Negative) represents the number of false negatives, that is, the number of samples classified as negative but actually positive; TN (True Negative) represents the number of true negatives, that is, the number of samples classified as negative but actually negative.

[0069] Example 2

[0070] The model structure of the improved initial residual network proposed in this embodiment is a model improved on the initial residual network. It is based on one of the hybrid deep neural network architectures of the initial network and the residual network. Generally, the safest way to improve the performance of the network model is to increase the depth and width of the network, but the deeper and wider the network often means a huge amount of parameters. When the amount of data is small, the trained network is easy to overfit, and when the network is very deep, the gradient disappears again. The initial network solves this problem very well. As mentioned above, very deep networks will have a gradient explosion during training, but the emergence of residual networks helps to solve this problem. While training very deep networks, it can also ensure good performance of the model. The initial residual network combines the advantages of these two models, and the improved model can accelerate training during the training process, improve the performance of the model, and reduce the number of network parameters and reduce the amount of calculation.

[0071] In this embodiment, the model structure diagram of the improved initial residual network is shown in FIG. 3. Figure 3 The parallel asymmetric input module 1 includes: a 3*(3×3) convolution layer, a maximum pooling layer branch structure, an asymmetric convolution kernel structure and a 2*(3×3) convolution layer. The output end of the 3*(3×3) convolution layer is respectively connected to the input end of the first maximum pooling layer branch structure and the input end of the first asymmetric convolution kernel structure. The output end of the first maximum pooling layer branch structure and the output end of the first asymmetric convolution kernel structure are fused and connected to the 2*(3×3) convolution layer; the maximum pooling layer branch structure is composed of a maximum pooling layer, and the asymmetric convolution kernel structure includes a 1×1 convolution layer, a 7×1 convolution layer, and a 1×7 convolution layer connected in sequence; the pathological image samples in the training sample set or the pathological image samples in the test sample set are input into the parallel asymmetric input module 1, and feature reuse is realized in the parallel asymmetric input module 1 based on the maximum pooling layer branch structure and the asymmetric convolution kernel structure. Here, the data processed by the parallel asymmetric input module 1 is a feature fusion after being processed by two channels at the same time, and the information contained is more comprehensive, so that the data obtained subsequently is more close to the original data sample.

[0072] The first initial module 2 includes a first ReLU layer, a parallel four-channel layer, a first 1×1 convolutional layer, and a second ReLU layer. The parallel four-channel layer includes a parallel downsampling layer, a second 1×1 convolutional layer, a (1×1+3×3) convolutional layer, and a [1×1+2*(3×3)] convolutional layer. The output end of the parallel asymmetric input module 1 is connected to the input end of the first ReLU layer, and the output end of the first ReLU layer is respectively connected to the downsampling layer, the second 1×1 convolutional layer, the (1×1+3×3) convolutional layer, and the [1×1+2*(3×3)] convolutional layer. The input end of the parallel asymmetric input module 1 is connected to the input end of the first 1×1 convolution layer and the input end of the second ReLU layer. The output end of the parallel four-channel layer is connected to the input end of the first 1×1 convolution layer. The output end of the first 1×1 convolution layer is fused with the output end of the first ReLU layer and then connected to the second ReLU layer. The pathological image sample output from the parallel asymmetric input module 1 is extracted with shallow features by the first ReLU layer and simultaneously input to the parallel four-channel layer, and then the deep features are extracted by convolution of the first 1×1 convolution layer. The deep features are fused with the shallow features and then input to the second ReLU layer. Here, a method of downsampling layer downsampling processing is introduced to reduce the size of the feature matrix of the data processing process. The module simultaneously inputs the data output by the parallel asymmetric input module 1 into the four channels including downsampling, and then adds the data processed by the four channels and the shallow data and inputs them into the first reduction module 3.

[0073] See also Figure 3 The first reduction module 3 includes: a downsampling layer, a (3×3) convolution layer, a [2*(3×3)+1×1] convolution layer and a maximum pooling layer. The downsampling layer, a (3×3) convolution layer, a [2*(3×3)+1×1] convolution layer and a maximum pooling layer are in parallel. The output end of the first initial module 2 is respectively connected to the input end of the downsampling layer, the input end of the (3×3) convolution layer, the input end of the [2*(3×3)+1×1] convolution layer and the input end of the maximum pooling layer. The output end of the downsampling layer, the output end of the (3×3) convolution layer, the output end of the [2*(3×3)+1×1] convolution layer and the output end of the maximum pooling layer are fused and connected to the input end of the second initial module 3.

[0074] A downsampling layer is also introduced in the first reduction module 3 to apply the downsampling technology. The data output by the first initial module 2 enters the four channels including downsampling for calculation, and then the data processed by all channels are fused and directly input into the second initial module 4.

[0075] The second initial module 4 includes: a splicing layer, a ReLU layer, a parallel three-channel layer and a 1×1 convolution layer, the parallel three-channel layer includes a parallel 1×1 convolution layer, a parallel (1×1+1×7+7×1) convolution layer and a first parallel downsampling layer, the output end of the first reduction module 3 is connected to the input end of the splicing layer, the output end of the splicing layer is connected to the input end of the ReLU layer, the output end of the ReLU layer is connected to the input end of the parallel 1×1 convolution layer in the parallel three-channel layer, the input end of the parallel (1×1+1×7+7×1) convolution layer and the input end of the first parallel downsampling layer, the output end of the parallel 1×1 convolution layer, the output end of the parallel (1×1+1×7+7×1) convolution layer and the output end of the first parallel downsampling layer are connected to the input end of the 1×1 convolution layer, and the output end of the 1×1 convolution layer is fused with the output end of the ReLU layer and connected to the second reduction module 5;

[0076] The second reduction module 5 includes: a ReLU layer and a parallel five-channel layer, wherein the parallel five-channel layer includes a second parallel downsampling layer, a parallel maximum pooling layer, a parallel first (1×1+3×3) convolution layer, a parallel second (1×1+3×3) convolution layer and a [1×1+2*(3×3)] convolution layer, and the output end of the second initial module 4 is connected to the input end of the ReLU layer, and the output end of the ReLU layer is respectively connected to the input end of the second parallel downsampling layer, the input end of the parallel maximum pooling layer, and the parallel The input end of the first (1×1+3×3) convolutional layer, the input end of the parallel second (1×1+3×3) convolutional layer and the input end of the [1×1+2*(3×3)] convolutional layer, the output end of the second parallel downsampling layer, the output end of the parallel maximum pooling layer, the output end of the parallel first (1×1+3×3) convolutional layer, the output end of the parallel second (1×1+3×3) convolutional layer and the output end of the [1×1+2*(3×3)] convolutional layer are fused and connected to the input end of the third initial module 6.

[0077] Downsampling layers are introduced into the second initial module 4 and the second reduction module 5 to apply downsampling processing technology. The data output from the first reduction module 3 is input into the second initial module 4 of block downsampling and three channels are added for calculation. Then, the processed data of each channel and the unprocessed shallow feature data are directly added to complete the fusion of multiple features. Finally, the processed data are input into the second reduction module 5 together including the five channels of downsampling for calculation. Then, the processed data of each channel are added to complete the fusion of multiple features, and the fused data is passed to the third initial module 6.

[0078] The third initial module 6 includes: a splicing layer, a ReLU layer, a parallel three-channel layer, and a 1×1 convolution layer. The parallel three-channel layer includes a parallel (1×1+1×3+3×1) convolution layer, a parallel 1×1 convolution layer and a parallel downsampling layer. The output end of the second reduction module 5 is connected to the input end of the splicing layer, the output end of the splicing layer is connected to the input end of the ReLU layer, the output end of the ReLU layer is respectively connected to the input end of the parallel (1×1+1×3+3×1) convolution layer, the input end of the parallel 1×1 convolution layer and the input end of the parallel downsampling layer, the output end of the parallel (1×1+1×3+3×1) convolution layer, the output end of the parallel 1×1 convolution layer and the output end of the parallel downsampling layer are fused and connected to the input end of the 1×1 convolution layer, and the output end of the 1×1 convolution layer is fused with the output end of the ReLU layer and connected to the output module; the output module 7 includes: an output ReLU layer, an average pooling layer, a regularization layer and a softmax layer, and the output ReLU layer, the average pooling layer, the regularization layer and the softmax layer are connected in sequence.

[0079] Here, a splicing layer is also introduced in the third initial module 6 to process data using downsampling technology. The third initial module 6 inputs the data output from the second initial module 4 into the three channels of downsampling for calculation, and then fuses the calculated data to complete the fusion of multiple features. The fused data is then passed through a convolution layer to fix the dimension of the data matrix. Finally, the processed data is added to the shallow data and input into the output module.

[0080] Example 3

[0081] See also Figure 4 The present invention also proposes a pathological image classification system based on an improved initial residual network, comprising:

[0082] An image sample acquisition and processing module is used to acquire original pathological image samples, perform preprocessing operations on the original pathological image samples, and randomly divide the preprocessed pathological image samples into a training sample set and a test sample set;

[0083] A model building module, used to build an improved initial residual network model, the improved initial residual network model includes a parallel asymmetric input module, a first initial module, a first reduction module, a second initial module, a second reduction module, a third initial module and an output module connected in sequence;

[0084] A training module, using the pathological image samples of the training sample set to train the improved initial residual network model, to obtain a trained improved initial residual network model;

[0085] The testing module inputs the pathological image samples of the test sample set into the trained improved initial residual network model for testing and outputs the classification results;

[0086] Evaluation module evaluates the classification results.

[0087] Obviously, the above embodiments of the present invention are only examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A pathological image classification method based on an improved initial residual network, characterized in that: The following steps are involved: S1. Collecting original pathological image samples, performing preprocessing operations on the original pathological image samples, and randomly dividing the preprocessed pathological image samples into a training sample set and a test sample set; S2. Building an improved initial residual network model, the improved initial residual network model includes a parallel asymmetric input module, a first initial module, a first reduction module, a second initial module, a second reduction module, a third initial module and an output module connected in sequence; The first initial module includes a first ReLU layer, a parallel four-channel layer, a first 1×1 convolutional layer, and a second ReLU layer. The parallel four-channel layer includes a parallel downsampling layer, a second 1×1 convolutional layer, a (1×1+3×3) convolutional layer, and a [1×1+2*(3×3)] convolutional layer. The output end of the parallel asymmetric input module is connected to the input end of the first ReLU layer, and the output end of the first ReLU layer is respectively connected to the downsampling layer, the second 1×1 convolutional layer, the (1×1+3×3) convolutional layer, and the [1×1+2*(3×3)] convolutional layer. The input end of the (3×3)] convolution layer and the input end of the second ReLU layer, the output end of the parallel four-channel layer is connected to the input end of the first 1×1 convolution layer, the output end of the first 1×1 convolution layer is fused with the output end of the first ReLU layer and then connected to the second ReLU layer; the pathological image samples output from the parallel asymmetric input module are extracted with shallow features by the first ReLU layer and simultaneously input to the parallel four-channel layer, and then convoluted by the first 1×1 convolution layer to extract deep features, and the deep features are fused with the shallow features and then input to the second ReLU layer; The first reduction module includes: a downsampling layer, a (3×3) convolution layer, a [2*(3×3)+1×1] convolution layer and a maximum pooling layer. The downsampling layer, a (3×3) convolution layer, a [2*(3×3)+1×1] convolution layer and a maximum pooling layer are in parallel. The output end of the first initial module is connected to the input end of the downsampling layer, the input end of the (3×3) convolution layer, the input end of the [2*(3×3)+1×1] convolution layer and the input end of the maximum pooling layer respectively. The output end of the downsampling layer, the output end of the (3×3) convolution layer, the output end of the [2*(3×3)+1×1] convolution layer and the output end of the maximum pooling layer are fused and connected to the input end of the second initial module. The second initial module includes: a splicing layer, a ReLU layer, a parallel three-channel layer and a 1×1 convolution layer, wherein the parallel three-channel layer includes a parallel 1×1 convolution layer, a parallel (1×1+1×7+7×1) convolution layer and a first parallel downsampling layer, the output end of the first reduction module is connected to the input end of the splicing layer, the output end of the splicing layer is connected to the input end of the ReLU layer, the output end of the ReLU layer is connected to the input end of the parallel 1×1 convolution layer in the parallel three-channel layer, the input end of the parallel (1×1+1×7+7×1) convolution layer and the input end of the first parallel downsampling layer, the output end of the parallel 1×1 convolution layer, the output end of the parallel (1×1+1×7+7×1) convolution layer and the output end of the first parallel downsampling layer are connected to the input end of the 1×1 convolution layer, and the output end of the 1×1 convolution layer is fused with the output end of the ReLU layer and connected to the second reduction module; The second reduction module includes: a ReLU layer and a parallel five-channel layer, wherein the parallel five-channel layer includes a second parallel downsampling layer, a parallel maximum pooling layer, a parallel first (1×1+3×3) convolution layer, a parallel second (1×1+3×3) convolution layer and a [1×1+2*(3×3)] convolution layer, the output end of the second initial module is connected to the input end of the ReLU layer, the output end of the ReLU layer is respectively connected to the input end of the second parallel downsampling layer, the input end of the parallel maximum pooling layer, the input end of the parallel first (1×1+3×3) convolution layer, the input end of the parallel second (1×1+3×3) convolution layer and the input end of the [1×1+2*(3×3)] convolution layer, the output end of the second parallel downsampling layer, the output end of the parallel maximum pooling layer, the output end of the parallel first (1×1+3×3) convolution layer, the output end of the parallel second (1×1+3×3) convolution layer and the output end of the [1×1+2*(3×3)] convolution layer are fused and connected to the input end of the third initial module; The third initial module includes: a splicing layer, a ReLU layer, a parallel three-channel layer, and a 1×1 convolution layer. The parallel three-channel layer includes a parallel (1×1+1×3+3×1) convolution layer, a parallel 1×1 convolution layer, and a parallel downsampling layer. The output end of the second reduction module is connected to the input end of the splicing layer, the output end of the splicing layer is connected to the input end of the ReLU layer, the output end of the ReLU layer is respectively connected to the input end of the parallel (1×1+1×3+3×1) convolution layer, the input end of the parallel 1×1 convolution layer, and the input end of the parallel downsampling layer. The output end of the parallel (1×1+1×3+3×1) convolution layer, the output end of the parallel 1×1 convolution layer, and the output end of the parallel downsampling layer are fused and connected to the input end of the 1×1 convolution layer. The output end of the 1×1 convolution layer is fused with the output end of the ReLU layer and connected to the output module; the output module includes: an output ReLU layer, an average pooling layer, a regularization layer, and a softmax layer. The output ReLU layer, the average pooling layer, the regularization layer, and the softmax layer are connected in sequence. S3. Using the pathological image samples of the training sample set to train the improved initial residual network model, to obtain a trained improved initial residual network model; S4. Input the pathological image samples of the test sample set into the trained improved initial residual network model for testing, and output the classification results; S5. Evaluate the classification results.

2. The pathological image classification method based on the improved initial residual network according to claim 1, characterized in that: The preprocessing operations performed on the original pathological image sample in step S1 include: scaling, flipping, shearing and rotating.

3. The pathological image classification method based on the improved initial residual network according to claim 1, characterized in that: The parallel asymmetric input module described in step S2 includes: a 3*(3×3) convolution layer, a maximum pooling layer branch structure, an asymmetric convolution kernel structure and a 2*(3×3) convolution layer. The output end of the 3*(3×3) convolution layer is respectively connected to the input end of the first maximum pooling layer branch structure and the input end of the first asymmetric convolution kernel structure. The output end of the first maximum pooling layer branch structure and the output end of the first asymmetric convolution kernel structure are fused and connected to the 2*(3×3) convolution layer; the maximum pooling layer branch structure is composed of a maximum pooling layer, and the asymmetric convolution kernel structure includes a 1×1 convolution layer, a 7×1 convolution layer, and a 1×7 convolution layer connected in sequence; the pathological image samples in the training sample set or the pathological image samples in the test sample set are input into the parallel asymmetric input module, and feature reuse is realized in the parallel asymmetric input module based on the maximum pooling layer branch structure and the asymmetric convolution kernel structure.

4. The pathological image classification method based on the improved initial residual network according to claim 1, characterized in that: The process of training the improved initial residual network model using the pathological image samples of the training sample set described in step S3 is: S31. Setting the parameters of the improved initial residual network model, including the learning rate, the number of training rounds M, and setting the image input format and data batch size when the training sample set is pre-input into the improved initial residual network model; S32. Assume that the loss function of the improved initial residual network model is L, input the training sample set into the improved initial residual network model, use L as the training guide, and use back propagation and gradient descent as the training method. Each time the improved initial residual network model is trained, the pathological image classification result predicted by the model is trained, and then the loss function L is updated; S33. Modifying the parameters of the improved initial residual network model according to the result of the loss function L; S34. Repeat S32 to S33 until the loss function no longer decreases in M ​​training rounds, and the training of the improved initial residual network model is completed, and the parameters of the improved initial residual network model are retained as the parameters of the trained improved initial residual network model.

5. The pathological image classification method based on the improved initial residual network according to claim 4, characterized in that: When evaluating the classification result in step S5, the evaluation criteria used include: accuracy, recall rate and F1 value.

6. A pathological image classification system based on an improved initial residual network, characterized in that: include: An image sample acquisition and processing module is used to acquire original pathological image samples, perform preprocessing operations on the original pathological image samples, and randomly divide the preprocessed pathological image samples into a training sample set and a test sample set; A model building module, used to build an improved initial residual network model, the improved initial residual network model includes a parallel asymmetric input module, a first initial module, a first reduction module, a second initial module, a second reduction module, a third initial module and an output module connected in sequence; The first initial module includes a first ReLU layer, a parallel four-channel layer, a first 1×1 convolutional layer, and a second ReLU layer. The parallel four-channel layer includes a parallel downsampling layer, a second 1×1 convolutional layer, a (1×1+3×3) convolutional layer, and a [1×1+2*(3×3)] convolutional layer. The output end of the parallel asymmetric input module is connected to the input end of the first ReLU layer, and the output end of the first ReLU layer is respectively connected to the downsampling layer, the second 1×1 convolutional layer, the (1×1+3×3) convolutional layer, and the [1×1+2*(3×3)] convolutional layer. The input end of the (3×3)] convolution layer and the input end of the second ReLU layer, the output end of the parallel four-channel layer is connected to the input end of the first 1×1 convolution layer, the output end of the first 1×1 convolution layer is fused with the output end of the first ReLU layer and then connected to the second ReLU layer; the pathological image samples output from the parallel asymmetric input module are extracted with shallow features by the first ReLU layer and simultaneously input to the parallel four-channel layer, and then convoluted by the first 1×1 convolution layer to extract deep features, and the deep features are fused with the shallow features and then input to the second ReLU layer; The first reduction module includes: a downsampling layer, a (3×3) convolution layer, a [2*(3×3)+1×1] convolution layer and a maximum pooling layer. The downsampling layer, a (3×3) convolution layer, a [2*(3×3)+1×1] convolution layer and a maximum pooling layer are in parallel. The output end of the first initial module is connected to the input end of the downsampling layer, the input end of the (3×3) convolution layer, the input end of the [2*(3×3)+1×1] convolution layer and the input end of the maximum pooling layer respectively. The output end of the downsampling layer, the output end of the (3×3) convolution layer, the output end of the [2*(3×3)+1×1] convolution layer and the output end of the maximum pooling layer are fused and connected to the input end of the second initial module. The second initial module includes: a splicing layer, a ReLU layer, a parallel three-channel layer and a 1×1 convolution layer, wherein the parallel three-channel layer includes a parallel 1×1 convolution layer, a parallel (1×1+1×7+7×1) convolution layer and a first parallel downsampling layer, the output end of the first reduction module is connected to the input end of the splicing layer, the output end of the splicing layer is connected to the input end of the ReLU layer, the output end of the ReLU layer is connected to the input end of the parallel 1×1 convolution layer in the parallel three-channel layer, the input end of the parallel (1×1+1×7+7×1) convolution layer and the input end of the first parallel downsampling layer, the output end of the parallel 1×1 convolution layer, the output end of the parallel (1×1+1×7+7×1) convolution layer and the output end of the first parallel downsampling layer are connected to the input end of the 1×1 convolution layer, and the output end of the 1×1 convolution layer is fused with the output end of the ReLU layer and connected to the second reduction module; The second reduction module includes: a ReLU layer and a parallel five-channel layer, wherein the parallel five-channel layer includes a second parallel downsampling layer, a parallel maximum pooling layer, a parallel first (1×1+3×3) convolution layer, a parallel second (1×1+3×3) convolution layer and a [1×1+2*(3×3)] convolution layer, the output end of the second initial module is connected to the input end of the ReLU layer, the output end of the ReLU layer is respectively connected to the input end of the second parallel downsampling layer, the input end of the parallel maximum pooling layer, the input end of the parallel first (1×1+3×3) convolution layer, the input end of the parallel second (1×1+3×3) convolution layer and the input end of the [1×1+2*(3×3)] convolution layer, the output end of the second parallel downsampling layer, the output end of the parallel maximum pooling layer, the output end of the parallel first (1×1+3×3) convolution layer, the output end of the parallel second (1×1+3×3) convolution layer and the output end of the [1×1+2*(3×3)] convolution layer are fused and connected to the input end of the third initial module; The third initial module includes: a splicing layer, a ReLU layer, a parallel three-channel layer, and a 1×1 convolution layer. The parallel three-channel layer includes a parallel (1×1+1×3+3×1) convolution layer, a parallel 1×1 convolution layer, and a parallel downsampling layer. The output end of the second reduction module is connected to the input end of the splicing layer, the output end of the splicing layer is connected to the input end of the ReLU layer, the output end of the ReLU layer is respectively connected to the input end of the parallel (1×1+1×3+3×1) convolution layer, the input end of the parallel 1×1 convolution layer, and the input end of the parallel downsampling layer. The output end of the parallel (1×1+1×3+3×1) convolution layer, the output end of the parallel 1×1 convolution layer, and the output end of the parallel downsampling layer are fused and connected to the input end of the 1×1 convolution layer. The output end of the 1×1 convolution layer is fused with the output end of the ReLU layer and connected to the output module; the output module includes: an output ReLU layer, an average pooling layer, a regularization layer, and a softmax layer. The output ReLU layer, the average pooling layer, the regularization layer, and the softmax layer are connected in sequence. A training module, using the pathological image samples of the training sample set to train the improved initial residual network model, to obtain a trained improved initial residual network model; The testing module inputs the pathological image samples of the test sample set into the trained improved initial residual network model for testing and outputs the classification results; Evaluation module evaluates the classification results.

Images