Pathological section image classification method and system integrating multiple multi-scale feature extraction

Abstract

The present invention relates to a pathology slice image classification method and system integrating multiple multi-scale feature extraction. The method comprises the following steps: in a training process, firstly performing image preprocessing on the pathology slice images for training to obtain a training data set, and then using the same training data set to train three pathology slice image classification networks Net1, Net2, and Net3 based on multi-scale feature extraction to obtain three corresponding trained models model_1, model_2, and model_3; in a testing process, performing image preprocessing on the pathology slice images for testing to obtain a testing data set, and then using the three trained models to predict the testing data set to obtain respective classification values score1, score2, and score3, and then voting on these three values in combination with weights to obtain a final classification value score f The method and system are beneficial to improving the accuracy of image classification.

Description

Technical Field

[0001] The present invention belongs to the field of image recognition technology, and in particular relates to a pathological slice image classification method and system integrating multiple multi-scale feature extractions. Background Art

[0002] Most existing studies on gastric pathology image classification and recognition have low accuracy rates, and the lack of publicly available datasets makes comparison difficult. Most accuracy rates range from 85% to 90%. Because pathology image features are complex and differences between cancer cells and normal cells are minimal, network models must extract sufficient image features. Analysis of existing studies revealed that most studies failed to effectively extract both deep and superficial features when extracting gastric pathology images. Kather et al. used HE slides as input to a ResNet18 network to classify microsatellite instability (MSI) and microsatellite stable (MSS) molecules in gastric cancer cells. This classification enabled them to identify MSI molecules and determine whether immunotherapy was effective for these patients. Because the network used only the ResNet18 network and did not extract sufficient image features, the final AUC was only 84%. Qu et al. constructed an automatic gastric tumor detection model using the VGG16 network, but achieved an accuracy of only 88.1%. This was due to their simple use of the VGG network for classification, without focusing on extracting features at different levels. Consequently, the network did not learn sufficient image features, resulting in suboptimal accuracy. Summary of the Invention

[0003] The purpose of the present invention is to provide a pathology slice image classification method and system that integrates multiple multi-scale feature extractions, which are conducive to improving the accuracy of image classification.

[0004] To achieve the above object, the technical solution adopted by the present invention is: a pathological section image classification method integrating multiple multi-scale feature extractions, comprising the following steps:

[0005] During the training process, the pathological slice images used for training are first preprocessed to obtain a training dataset. Then, the same training dataset is used to train three pathological slice image classification networks Net1, Net2, and Net3 based on multi-scale feature extraction to obtain the corresponding three trained models model_1, model_2, and model_3.

[0006] During the test, the pathological slice images used for the test are also preprocessed to obtain the test data set testdataset. Then, the three trained models model_1, model_2, and model_3 are used to predict the test data set to obtain their respective classification values score1, score2, and score3. Then, the three values are voted on with the weights to obtain the final classification value score. f .

[0007] Furthermore, the classification value score f The calculation formula is shown in formula (1):

[0008]

[0009] Among them, a m is the weight, score m The classification values predicted by the three trained models for the test data set, m = 1, 2, 3; weight a m The calculation formula is shown in formula (2):

[0010]

[0011] Among them, e m is the error rate, R is the number of sample categories; the pathology section image classification network obtains the classification accuracy of the network on the training data set during the training process, and the corresponding error rate e is obtained through the classification accuracy m , the smaller the error rate, the greater the weight.

[0012] Furthermore, the three pathological section image classification networks Net1, Net2, and Net3 based on multi-scale feature extraction are FPNINet, ResNetv2p, and ASPPoolNet.

[0013] Furthermore, in the FPNINet network, the input image is extracted by the FPN module to obtain four feature results of different scales. The feature maps obtained by the four feature results are from shallow to deep and of different sizes. The shallow feature map is reduced in size and image features are extracted by a NIN network layer, and then connected and fused with the feature map of the previous layer of the same size. The operation of extracting and fusing the feature map of the previous layer is repeated until all feature maps are fused; at this time, the output result of the network is a collection of all feature images; the above data is processed by the dropout layer and the Highway Network, and the dropout layer is used to randomly remove part of the information value, and the feature information is extracted by the Highway Network, and then input into the classifier, and finally classified by the classifier.

[0014] Furthermore, the ResNetv2p network is based on the ResNetv2 structure and inserts the CBAM attention mechanism into the network structure to enhance the weights of important channel information and spatial information; the ResNetv2p network inserts the PyConv module into ResNetv2 to replace a convolutional layer on the residual branch. The PyConv module inputs image features into four convolution kernels of different sizes respectively, and then fuses the image features of different depths output by the four convolution kernels, and outputs the results through the ResNetv2 network.

[0015] Furthermore, the ASPPoolNet network includes four ResNet network layers L1, L2, L3, and L4 using void convolution and two CBAM attention mechanism modules. The input image first passes through an initialization convolution layer, an L1 network layer, and a CBAM attention mechanism module, and then is divided into two paths. One path extracts the global features of the current feature map through the global average pooling and upsampling layer, and the other path passes through the L2, L3, and L4 network layers and the ASPP module. The ASPP module fuses the features of different levels extracted by void convolutions of different sizes into a feature map through a pooling layer, and then fuses them with the results calculated by 1×1 convolution. After processing through the NIN network layer, the CBAM attention mechanism module, and the pooling layer, it is fused with the global features extracted by the other path, and then the final result is output through the classifier.

[0016] The present invention also provides a pathological section image classification system that integrates multiple multi-scale feature extractions, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the above-mentioned method steps can be implemented.

[0017] Compared with the existing technology, the present invention has the following beneficial effects: the method and system use three multi-scale feature extraction networks to extract the deep and shallow features of gastric pathology slice images as much as possible, and then fuse the three networks through a voting network, thereby avoiding prediction errors to the greatest extent and greatly improving the accuracy of image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic diagram of the overall framework of an embodiment of the present invention.

[0019] Figure 2 Schematic diagram of the voting network structure in an embodiment of the present invention.

[0020] Figure 3 Schematic diagram of the structure of the FPNINet network in an embodiment of the present invention.

[0021] Figure 4Schematic diagram of the structure of the ResNetv2p network in an embodiment of the present invention.

[0022] Figure 5 Schematic diagram of the structure of the PyConv module in an embodiment of the present invention.

[0023] Figure 6 Schematic diagram of the structure of the ASPPoolNet network in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0025] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present application belongs.

[0026] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0027] like Figure 1 As shown, this embodiment provides a pathological slice image classification method integrating multiple multi-scale feature extractions, including the following steps:

[0028] During the training process, the pathological slice images used for training are first preprocessed to obtain a training dataset. Then, the same training dataset is used to train three pathological slice image classification networks Net1, Net2, and Net3 based on multi-scale feature extraction to obtain the corresponding three trained models model_1, model_2, and model_3.

[0029] During the test, the pathological slice images used for the test are also preprocessed to obtain the test data set testdataset. Then, the three trained models model_1, model_2, and model_3 are used to predict the test data set to obtain their respective classification values score1, score2, and score3. Then, the three values are voted on with the weights to obtain the final classification value score. f .

[0030] The voting network structure of this method is as follows Figure 2As shown, the three trained network models model_1, model_2, and model_3 predict the input image respectively and obtain their respective prediction scores. m (m=1,2,3), the weight a m and predicted value score m Calculate and get the final prediction result score f , score f The calculation formula is shown in formula (1), which is to multiply the prediction result and the weight and then add them together:

[0031]

[0032] Among them, the weight a m The calculation formula is shown in formula (2):

[0033]

[0034] Among them, e m is the error rate, R is the number of sample categories, and in this embodiment, it is 2. During the training process, the classification accuracy of the pathology slice image classification network on the training data set can be obtained, and the corresponding error rate e can be obtained through the classification accuracy. m The smaller the error rate, the greater the weight. The three network classifiers are in parallel, and the prediction processes are independent of each other and do not affect each other. In addition, the mechanism of the three classifiers predicting and voting separately can minimize prediction errors.

[0035] In this embodiment, the three pathological slice image classification networks Net1, Net2, and Net3 based on multi-scale feature extraction are FPNINet, ResNetv2p, and ASPPoolNet.

[0036] The FPNINet network structure is as follows Figure 3As shown. The input image is passed through the FPN module to extract four feature results of different scales. The feature maps obtained by the four feature results are from shallow to deep and of different sizes. The shallow feature map is reduced in size and image features are extracted through a NIN network layer, and then connected and fused with the feature map of the previous layer of the same size. The operation of extracting and fusing the feature map of the previous layer is repeated until all feature maps are fused. At this point, the output result of the network is a collection of all feature images. The amount of data is large and complex. The above data is processed by the dropout layer and the Highway Network. The dropout layer is used to randomly remove some information values, and the simple structure of the Highway Network is used instead of the deep network to extract feature information. Compared with the deep network, the Highway Network training process is simple, which avoids the problems of gradient disappearance and gradient explosion to a certain extent. Then it is input into the classifier, and finally classified by the classifier.

[0037] The ResNetv2p network structure is as follows Figure 4 shown. Figure 4 The structure of (a) is the main structure of the ResNetv2p network. This structure is based on the ResNetv2 structure and inserts the CBAM attention mechanism into the network structure to enhance the weight of important channel information and spatial information. Figure 4 As shown in (b), the main feature of the ResNetv2p network is that the PyConv module is inserted into ResNetv2 to replace a convolutional layer on the residual branch. The PyConv module structure is as follows Figure 5 As shown in the figure, this module inputs image features into four convolution kernels of different sizes and then fuses the image features at different depths output by these four kernels. In summary, this network uses a convolutional pyramid in the residual branch to extract image features at different depths, directly fuses them, and outputs the results through the ResNetv2 network. Furthermore, the inclusion of an attention mechanism within the overall network ensures that spatial and channel information is not lost in subsequent convolution calculations.

[0038] The ASPPoolNet network structure is as follows Figure 6As shown in the figure, the network includes four ResNet network layers L1, L2, L3, and L4 that use dilated convolutions, and two CBAM attention mechanism modules to enhance the weight of important channel and spatial information. After the image is input into the network, it first passes through an initialization convolution layer, an L1 network layer, and a CBAM attention mechanism module. It is then divided into two paths. One path extracts the global features of the current feature map through global average pooling and an upsampling layer, and the other path passes through the L2, L3, and L4 network layers and the ASPP module. The ASPP module uses a pooling layer to fuse the features of different levels extracted by dilated convolutions of different sizes into a feature map. The features are then fused with the results of 1×1 convolution calculations. After processing through the NIN network layer, the CBAM attention mechanism module, and the pooling layer, they are fused with the global features extracted from the other path. The final result is then output through the classifier.

[0039] This embodiment also provides a pathological slice image classification system that integrates multiple multi-scale feature extractions, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the above-mentioned method steps can be implemented.

[0040] This embodiment conducts comparative experiments on complete pathological slice images using various network models and the method of the present invention. The comparative experimental results are shown in Tables 1 and 2 below.

[0041] Table 1 Comparison of various indicators of various networks

[0042]

[0043] As shown in Table 1, the method of the present invention achieves the best classification results. By using multiple networks to predict results in parallel and combining different weights for calculation, it can produce better prediction results than a single network alone. ResNet18's accuracy in classifying gastric cancer images is second only to ASPPoolNet and the method of the present invention, but its accuracy in classifying normal images is lower, resulting in poor overall network performance. Inserting the PyConv module into ResNetv2 allows for more image features to be captured during convolution. When an image is input into the FPNINet network, the FPN module extracts features at different levels. While this network's overall accuracy is inferior to that of ResNetv2, it performs better than ResNetv2 in classifying forward images. Unlike other networks, the ASPPoolNet network uses dilated convolutions to extract different features and then pool them into a single feature map, integrating global features. Consequently, this network outperforms other networks in all metrics, with the highest regression value of all networks, indicating that it performs best in classifying sections containing cancer cells. The F1 score demonstrates that the stability of this network is comparable to that of the method of the present invention.

[0044] Table 2 Test results using complete pathological sections

[0045]

[0046] As shown in Table 2, the method proposed in this paper achieved the best test results, while Inceptionv3 achieved the lowest. ResNetv2p and ASPPoolNet performed slightly better than the other networks. Since this test program does not utilize a segmentation network, it first divides the entire image into small blocks. The resulting sum of the blocks is then divided by the number of blocks. If the calculated sum is greater than 0.4, the sample is classified as gastric cancer; if it is less than 0.4, the sample is classified as normal. This value is set to 0.4 to ensure the accuracy of the final result in the event that the network misclassifies small images.

[0047] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0048] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0049] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0050] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0051] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other manner. Any person skilled in the art may utilize the above-disclosed technical content to modify or modify the present invention into equivalent embodiments. However, any simple modifications, equivalent variations, and modifications to the above embodiments that do not depart from the technical content of the present invention and are based on the technical essence of the present invention remain within the scope of protection of the present invention.

Claims

1. A pathological section image classification method integrating multiple multi-scale feature extractions, characterized in that: The following steps are involved: During the training process, the pathological slice images used for training are first preprocessed to obtain a training dataset traindataset. Then, the same training dataset is used to train three pathological slice image classification networks Net1, Net2, and Net3 based on multi-scale feature extraction to obtain the corresponding three trained models model_1, model_2, and model_3. During the test, the pathological slice images used for the test are also preprocessed to obtain the test data set testdataset. Then, the three trained models model_1, model_2, and model_3 are used to predict the test data set to obtain their respective classification values score1, score2, and score3. Then, the three values are voted on with the weights to obtain the final classification value score. f ; The three pathological section image classification networks Net1, Net2, and Net3 based on multi-scale feature extraction are FPNINet, ResNetv2p, and ASPPoolNet; In the FPNINet network, the input image is passed through the FPN module to extract four feature results of different scales. The feature maps obtained by the four feature results are from shallow to deep and of different sizes. The shallow feature map is reduced in size and image features are extracted through a NIN network layer, and then connected and fused with the feature map of the previous layer of the same size. The operation of extracting and fusing the feature map of the previous layer is repeated until all feature maps are fused; at this point, the output result of the network is a collection of all feature images; the above data is processed through the dropout layer and the Highway Network, and the dropout layer is used to randomly remove some information values. The feature information is extracted through the Highway Network, and then input into the classifier, and finally classified by the classifier; The ResNetv2p network is based on the ResNetv2 structure and inserts the CBAM attention mechanism into the network structure to increase the weight of important channel information and spatial information. The ResNetv2p network inserts the PyConv module into ResNetv2 to replace a convolution layer on the residual branch. The PyConv module inputs image features into four convolution kernels of different sizes, then fuses the image features of different depths output by the four convolution kernels, and outputs the result through the ResNetv2 network. The ASPPoolNet network includes four ResNet network layers L1, L2, L3, and L4 using void convolution and two CBAM attention mechanism modules. The input image first passes through an initialization convolution layer, an L1 network layer, and a CBAM attention mechanism module, and then is divided into two paths. One path extracts the global features of the current feature map through the global average pooling and upsampling layer, and the other path passes through the L2, L3, and L4 network layers and the ASPP module. The ASPP module fuses the features of different levels extracted by void convolution of different sizes into a feature map through a pooling layer, and then fuses them with the results after 1×1 convolution calculation. After processing through the NIN network layer, the CBAM attention mechanism module, and the pooling layer, it is fused with the global features extracted by the other path, and then outputs the final result through the classifier.

2. The pathological section image classification method integrating multiple multi-scale feature extractions according to claim 1 is characterized in that: The classification value score f The calculation formula is shown in formula (1): Among them, a m is the weight, score m The classification values predicted by the three trained models for the test data set, m = 1, 2, 3; weight a m The calculation formula is shown in formula (2): Among them, e m is the error rate, R is the number of sample categories; the pathology section image classification network obtains the classification accuracy of the network on the training data set during the training process, and the corresponding error rate e is obtained through the classification accuracy m , the smaller the error rate, the greater the weight.

3. A pathology slice image classification system integrating multiple multi-scale feature extractions, characterized in that: The method comprises a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method according to any one of claims 1 to 2 can be implemented.

Images