A pathological image classification device and method based on deep feature fusion and a use method of the device

Abstract

The application discloses a pathological image classification device based on deep feature fusion, which comprises an image data set acquisition unit for acquiring small block image data sets; a specific number of integrated units for receiving small block images in the small block image data sets, making the small block images pass through the operation of a convolution kernel and a weighted channel, dimension deepening cascade, repeatedly acquiring further weighted and deepened image convolution features, and extracting deep image convolution features and shallow image convolution features according to set network training parameters; a feature fusion unit for performing operation on the acquired deep image convolution features and shallow image convolution features, and then performing cascade on the vectorized deep image convolution features to obtain image deep fusion features; and a classifier for obtaining the classification label of the image according to the image deep fusion features acquired by the feature fusion module and combining preset classification standards. The application further discloses a pathological image classification method based on deep feature fusion and a use method of the pathological image classification device based on deep feature fusion. The application can realize more accurate pathological image classification.

Description

Technical Field

[0001] The present invention relates to the field of pathological image classification, and more specifically, to a pathological image classification technology based on deep feature fusion. Background Art

[0002] In recent years, with the development of deep learning, pathological image classification methods based on convolutional neural networks have made certain progress and have been applied to some computer-aided diagnosis systems. However, due to the limitations of collecting and annotating pathological image data, small data volume, noisy data and other problems, the accuracy and reliability of the model are seriously affected.

[0003] Therefore, experts in the field have conducted a series of explorations and research on pathology image classification methods based on convolutional neural networks. In 2016, H. Kallen used convolutional features extracted by convolutional neural networks to train random forest and support vector machine (SVM) classifiers to classify prostate cancer pathology images, reducing the workload of doctors. In 2018, K. Nagpal used the Inception network to classify prostate cancer pathology images, further improving the model's accuracy. In 2018, E. Arvaniti trained the lightweight MobileNet network using small image patches, achieving even higher classification accuracy. In 2019, J. Wang proposed a weakly supervised method for grading tissue microarrays using graph convolutional networks. This method models the spatial organization of cells as a graph to better capture tumor cell proliferation and community structure. The convolutional neural networks used in these methods only map the pathology images layer by layer, aiming to improve classification accuracy. However, the convolutional features are not fused with the pathology images, which limits the model's classification performance and accuracy.

[0004] Therefore, a new technology that fuses deep features with pathological images to obtain more discriminative feature representations and thus improve the classification accuracy of the model is urgently needed. Summary of the Invention

[0005] The purpose of this invention is to provide a pathological image classification technology based on deep feature fusion, so as to construct a deep convolutional neural network and extract convolution features under different fields of view to obtain a technical means of high classification accuracy model.

[0006] In order to achieve the above objectives, the present invention discloses a pathological image classification device based on deep feature fusion, the device comprising:

[0007] An image data set acquisition unit is used to sample characteristic areas in digital slices of prostate pathological image morphology to acquire a small image data set;

[0008] a specific number of integrated units, configured to receive small image blocks from the small image block dataset acquired by the image dataset acquisition unit, subject the small image blocks to operations on convolution kernels and weighted channels, and cascade dimension deepening according to set network training parameters, repeatedly acquire further weighted and deepened image convolution features, and extract deep image convolution features and shallow image convolution features;

[0009] A feature fusion unit, configured to perform operations on the deep image convolution features and the shallow image convolution features obtained by the integration unit and then concatenate them with the vectorized deep image convolution features to obtain image deep fusion features;

[0010] The classifier is used to obtain the classification label of the image based on the image depth fusion features obtained by the feature fusion module and the preset classification criteria.

[0011] The integrated unit further comprises:

[0012] A specific number of attention mechanism modules are used to make the small image block pass the operation of convolution kernel and weighted channel to obtain repeatedly weighted image convolution features;

[0013] A downsampling module is used to extract features from different fields of view on the image convolution features that have been repeatedly weighted by the specific number of attention mechanism modules, and to fuse the features in a cascade manner to obtain a deeper image convolution feature;

[0014] The specific number of integrated units are stacked in sequence, and the convolution features extracted by the previous integrated unit are extracted again. The convolution features extracted by the last integrated unit are deep image convolution features, and the convolution features extracted by the penultimate integrated unit are shallow image convolution features.

[0015] Preferred:

[0016] The feature fusion module converts the deep image convolution features and the shallow image convolution features into feature matrices of the same dimension and then performs an outer product operation. The deep image convolution features are converted into full convolution vectors through the convolution operation. The full convolution vector is cascaded with the result of the outer product operation to obtain the image deep fusion features.

[0017] In order to improve the accuracy of the device, it is preferred that:

[0018] The device also includes an error analysis unit and a parameter setting unit;

[0019] The error analysis unit is used to calculate the classification error according to the classification result of the classifier;

[0020] The parameter setting unit is used for updating the network training parameter according to the classification error calculated by the error analysis unit.

[0021] The small block image data set obtained by the image data set acquisition unit is a training set and a test set.

[0022] Specifically,

[0023] Different classification standards are preset for different classifiers, and the corresponding classifier is used to classify the image deep fusion features according to the judgment requirement.

[0024] The application further discloses a pathological image classification method based on deep feature fusion.

[0025] The feature region in the prostate pathological image morphological digital slice is sampled to obtain a small block image data set.

[0026] According to the network training parameter, the small block image in the small block image data set is repeatedly obtained through the operation of the convolution kernel and the weighted channel, the dimension deepening cascade, the further weighting and deepening image convolution features, the deep image convolution features and the shallow convolution features are extracted.

[0027] After the operation of the deep image convolution features and the shallow convolution image features, the vectorized deep image convolution features are cascaded to obtain the image deep fusion features.

[0028] According to the obtained image deep fusion features, the image classification label is obtained in combination with the preset classification standard.

[0029] Further, the method for repeatedly obtaining the further weighting and deepening image convolution features, extracting the deep image convolution features and the shallow convolution features according to the network training parameter is specifically:

[0030] The small block image is operated through the convolution kernel and the weighted channel to obtain the repeatedly weighted image convolution features.

[0031] The repeatedly weighted image convolution features are extracted in different fields of view, and the features are fused through the cascade mode to obtain the deepened image convolution features.

[0032] The image convolution features are extracted again, the last extracted convolution features are the deep image convolution features, and the second last extracted convolution features are the shallow image convolution features.

[0033] Preferably,

[0034] The deep image convolution features and the shallow image convolution features are converted into feature matrices of the same dimension and then an outer product operation is performed. The deep image convolution features are converted into full convolution vectors through convolution operation. The full convolution vectors are cascaded with the results of the outer product operation to obtain image depth fusion features.

[0035] In order to improve the accuracy of classification, preferably, the method further comprises:

[0036] Calculating a classification error based on the classification labels of the obtained images;

[0037] Update and set the network training parameters according to the error;

[0038] The small image datasets are a training set and a test set.

[0039] Specifically:

[0040] Different classification standards are preset, and the image depth fusion features are classified using corresponding classification standards according to the judgment requirements.

[0041] The present invention also discloses a method for using a pathological image classification device based on deep feature fusion, according to which the device can be trained to be a high-precision classification device, the method comprising:

[0042] The image data set acquisition unit samples characteristic areas in the digital slices of the prostate pathological image morphology to acquire a small image data set, and divides the small image data set into a training set and a test set according to a specific ratio;

[0043] A specific number of integrated units receive the small image blocks in the training set from the small image block dataset acquired by the image dataset acquisition unit, and subject the small image blocks to operations on convolution kernels and weighted channels and cascades of dimensional deepening according to set initialization network training parameters, repeatedly obtaining further weighted and deepened image convolution features, and extracting deep image convolution features and shallow image convolution features;

[0044] The feature fusion unit operates on the deep image convolution feature and the shallow image convolution feature obtained by the integration unit, and then cascades them with the vectorized deep image convolution feature to obtain an image deep fusion feature;

[0045] The classifier obtains the classification label of the image based on the image depth fusion features obtained by the feature fusion module and the preset classification criteria;

[0046] The error analysis unit calculates the classification error based on the image classification label result of the classifier;

[0047] The parameter setting unit updates and sets the network training parameters according to the classification error calculated by the error analysis unit;

[0048] Repeat the above steps until the corresponding number of training generations set in the network training parameters are completed.

[0049] The present invention also discloses a method for using a pathological image classification device based on deep feature fusion. By using this method, pathological images can be accurately classified by the device. The method comprises:

[0050] The image data set acquisition unit samples characteristic areas in the digital slices of the prostate pathological image morphology to acquire a small image data set, and divides the small image data set into a training set and a test set according to a specific ratio;

[0051] A specific number of integrated units receive the small image blocks in the test set from the small image block dataset acquired by the image dataset acquisition unit, and, based on the network training parameters finally updated and set by the parameter setting unit, cause the small image blocks to pass through the convolution kernel and weighted channel operations and the cascade of dimensional deepening, repeatedly obtaining further weighted and deepened image convolution features, and extracting deep image convolution features and shallow image convolution features;

[0052] The feature fusion unit operates on the deep image convolution feature and the shallow image convolution feature obtained by the integration unit, and then cascades them with the vectorized deep image convolution feature to obtain an image deep fusion feature;

[0053] The classifier obtains the classification label of the image based on the image depth fusion features obtained by the feature fusion module and the preset classification standards.

[0054] The application discloses a pathological image classification technology based on deep feature fusion, which better combines a convolution model and image classification in the prior art, and effectively improves the classification precision of prostate pathological images. The morphological digital slices in the data are divided into a training set and a test set by constructing a data set; according to the labeling information, the labeled regions of the morphological digital slices are cropped to obtain small block images, which are used for training and testing a deep convolution neural network; an attention mechanism module, a down-sampling module and a feature fusion module are constructed to stack the modules to construct the deep convolution neural network; the deep convolution neural network is trained by means of a stochastic gradient descent method to optimize the parameters of the deep convolution neural network; the convolution features extracted by the deep convolution neural network are subjected to an outer product operation to calculate and obtain bilinear features; the convolution features are subjected to a convolution operation to obtain a convolution feature vector; the convolution feature vector and the bilinear features are concatenated to obtain a concatenated vector, and the concatenated vector is input into a classifier to obtain a predicted label. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0056] Figure 1 A structural schematic diagram of a pathological image classification device based on deep feature fusion is provided for the embodiments of the present application.

[0057] Figure 2 A structural schematic diagram is provided for the second embodiment of the present application.

[0058] Figure 3-1 An attention mechanism module structural schematic diagram is provided for the third embodiment of the present application.

[0059] Figure 3-2 A down-sampling module structural schematic diagram is provided for the third embodiment of the present application.

[0060] Figure 3-3 A feature fusion unit structural schematic diagram is provided for the third embodiment of the present application.

[0061] Figure 4 A flowchart of a pathological image classification method based on deep feature fusion is provided for the fourth embodiment of the present application.

[0062] Figure 5A schematic diagram of the method flow provided in Example 5 of this application;

[0063] Figure 6 A flowchart illustrating a method for using a pathological image classification device based on deep feature fusion provided in Example 6 of the present application;

[0064] Figure 7 A flowchart of a method for using a pathological image classification device based on deep feature fusion provided in Example 7 of the present application. DETAILED DESCRIPTION

[0065] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without creative work are within the scope of protection of the present invention.

[0066] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0067] like Figure 1 As shown, a pathological image analysis device based on deep feature fusion, the device comprising:

[0068] The image data set acquisition unit 1 is used to sample characteristic areas in the digital slices of the prostate pathological image morphology to acquire a small image data set.

[0069] A digital slide image (WSI) utilizes a fully automated microscope scanning system, combined with a virtual slide software system, to scan and seamlessly stitch traditional glass slides to create a complete, full-view (Whole Slide Image) digital slide, referred to as a WSI. Feature regions in a morphological digital slide are areas of interest to the user, and are labeled as Gleason grade types 0, 3, 4, or 5. Of course, other labeling methods can also be used to identify and sample feature regions. The present invention does not limit the methods for identifying and sampling feature regions.

[0070] In general, the morphological digital slices are sampled according to the feature area to construct the image dataset required for the neural network model. The morphological digital slices are tiled and sampled with a sliding window of size l and step length s. When the side length of the center of the sampled image is When the area of ​​the region belonging to the labeled region reaches the ratio coefficient r, the sampled image is put into the dataset X and the label is obtained, X=[x1,x2,...,x N] represents the sample set consisting of all images, each image is denoted as x i ,{i=1,2,...,N, where N is the number of image samples; Y=[y1,y2,...,y M ] represents the label corresponding to the image dataset X. The label here is used to record the annotation of the feature area.

[0071] A specific number of integrated units 2 are used to receive small-block images in the small-block image data set acquired by the image data set acquisition unit, and according to the set network training parameters, the small-block images are subjected to the operation of convolution kernels and weighted channels, and the cascade of dimensional deepening, to repeatedly acquire further weighted and deepened image convolution features, and extract deep image convolution features and shallow image convolution features.

[0072] In order to clearly describe the relationship between the integrated units, multiple integrated units are shown in the figure, and the last two integrated units are the N-1th integrated unit and the Nth integrated unit (N is a positive integer).

[0073] To obtain rich, multi-layered image features with a certain depth, multiple layers of features are used to further extract convolutional features. These integrated units are stacked sequentially, further extracting the convolutional features extracted by the previous integrated unit. The features extracted multiple times by multiple integrated units are more abstract and representative, laying a good data foundation for subsequent feature fusion.

[0074] The integrated unit applies convolution kernels and weighted channels to small image blocks, weighting the image features. This is then combined with a dimensional deepening cascade to increase the image dimension and enrich the feature expression. In this context, concatenation refers to concatenating vectors or matrices along a specific dimension.

[0075] The network training parameters can be initialized based on actual experience, or updated based on classification errors to improve the accuracy of the entire neural network.

[0076] The network training parameters mainly include: learning rate, which can be set to 0.0001, momentum, which can be set to 0.9, and training generations, which can be set to 50. It should be noted that the values ​​of the above network training parameters can be modified according to actual conditions and are not limited to the parameter values ​​listed in this article.

[0077] The feature fusion unit 3 is used to perform operations on the deep image convolution features and the shallow image convolution features obtained by the integration unit and cascade them with the vectorized deep image convolution features to obtain image depth fusion features.

[0078] Shallow image convolution features have poor representation capabilities for pathological images, so shallow image convolution features and deep image convolution features are used together to obtain image deep fusion features.

[0079] The classifier 4 is used to obtain a classification label of the image based on the image depth fusion features obtained by the feature fusion module and a preset classification standard.

[0080] The classifier can preset classification standards, and if it meets the classification standards, it will output the corresponding classification label.

[0081] From this, it can be seen that this embodiment performs multiple multi-dimensional feature extractions on small-block images, and realizes feature fusion through the operation of shallow convolution features and deep convolution features. Then, the classifier outputs the corresponding classification labels, and combines the pathological images after deep feature fusion with the classification method to achieve accurate classification results.

[0082] In order to better illustrate the present invention, a second embodiment is given to explain in detail the working principles of each unit and module. Figure 2 shown.

[0083] The image data set acquisition unit 1 is used to sample characteristic areas in the digital slices of the prostate pathological image morphology to acquire a small image data set.

[0084] The small image datasets acquired by the image dataset acquisition unit are the training set and the test set. The training set is denoted as X. train , the test set is denoted as X test .

[0085] The test and validation sets are randomly divided into WSIs according to a certain ratio. For example, 35 morphological digital slices can be used to construct the training set, and the remaining 9 morphological digital slices can be used to construct the test set. Of course, other numbers of digital slices can also be used to construct the training and test sets. There are many options based on actual needs.

[0086] Based on the annotation information, morphological digital slices are sampled to form an image dataset. This dataset can be divided into a training set and a test set according to a specific ratio. When the training set is used as the target data for the device, the neural network is trained according to the set network training parameters. Once a certain level of accuracy is achieved, the network training parameters are fixed to form a fixed neural network model. When the test set is used as the target data for the device, the fixed neural network model is used to classify small blocks of data in the test set to obtain accurate classification results.

[0087] A specific number of integrated units 2:

[0088] The number of integrated units is flexibly set according to needs and needs to match the information contained in the image, with the ability to clearly express the image features as the setting standard.

[0089] For the sake of brevity, Figure 2 Only the structure of one integrated unit is shown, and these integrated units all have the same structure.

[0090] The integrated unit 2 further comprises:

[0091] A specific number of attention mechanism modules 21 are used to make the small block of image pass through the operation of convolution kernel and weighted channel to obtain repeatedly weighted image convolution features.

[0092] The attention mechanism module extracts the salient features of the image and ignores the non-salient parts of the image. Through a certain number of attention mechanism modules, the salient features in the image are weighted, that is, made more significant.

[0093] The number of attention mechanism modules can also be flexibly set according to actual conditions.

[0094] The attention mechanism module adds a linear mapping of the image features after convolution and maximum pooling the image. After obtaining the weighted image features, it multiplies them with the convolution features and finally adds them to obtain the weighted image convolution features.

[0095] The downsampling module 22 is used to extract features of different fields of view from the image convolution features that have been repeatedly weighted by the specific number of attention mechanism modules, and to fuse the features in a cascade manner to obtain deepened image convolution features.

[0096] The convolution layer and maximum pooling layer of the downsampling module extract features of different fields of view from the image convolution features extracted by multiple attention mechanism modules, and then fuse the extracted results in a cascade manner to obtain deepened image convolution features.

[0097] The specific number of integrated units are stacked in sequence, and the convolution features extracted by the previous integrated unit are extracted again. The convolution features extracted by the last integrated unit are deep image convolution features, and the convolution features extracted by the penultimate integrated unit are shallow image convolution features.

[0098] Feature fusion unit 3, the feature fusion module converts the deep image convolution feature and the shallow image convolution feature into feature matrices of the same dimension and then performs an outer product operation, converts the deep image convolution feature into a full convolution vector through the convolution operation, and cascades the full convolution vector with the result of the outer product operation to obtain the image deep fusion feature.

[0099] Deep convolutional feature hd and shallow convolutional features h s Entering the feature fusion unit, it is converted into a feature matrix M of the same dimension through the convolution layer and the pooling layer. d and M s , the bilinear feature M is obtained by calculating the bilinear pooling layer b , which is calculated as follows:

[0100]

[0101] Bilinear feature M b Rearrange to a bilinear vector V b . Deep convolutional feature h d Transformed into a full convolution vector V through the convolution layer fc . Bilinear vector V b and the fully convolution vector V fc The cascade vector V is obtained, and its calculation formula is as follows:

[0102] V=[V fc ,V b ]

[0103] This completes feature fusion.

[0104] The classifier 4 is used to obtain a classification label of the image based on the image depth fusion features obtained by the feature fusion module and a preset classification standard.

[0105] The error analysis unit 5 is used to calculate the classification error according to the classification result of the classifier.

[0106] Usually, the softmax loss of the classifier is used as the loss function, and the stochastic gradient descent method is used to train the deep convolutional neural network to obtain the optimal parameters.

[0107] For example, the 2×1 concatenated vector output by the feature fusion unit, corresponding to the values ​​of two class labels, is processed by Softmax to become a 2×1 prediction probability vector, and the Softmax loss is calculated. Alternatively, the concatenated vector V is mapped to a 3×1 vector, corresponding to the values ​​of three class labels, and processed by Softmax to become a 3×1 prediction probability vector, and the Softmax loss is calculated.

[0108] The parameter setting unit 6 is used to update and set the network training parameters according to the classification error calculated by the error analysis unit.

[0109] Backpropagation is performed based on the initial error to update the network training parameters. During the iteration process, the error rate gradually decreases as the number of training generations in the network training parameters increases. During the training process, the network training parameters are fine-tuned until convergence is achieved, and the optimal deep convolutional neural network is determined.

[0110] In order to explain the working principle and working process of the present device in more detail, the following is a third embodiment of the present invention, which is described in conjunction with an example:

[0111] This example uses the TCGA image dataset as an example dataset. Morphological digital slides from 44 patients in this dataset are selected, and the regions of interest in these images are annotated into four categories: G0, G3, G4, and G5. This example first determines the presence of cancer using a deep convolutional neural network for cancer classification constructed using a pathology image classification device based on deep feature fusion. The cancer type is then determined using a deep convolutional neural network for cancer classification constructed using a pathology image classification device based on deep feature fusion.

[0112] The image data acquisition unit acquires images and divides the data set.

[0113] The image data acquisition unit selected 44 high-quality morphological digital slices from the prostate cancer public dataset TCGA, of which 35 morphological digital slices were used to construct the training set and the remaining 9 morphological digital slices were used to construct the test set.

[0114] Small image patches are sampled using a sliding window of size 1200 and step size 300. When the central area of ​​the small image patch with a side length of 600 is located in an area greater than 90% of the annotated area, the small image patch will be placed in the training set or test set. The training set is used for model training, and the test set images are used for model testing. The doctor annotates the areas on the pathological sections and gives category labels. The labels of the collected small image patches are the same as the labels of their corresponding annotated areas. The labels here use Gleason grading: category 0, category 3, category 4, category 5, four labels, and a total of 61,854 images were collected. To adapt to the convolutional neural network, the images are scaled to 224×224. The image dataset X=[x1,x2,…,x 61854 ], the four category labels of the dataset X are represented as Y=[y1,y2,y3,y4], and each sample x in X i ,{i=1,2,…,61854} all contain image features and labels.

[0115] A specific number of integration units, feature fusion units, and classifiers constitute a deep convolutional neural network model. In this embodiment, five integration units, one feature fusion unit, and one classifier are used to form a deep convolutional neural network model, where each integration unit contains two attention mechanism modules and one downsampling module.

[0116] Two pathological image classification devices based on deep feature fusion are constructed, that is, two deep convolutional neural networks are constructed, the first deep convolutional neural network is used for determining whether there is cancer, and the second deep convolutional neural network is used for determining the type of cancer. It should be noted that the structures of the two pathological image classification devices based on deep feature fusion are completely same, the difference lies in that the set network training parameters are different, the preset classification standards of the classifiers are different, the first deep convolutional neural network adopts a cancer or no cancer classifier, and the second deep convolutional neural network adopts a cancer type classifier.

[0117] The integration unit of the deep convolutional neural network is stacked by two attention mechanism modules and a down-sampling module.

[0118] The number of 3*3 convolution kernels of the first attention mechanism module is set to 64. The convolution layer and the maximum pooling layer of the down-sampling module extract features with a field of view of 3*3 and 2*2 respectively, and the features are fused in a cascading manner. The attention mechanism module is as shown in Figure 3-1 , and the down-sampling module is as shown in Figure 3-2 .

[0119] Five integration units and a feature fusion unit are stacked in sequence. The feature fusion unit is as shown in Figure 3-3 . The five integration units are respectively named as a first integration unit, a second integration unit, a third integration unit, a fourth integration unit and a fifth integration unit. The five integration units are stacked once, and the results output by the previous integration unit are operated again. The result output by the fourth integration unit is a shallow image convolution feature, the result output by the fifth integration unit is a deep image convolution feature, and the feature fusion unit performs outer product operation on the convolution features extracted by the fourth integration unit and the fifth integration unit, and then cascades with the full convolution vector to realize the fusion of the convolution features. Because the features extracted by the later integration units are more abstract and have stronger feature representation ability. Experiments are conducted by using the features extracted by different integration units for outer product operation, and it is found that the best classification effect of the model is obtained by using the features extracted by the fourth integration unit (the second last one) and the fifth integration unit (the last one) for outer product operation.

[0120] The deep convolutional neural network is trained by using the training set, the initialization parameters are set, the network training parameters are adjusted by the parameter setting unit according to the training error, and the model reaches a convergent state.

[0121] (1) Network training parameter initialization, setting the learning rate of training to 10 -4 , the weight drop rate is 0.9, the training batch is set to 100, the training iteration is 50, and the image feature is forward propagated.

[0122] The training set X trainThe small block image in is input into the deep convolutional neural network, and the convolution features of integrated unit 4 and integrated unit 5 are extracted and mapped into a 1×1024-dimensional bilinear vector and a 1×2408-dimensional full convolution vector. The two are cascaded to obtain a 1×3425-dimensional cascade vector V, which is input into the classifier to obtain the predicted label.

[0123] (2) Error back propagation:

[0124] In the deep convolutional neural network used to determine the presence or absence of cancer, the fully connected layer maps the concatenated vector V to a 2×1 vector, corresponding to the values ​​of two class labels. After Softmax processing, it is converted into a 2×1 predicted probability vector, and the Softmax loss is calculated.

[0125] In a deep convolutional neural network used to determine cancer types, the fully connected layer maps the concatenated vector V to a 3×1 vector corresponding to the values ​​of the three class labels. This vector is then processed by Softmax to form a 3×1 predicted probability vector, and the Softmax loss is calculated. In this example, the number of training generations is set to 50.

[0126] Backpropagation is then performed based on this initial error to update the network training parameters. During the iterative process, the error rate gradually decreases as the number of training generations increases. During training, the network training parameters are fine-tuned until convergence is achieved, determining the optimal deep convolutional neural network.

[0127] The deep convolutional neural network is tested on the test set. The network training parameters used are those determined by the parameter setting unit after adjusting the training set error. They can be used directly.

[0128] Input a small patch of images from the test set into a deep convolutional neural network to obtain a predicted label for each patch. Calculate the classification accuracy of the device. This is the number of images in the test set correctly classified by the device divided by the total number of images in the test set.

[0129] The image-level classification accuracy is calculated based on whether the predicted labels for the test set samples are consistent with the true labels. Table 1 below shows the classification accuracy of the model. Compared with the ResNet-50-based method on the TCGA dataset, the model's results are more advantageous.

[0130] Table 1 Classification accuracy on TCGA dataset

[0131]

[0132] The fourth embodiment of the present invention discloses a pathological image classification method based on deep feature fusion, such as Figure 4 shown.

[0133] Step S41: sampling the characteristic regions in the digital slices of the prostate pathological image to obtain a small image data set.

[0134] Step S42: According to the set network training parameters, the small-block images in the small-block image data set are subjected to the operation of convolution kernel and weighted channel, and the cascade of dimensional deepening, and further weighted and deepened image convolution features are repeatedly obtained to extract deep image convolution features and shallow convolution features.

[0135] Step S43: After performing calculations on the deep image convolution features and the shallow image convolution features, the deep image convolution features are cascaded with the vectorized deep image convolution features to obtain image depth fusion features.

[0136] Step S44: Obtain a classification label for the image based on the acquired image depth fusion features and a preset classification standard.

[0137] In order to better illustrate the working principle of each step, the fifth embodiment of the present invention is given. Figure 5 shown.

[0138] Step S51: sampling the characteristic regions in the digital slices of the prostate pathological image to obtain a small image data set.

[0139] The small image datasets are a training set and a test set.

[0140] Step S521: the small block image is subjected to the operation of the convolution kernel and the weighted channel to obtain the repeatedly weighted image convolution feature.

[0141] Step S522: extracting features of different fields of view from the repeatedly weighted image convolution features, and fusing the features in a cascade manner to obtain a deepened image convolution feature.

[0142] Step S523: extracting the convolution features of the image again. The convolution features extracted for the last time are deep image convolution features, and the convolution features extracted for the second to last time are shallow image convolution features.

[0143] Step S53: Convert the deep image convolution features and the shallow image convolution features into feature matrices of the same dimension and perform outer product operation, convert the deep image convolution features into full convolution vectors through convolution operation, and cascade the full convolution vectors with the results of the outer product operation to obtain image depth fusion features.

[0144] Step S54: presetting different classification standards, and using the corresponding classification standards to classify the image depth fusion features according to the determination requirements.

[0145] Step S55: calculating a classification error according to the classification label of the obtained image.

[0146] Step S56: updating the network training parameter according to the error.

[0147] The invention content of the method part is similar to the foregoing device part, and the specific description can refer to the foregoing device part, which will not be described here again.

[0148] In order to specifically describe the working principle of the pathological image classification device based on deep feature fusion for the training set, the embodiment six of the present application is given, which comprises the following steps:

[0149] Step 61: the image data set acquisition unit samples the feature region in the prostate pathological image form digital slice, acquires a small block image data set, and divides the small block image data set into a training set and a test set according to a specific proportion.

[0150] Step 62: a specific number of integration units receive the small block image in the training set in the small block image data set acquired by the image data set acquisition unit, make the small block image pass through the operation of convolution kernel and weighted channel, dimension deepening cascade according to the set initialization network training parameter, repeatedly obtain further weighted and deepened image convolution features, and extract deep image convolution features and shallow image convolution features.

[0151] Step 63: the feature fusion unit is cascaded with the vectorized deep image convolution feature after the operation of the deep image convolution feature and the shallow image convolution feature acquired by the integration unit, and the image deep fusion feature is obtained.

[0152] Step 64: the classifier obtains the classification label of the image according to the image deep fusion feature acquired by the feature fusion module, and combines the preset classification standard.

[0153] Step 65: the error analysis unit calculates the classification error according to the image classification label result of the classifier.

[0154] Step 66: the parameter setting unit updates and sets the network training parameter according to the classification error calculated by the error analysis unit.

[0155] Step 67: it is judged whether the corresponding number of times of the training algebra set in the network training parameter is executed, if not, returning to step 62, if yes, ending.

[0156] This embodiment uses a pathological image classification device based on deep feature fusion to complete the training process. By using the data of the training set and the initialized network training parameters, the images are classified using deep feature fusion, and the network training parameters are adjusted with reference to the classification error. Ultimately, the network training parameters of the pathological image classification device based on deep feature fusion are in the optimal state, and the classification result has the smallest error and the highest accuracy.

[0157] In order to explain in detail the working principle of a pathological image classification device based on deep feature fusion for a test set, a seventh embodiment of the present invention is provided, which includes the following steps:

[0158] Step S71: the image data set acquisition unit samples the characteristic areas in the digital slices of the prostate pathological image morphology to obtain a small image data set, and divides the small image data set into a training set and a test set according to a specific ratio.

[0159] Step S72: A specific number of integrated units receive the small-block images in the test set in the small-block image dataset obtained by the image dataset acquisition unit, and finally update the set network training parameters according to the parameter setting unit so that the small-block images pass through the convolution kernel and weighted channel operations and the cascade of dimensional deepening, repeatedly obtain further weighted and deepened image convolution features, and extract deep image convolution features and shallow image convolution features.

[0160] Step S73: The feature fusion unit operates on the deep image convolution feature and the shallow image convolution feature obtained by the integration unit, and then cascades them with the vectorized deep image convolution feature to obtain an image depth fusion feature.

[0161] Step S74: The classifier obtains a classification label for the image based on the image depth fusion features obtained by the feature fusion module and a preset classification standard.

[0162] This embodiment mainly tests a pathology image classification device based on deep feature fusion using a test set. According to the training steps in the sixth embodiment above, the pathology image classification device based on deep feature fusion has reached a relatively accurate state. Its accuracy is tested using images in the test set. The test examples are listed in the above embodiment and will not be repeated here.

[0163] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0164] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein can be implemented directly with hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. The terms "first", "second", "third", "fourth", etc. (if present) in the specification and claims and the above-mentioned drawings are used to distinguish similar parts and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate so that the embodiments of the present application described herein can be implemented in a sequence other than that illustrated here.

[0165] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A pathological image classification device based on deep feature fusion, characterized in that: The device comprises: An image data set acquisition unit is used to sample characteristic areas in digital slices of prostate pathological image morphology to acquire a small image data set; a plurality of integrated units, configured to receive small image blocks from the small image block dataset acquired by the image dataset acquisition unit, and subject the small image blocks to operations on convolution kernels and weighted channels, and cascade dimension deepening according to set network training parameters, to repeatedly acquire further weighted and deepened image convolution features, and to extract deep image convolution features and shallow image convolution features; A feature fusion module is used to perform an operation on the deep image convolution feature and the shallow image convolution feature obtained by the integration unit and then cascade them with the vectorized deep image convolution feature to obtain an image depth fusion feature, including converting the deep image convolution feature and the shallow image convolution feature into feature matrices of the same dimension and then performing an outer product operation, converting the deep image convolution feature into a full convolution vector through the convolution operation, and cascading the full convolution vector with the result of the outer product operation to obtain the image depth fusion feature; A classifier is used to obtain a classification label for the image based on the image depth fusion features obtained by the feature fusion module and a preset classification standard; The integrated unit further comprises: Multiple attention mechanism modules are used to make the small block of image pass the operation of convolution kernel and weighted channel to obtain repeatedly weighted image convolution features; A downsampling module is used to extract features of different fields of view from the image convolution features that have been repeatedly weighted by the multiple attention mechanism modules, and to fuse the features in a cascade manner to obtain a deeper image convolution feature; The multiple integrated units are stacked in sequence, and the convolution features extracted by the previous integrated unit are extracted again. The convolution features extracted by the last integrated unit are deep image convolution features, and the convolution features extracted by the penultimate integrated unit are shallow image convolution features.

2. The device according to claim 1, characterized in that: The device also includes an error analysis unit and a parameter setting unit; The error analysis unit is used to calculate the classification error according to the classification result of the classifier; The parameter setting unit is used to update and set the network training parameters according to the classification error calculated by the error analysis unit; The small image datasets acquired by the image dataset acquisition unit are a training set and a test set.

3. The device according to claim 2, characterized in that: Different classification standards are preset for different classifiers, and the image depth fusion features are classified using corresponding classifiers according to the judgment requirements.

4. A pathological image classification method based on deep feature fusion, characterized in that: The method comprises: Sampling characteristic areas in digital slices of prostate pathological images to obtain a small image dataset; According to the set network training parameters, the small blocks of images in the small block image data set are subjected to the operation of convolution kernel and weighted channel and the cascade of dimension deepening, and further weighted and deepened image convolution features are repeatedly obtained to extract deep image convolution features and shallow image convolution features; The deep image convolution feature and the shallow image convolution feature are operated and then cascaded with the vectorized deep image convolution feature to obtain an image depth fusion feature, including converting the deep image convolution feature and the shallow image convolution feature into feature matrices of the same dimension and then performing an outer product operation, converting the deep image convolution feature into a full convolution vector through the convolution operation, and cascading the full convolution vector with the result of the outer product operation to obtain the image depth fusion feature; According to the image depth fusion feature, combined with the preset classification standard, the classification label of the image is obtained; according to the set network training parameters, the small block images in the small block image data set are subjected to the operation of the convolution kernel and the weighted channel, and the cascade of the dimension deepening, and the further weighted and deepened image convolution features are repeatedly obtained, and the deep image convolution features and the shallow image convolution features are extracted specifically as follows: The small image is subjected to a convolution kernel and a weighted channel operation to obtain a repeatedly weighted image convolution feature; Extracting features of different fields of view from the repeatedly weighted image convolution features, and fusing the features in a cascade manner to obtain deepened image convolution features; The convolution feature of the image is extracted again, the convolution feature extracted for the last time is the deep image convolution feature, and the convolution feature extracted for the second to last time is the shallow image convolution feature.

5. The method according to claim 4, characterized in that The method further comprises: Calculating a classification error based on the classification label of the image; Update and set the network training parameters according to the error; The small image datasets are a training set and a test set.

6. The method according to claim 5, characterized in that: Different classification standards are preset, and the image depth fusion features are classified using corresponding classification standards according to the judgment requirements.

7. A method for using a pathological image classification device based on deep feature fusion, characterized in that: The method comprises: Step 1: The image dataset acquisition unit samples characteristic areas in the digital slices of the prostate pathological image to obtain a small image dataset, and divides the small image dataset into a training set and a test set according to a preset ratio; In step 2, multiple integrated units receive the small image blocks in the training set in the small image block data set obtained by the image data set acquisition unit, and make the small image blocks pass through the convolution kernel and weighted channel operation and dimension deepening cascade according to the set initialization network training parameters, repeatedly obtain further weighted and deepened image convolution features, and extract deep image convolution features and shallow image convolution features, which is specifically as follows: The small image is subjected to a convolution kernel and a weighted channel operation to obtain a repeatedly weighted image convolution feature; Extracting features of different fields of view from the repeatedly weighted image convolution features, and fusing the features in a cascade manner to obtain deepened image convolution features; The convolution feature of the image is extracted again, the convolution feature extracted for the last time is the deep image convolution feature, and the convolution feature extracted for the second to last time is the shallow image convolution feature; Step three, the feature fusion module operates on the deep image convolution feature and the shallow image convolution feature obtained by the integration unit and then cascades them with the vectorized deep image convolution feature to obtain image depth fusion feature, including converting the deep image convolution feature and the shallow image convolution feature into feature matrices of the same dimension and then performing outer product operation, converting the deep image convolution feature into a full convolution vector through convolution operation, and cascading the full convolution vector with the result of the outer product operation to obtain image depth fusion feature; Step 4: The classifier obtains the classification label of the image based on the image depth fusion features obtained by the feature fusion module and the preset classification criteria; Step 5: the error analysis unit calculates the classification error based on the image classification label result of the classifier; Step 6: The parameter setting unit updates and sets the network training parameters according to the classification error calculated by the error analysis unit; Repeat steps 2 to 6 until the corresponding number of training generations set in the network training parameters are completed.

8. A method for using a pathological image classification device based on deep feature fusion, characterized in that: The method comprises: The plurality of integrated units receive the small image blocks in the test set of the small image block data set obtained by the image data set acquisition unit, and finally update the network training parameters set by the parameter setting unit so that the small image blocks pass through the convolution kernel and weighted channel operation and the cascade of dimensional deepening, repeatedly obtaining further weighted and deepened image convolution features, and extracting deep image convolution features and shallow image convolution features, which are specifically as follows: The small image is subjected to a convolution kernel and a weighted channel operation to obtain a repeatedly weighted image convolution feature; The repeatedly weighted image convolution features are subjected to feature extraction from different fields of view, and the features are fused in a cascade manner to obtain a deepened image convolution feature, including converting the deep image convolution feature and the shallow image convolution feature into feature matrices of the same dimension and then performing an outer product operation, converting the deep image convolution feature into a full convolution vector through the convolution operation, and cascading the full convolution vector with the result of the outer product operation to obtain a deep image fusion feature; The convolution feature of the image is extracted again, the convolution feature extracted for the last time is the deep image convolution feature, and the convolution feature extracted for the second to last time is the shallow image convolution feature; The classifier obtains the classification label of the image based on the image depth fusion features obtained by the feature fusion module and the preset classification criteria.

Images