Intelligent identification and subtype classification method for renal cell carcinoma heteromorphism based on deep learning

Abstract

The present application relates to the technical field of image processing, and especially relates to a renal cell carcinoma atypia intelligent identification and subtype classification method based on deep learning. The present application inputs a renal cell image into a deep learning subtype classification model, obtains image network features, and obtains network features of cells in the renal cell image; morphological nuclear features and texture features of cells in the renal cell image are extracted, cells are graded by using different features in the morphological nuclear features, and an overlapping cell set between adjacent grades, a morphological nuclear feature overlapping degree and an overlapping feature area distinguishing degree are determined; based on the morphological nuclear feature overlapping degree and the texture features of cells in the overlapping cell set, network features of cells in the overlapping cell set are enhanced, enhanced network features are obtained, and atypia identification and subtype classification of cells are performed. The present application optimizes and enhances network features of cells with high overlapping degrees, and effectively improves the accuracy of cell atypia identification and subtype classification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a renal cell carcinoma heterotypia intelligent identification and subtype classification method based on deep learning. BACKGROUND

[0002] Renal cell carcinoma is one of the most common malignant tumors in the urinary system, and its pathological subtypes are diverse, with significant differences in biological behavior and prognosis. Moreover, due to the extensive spatial heterogeneity of tumor tissues in renal cell carcinoma, multiple different subtype cells can exist in the same tumor region, and the treatment plans for different subtype cancer cells differ. Therefore, accurate subtype classification and heterotypia evaluation are crucial for developing individualized treatment plans and judging patient prognosis.

[0003] In the pathological examination of renal cell carcinoma using artificial intelligence technology represented by deep learning, HE staining is usually used as the basis, and slice staining images are obtained through an optical microscope, and then a pre-trained deep learning model is used to realize heterotypia identification and subtype classification. Since the weight matrix in the trained deep learning model is fixed, the extraction of cell features in each region of the slice image can be regarded as being extracted under the same reference. However, due to the differences in cell morphological nuclear features in different regions, the feature extraction scheme under the same reference can cause the features of some low-grade heterotypia tumor cell regions to be easily submerged in noise, and the feature distribution of some cell regions with high overlap of morphological and texture features is relatively dense, resulting in low accuracy of identification and classification. SUMMARY

[0004] To solve the problem of inaccurate renal cell carcinoma heterotypia identification and subtype classification due to the relatively dense feature distribution of some cell regions with high overlap of morphological and texture features in the prior art, the present application aims to provide a renal cell carcinoma heterotypia intelligent identification and subtype classification method based on deep learning.

[0005] In the present application, the renal cell image is input into a deep learning subtype classification model to obtain image network features and the network features of cells in the renal cell image. The morphological nuclear features and texture features of cells in the renal cell image are extracted, and the cells are classified by different features in the morphological nuclear features to determine the overlapping cell set between adjacent grades and the morphological nuclear feature overlap degree. Based on the morphological nuclear feature overlap degree and the texture features of cells in the overlapping cell set, the network features of cells in the overlapping cell set are enhanced to obtain enhanced network features, and the heterotypia identification and subtype classification of cells are performed. By optimizing and enhancing the network features of cells with high overlap, the accuracy of renal cell carcinoma heterotypia identification and subtype classification is effectively improved.

[0006] The object of the present application can be achieved by the following technical solutions:

[0007] The first object of the present application is to provide a deep learning-based renal cell carcinoma heterotypic intelligent identification and subtype classification method, comprising the following steps:

[0008] S1, constructing a deep learning subtype classification model containing a feature optimization enhancement layer through a convolutional neural network;

[0009] S2, preprocessing a standard renal cell carcinoma image to obtain a preprocessed standard renal cell carcinoma image;

[0010] Cutting the preprocessed standard renal cell carcinoma image through an image segmentation method to extract morphological nuclear features and texture features of each renal cell carcinoma;

[0011] S3, pre-training the deep learning subtype classification model of step S1 using the morphological nuclear features of each renal cell carcinoma obtained in step S2 to obtain a pre-trained hierarchical network model based on morphological nuclear feature single features and a pre-trained hierarchical network model based on morphological nuclear feature multiple features;

[0012] Pre-training the deep learning subtype classification model of step S1 using the texture features of each renal cell carcinoma obtained in step S2 to obtain a pre-trained hierarchical network model based on texture feature single features and a pre-trained hierarchical network model based on texture feature multiple features;

[0013] S4, preprocessing a renal cell image to be analyzed to obtain a preprocessed renal cell image;

[0014] Cutting the preprocessed renal cell image through an image segmentation method to extract morphological nuclear features and texture features of each renal cell;

[0015] S5, inputting the morphological nuclear features of each renal cell in step S4 into the pre-trained hierarchical network model based on morphological nuclear feature single features and the pre-trained hierarchical network model based on morphological nuclear feature multiple features in step S3 to obtain morphological nuclear feature single feature classification and morphological nuclear feature multiple feature classification of each renal cell, and determine overlapping cell sets between adjacent classifications and morphological nuclear feature overlap degrees between adjacent classifications;

[0016] S6, inputting the texture features of each renal cell in step S4 into the pre-trained hierarchical network model based on texture feature single features and the pre-trained hierarchical network model based on texture feature multiple features in step S3 to obtain texture feature single feature classification and texture feature multiple feature classification of each renal cell, and determine overlapping cell sets between adjacent classifications and texture feature overlap degrees between adjacent classifications;

[0017] S7, determining the overlapping feature discriminativeness between adjacent hierarchical levels using the morphological nuclear feature overlapping degree between adjacent hierarchical levels obtained in step S5 and the texture feature overlapping degree between adjacent hierarchical levels obtained in step S6;

[0018] S8, obtaining enhanced network features of each cell based on the overlapping cell set between adjacent hierarchical levels obtained in step S5, the overlapping cell set and the overlapping feature discriminativeness between adjacent hierarchical levels obtained in steps S6 and S7;

[0019] S9, obtaining enhanced image network features based on the enhanced network features of each cell obtained in step S8, and performing cell atypia recognition and subtype classification through the deep learning subtype classification model in step S1.

[0020] Since renal cell carcinoma subtype classification usually needs to rely on both the tissue arrangement structure of tumor cells and the morphological features of single tumor cells, both of which are indispensable, for tumor cells with low atypia, the morphological nuclear features are similar to those of normal tissue cells, such as smaller cell nuclei and higher nuclear roundness. Since the features extracted by the traditional network feature extraction layer are obtained under a global weight sharing mechanism, that is, the extracted features can be regarded as being obtained based on the same benchmark, the image features provided by the traditional network feature extraction layer are difficult to directly reflect effective cancer cell features in the case of tumor cells with low atypia, because the nuclear morphological hierarchical levels of different cells are similar in performance and have high overlapping degree, and the texture features are also similar in difference, resulting in low accuracy of final cell atypia recognition and subtype classification.

[0021] Therefore, in step S1, the deep learning subtype classification model is improved based on an existing model architecture (including an input layer, a feature extraction layer and a classification layer), and a feature optimization enhancement layer is added between the feature extraction layer and the classification layer; that is, the basic architecture of the deep learning subtype classification model includes an input layer, a feature extraction layer, a feature optimization enhancement layer and a classification layer.

[0022] In an embodiment of the present application, in step S2, the preprocessing includes color normalization, artifact elimination and image enhancement.

[0023] In step S4, the preprocessing includes color normalization, artifact elimination and image enhancement.

[0024] In an embodiment of the present application, in step S2, the standard renal cell carcinoma image is a renal cell carcinoma pathological tissue section image.

[0025] The local features in the network features (morphological nuclear features and texture features) of the standard renal cell carcinoma image are mapped to a local area in the standard renal cell carcinoma image using a multi-layer backpropagation neural network structure or a receptive field algorithm.

[0026] Based on the distribution of the cell regions of each cell segmented from the standard renal cell carcinoma image in the local region, and in combination with the local features corresponding to the region, network features of each cell are determined;

[0027] In step S4, the renal cell image to be analyzed is a renal cell pathological tissue section image to be analyzed;

[0028] The local features in the network features (morphological core features and texture features) of the renal cell image to be analyzed are mapped to one local region in the receptive field by using a multi-layer reverse neural network structure or a receptive field algorithm;

[0029] Based on the distribution of the cell regions of each cell segmented from the standard renal cell carcinoma image in the local region, and in combination with the local features corresponding to the region, network features of each cell are determined;

[0030] In an embodiment of the present application, the morphological core features include the core area, the core-to-cytoplasm ratio, and the core roundness of each stained cell;

[0031] The texture features include contrast, roughness, directionality, linearity, regularity, and coarseness.

[0032] In the same pathological section image, the morphological core features of tumor cells show differences, that is, there are both high atypia tumor cells with large core volume, abnormal shape, and high core-to-cytoplasm ratio, and low-grade atypia tumor cells similar to normal cell cores. Therefore, when different morphological core features are used to grade cells, differences in grading will occur, and the more obvious the differences between single-feature and multi-feature grading, the higher the overlap degree of grading.

[0033] In an embodiment of the present application, in step S5, the determination of the overlapping cell set between adjacent grades is as follows:

[0034] In adjacent grades, cells with unequal morphological core feature single-feature grading and multi-feature grading are determined as overlapping cells, and the overlapping cell set between adjacent grades is obtained based on all overlapping cells between adjacent grades.

[0035] The determination of the morphological core feature overlap degree between adjacent grades is as follows:

[0036] In adjacent grades, the number of cells with unequal arbitrary single-feature grading and multi-feature grading is determined, and the morphological feature overlap degree between adjacent grades is obtained by normalization processing after accumulating all cell numbers.

[0037] In an embodiment of the present application, the morphological core feature overlap degree between adjacent grades is calculated by the following formula:

[0038]

[0039] wherein: α i,i+1 denotes the morphological kernel feature overlap between the i-th level and the (i+1)-th level; num denotes the number of categories of all features in the morphological kernel feature; r i,i+1,k denotes the number of cells with different levels between the k-th feature and the multi-feature of the i-th level and the (i+1)-th level; norm denotes a standard normalization function for normalizing a numerical value to the range of [0, 1].

[0040] In an embodiment of the present application, in step S6, the determination of the overlapping cell set between adjacent levels is specifically as follows:

[0041] In adjacent levels, the cells with different levels of the texture feature single feature level and the multi-feature level are determined as overlapping cells; and based on all overlapping cells between adjacent levels, the overlapping cell set between adjacent levels is obtained;

[0042] The determination of the texture feature overlap between adjacent levels is specifically as follows:

[0043] Based on the mean square error distance of the texture vector composed of the texture features between any two cells in the overlapping cell set between adjacent levels, all mean square error distances are accumulated and then subjected to negative correlation normalization processing to obtain the texture feature overlap between adjacent levels;

[0044] In step S7, the overlapping feature discrimination between adjacent levels is the product of the morphological kernel feature overlap between adjacent levels and the texture feature overlap between adjacent levels.

[0045] In an embodiment of the present application, the texture feature overlap between adjacent levels is obtained by the following method:

[0046]

[0047] wherein: β i,i+1 denotes the texture feature overlap between the i-th level and the (i+1)-th level; s(k,m) denotes the overlapping cell set U i,i+1 the mean square error distance of the texture vector composed of the texture features of the cells k and m; norm denotes a standard normalization function for normalizing a numerical value to the range of [0, 1].

[0048] In one embodiment of the present application, in step S8, the product of each component in the network feature of each cell in the overlapping cell set and the overlapping feature discriminant is determined to obtain a component increment; the sum of each component in the network feature of each cell in the overlapping cell set and the corresponding component increment is determined to obtain an enhanced component; and the enhanced network feature of each cell in the overlapping cell set is obtained based on the enhanced component corresponding to the used component in the network feature of each cell in the overlapping cell set.

[0049] In step S9, the enhanced image network feature is obtained based on the enhanced network feature of each cell in the overlapping cell set.

[0050] A second object of the present application is to provide a deep learning-based intelligent recognition and subtype classification device for renal cell carcinoma heteromorphism, comprising an image feature extraction module for inputting a renal cell image into a deep learning subtype classification model to obtain image network features extracted by the deep learning subtype classification model when the deep learning subtype classification model recognizes and classifies the heteromorphism of cells in the renal cell image;

[0051] A morphological texture feature extraction module for segmenting cells in the renal cell image and extracting morphological nuclear features and texture features of each cell;

[0052] A grade division module for dividing each cell into grades based on different features in the morphological nuclear features to obtain feature grades of each cell;

[0053] An overlapping degree analysis module for determining an overlapping cell set and morphological nuclear feature overlapping degree between adjacent grades based on the difference between the feature grades corresponding to the same cell;

[0054] A feature division module for determining network features of each cell in the renal cell image based on the image network features;

[0055] A feature enhancement module for enhancing the network features of each cell in the overlapping cell set based on the morphological nuclear feature overlapping degree and the texture feature difference between cells in the overlapping cell set to obtain enhanced network features;

[0056] An identification module for obtaining enhanced image network features based on the enhanced network features and recognizing the heteromorphism and classifying the subtype of cells based on the enhanced image network features.

[0057] A third object of the present application is to provide a deep learning-based intelligent recognition and subtype classification system for renal cell carcinoma heteromorphism, comprising a memory and a processor.

[0058] The memory is configured to store executable computer program code, and the processor is configured to call and run the executable computer program code from the memory, so that the system executes the method described above.

[0059] A fourth object of the present application is to provide a computer program product comprising computer program code.

[0060] When the computer program code is run on a computer, the computer is caused to execute the method described above.

[0061] A fifth object of the present application is to provide a computer-readable storage medium storing computer program code.

[0062] When the computer program code is run on a computer, the computer is caused to execute the method described above.

[0063] Compared with the prior art, the present application has the following beneficial effects:

[0064] The present application inputs the kidney cell image into the deep learning subtype classification model, obtains the image network features extracted by the deep learning subtype classification model when identifying the atypicality of the cells in the kidney cell image and classifying the subtypes, and determines the network features of each cell in the kidney cell image; different features in the morphological nuclear features of each cell are used to grade each cell, to determine the overlapping cell set and the morphological nuclear feature overlap degree; then, based on the morphological nuclear feature overlap degree and the texture feature difference between different cells in the overlapping cell set, the network features of each cell in the overlapping cell set are enhanced to obtain enhanced network features, and further enhanced image network features of the whole image are obtained; finally, based on the enhanced image network features, the atypicality of the cells is identified and the subtypes are classified. The present application optimizes and enhances the network features of cells with high feature overlap degree, converts the relatively dense features into relatively sparse features, effectively avoids the influence of subtle noise on the features, and effectively improves the accuracy of atypicality identification and subtype classification of the cells. DETAILED DESCRIPTION

[0065] The present application will be described in detail below in combination with specific embodiments.

[0066] Firstly, the application scenario of the present application is described. The specific application scenario that the present application aims at is the atypia recognition and subtype classification of renal cell carcinoma tissue sections. Among them, atypia refers to tumor cells similar in morphology to normal tissue cells under a microscope, which grow relatively slowly, have low invasiveness and relatively good prognosis; the subtype refers to tumor cells that exhibit typical morphological nuclear features, such as clear cell renal cell carcinoma, papillary renal cell carcinoma, etc. Atypia determines the degree of malignancy (grade) of cells, and subtype determines the classification of cells, and the two overlap and do not overlap, such as for cells with high atypia and loss of subtype (sarcomatous cells), the two do not overlap. The renal tumor cells that the present application aims at are low-grade tumor cells with typical subtype and low atypia.

[0067] In order to solve the problem that the existing renal cell carcinoma atypia recognition and subtype classification is not accurate enough, the present application provides a renal cell carcinoma atypia intelligent recognition and subtype classification method based on deep learning, which extracts image network features when the deep learning subtype classification model recognizes atypia and classifies subtypes of cells in a renal cell image, and optimizes network features with high feature overlap degree, so that relatively dense features are converted into relatively sparse features, which can effectively avoid the influence of subtle noise on features, optimize and enhance the feature extraction capability of cells, and improve the accuracy of low atypia cell recognition and subtype classification.

[0068] Embodiment 1

[0069] The present embodiment provides a renal cell carcinoma atypia intelligent recognition and subtype classification method based on deep learning, which comprises the following steps:

[0070] S1, a deep learning subtype classification model (including an input layer, a feature extraction layer, a feature optimization enhancement layer and a classification layer) containing a feature optimization enhancement layer is constructed by a convolutional neural network;

[0071] Specifically, a deep learning subtype classification model is constructed by using a convolutional neural network (CNN), and a feature optimization enhancement layer is added between the feature extraction layer and the classification layer based on the existing deep learning subtype classification model (a classification network composed of a deep neural network) architecture (including an input layer, a feature extraction layer and a classification layer). The feature optimization enhancement layer matches the network features (morphological nuclear features and texture features) corresponding to each cell, and according to the morphological and texture features of a single cell, the network features of cells with low discrimination are moderately enhanced and converted to image network features, and finally the atypia index and subtype classification of each region in the image are output through the classification layer.

[0072] S2, pre-process a large number of renal cell carcinoma images (surface images of renal cell carcinoma pathological tissue sections: after staining the renal cell carcinoma pathological tissue sections, then using a high-power microscope to obtain standard renal cell carcinoma images) (including color normalization, artifact removal and image enhancement) to obtain pre-processed renal cell carcinoma images (subsequently used as a training set, and the training set is manually labeled with labels, and the labels are atypicality indicators and subtype classifications of different regions);

[0073] Among them, the color normalization adopts a Vahadane algorithm (a sparse decomposition staining normalization algorithm) to unify the H&E staining color tone; the artifact removal adopts an opening / closing operation to remove tissue folding, bubbles, and mark out the background area (a repair algorithm (such as Inpainting) can also be used); and the image enhancement adopts a CLAHE (contrast limited adaptive histogram equalization) algorithm to improve the clarity of the cell boundary.

[0074] The pre-processed renal cell carcinoma images are cut by an image segmentation method to extract morphological nuclear features (including the nuclear area, nuclear-cytoplasmic ratio and nuclear roundness of each stained cell) and texture features (including contrast, roughness, directionality, linearity, regularity and coarseness) of each renal cell carcinoma.

[0075] S3, first, different features or all features in the morphological nuclear features are used to manually label and grade each cell to obtain feature grades of each cell; then, the morphological nuclear features of each renal cell carcinoma obtained in step S2 are used to pre-train the deep learning subtype classification model (a grading network model based on a single feature of morphological nuclear features and a grading network model based on multiple features of morphological nuclear features) of step S1 to obtain a pre-trained grading network model based on a single feature of morphological nuclear features and a pre-trained grading network model based on multiple features of morphological nuclear features;

[0076] First, different features or all features in the texture features are used to manually label and grade each cell to obtain feature grades of each cell; the texture features of each renal cell carcinoma obtained in step S2 are used to pre-train the deep learning subtype classification model (a grading network model based on a single feature of texture features and a grading network model based on multiple features of texture features) of step S1 to obtain a pre-trained grading network model based on a single feature of texture features and a pre-trained grading network model based on multiple features of texture features;

[0077] Further, in each pre-training, a plurality of set grades are pre-set, for example, the set grades are set to 1, 2, 3 and 4, and the higher the grade, the greater and more extreme the nuclear features of the cell, and the greater the malignant degree and prognosis risk; the training set is manually labeled with labels, and the labels are atypicality indicators and subtype classifications of different regions;

[0078] The loss function used in the pre-training process is cross-entropy loss, and cross-validation is used for verification.

[0079] S4, pre-processing the kidney cell image to be analyzed (i.e., the surface image of the kidney cell pathological tissue section to be analyzed: after staining the kidney cell pathological tissue section to be analyzed, then using a high-power microscope to obtain the kidney cell image to be analyzed) to obtain a pre-processed kidney cell image, including color normalization, artifact removal, and image enhancement;

[0080] Among them, the color normalization adopts the Vahadane algorithm (sparse decomposition staining normalization algorithm) to unify the H&E staining tone; the artifact removal adopts the opening / closing operation to remove tissue folding and bubbles, and mark the background area (a repair algorithm such as Inpainting can also be used); the image enhancement adopts the CLAHE (contrast limited histogram equalization) algorithm to improve the clarity of the cell boundary.

[0081] The pre-processed kidney cell image is cut by image segmentation method, and the morphological nuclear features (including the nuclear area, nuclear-cytoplasmic ratio, and nuclear roundness of each stained cell) and texture features (including contrast, roughness, directionality, linearity, regularity, and coarseness) of each kidney cell are extracted.

[0082] S5, inputting the morphological nuclear features of each kidney cell in step S4 into the morphological nuclear feature single-feature-based hierarchical network model pre-trained in step S3, outputting the set grade of each cell corresponding to the single feature, and taking the set grade as the morphological nuclear feature single-feature grading of each cell.

[0083] S5, inputting the morphological nuclear features of each kidney cell in step S4 into the morphological nuclear feature single-feature-based hierarchical network model pre-trained in step S3, outputting the set grade of each cell corresponding to the single feature, and taking the set grade as the morphological nuclear feature single-feature grading of each cell.

[0084] Determine the overlapping cell set between adjacent hierarchical levels and the morphological nuclear feature overlap degree between adjacent hierarchical levels.

[0085] Among them, the determination of the overlapping cell set between adjacent hierarchical levels is as follows:

[0086] Among adjacent hierarchical levels, determine the cells with unequal morphological nuclear feature single-feature grading and multi-feature grading as overlapping cells; based on all overlapping cells between any adjacent hierarchical levels, count the overlapping cells to obtain the overlapping cell set between adjacent hierarchical levels.

[0087] Specifically, the difference between different classification of various characteristics of each cell is analyzed, if there are more cells with different classification in adjacent classification, it means that the overlap between the classification is higher. For example, between 1 and 2, when using single feature nuclear area classification, for each cell A, B, C, D, E, F, the classification result is 1: A, B, C, D; the classification result is 2: E, F. When using multi-feature classification (three features), the classification result is 1: A, B, C; the classification result is 2: D, E, F. Among them, cell D is the overlapping cell in the two classification processes, indicating that there is feature overlap between 1 and 2 in the nuclear area feature, which is not easy to distinguish.

[0088] For any cell between any two adjacent classifications, if there is a single feature classification and multi-feature classification that is not equal, the cell is considered as an overlapping cell, otherwise, if all single feature classifications and multi-feature classifications are equal, the cell is not considered as an overlapping cell.

[0089] The determination of the morphological nuclear feature overlap between adjacent classifications is as follows:

[0090] Between adjacent classifications, the number of cells with different single feature classification and multi-feature classification is determined, and the number of all cells is accumulated and normalized to obtain the morphological feature overlap between adjacent classifications.

[0091] Further, the morphological nuclear feature overlap between adjacent classifications is calculated by the following formula:

[0092]

[0093] In the formula: α i,i+1 represents the morphological nuclear feature overlap between the i-th classification and the i+1-th classification; num represents the number of all features in the morphological nuclear feature; r i,i+1,k represents the number of cells with different classification between the k-th feature of the i-th classification and the i+1-th classification and multi-feature classification; norm represents a standard normalization function for normalizing the value to the range of [0, 1];

[0094] The number of cells that appear twice in the set of any single feature classification and the set of multi-feature classification, but are classified in adjacent different sets, when the number is higher, it means that the morphological nuclear feature overlap between adjacent classifications is higher when using multi-feature classification.

[0095] The image network features extracted by the network feature extraction layer in the deep learning subtype classification model usually fuse multiple parameters such as texture and morphology, so it is necessary to evaluate the overlap of the texture under the condition of high morphological overlap. If the overlap of both is high, it means that the network features of the cells in this grading interval have low discrimination, and the network features of the cells in this interval need to be optimized in the same proportion to make the feature difference more obvious, so as to improve the accuracy of classification.

[0096] S6, input the texture features of each kidney cell in step S4 into the pre-trained single-feature grading network model based on texture features in step S3, output the set grade of each cell corresponding to the single feature, and take the set grade as the texture feature single-feature grading of each cell;

[0097] S6, input the texture features of each kidney cell in step S4 into the pre-trained single-feature grading network model based on texture features in step S3, output the set grade of each cell corresponding to the single feature, and take the set grade as the texture feature single-feature grading of each cell;

[0098] Determine the overlapping cell set between adjacent grades and the texture feature overlap between adjacent grades;

[0099] The determination of the overlapping cell set between adjacent grades is as follows:

[0100] Determine the cells with unequal texture feature single-feature grading and multi-feature grading between adjacent grades as overlapping cells; based on all overlapping cells between any adjacent grades, the overlapping cells are counted to obtain the overlapping cell set between adjacent grades;

[0101] The determination of the texture feature overlap between adjacent grades is as follows:

[0102] Based on the mean square error distance of the texture vector composed of the texture features between any two cells in the overlapping cell set between adjacent grades, all the mean square error distances are accumulated and then subjected to negative correlation normalization to obtain the texture feature overlap between adjacent grades;

[0103] The texture feature overlap between adjacent grades is calculated by the following method:

[0104] For example, let the overlapping cell set of the i-th grade and the i+1-th grade be U i,i+1 Based on the texture vector composed of the texture features of any two cells in the overlapping cell set U i,i+1 Determine the texture feature overlap between the i-th grade and the i+1-th grade:

[0105]

[0106] wherein: β i,i+1 represents the texture feature overlap degree between the i-th level and the i+1-th level; s(k,m) represents the overlapping cell set is U i,i+1 the mean square error distance of the texture vector composed of the texture features of cell k and cell m in the i-th level; norm represents the standard normalization function, which is used to normalize the numerical value to the range of [0, 1].

[0107] S7, using the morphological kernel feature overlap degree between adjacent levels obtained in step S5 and the texture feature overlap degree between adjacent levels obtained in step S6 to determine the overlapping feature discriminability between adjacent levels (which is the product of the morphological kernel feature overlap degree between adjacent levels and the texture feature overlap degree between adjacent levels);

[0108] For the overlapping cell set between any two adjacent levels, when the morphological kernel feature overlap degree between adjacent levels and the texture feature overlap degree between adjacent levels are both high, it can be considered that the overlapping feature discriminability between adjacent levels of the cells in the overlapping cell set is low, and the original network feature components need to be enlarged.

[0109] For the i-th level and the i+1-th level, based on the corresponding morphological kernel feature overlap degree α i,i+1 and the texture feature overlap β i,i+1 , the overlapping feature discriminability between the i-th level and the i+1-th level is determined r i,i+1 , at this time r i,i+1 = α i,i+1 × β i,i+1 When the overlapping feature discriminability r i,i+1 is higher, it means that the cells in the overlapping cell set are more difficult to distinguish features, and the distribution density of the features is higher.

[0110] For the overlapping cell set between any two adjacent levels, the interval range of the network feature same component of all cells in the overlapping cell set is obtained, and the range length is enlarged according to the overlapping feature discriminability, and the original data is converted according to the original proportion, so that the feature enhancement can be realized.

[0111] S8, based on the overlapping cell set between adjacent hierarchies obtained in step S5, the overlapping cell set obtained in step S6 and the overlapping feature discriminant between adjacent hierarchies obtained in step S7, enhanced network features of each cell are obtained, specifically as follows:

[0112] The product of each component (each component value in the morphological feature and the texture feature is a scalar) in the network features (morphological nuclear features and texture features) of each cell in the overlapping cell set and the overlapping feature discriminant is determined to obtain a component increment;

[0113] The addition value of each component in the network features of each cell in the overlapping cell set and the corresponding component increment is determined to obtain an enhanced component;

[0114] Based on the enhanced components corresponding to the used components in the network features of each cell in the overlapping cell set, enhanced network features of each cell in the overlapping cell set are obtained;

[0115] For example: for the i-th hierarchy and the i+1-th hierarchy, the overlapping cell set U i,i+1 The CNN feature length of each cell in the overlapping cell set is consistent, and the length range of the k-th component is:

[0116] △ w i,i+1,k = w mx,k - w mi,k

[0117] In the formula: △ w i,i+1,k represents the initial length range of the k-th component of the network features of the cells in the overlapping cell set U i,i+1 , w mx,k , w mi,k respectively represent the maximum value and the minimum value of the k-th component in the network features of all cells in the overlapping cell set U i,i+1 .

[0118] Therefore, the length range of the k-th component after growth is:

[0119] △ w ’ i,i+1,k =△ w i,i+1,k ×(1+ r i,i+1 )

[0120] In the formula, △ w’ i,i+1,k This indicates the length range of the k-th component after enhancement; r i,i+1 This represents the overlapping feature discrimination between the i-th level and the (i+1)-th level.

[0121] For overlapping cell sets U i,i+1 The enhanced component value of the k-th component of the network feature of cell a is:

[0122] w ’ i,i+1,a,k =( w i,i+1,a,k / △ w i,i+1,k )×△ w ’ i,i+1,k

[0123] In the formula, w ’ i,i+1,a,k Represents an overlapping set of cells U i,i+1 The enhanced component of the k-th component in the network features of cell a; w i,i+1,a,k Represents an overlapping set of cells U i,i+1 The k-th component in the network features of cell a.

[0124] For overlapping cell sets U i,i+1 Each component of each cell is enhanced in the manner described above, thereby obtaining enhanced network features in areas with high overlap in morphological and textural features. This transforms the original network features with low discriminative power into features with higher discriminative power. For example, if the network features of cell A are [0.1, 0.3, 0.6] and the network features of cell B are [0.15, 0.35, 0.55], and the discriminative interval lengths of the small components when features overlap are 1, 3, 1 respectively, with a discriminative power of 0.5, then the new discriminative lengths are 1.5, 4.5, 1.5. The enhanced features of cell A are then [0.15, 0.45, 0.9], and the enhanced features of cell B are [0.225, 0.525, 0.825]. If the feature distance between the original cells A and B is calculated using a mean square function, the enhanced feature distance of the latter is clearly larger, effectively transforming the original high-density features into low-density features, facilitating subsequent discrimination.

[0125] In order to keep consistent with the dimension of the preprocessed kidney cell image, the enhanced network features of these cells need to be combined and restored to the dimension of the preprocessed kidney cell image, so that based on the restored enhanced image network features, the deep learning subtype classification model is used for cell heteromorphism recognition and subtype classification.

[0126] S9, based on the enhanced network features of each cell obtained in step S8, the enhanced image network features are restored, and the deep learning subtype classification model in step S1 is used for cell heteromorphism recognition and subtype classification, as follows:

[0127] Firstly, the enhanced network features of each cell and the area proportion of each cell in different local regions are determined;

[0128] Secondly, the enhanced network features of each local region are restored according to the area proportion relationship through the cells contained in the preprocessed kidney cell image. If the cell is completely in the local region, the enhanced network features of the cell are directly selected as the mean value summation component; if the cell is partially in the local region, the area proportion in the local region is multiplied by the enhanced network features of the cell as the mean value summation component.

[0129] Then, based on the mean value summation component determined by the enhanced network features of each cell in each local region, the enhanced network features of each local region are determined:

[0130]

[0131] In the formula: DCNN a represents the enhanced network features of the local region a in the preprocessed kidney cell image; n a represents the number of cells in the local region a; η i represents the area proportion of the i-th cell in the local region a, for example, completely in the local region a η i is 1, 30% of the area is in the local region a η i then 0.3; DCNN i represents the enhanced network features of the i-th cell in the local region a.

[0132] Finally, according to the enhanced network features of each local region, according to the relationship between each network feature and the local region in the pre-processed kidney cell image, the network features of each local region in the pre-processed kidney cell image network features are replaced by the enhanced network features of each local region, and thus the enhanced image network features (i.e. "optimized kidney cell image network features to be analyzed") are obtained.

[0133] Based on the optimized kidney cell image network features to be analyzed, the deep learning subtype classification model is used for cell atypia recognition and subtype classification, to realize accurate atypia recognition and subtype classification.

[0134] Embodiment 2

[0135] The embodiment provides a deep learning-based renal cell carcinoma atypia intelligent recognition and subtype classification device, which comprises an image feature extraction module for inputting a kidney cell image into a deep learning subtype classification model to obtain image network features extracted by the deep learning subtype classification model when the deep learning subtype classification model performs atypia recognition and subtype classification on cells in the kidney cell image.

[0136] A morphological texture feature extraction module is configured to segment cells in the kidney cell image and extract morphological nuclear features and texture features of each cell.

[0137] A grade division module is configured to divide each cell into a feature grade by using different features in the morphological nuclear features.

[0138] An overlap degree analysis module is configured to determine an overlapping cell set and a morphological nuclear feature overlap degree between adjacent grades based on differences between feature grades corresponding to the same cell.

[0139] A feature division module is configured to determine network features of each cell in the kidney cell image based on the image network features.

[0140] A feature enhancement module is configured to enhance network features of each cell in the overlapping cell set based on the morphological nuclear feature overlap degree and texture feature differences between cells in the overlapping cell set, to obtain enhanced network features.

[0141] An identification module is configured to obtain enhanced image network features based on the enhanced network features, and perform atypia recognition and subtype classification of cells based on the enhanced image network features.

[0142] Embodiment 3

[0143] The embodiment provides a deep learning-based renal cell carcinoma atypia intelligent recognition and subtype classification system, which comprises a memory and a processor.

[0144] wherein the memory is configured to store executable computer program code, the processor is configured to call and run the executable computer program code from the memory, so that the system executes the method described in Embodiment 1.

[0145] Embodiment 4

[0146] This embodiment provides a computer program product, comprising: computer program code;

[0147] When the computer program code runs on the computer, the computer executes the method described in Embodiment 1.

[0148] Embodiment 5

[0149] This embodiment provides a computer readable storage medium, which stores computer program code;

[0150] When the computer program code runs on the computer, the computer executes the method described in Embodiment 1.

[0151] The above description of the embodiments is to facilitate those skilled in the art to understand and use the invention. Those skilled in the art can easily make various modifications to these embodiments, and apply the general principles described herein to other embodiments without creative labor. Therefore, the invention is not limited to the above embodiments, and the improvements and modifications made by those skilled in the art within the scope of the invention should be within the protection scope of the invention.

Claims

1. A deep learning-based intelligent identification and subtype classification method for renal cell carcinoma atypia, characterized in that, Includes the following steps: S1. Construct a deep learning subtype classification model with a feature optimization and enhancement layer using a convolutional neural network; S2. Preprocess the standard renal cell carcinoma image to obtain a preprocessed standard renal cell carcinoma image; The preprocessed standard renal cell carcinoma images were segmented, and the morphological nuclear features and texture features of each renal cell carcinoma were extracted. S3. Using the morphological nuclear features of each renal cell carcinoma obtained in step S2, the deep learning subtype classification model in step S1 is pre-trained to obtain a pre-trained hierarchical network model based on single morphological nuclear features and a pre-trained hierarchical network model based on multiple morphological nuclear features. The deep learning subtype classification model obtained in step S1 is pre-trained using the texture features of each renal cell carcinoma obtained in step S2, resulting in a pre-trained hierarchical network model based on single texture features and a pre-trained hierarchical network model based on multiple texture features. S4. Preprocess the kidney cell images to be analyzed to obtain preprocessed kidney cell images; The preprocessed kidney cell images were segmented to extract the morphological nuclear features and texture features of each kidney cell. S5. Input the morphological nuclear features of each kidney cell in step S4 into the pre-trained hierarchical network model based on single morphological nuclear features and the pre-trained hierarchical network model based on multiple morphological nuclear features in step S3 to obtain the single morphological nuclear feature hierarchies and multiple morphological nuclear feature hierarchies of each kidney cell, and determine the overlapping cell sets between adjacent hierarchies and the degree of overlap of morphological nuclear features between adjacent hierarchies. S6. Input the texture features of each kidney cell in step S4 into the pre-trained hierarchical network model based on single texture features and the pre-trained hierarchical network model based on multiple texture features in step S3 to obtain the single texture feature hierarchy and multiple texture feature hierarchy of each kidney cell, and determine the overlapping cell set between adjacent hierarchy and the texture feature overlap between adjacent hierarchy. S7. Use the overlap of morphological kernel features between adjacent levels obtained in step S5 and the overlap of texture features between adjacent levels obtained in step S6 to determine the distinguishability of overlapping features between adjacent levels. S8. Based on the overlapping cell set between adjacent levels obtained in step S5, the overlapping cell set obtained in step S6 and the overlapping feature discrimination between adjacent levels obtained in step S7, the enhanced network features of each cell are obtained. The network features are morphological kernel features and texture features; S9. Based on the enhanced network features of each cell obtained in step S8, obtain enhanced image network features, and use the deep learning subtype classification model in step S1 to identify cell heterogeneity and classify subtypes.

2. The method for intelligent identification and subtype classification of renal cell carcinoma atypia based on deep learning according to claim 1, characterized in that, In step S1, the basic architecture of the deep learning subtype classification model includes an input layer, a feature extraction layer, a feature optimization and enhancement layer, and a classification layer.

3. The method for intelligent identification and subtype classification of renal cell carcinoma atypia based on deep learning according to claim 1, characterized in that, In step S2, the preprocessing includes color normalization, artifact removal, and image enhancement; In step S4, the preprocessing includes color normalization, artifact removal, and image enhancement.

4. The method for intelligent identification and subtype classification of renal cell carcinoma atypia based on deep learning according to claim 1, characterized in that, In step S5, the determination of the overlapping cell sets between adjacent levels is as follows: Between adjacent grades, cells with unequal morphological nuclear features in single-feature and multi-feature grades are identified as overlapping cells; based on all overlapping cells between adjacent grades, a set of overlapping cells between adjacent grades is obtained. The determination of the overlap of morphological nuclear features between adjacent grades is as follows: Between adjacent grades, the number of cells in any single-feature grade and multi-feature grade is determined to be unequal. After summing all cell numbers, the morphological feature overlap between adjacent grades is obtained through normalization.

5. The method for intelligent identification and subtype classification of renal cell carcinoma atypia based on deep learning according to claim 1, characterized in that, In step S6, the determination of the overlapping cell sets between adjacent levels is as follows: Between adjacent levels, cells with unequal texture feature single-feature levels and multi-feature levels are identified as overlapping cells; based on all overlapping cells between adjacent levels, a set of overlapping cells between adjacent levels is obtained. The determination of the overlap of texture features between adjacent levels is as follows: Based on the mean squared distance of the texture vector formed by the texture features between any two cells in the overlapping cell set between adjacent levels, the mean squared distances are accumulated and then negatively correlated and normalized to obtain the texture feature overlap between adjacent levels. In step S7, the overlapping feature discrimination between adjacent levels is the product of the morphological kernel feature overlap between adjacent levels and the texture feature overlap between adjacent levels.

6. The method for intelligent identification and subtype classification of renal cell carcinoma atypia based on deep learning according to claim 1, characterized in that, In step S8, the product of the network features of each cell in the overlapping cell set and the discriminative power of the overlapping features is determined to obtain the component increment; the sum of the network features of each cell in the overlapping cell set and the corresponding component increment is determined to obtain the enhanced component. Based on the enhanced components corresponding to the components used in the network features of each cell in the overlapping cell set, the enhanced network features of each cell in the overlapping cell set are obtained; In step S9, enhanced image network features are obtained by restoring the enhanced network features of each cell in the overlapping cell set.

7. A deep learning-based intelligent identification and subtype classification device for renal cell carcinoma atypia, characterized in that, It includes an image feature extraction module, which is used to input kidney cell images into a deep learning subtype classification model to obtain image network features extracted by the deep learning subtype classification model when performing heterogeneity identification and subtype classification of cells in the kidney cell images; The morphological and textural feature extraction module is used to segment cells in kidney cell images and extract the morphological and textural features of each cell. The grading module is used to classify each cell into different grades based on different features in the morphological nuclear features, and to obtain the feature grading of each cell. The overlap analysis module is used to determine the overlapping cell sets and morphological nuclear feature overlap between adjacent grades based on the differences between the feature grades corresponding to the same cell. The feature segmentation module is used to determine the network features of each cell in the kidney cell image based on the image network features; The feature enhancement module is used to enhance the network features of each cell in the overlapping cell set based on the overlap of the morphological kernel features and the differences in texture features between cells in the overlapping cell set, so as to obtain enhanced network features. The recognition module is used to acquire enhanced image network features based on the enhanced network features, and to perform cell heterogeneity recognition and subtype classification based on the enhanced image network features; The network features are morphological kernel features and texture features.

8. A deep learning-based intelligent identification and subtype classification system for renal cell carcinoma atypia, characterized in that, Including memory and processor; The memory is used to store executable computer program code, and the processor is used to call and run the executable computer program code from the memory, so that the system performs the method according to any one of claims 1 to 6.

9. A computer program product, characterized in that, include: Computer program code; When the computer program code is run on a computer, it causes the computer to perform the method described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program code; When the computer program code is run on a computer, it causes the computer to perform the method described in any one of claims 1 to 6.

Images