Gleason grading scoring method and device, equipment and storage medium

Abstract

The invention discloses a Gleason grading scoring method, device and equipment and a storage medium, and the method comprises the steps: collecting a pathological image of a prostate cancer patient, and forming an image data set; training a self-supervised learning model by using the image data set to form a block coding feature data set; training a gland segmentation model by using the block coding feature data set to obtain a gland segmentation image; performing secondary segmentation on each single gland in the gland segmentation image to obtain a plurality of single gland images, and forming a single gland data set; processing each single gland image in the single gland data set to obtain a plurality of gland fusion features, and forming a fusion feature data set; training a gland scoring model by using the fusion feature data set; screening the primary grading area and the secondary grading area to obtain a final grading score; and applying the trained model. According to the method, automatic Gleason grading scoring of the pathological images can be realized, the workload of pathologists is greatly reduced, and the working efficiency and the detection capability are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of pathological image processing, and in particular relates to a Gleason grading method, device, equipment and storage medium. Background Art

[0002] Prostate cancer is one of the most common malignant tumors in men, and its incidence rate is increasing year by year. The diagnosis and treatment of prostate cancer rely on the accurate analysis and grading of its pathological images, among which the Gleason Score system is currently the most widely used and recognized prostate cancer grading method. The Gleason grading classifies cancer into different grades based on the glandular structure and cell morphology characteristics by observing the stained images of prostate cancer tissue sections to evaluate the aggressiveness and prognosis of cancer.

[0003] Although it has been widely used in clinical practice, it also has some defects and limitations.

[0004] First, the traditional pathological evaluation method relies on the pathologist's microscopic observation and subjective judgment, which has a certain degree of error and subjectivity. The repeatability of Gleason grading will vary depending on the scales used by the pathologist.

[0005] Second, traditional pathological assessment methods lack automation, and pathologists must use microscopes to carefully observe the morphological characteristics and cell structure of tissue sections to provide accurate pathological assessments.

[0006] Third, in the analysis of prostate pathological sections, there may be heterogeneity between different glandular regions. However, existing research methods often fail to conduct in-depth and detailed analysis of this. This heterogeneity may lead to inconsistency in the prediction of Gleason scores, thereby affecting the accuracy of the scores.

[0007] Fourth, with the development of artificial intelligence, a series of analysis methods based on artificial intelligence technology and pathological images have begun to emerge, which to some extent make up for the shortcomings of traditional methods. However, they do not fully consider the correlation between the tissue structure of the tumor and the Gleason grade, and lack clinical interpretability.

[0008] Fifth, Gleason grading is not only related to glandular structure but may also be related to cell structure. However, existing Gleason grading methods only conduct experiments on low-resolution images or extract image blocks on high-resolution images, and do not combine global information with local information. Summary of the invention

[0009] In order to solve the above technical problems, the present invention proposes a Gleason grading method, device, equipment and storage medium.

[0010] In order to achieve the above object, the technical solution of the present invention is as follows: In a first aspect, the present invention discloses a Gleason grading method, comprising: Step S1: collecting pathological images of prostate cancer patients and their corresponding annotation information; Step S2: preprocessing the collected pathological images to form an image data set; Step S3: using the image data set to train a self-supervised learning model, the self-supervised learning model is used to divide each pathological image in the image data set into a number of image blocks, and extract block coding features of each image block to form a block coding feature data set; Step S4: using the block coding feature data set to train a gland segmentation model, the gland segmentation model is used to segment the glands in the pathological image to obtain a gland segmentation image; Step S5: performing secondary segmentation on each single gland in the gland segmentation image to obtain a number of single gland images to form a single gland data set; Step S6: sampling each single gland image in the single gland data set to obtain single gland images at different resolutions, and performing feature extraction and feature fusion on the single gland images to obtain a number of gland fusion features to form a fusion feature data set; Step S7: using the fused feature dataset to train a gland scoring model, the gland scoring model is used to predict the Gleason score of each single gland; Step S8: screening the single gland region with the largest gland area in the pathological image as the primary grading area, and the single gland region with the highest predicted Gleason score outside the primary grading area as the secondary grading area; Step S9: Adding the predicted Gleason score of the primary grading area and the predicted Gleason score of the secondary grading area to obtain a final Gleason grading score; Step S10: Apply the trained model to the pathological image to be analyzed to obtain the Gleason grading score.

[0011] Based on the above technical solution, the following improvements can be made: As a preferred solution, the preprocessing operation in step S2 includes one or more of the following: removing image noise, removing image artifacts, and data augmentation.

[0012] As a preferred solution, step S6 includes: Step S6.1: sampling any single gland image in the single gland data set to obtain corresponding single gland images at different high resolutions; Step S6.2: extracting features from the original single gland image to obtain the global features of the gland; Step S6.3: dividing the single gland image at high resolution into a number of image blocks, and extracting the local features of the gland in each image block; Step S6.4: Aggregate all local features of the gland to obtain glandular structural features; Step S6.5: Fusing the gland global feature with the gland structural feature to obtain the gland fusion feature; Step S6.6: Repeat steps S6.1 to S6.5 until each single gland image in the single gland dataset is processed to obtain the corresponding gland fusion features.

[0013] As a preferred solution, step S10 includes: Step S10.1: collecting pathological images of prostate cancer patients to be analyzed; Step S10.2: preprocessing the pathological image to be analyzed; Step S10.3: using the trained self-supervised learning model to extract block coding features of the pre-processed pathological image to be analyzed, to form a block coding feature data set; Step S10.4: Based on the block coding feature data set obtained in step S10.3, the trained gland segmentation model is used to segment the glands in the pathological image to be analyzed to obtain a gland segmentation image; Step S10.5: performing secondary segmentation on each single gland in the gland segmentation image obtained in step S10.4 to obtain a number of single gland images to form a single gland data set; Step S10.6: sampling each single gland image in the single gland data set obtained in step S10.5 to obtain single gland images at different resolutions, and performing feature extraction and feature fusion on the images to obtain a number of gland fusion features to form a fusion feature data set; Step S10.7: using the trained gland scoring model to predict the Gleason score of each single gland in the fusion feature dataset obtained in step S10.6; Step S10.8: selecting the single gland region with the largest gland area in the pathological image to be analyzed as the primary grading region, and the single gland region with the highest predicted Gleason score outside the primary grading region as the secondary grading region; Step S10.9: Add the predicted Gleason score of the primary grading area and the predicted Gleason score of the secondary grading area to obtain a final Gleason grading score.

[0014] In a second aspect, the present invention discloses a Gleason grading and scoring device, comprising: A pathological image collection module is used to collect pathological images of prostate cancer patients and their corresponding annotation information; An image preprocessing module is used to preprocess the collected pathological images to form an image data set; A self-supervised learning model training module is used to train a self-supervised learning model using an image data set, wherein the self-supervised learning model is used to divide each pathological image in the image data set into a number of image blocks, and extract block coding features of each image block to form a block coding feature data set; A gland segmentation model training module is used to train a gland segmentation model using a block coding feature data set. The gland segmentation model is used to segment the glands in the pathological image to obtain a gland segmentation image. The gland secondary segmentation module is used to perform secondary segmentation on each single gland in the gland segmentation image to obtain a number of single gland images and form a single gland data set; The gland fusion feature acquisition module is used to sample each single gland image in the single gland data set to obtain single gland images at different resolutions, and perform feature extraction and feature fusion on them to obtain a number of gland fusion features to form a fusion feature data set; The gland scoring model training module is used to train the gland scoring model using the fusion feature dataset. The gland scoring model is used to predict the Gleason score of each single gland. The gland region division module is used to select the single gland region with the largest gland area in the pathological image as the primary grading region, and the single gland region with the highest predicted Gleason score outside the primary grading region as the secondary grading region; A grading score calculation module, used to add the predicted Gleason score of the primary grading area and the predicted Gleason score of the secondary grading area to obtain a final Gleason grading score; The model application module is used to apply the trained model to the pathological image to be analyzed to obtain the Gleason grading score.

[0015] As a preferred solution, the preprocessing operations in the image preprocessing module include one or more of the following: removing image noise, removing image artifacts, and data augmentation.

[0016] As a preferred solution, the gland fusion feature acquisition module includes: An image sampling unit, used for sampling any single gland image in the single gland data set to obtain corresponding single gland images at different high resolutions; The gland global feature extraction unit is used to extract features from the original single gland image to obtain the gland global features; A gland local feature extraction unit, used for dividing a single gland image at high resolution into a plurality of image blocks, and extracting a gland local feature of each image block; A gland structure feature acquisition unit, used for aggregating all the local features of the gland to obtain the gland structure feature; A gland fusion feature acquisition unit, used for fusing the gland global feature with the gland structural feature to obtain the gland fusion feature; The repeated execution unit is used to repeatedly execute the methods in the image sampling unit, the gland global feature extraction unit, the gland local feature extraction unit, the gland structure feature acquisition unit, and the gland fusion feature acquisition unit until each single gland image in the single gland data set is processed to obtain the corresponding gland fusion feature.

[0017] As a preferred solution, the model application module includes: A collection unit is used to collect pathological images of prostate cancer patients to be analyzed; An application preprocessing unit is used to preprocess the pathological image to be analyzed; A feature extraction unit is used to extract block coding features of the pre-processed pathological image to be analyzed by using the trained self-supervised learning model to form a block coding feature data set; Applying a gland segmentation unit, for segmenting the glands in the pathological image to be analyzed using a trained gland segmentation model based on the block coding feature data set obtained by applying the feature extraction unit, to obtain a gland segmentation image; Applying a secondary segmentation unit to perform secondary segmentation on each single gland in the gland segmentation image obtained by applying the gland segmentation unit, so as to obtain a plurality of single gland images and form a single gland data set; A feature fusion unit is used to sample each single gland image in the single gland data set obtained by applying the secondary segmentation unit to obtain single gland images at different resolutions, and perform feature extraction and feature fusion on the images to obtain a number of gland fusion features to form a fusion feature data set; A gland scoring unit is used to predict the Gleason score of each single gland in the fused feature data set obtained by the feature fusion unit using the trained gland scoring model; The gland region division unit is used to select the single gland region with the largest gland area in the pathological image to be analyzed as the main grading region, and the single gland region with the highest predicted Gleason score outside the main grading region as the secondary grading region; A grading score calculation unit is applied to add the predicted Gleason score of the primary grading area and the predicted Gleason score of the secondary grading area to obtain a final Gleason grading score.

[0018] In a third aspect, the present invention discloses a computing device, comprising: one or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, and the one or more programs include instructions for any of the above-mentioned Gleason grading methods.

[0019] In a fourth aspect, the present invention discloses a storage medium storing one or more computer-readable programs, wherein the one or more programs include instructions suitable for being loaded by a memory and executing any of the above-mentioned Gleason grading methods.

[0020] The present invention discloses a Gleason grading method, device, equipment and storage medium, which have the following beneficial effects: First, the present invention adopts a self-supervised learning model as a model encoder, which improves the representation ability of pathological images, reduces the dependence on labeled data, and enhances the generalization ability of the model.

[0021] Second, the present invention trains the gland segmentation model based on block coding features, which can effectively separate the gland structure from the pathological image and effectively solve the variability problem between different gland regions in the same pathological image. Through precise gland segmentation, accurate image features are provided for subsequent grading prediction work, thereby improving the reliability and prediction accuracy of the Gleason grading score.

[0022] Third, based on multi-resolution single gland images, the present invention focuses on the global and local features of the gland, can capture the morphology between cells, increases the interpretability of subsequent Gleason grading scores, and significantly improves the prediction accuracy.

[0023] Fourth, the present invention uses glandular fusion features to train the glandular scoring model, which more accurately reflects the specific pathological characteristics at the glandular level and the cellular level, and improves the interpretability and accuracy of the scoring.

[0024] Fifth, the present invention determines the final Gleason grading score by comprehensively considering the primary grading area and the secondary grading area in the pathological image, which can comprehensively reflect the aggressiveness and heterogeneity of the tumor and provide more comprehensive and accurate pathological information for clinical decision-making.

[0025] In summary, the present invention can realize the automated Gleason grading of pathological images, which can greatly reduce the workload of pathologists, improve the work efficiency and detection ability of the pathology department, and has great clinical significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 A flow chart of a Gleason grading method provided in an embodiment of the present invention.

[0028] Figure 2 A schematic diagram of the flow of gland segmentation provided in an embodiment of the present invention.

[0029] Figure 3 A schematic diagram of a single gland scoring process provided in an embodiment of the present invention.

[0030] Figure 4 A visualization result diagram of the primary grading area and the secondary grading area provided in an embodiment of the present invention.

[0031] Figure 5 The Gleason grading confusion matrix provided by the embodiment of the present invention.

[0032] Figure 6 A block diagram of a Gleason grading and scoring device provided in an embodiment of the present invention.

[0033] Figure 7 A block diagram of a computing device provided for an embodiment of the present invention.

[0034] Among them: 201-pathological image collection module, 202-image preprocessing module, 203-self-supervised learning model training module, 204-gland segmentation model training module, 205-gland secondary segmentation module, 206-gland fusion feature acquisition module, 207-gland scoring model training module, 208-gland area division module, 209-grading scoring calculation module, 210-model application module, 301-processor, 302-memory. DETAILED DESCRIPTION

[0035] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] The expression of “comprising” an element is an “open” expression, which merely means that a corresponding component or step exists, and should not be interpreted as excluding additional components or steps.

[0038] In order to achieve the purpose of the present invention, some embodiments of the Gleason grading method, such as Figure 1 As shown, the Gleason grading method includes: Step S101: collecting pathological images of prostate cancer patients and their corresponding annotation information; Step S102: preprocessing the collected pathological images to form an image data set; Step S103: using the image data set to train a self-supervised learning model, the self-supervised learning model is used to divide each pathological image in the image data set into a number of image blocks, and extract block coding features of each image block to form a block coding feature data set; Step S104: using the block coding feature data set to train a gland segmentation model, the gland segmentation model is used to segment the glands in the pathological image to obtain a gland segmentation image; Step S105: performing secondary segmentation on each single gland in the gland segmentation image to obtain a number of single gland images to form a single gland data set; Step S106: sampling each single gland image in the single gland data set to obtain single gland images at different resolutions, and performing feature extraction and feature fusion on the single gland images to obtain a number of gland fusion features to form a fusion feature data set; Step S107: using the fused feature data set to train a gland scoring model, the gland scoring model is used to predict the Gleason score of each single gland; Step S108: screening the single gland region with the largest gland area in the pathological image as the primary grading region, and the single gland region with the highest predicted Gleason score outside the primary grading region as the secondary grading region; Step S109: Adding the predicted Gleason score of the primary grading area and the predicted Gleason score of the secondary grading area to obtain a final Gleason grading score; Step S110: Apply the trained model to the pathological image to be analyzed to obtain the Gleason grading score.

[0039] The above steps are described in detail below.

[0040] Step S101 is used to collect H&E stained pathological images (Whole Slide Images, WSI) of prostate cancer patients.

[0041] In this embodiment, the public dataset Gleason 2019 is used for experiments. The dataset contains 331 annotated pathological images. The dataset is divided into 70% training set and 30% test set, with the training set containing 231 images and the test set containing 100 images.

[0042] Step S102 is used to preprocess the collected pathological images. The preprocessing operations include but are not limited to: removing image noise, removing image artifacts, data augmentation, etc.

[0043] For example, image noise and image artifacts are removed through the threshold segmentation algorithm to reduce the impact of holes caused by cell cancer and blank areas in the tissue section preparation process on the segmentation results. The training set is then augmented through methods such as image overlapping sliding window cutting, random image rotation and flipping, which increases the amount of data while simulating the random direction of actual histological analysis to prevent model overfitting.

[0044] The augmented training set consists of 1,317 images. The augmented training set and test set form an image dataset.

[0045] like Figure 2 As shown, step S103 uses the image data set to train the self-supervised learning model to obtain a block coding feature data set.

[0046] Specifically, the training set is used to train the self-supervised learning model, and the model uses the Dinov2 model as the backbone network.

[0047] Input an image of size w×h , , w represents the image width, h represents the image height. The image is scaled by the scaling function to obtain the scaled image .

[0048]

[0049] Resize represents a scaling function. In the embodiment, the scaled image The size is 224×224.

[0050] The scaled image is then passed through a self-supervised learning model to generate block encoding features , , where: 196 represents the number of flattened image blocks and 1536 represents the feature dimension.

[0051] The transformation function T rearranges the block-encoded features V into a block-encoded feature map of size , .

[0052] The conversion function is expressed as:

[0053] Where: F represents the block coding feature map, and V represents the block coding feature.

[0054] The model with the best performance on the test set is saved, and the saved best model is used to extract the block coding features of the image dataset to obtain the block coding feature map.

[0055] Compared with traditional image color features, block coding feature maps aggregate image details through neural networks and have better expressiveness. Compared with classic neural networks such as Inception v3 and VGG16, block coding feature maps retain the spatial information of the image and add an attention mechanism within the network. They have stronger expressiveness and interpretability, which is helpful for segmentation model training.

[0056] For step S103, the present invention improves the representation ability of pathological images through self-supervised learning, reduces the dependence on labeled data, and enhances the generalization ability of the model. The block coding features contain image location information, detailed information and even rich information inside the cell, which is helpful for segmentation model training. Self-supervised learning can learn useful feature representations from unlabeled data, which can help the model learn effective features even with only a small amount of labeled data.

[0057] Step S104 uses the block encoding feature data set to train the gland segmentation model.

[0058] The glands in the training set are labeled, and the block encoding feature maps of the training set are used to train the gland segmentation model. The gland segmentation model consists of two core parts: downsampling (encoder) and upsampling (decoder).

[0059] The downsampling part mainly consists of a convolutional neural network (CNN), where each convolution layer is followed by an activation function (such as ReLU) and a batch normalization layer, which is responsible for extracting image features.

[0060] The convolutional layer can be expressed as:

[0061] in: is the input feature map, W is the weight of the convolution kernel, b is the bias term, RELU represents the activation function, represents the ratio normalization function, Represents the convolution function.

[0062] The upsampling part adopts a deconvolution structure with skip connections to preserve image details.

[0063] The final output layer can be expressed as follows:

[0064] Where: x is the output of the last layer of the decoder, W is the weight of the convolution kernel, is a sigmoid activation function used to convert the output into a probability value. Finally, the feature map is converted into an image probability of size 224×224, where the value of each pixel represents the probability of belonging to the gland category.

[0065] During the training process, the gradient is updated through DiceLoss, and the formula is as follows:

[0066] in, and They represent the label value and predicted value of pixel i in the segmentation task respectively, and N is the total number of pixels.

[0067] Evaluate on the test set to select the best gland segmentation model.

[0068] The glands in the image dataset are segmented using the optimal gland segmentation model to obtain a gland segmentation image.

[0069] In view of the close connection between Gleason grading and glandular structure, the present invention proposes a glandular segmentation model based on block coding features and a small amount of annotated glandular data sets. The model can effectively separate the glandular structure from the pathological image through deep learning technology. The glandular segmentation model effectively solves the variability problem between different glandular regions in the same pathological image. Through precise glandular segmentation, it provides more accurate image features for subsequent grading prediction, thereby improving the reliability of Gleason score and the accuracy of prediction.

[0070] In step S105, a morphological algorithm is used to perform secondary segmentation on each single gland in the gland segmentation image to obtain a number of single gland images, forming a new data set, namely, a single gland data set.

[0071] The single gland dataset contains a total of 2678 single gland images, divided into training set and test set. The training set has 2362 single gland images, and the test set has 316 single gland images.

[0072] like Figure 3 As shown, step S106 includes: Step S106.1: sampling any single gland image in the single gland data set to obtain corresponding single gland images at different high resolutions; Step S106.2: extracting features from the original single gland image to obtain the global features of the gland; Step S106.3: Divide the single gland image at high resolution into a number of image blocks, and extract the local features of the gland in each image block; Step S106.4: Aggregate all local features of the gland to obtain glandular structural features; Step S106.5: Fusing the gland global feature with the gland structural feature to obtain the gland fusion feature; Step S106.6: Repeat steps S106.1 to S106.5 until each single gland image in the single gland dataset is processed to obtain the corresponding gland fusion features.

[0073] Specifically, the coordinates of a single gland in a single gland image are converted or mapped to a coordinate system at a high magnification, and the coordinates of the gland in a low magnification image are converted to a coordinate system at a high magnification. The magnification of the high-magnification image relative to the low-magnification image is expressed by The coordinates in the high-magnification image can be obtained by the following formula:

[0074]

[0075] in: , They represent the corresponding coordinates in the high-magnification image.

[0076] In this embodiment, single gland images at 10× and 20× magnifications are obtained respectively. Then, each single gland image at high magnification is divided into image blocks, and the size of each image block is defined as 224×224 pixels to ensure that the image block contains enough cell structure information.

[0077] For glands of different sizes, calculate the number of image blocks that can be segmented at high magnification. Suppose the size of the gland is (at low magnification), the size of each image block is Pixels. The magnification is , the size of the gland at high magnification becomes .

[0078] The number of image blocks It can be calculated by the following formula:

[0079]

[0080] in: Indicates rounding down. Indicates the number of horizontal image blocks, Indicates the number of vertical image blocks.

[0081] The total number of image blocks for:

[0082] In this embodiment, 55 image blocks at a 10× magnification and 220 image blocks at a 20× magnification are obtained based on the single gland image.

[0083] Furthermore, feature extraction is performed on the original single gland image to obtain the global features of the gland , , the global features of the gland are used to explain the overall structure of the gland.

[0084] Feature extraction is performed on each image block of a single gland image at high magnification to obtain the local features of the gland , , the local features of the gland are used to explain the local features of the glandular cell structure. It is worth noting that here, the self-supervised learning model as in step S103 can be used but is not limited to be used to extract the corresponding image features.

[0085] Use aggregation algorithm to analyze local features of glands Aggregation is performed, and the aggregation algorithm may include group statistics and feature engineering-based methods. In this embodiment, taking the mean algorithm as an example, the specific formula is as follows:

[0086] in: It represents the glandular structure features of 196×1536 dimensions obtained by aggregating k×196×1536 features through the feature aggregation algorithm.

[0087] Finally, the gland global features are fused with the gland structural features to obtain the gland fusion features. , , which can be expressed as:

[0088] Step S107 uses the fused feature data set to train a gland scoring model.

[0089] The present invention divides the growth pattern of prostate cancer tissue into 5 levels, as follows: 1) Gleason grade 1: Cancer cells are evenly distributed and present a small glandular structure, similar to normal prostate tissue.

[0090] 2) Gleason grade 2: Cancer cells still maintain glandular structure, but are densely distributed and slightly uneven in size.

[0091] 3) Gleason grade 3: Cancer cells destroy the glandular structure and are scattered and irregular in shape.

[0092] 4) Gleason 4: Cancer cells are clustered together, the glandular structure is obviously destroyed, and there is obvious fusion between cells.

[0093] 5) Gleason 5: Cancer cells completely lose their glandular structure, grow diffusely, and the cells are extremely irregular.

[0094] Considering that Gleason grading is not only related to glandular structure, but also to cell morphology. Generally speaking, the diameter of cancer cells is about 10um, and it is difficult to capture the morphology of cells only by low-resolution glandular images. Therefore, the invention trains a glandular scoring model by combining multi-resolution glandular fusion features to predict the glandular Gleason score.

[0095] The following are the training and prediction steps for the gland scoring model: First, the glandular fusion features are processed by maximum pooling to extract key features and obtain feature vectors .

[0096] Then, the attention mechanism will be applied to the feature vector after the maximum pooling Weighting is performed to increase the model's focus on the glandular region.

[0097] set up is a weight matrix, is a bias vector, the weighted eigenvector Calculated by the following formula:

[0098] Where: σ is an activation function, such as sigmoid or softmax, which is used to convert the weighted feature vector into a probability distribution, indicating the degree of attention the model pays to each feature.

[0099] Finally, the weighted feature vector is processed using a multi-layer perceptron (MLP). Comprehensive analysis was performed to achieve prediction of glandular scores.

[0100] During the training process, glandular grades 1 to 5 were defined to represent Gleason grade 1 to Gleason grade 5.

[0101] The training parameters include: batch size is 32, learning rate is 0.001, optimizer is Adam optimizer, loss function is cross entropy loss function, and the number of training rounds is 100.

[0102]

[0103] in: represents the cross entropy loss function, Represents the output of the classification model, Y represents the true label, C represents the number of categories, and n represents the number of samples.

[0104] The present invention discloses a gland scoring model with a single gland as a basic unit, which is used to score glands in prostate pathological images. The Gleason grading method evaluates the invasiveness of prostate cancer based on the degree of differentiation of the glandular region and the degree of dispersion of cells. Therefore, by combining multi-resolution images, a specific analysis of the glandular region and even the cell morphology is achieved, which more accurately reflects the specific pathological characteristics of the glandular level and the cell level, improves the interpretability and accuracy of the scoring, and provides pathologists with more consistent and reliable scoring results.

[0105] Step S108 selects the single gland region with the largest gland area in the pathological image as the primary grading region, and the single gland region with the highest predicted Gleason score outside the primary grading region as the secondary grading region.

[0106] In this embodiment, the gland region of the pathological image is divided into three single gland regions, and the grading scores predicted by the gland scoring model are level 3, level 3, and level 4, respectively.

[0107] like Figure 4 As shown, the yellow covered part represents the area predicted to be level 3, and the red covered part represents the area predicted to be level 4.

[0108] In determining primary and secondary classification areas, the following guidelines apply: Main grading area: The single gland area with the largest gland area is selected as the main grading area. Figure 4 The middle main grading area is a single gland area with a predicted score of 3, so the main grading area is predicted to have a Gleason score of 3.

[0109] Secondary grading area: The single gland area with the highest predicted Gleason score outside the primary grading area is selected as the secondary grading area. Figure 4 The intermediate secondary grade zone is a single gland area with a predicted score of 4, so the predicted Gleason score of the secondary grade zone is 4.

[0110] Step S109 adds the predicted Gleason score of the primary grading area and the predicted Gleason score of the secondary grading area to obtain a final Gleason grading score.

[0111] In the above example, the final Gleason grading score is 7 (ie, 3+4=7), which means that the cancer progresses rapidly and more aggressive treatment is required.

[0112] Finally, the test on the test set divided by Gleason2019 achieved an accuracy of 96%. The confusion matrix of Gleason score on the test set is as follows: Figure 5 shown.

[0113] It is worth noting that steps S101 to S109 are the training phase, and step S110 applies the trained models in the application phase, which specifically includes: Step S110.1: collecting pathological images of prostate cancer patients to be analyzed; Step S110.2: pre-processing the pathological image to be analyzed; Step S110.3: using the trained self-supervised learning model to extract block coding features of the pre-processed pathological image to be analyzed, to form a block coding feature data set; Step S110.4: Based on the block coding feature data set obtained in step S110.3, the trained gland segmentation model is used to segment the glands in the pathological image to be analyzed to obtain a gland segmentation image; Step S110.5: performing secondary segmentation on each single gland in the gland segmentation image obtained in step S110.4 to obtain a number of single gland images to form a single gland data set; Step S110.6: sampling each single gland image in the single gland data set obtained in step S110.5 to obtain single gland images at different resolutions, and performing feature extraction and feature fusion on the images to obtain a number of gland fusion features to form a fusion feature data set; Step S110.7: using the trained gland scoring model to predict the Gleason score of each single gland in the fused feature data set obtained in step S110.6; Step S110.8: selecting the single gland region with the largest gland area in the pathological image to be analyzed as the primary grading region, and the single gland region with the highest predicted Gleason score outside the primary grading region as the secondary grading region; Step S110.9: Add the predicted Gleason score of the primary grading area and the predicted Gleason score of the secondary grading area to obtain a final Gleason grading score.

[0114] The present invention can effectively improve the accuracy and efficiency of Gleason grading, reduce the problem of strong subjectivity, and improve the stability and consistency of grading results. The automated image analysis and grading process significantly reduces the time and human resources required for manual evaluation, especially in areas where pathologist resources are in short supply, and can more effectively meet the growing demand for prostate cancer diagnosis.

[0115] In other embodiments, Figure 6As shown, the present invention discloses a Gleason grading and scoring device, comprising: Pathological image collection module 201, used to collect pathological images of prostate cancer patients and their corresponding annotation information; An image preprocessing module 202 is used to preprocess the collected pathological images to form an image data set; A self-supervised learning model training module 203 is used to train a self-supervised learning model using an image data set, wherein the self-supervised learning model is used to divide each pathological image in the image data set into a plurality of image blocks, and extract block coding features of each image block to form a block coding feature data set; The gland segmentation model training module 204 is used to train the gland segmentation model using the block coding feature data set, and the gland segmentation model is used to segment the glands in the pathological image to obtain a gland segmentation image; The gland secondary segmentation module 205 is used to perform secondary segmentation on each single gland in the gland segmentation image to obtain a plurality of single gland images and form a single gland data set; The gland fusion feature acquisition module 206 is used to sample each single gland image in the single gland data set to obtain single gland images at different resolutions, and perform feature extraction and feature fusion on the single gland images to obtain a number of gland fusion features to form a fusion feature data set; A gland scoring model training module 207 is used to train a gland scoring model using the fused feature data set, and the gland scoring model is used to predict the Gleason score of each single gland; The gland region division module 208 is used to select the single gland region with the largest gland area in the pathological image as the primary grading region, and the single gland region with the highest predicted Gleason score outside the primary grading region as the secondary grading region; A grading score calculation module 209 is used to add the predicted Gleason score of the primary grading area and the predicted Gleason score of the secondary grading area to obtain a final Gleason grading score; The model application module 210 is used to apply the trained model to the pathological image to be analyzed to obtain the Gleason grading score.

[0116] Furthermore, the preprocessing operations in the image preprocessing module include one or more of the following: removing image noise, removing image artifacts, and data augmentation.

[0117] Furthermore, the gland fusion feature acquisition module includes: An image sampling unit, used for sampling any single gland image in the single gland data set to obtain corresponding single gland images at different high resolutions; The gland global feature extraction unit is used to extract features from the original single gland image to obtain the gland global features; A gland local feature extraction unit, used for dividing a single gland image at high resolution into a plurality of image blocks, and extracting a gland local feature of each image block; A gland structure feature acquisition unit, used for aggregating all the local features of the gland to obtain the gland structure feature; A gland fusion feature acquisition unit, used for fusing the gland global feature with the gland structural feature to obtain the gland fusion feature; The repeated execution unit is used to repeatedly execute the methods in the image sampling unit, the gland global feature extraction unit, the gland local feature extraction unit, the gland structure feature acquisition unit, and the gland fusion feature acquisition unit until each single gland image in the single gland data set is processed to obtain the corresponding gland fusion feature.

[0118] Furthermore, the model application module includes: A collection unit is used to collect pathological images of prostate cancer patients to be analyzed; An application preprocessing unit is used to preprocess the pathological image to be analyzed; A feature extraction unit is used to extract block coding features of the pre-processed pathological image to be analyzed by using the trained self-supervised learning model to form a block coding feature data set; Applying a gland segmentation unit, for segmenting the glands in the pathological image to be analyzed using a trained gland segmentation model based on the block coding feature data set obtained by applying the feature extraction unit, to obtain a gland segmentation image; Applying a secondary segmentation unit to perform secondary segmentation on each single gland in the gland segmentation image obtained by applying the gland segmentation unit, so as to obtain a plurality of single gland images and form a single gland data set; A feature fusion unit is used to sample each single gland image in the single gland data set obtained by applying the secondary segmentation unit to obtain single gland images at different resolutions, and perform feature extraction and feature fusion on the images to obtain a number of gland fusion features to form a fusion feature data set; A gland scoring unit is used to predict the Gleason score of each single gland in the fused feature data set obtained by the feature fusion unit using the trained gland scoring model; The gland region division unit is used to select the single gland region with the largest gland area in the pathological image to be analyzed as the main grading region, and the single gland region with the highest predicted Gleason score outside the main grading region as the secondary grading region; A grading score calculation unit is applied to add the predicted Gleason score of the primary grading area and the predicted Gleason score of the secondary grading area to obtain a final Gleason grading score.

[0119] Further, it should be noted that: when the Gleason grading and scoring device provided in the above embodiment performs Gleason grading and scoring prediction, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the Gleason grading and scoring device is divided into different functional modules to complete all or part of the functions described above.

[0120] In addition, the Gleason grading and scoring device provided in the above embodiment and the embodiment of the Gleason grading and scoring method belong to the same concept, and the specific implementation process thereof is detailed in the method embodiment and will not be repeated here.

[0121] In addition, in some other embodiments, Figure 7 As shown, the present invention also discloses a computing device, including: One or more processors 301; Memory 302; and one or more programs, wherein the one or more programs are stored in the memory 302 and configured to be executed by the one or more processors 301, and the one or more programs include instructions for the Gleason grading method disclosed in the above embodiment.

[0122] The processor 301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 301 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 301 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 301 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0123] The memory 302 may include one or more computer-readable storage media, which may be non-transitory. The memory 302 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 302 is used to store at least one instruction, which is used to be executed by the processor 301 to implement the Gleason grading method provided in the method embodiment of the present invention.

[0124] In addition, the computing device may also optionally include: a peripheral device interface and at least one peripheral device. The processor 301, the memory 302 and the peripheral device interface may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface via a bus, a signal line or a circuit board. Schematically, the peripheral devices include but are not limited to: a radio frequency circuit, a touch display screen, an audio circuit, and a power supply.

[0125] Of course, the computing device may also include fewer or more components, which is not limited in this embodiment.

[0126] In addition, in some other embodiments, the present invention further discloses a storage medium, which stores one or more computer-readable programs, and the one or more programs include instructions, and the instructions are suitable for being loaded by a memory and executing the Gleason grading method disclosed in the above embodiment.

[0127] The present invention discloses a Gleason grading method, device, equipment and storage medium, which have the following beneficial effects: First, the present invention adopts a self-supervised learning model as a model encoder, which improves the representation ability of pathological images, reduces the dependence on labeled data, and enhances the generalization ability of the model.

[0128] Second, the present invention trains the gland segmentation model based on block coding features, which can effectively separate the gland structure from the pathological image and effectively solve the variability problem between different gland regions in the same pathological image. Through precise gland segmentation, accurate image features are provided for subsequent grading prediction work, thereby improving the reliability and prediction accuracy of the Gleason grading score.

[0129] Third, based on multi-resolution single gland images, the present invention focuses on the global and local features of the gland, can capture the morphology between cells, increases the interpretability of subsequent Gleason grading scores, and significantly improves the prediction accuracy.

[0130] Fourth, the present invention uses glandular fusion features to train the glandular scoring model, which more accurately reflects the specific pathological characteristics at the glandular level and the cellular level, and improves the interpretability and accuracy of the scoring.

[0131] Fifth, the present invention determines the final Gleason grading score by comprehensively considering the primary grading area and the secondary grading area in the pathological image, which can comprehensively reflect the aggressiveness and heterogeneity of the tumor and provide more comprehensive and accurate pathological information for clinical decision-making.

[0132] In summary, the present invention can realize the automated Gleason grading of pathological images, which can greatly reduce the workload of pathologists, improve the work efficiency and detection ability of the pathology department, and has great clinical significance.

[0133] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for illustrating the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which shall fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. Gleason grading method, characterized in that: include: Step S1: Collect pathological images of prostate cancer patients and their corresponding annotation information; Step S2: preprocessing the collected pathological images to form an image data set; Step S3: using the image data set to train a self-supervised learning model, wherein the self-supervised learning model is used to divide each pathological image in the image data set into a plurality of image blocks, and extract block coding features of each image block to form a block coding feature data set; Step S4: using the block coding feature data set to train a gland segmentation model, wherein the gland segmentation model is used to segment the glands in the pathological image to obtain a gland segmentation image; Step S5: performing secondary segmentation on each single gland in the gland segmentation image to obtain a number of single gland images to form a single gland data set; Step S6: sampling each single gland image in the single gland data set to obtain single gland images at different resolutions, and performing feature extraction and feature fusion on the single gland images to obtain a number of gland fusion features to form a fusion feature data set; Step S7: training a gland scoring model using the fused feature data set, wherein the gland scoring model is used to predict the Gleason score of each single gland; Step S8: screening the single gland region with the largest gland area in the pathological image as the primary grading area, and the single gland region with the highest predicted Gleason score outside the primary grading area as the secondary grading area; Step S9: Adding the predicted Gleason score of the primary grading area and the predicted Gleason score of the secondary grading area to obtain a final Gleason grading score; Step S10: Apply the trained model to the pathological image to be analyzed to obtain the Gleason grading score.

2. The Gleason grading method according to claim 1, characterized in that: The preprocessing operation in step S2 includes one or more of the following: removing image noise, removing image artifacts, and data augmentation.

3. The Gleason grading method according to claim 1, characterized in that: The step S6 comprises: Step S6.1: sampling any single gland image in the single gland data set to obtain corresponding single gland images at different high resolutions; Step S6.2: extracting features from the original single gland image to obtain the global features of the gland; Step S6.3: dividing the single gland image at high resolution into a number of image blocks, and extracting the local features of the gland in each image block; Step S6.4: Aggregate all local features of the gland to obtain glandular structural features; Step S6.5: Fusing the gland global feature with the gland structural feature to obtain the gland fusion feature; Step S6.6: Repeat steps S6.1 to S6.5 until each single gland image in the single gland dataset is processed to obtain the corresponding gland fusion features.

4. The Gleason grading method according to claim 1, characterized in that: The step S10 comprises: Step S10.1: collecting pathological images of prostate cancer patients to be analyzed; Step S10.2: preprocessing the pathological image to be analyzed; Step S10.3: using the trained self-supervised learning model to extract block coding features of the pre-processed pathological image to be analyzed, to form a block coding feature data set; Step S10.4: Based on the block coding feature data set obtained in step S10.3, the trained gland segmentation model is used to segment the glands in the pathological image to be analyzed to obtain a gland segmentation image; Step S10.5: performing secondary segmentation on each single gland in the gland segmentation image obtained in step S10.4 to obtain a number of single gland images to form a single gland data set; Step S10.6: sampling each single gland image in the single gland data set obtained in step S10.5 to obtain single gland images at different resolutions, and performing feature extraction and feature fusion on the images to obtain a number of gland fusion features to form a fusion feature data set; Step S10.7: using the trained gland scoring model to predict the Gleason score of each single gland in the fusion feature dataset obtained in step S10.6; Step S10.8: selecting the single gland region with the largest gland area in the pathological image to be analyzed as the primary grading region, and the single gland region with the highest predicted Gleason score outside the primary grading region as the secondary grading region; Step S10.9: Add the predicted Gleason score of the primary grading area and the predicted Gleason score of the secondary grading area to obtain a final Gleason grading score.

5. Gleason grading device, characterized in that: include: A pathological image collection module is used to collect pathological images of prostate cancer patients and their corresponding annotation information; An image preprocessing module is used to preprocess the collected pathological images to form an image data set; A self-supervised learning model training module, used to train a self-supervised learning model using an image data set, wherein the self-supervised learning model is used to divide each pathological image in the image data set into a plurality of image blocks, and extract block coding features of each image block to form a block coding feature data set; A gland segmentation model training module, used to train a gland segmentation model using a block coding feature data set, wherein the gland segmentation model is used to segment the glands in the pathological image to obtain a gland segmentation image; The gland secondary segmentation module is used to perform secondary segmentation on each single gland in the gland segmentation image to obtain a number of single gland images and form a single gland data set; The gland fusion feature acquisition module is used to sample each single gland image in the single gland data set to obtain single gland images at different resolutions, and perform feature extraction and feature fusion on them to obtain a number of gland fusion features to form a fusion feature data set; A gland scoring model training module, used for training a gland scoring model using the fused feature data set, wherein the gland scoring model is used for predicting the Gleason score of each single gland; The gland region division module is used to select the single gland region with the largest gland area in the pathological image as the primary grading region, and the single gland region with the highest predicted Gleason score outside the primary grading region as the secondary grading region; A grading score calculation module, used to add the predicted Gleason score of the primary grading area and the predicted Gleason score of the secondary grading area to obtain a final Gleason grading score; The model application module is used to apply the trained model to the pathological image to be analyzed to obtain the Gleason grading score.

6. The Gleason grading device according to claim 5, characterized in that: The preprocessing operations in the image preprocessing module include one or more of the following: removing image noise, removing image artifacts, and data augmentation.

7. The Gleason grading device according to claim 5, characterized in that: The gland fusion feature acquisition module comprises: An image sampling unit, used for sampling any single gland image in the single gland data set to obtain corresponding single gland images at different high resolutions; The gland global feature extraction unit is used to extract features from the original single gland image to obtain the gland global features; A gland local feature extraction unit, used for dividing a single gland image at high resolution into a plurality of image blocks, and extracting a gland local feature of each image block; A gland structure feature acquisition unit, used for aggregating all the local features of the gland to obtain the gland structure feature; A gland fusion feature acquisition unit, used for fusing the gland global feature with the gland structural feature to obtain the gland fusion feature; The repeated execution unit is used to repeatedly execute the methods in the image sampling unit, the gland global feature extraction unit, the gland local feature extraction unit, the gland structure feature acquisition unit, and the gland fusion feature acquisition unit until each single gland image in the single gland data set is processed to obtain the corresponding gland fusion feature.

8. The Gleason grading device according to claim 5, characterized in that: The model application module includes: A collection unit is used to collect pathological images of prostate cancer patients to be analyzed; An application preprocessing unit is used to preprocess the pathological image to be analyzed; A feature extraction unit is used to extract block coding features of the pre-processed pathological image to be analyzed by using the trained self-supervised learning model to form a block coding feature data set; Applying a gland segmentation unit, for segmenting the glands in the pathological image to be analyzed using a trained gland segmentation model based on the block coding feature data set obtained by applying the feature extraction unit, to obtain a gland segmentation image; Applying a secondary segmentation unit to perform secondary segmentation on each single gland in the gland segmentation image obtained by applying the gland segmentation unit, so as to obtain a plurality of single gland images and form a single gland data set; A feature fusion unit is used to sample each single gland image in the single gland data set obtained by applying the secondary segmentation unit to obtain single gland images at different resolutions, and perform feature extraction and feature fusion on the images to obtain a number of gland fusion features to form a fusion feature data set; A gland scoring unit is used to predict the Gleason score of each single gland in the fused feature data set obtained by the feature fusion unit using the trained gland scoring model; The gland region division unit is used to select the single gland region with the largest gland area in the pathological image to be analyzed as the main grading region, and the single gland region with the highest predicted Gleason score outside the main grading region as the secondary grading region; A grading score calculation unit is applied to add the predicted Gleason score of the primary grading area and the predicted Gleason score of the secondary grading area to obtain a final Gleason grading score.

9. A computing device, characterized in that include: one or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, and the one or more programs include instructions for the Gleason grading method described in any one of claims 1 to 4.

10. A storage medium, characterized in that The storage medium stores one or more computer-readable programs, wherein the one or more programs include instructions, and the instructions are suitable for being loaded by the memory and executing the Gleason grading method described in any one of claims 1 to 4.

Images