Gleason Grading Scoring Method, Device, Equipment and Storage Medium

Abstract

The present invention discloses a Gleason grading and scoring method, device, equipment and storage medium, including: collecting pathological images of prostate cancer patients to form an image data set; training a self-supervised learning model using the image data set to form a block coding feature data set; training a gland segmentation model using the block coding feature data set to obtain gland segmentation images; performing secondary segmentation on each single gland in the gland segmentation images to obtain a number of single gland images, forming a single gland data set; processing each single gland image in the single gland data set to obtain a number of gland fusion features, forming a fusion feature data set; training a gland scoring model using the fusion feature data set; screening the main grading area and the secondary grading area to obtain the final grading score; and applying the trained model. The present invention can realize the automatic Gleason grading and scoring of pathological images, greatly reducing the workload of pathologists and improving work efficiency and detection ability.

Description

Technical Field

[0001] The present invention belongs to the technical field of pathological image processing, and particularly relates to a Gleason grading method, device, equipment and storage medium. Background Art

[0002] Prostate cancer is one of the most common malignant tumors in men, and its incidence rate increases year by year. The diagnosis and treatment of prostate cancer rely on the accurate analysis and grading of its pathological images. Among them, the Gleason scoring system is the most widely used and recognized prostate cancer grading method at present. Gleason grading observes the stained images of prostate cancer tissue sections, and classifies the cancer into different grades according to the glandular structure and cell morphological characteristics to evaluate the invasiveness and prognosis of the cancer.

[0003] Although it has been widely used clinically, there are still some defects and limitations.

[0004] First, traditional pathological assessment methods rely on the microscopic observation and subjective judgment of pathologists, resulting in certain error subjectivity. The repeatability of Gleason grading may vary due to different scales mastered by pathologists.

[0005] Second, traditional pathological assessment methods lack automation. Pathologists must rely on microscopes to carefully observe the morphological characteristics and cell structures of tissue sections, and then provide accurate pathological assessments.

[0006] Third, in the analysis of prostate pathological sections, there may be heterogeneity between different glandular regions. However, existing research methods often fail to conduct in-depth refined analysis on this. This heterogeneity may lead to inconsistencies in Gleason score prediction, thereby affecting the accuracy of the score.

[0007] Fourth, with the development of artificial intelligence, a series of analysis methods based on artificial intelligence technology and pathological images have emerged, which to a certain extent make up for the deficiencies of traditional methods. However, they do not fully consider the correlation between the tumor tissue structure and Gleason grading, and lack clinical interpretability.

[0008] Fifth, Gleason grading is not only related to glandular structure but may also be related to cell structure. However, existing Gleason grading methods only extract image patches on low-resolution images or high-resolution images for experiments, and do not combine global information and local information. Summary of the Invention

[0009] In order to solve the above technical problems, the present invention proposes a Gleason grading method, device, equipment and storage medium.

[0010] To achieve the above object, the technical solution of the present invention is as follows:

[0011] In a first aspect, the present invention discloses a Gleason grading and scoring method, including:

[0012] Step S1: Collect the pathological images of prostate cancer patients and their corresponding annotation information;

[0013] Step S2: Preprocess the collected pathological images to form an image data set;

[0014] Step S3: Use the image data set to train a self-supervised learning model, which is used to divide each pathological image in the image data set into several image patches, and extract the patch coding features of each image patch to form a patch coding feature data set;

[0015] Step S4: Use the patch coding feature data set to train a gland segmentation model, which is used to segment the glands in the pathological image to obtain a gland segmentation image;

[0016] Step S5: Perform secondary segmentation on each single gland in the gland segmentation image to obtain several single gland images, forming a single gland data set;

[0017] Step S6: Sample each single gland image in the single gland data set to obtain single gland images at different resolutions, and perform feature extraction and feature fusion on them to obtain several gland fusion features, forming a fusion feature data set;

[0018] Step S7: Use the fusion feature data set to train a gland scoring model, which is used to predict the Gleason score of each single gland;

[0019] Step S8: Screen the single gland region with the largest gland area in the pathological image as the main grading area, and the single gland region with the highest predicted Gleason score outside the main grading area as the secondary grading area;

[0020] Step S9: Add the predicted Gleason score of the main grading area and the predicted Gleason score of the secondary grading area to obtain the final Gleason grading score;

[0021] Step S10: Apply the trained model to the pathological image to be analyzed to obtain the Gleason grading score.

[0022] Based on the above technical solution, the following improvements can also be made:

[0023] As a preferred solution, the preprocessing operation in step S2 includes one or more of the following: removing image noise, removing image artifacts, and data augmentation.

[0024] As a preferred solution, step S6 includes:

[0025] Step S6.1: Sample any single-gland image in the single-gland dataset to obtain corresponding single-gland images at different high resolutions;

[0026] Step S6.2: Extract features from the original single-gland image to obtain global gland features;

[0027] Step S6.3: Divide the single-gland image at high resolution into several image patches, and extract local gland features of each image patch;

[0028] Step S6.4: Aggregate all local gland features to obtain gland structure features;

[0029] Step S6.5: Fuse the global gland features and the gland structure features to obtain gland fusion features;

[0030] Step S6.6: Repeat steps S6.1 - S6.5 until each single-gland image in the single-gland dataset is processed to obtain corresponding gland fusion features.

[0031] As a preferred solution, step S10 includes:

[0032] Step S10.1: Collect pathological images of prostate cancer patients to be analyzed;

[0033] Step S10.2: Preprocess the pathological images to be analyzed;

[0034] Step S10.3: Use the trained self-supervised learning model to extract patch coding features of the preprocessed pathological images to be analyzed, forming a patch coding feature dataset;

[0035] Step S10.4: Based on the patch coding feature dataset obtained in step S10.3, use the trained gland segmentation model to segment the glands in the pathological images to be analyzed, obtaining gland segmentation images;

[0036] Step S10.5: Perform secondary segmentation on each single gland in the gland segmentation images obtained in step S10.4 to obtain several single-gland images, forming a single-gland dataset;

[0037] Step S10.6: Sample each single-gland image in the single-gland dataset obtained in step S10.5 to obtain single-gland images at different resolutions, and perform feature extraction and feature fusion on them to obtain several gland fusion features, forming a fusion feature dataset;

[0038] Step S10.7: Use the trained gland scoring model to predict the Gleason score of each single gland in the fused feature dataset obtained in step S10.6;

[0039] Step S10.8: Screen the single gland region with the largest gland area in the pathological image to be analyzed as the main grading area, and the single gland region with the highest predicted Gleason score outside the main grading area as the secondary grading area;

[0040] Step S10.9: Add the predicted Gleason score of the main grading area and the predicted Gleason score of the secondary grading area to obtain the final Gleason grading score.

[0041] In a second aspect, the present invention discloses a Gleason grading score device, including:

[0042] A pathological image collection module, configured to collect pathological images of prostate cancer patients and their corresponding annotation information;

[0043] An image preprocessing module, configured to preprocess the collected pathological images to form an image dataset;

[0044] A self-supervised learning model training module, configured to train a self-supervised learning model using the image dataset. The self-supervised learning model is used to divide each pathological image in the image dataset into several image patches, and extract the patch coding features of each image patch to form a patch coding feature dataset;

[0045] A gland segmentation model training module, configured to train a gland segmentation model using the patch coding feature dataset. The gland segmentation model is used to segment the glands in the pathological image to obtain a gland segmentation image;

[0046] A gland secondary segmentation module, configured to perform secondary segmentation on each single gland in the gland segmentation image to obtain several single gland images, forming a single gland dataset;

[0047] A gland fusion feature acquisition module, configured to sample each single gland image in the single gland dataset to obtain single gland images at different resolutions, and perform feature extraction and feature fusion on them to obtain several gland fusion features, forming a fused feature dataset;

[0048] A gland scoring model training module, configured to train a gland scoring model using the fused feature dataset. The gland scoring model is used to predict the Gleason score of each single gland;

[0049] A gland region division module, configured to screen the single gland region with the largest gland area in the pathological image as the main grading area, and the single gland region with the highest predicted Gleason score outside the main grading area as the secondary grading area;

[0050] A grading score calculation module, which is used to add the predicted Gleason score of the main grading area and the predicted Gleason score of the secondary grading area to obtain the final Gleason grading score;

[0051] A model application module, which is used to apply the trained model to the pathological image to be analyzed to obtain the Gleason grading score.

[0052] As a preferred solution, the preprocessing operations in the image preprocessing module include one or more of the following: removing image noise, removing image artifacts, and data augmentation.

[0053] As a preferred solution, the gland fusion feature acquisition module includes:

[0054] An image sampling unit, which is used to sample any single gland image in the single gland data set to obtain the corresponding single gland images at different high resolutions;

[0055] A gland global feature extraction unit, which is used to extract features from the original single gland image to obtain gland global features;

[0056] A gland local feature extraction unit, which is used to divide the single gland image at high resolution into several image blocks and extract the gland local features of each image block;

[0057] A gland structure feature acquisition unit, which is used to aggregate all the gland local features to obtain gland structure features;

[0058] A gland fusion feature acquisition unit, which is used to fuse the gland global features and the gland structure features to obtain gland fusion features;

[0059] A repeated execution unit, which is used to repeatedly execute the methods in the image sampling unit, the gland global feature extraction unit, the gland local feature extraction unit, the gland structure feature acquisition unit, and the gland fusion feature acquisition unit until each single gland image in the single gland data set is processed to obtain the corresponding gland fusion features.

[0060] As a preferred solution, the model application module includes:

[0061] An application collection unit, which is used to collect the pathological images of prostate cancer patients to be analyzed;

[0062] An application preprocessing unit, which is used to preprocess the pathological images to be analyzed;

[0063] An application feature extraction unit, which is used to extract the block coding features of the preprocessed pathological images to be analyzed by using the trained self-supervised learning model to form a block coding feature data set;

[0064] An application gland segmentation unit is used to segment the glands in the pathological image to be analyzed based on the block coding feature dataset obtained by the application feature extraction unit by using the trained gland segmentation model, so as to obtain a gland segmentation image;

[0065] An application secondary segmentation unit is used to perform secondary segmentation on each single gland in the gland segmentation image obtained by the application gland segmentation unit to obtain a number of single gland images, forming a single gland dataset;

[0066] An application feature fusion unit is used to sample each single gland image in the single gland dataset obtained by the application secondary segmentation unit to obtain single gland images at different resolutions, and perform feature extraction and feature fusion on them to obtain a number of gland fusion features, forming a fusion feature dataset;

[0067] An application gland scoring unit is used to predict the Gleason score of each single gland in the fusion feature dataset obtained by the application feature fusion unit by using the trained gland scoring model;

[0068] An application gland region division unit is used to screen the single gland region with the largest gland area in the pathological image to be analyzed as the main grading area, and the single gland region with the highest predicted Gleason score outside the main grading area as the secondary grading area;

[0069] An application grading score calculation unit is used to add the predicted Gleason score of the main grading area and the predicted Gleason score of the secondary grading area to obtain the final Gleason grading score.

[0070] In a third aspect, the present invention discloses a computing device, including:

[0071] One or more processors;

[0072] A memory;

[0073] And one or more programs, where one or more programs are stored in the memory and are configured to be executed by one or more processors, and one or more programs include instructions for any of the above Gleason grading score methods.

[0074] In a fourth aspect, the present invention discloses a storage medium, and the storage medium stores one or more computer-readable programs, and one or more programs include instructions, and the instructions are adapted to be loaded and executed by the memory for any of the above Gleason grading score methods.

[0075] The present invention discloses a Gleason grading score method, device, device and storage medium, which have the following beneficial effects:

[0076] First, the present invention uses a self-supervised learning model as the model encoder, which improves the representation ability of pathological images, reduces the dependence on labeled data, and enhances the generalization ability of the model.

[0077] Second, the present invention trains a gland segmentation model based on block-encoded features, which can effectively separate glandular structures from pathological images, can effectively solve the variability problem existing between different glandular regions in the same pathological image, and provides accurate image features for subsequent grading prediction work through precise gland segmentation, improving the reliability and prediction accuracy of Gleason grading scores.

[0078] Third, the present invention focuses on the global and local features of glands based on multi-resolution single-gland images, can capture the morphology between cells, increases the interpretability of subsequent Gleason grading scores, and significantly improves the prediction accuracy.

[0079] Fourth, the present invention trains a gland scoring model using gland fusion features, which can more accurately reflect the specific pathological features of gland levels and cell levels, improving the interpretability and accuracy of scoring.

[0080] Fifth, the present invention determines the final Gleason grading score by comprehensively considering the main grading region and the secondary grading region in the pathological image, which can comprehensively reflect the invasiveness and heterogeneity of the tumor, and provides more comprehensive and accurate pathological information for clinical decision-making.

[0081] In summary, the present invention can achieve automatic Gleason grading scoring of pathological images, can greatly reduce the workload of pathologists, improve the work efficiency and detection ability of the pathology department, and has great clinical significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0083] Figure 1 It is a flowchart of the Gleason grading scoring method provided by the embodiment of the present invention.

[0084] Figure 2 It is a schematic flowchart of gland segmentation provided by the embodiment of the present invention.

[0085] Figure 3 It is a schematic flowchart of the scoring of a single gland provided by the embodiment of the present invention.

[0086] Figure 4 Visualization result diagram of the main grading area and the secondary grading area provided by the embodiment of the present invention.

[0087] Figure 5 Gleason grading confusion matrix provided by the embodiment of the present invention.

[0088] Figure 6 Block diagram of the Gleason grading scoring device provided by the embodiment of the present invention.

[0089] Figure 7 Block diagram of the computing device provided by the embodiment of the present invention.

[0090] Wherein: 201 - pathological image collection module, 202 - image preprocessing module, 203 - self-supervised learning model training module, 204 - gland segmentation model training module, 205 - gland secondary segmentation module, 206 - gland fusion feature acquisition module, 207 - gland scoring model training module, 208 - gland region division module, 209 - grading score calculation module, 210 - model application module, 301 - processor, 302 - memory. Detailed implementation manners

[0091] The preferred implementation manners of the present invention will be described in detail below with reference to the accompanying drawings.

[0092] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without making creative efforts belong to the scope of protection of the present invention.

[0093] The expression of "including" an element is an "open" expression, which only means that there are corresponding components or steps, and should not be construed as excluding additional components or steps.

[0094] In order to achieve the purpose of the present invention, in some embodiments of the Gleason grading scoring method, such as Figure 1 As shown, the Gleason grading scoring method includes:

[0095] Step S101: Collect pathological images of prostate cancer patients and their corresponding annotation information;

[0096] Step S102: Preprocess the collected pathological images to form an image data set;

[0097] Step S103: Train a self-supervised learning model using an image dataset. The self-supervised learning model is used to divide each pathological image in the image dataset into several image patches, and extract the patch encoding features of each image patch to form a patch encoding feature dataset;

[0098] Step S104: Train a gland segmentation model using the patch encoding feature dataset. The gland segmentation model is used to segment the glands in the pathological image to obtain a gland segmentation image;

[0099] Step S105: Perform secondary segmentation on each single gland in the gland segmentation image to obtain several single gland images, forming a single gland dataset;

[0100] Step S106: Sample each single gland image in the single gland dataset to obtain single gland images at different resolutions, and perform feature extraction and feature fusion on them to obtain several gland fusion features, forming a fusion feature dataset;

[0101] Step S107: Train a gland scoring model using the fusion feature dataset. The gland scoring model is used to predict the Gleason score of each single gland;

[0102] Step S108: Screen the single gland region with the largest gland area in the pathological image as the main grading region, and the single gland region with the highest predicted Gleason score outside the main grading region as the secondary grading region;

[0103] Step S109: Add the predicted Gleason score of the main grading region and the predicted Gleason score of the secondary grading region to obtain the final Gleason grading score;

[0104] Step S110: Apply the trained model to the pathological image to be analyzed to obtain the Gleason grading score.

[0105] The above steps will be elaborated in detail below.

[0106] Step S101 is used to collect H&E stained pathological images (Whole Slide Images, WSI) of prostate cancer patients.

[0107] In this embodiment, the public dataset Gleason 2019 is used for experiments. This dataset contains 331 annotated pathological images. It is divided according to the ratio of 70% training set and 30% test set, where the training set contains 231 images and the test set contains 100 images.

[0108] Step S102 is used to preprocess the collected pathological images. The preprocessing operations include but are not limited to: removing image noise, removing image artifacts, data augmentation, etc.

[0109] For example, the image noise and artifacts are removed by a threshold segmentation algorithm to reduce the influence of the holes caused by cell carcinogenesis and the blank areas in the tissue section preparation process on the segmentation result. Then, data augmentation is performed on the training set by methods such as image overlapping sliding window cutting, image random rotation and flipping. While increasing the data volume, it simulates the random directions in actual histological analysis to prevent the model from overfitting.

[0110] The augmented training set contains 1317 images. The augmented training set and the test set form an image dataset.

[0111] As Figure 2 shown in

[0112] Specifically, the self-supervised learning model is trained using the training set, and the Dinov2 model is used as the backbone network of the model.

[0113] An image with a size of w×h is input , , where w represents the image width and h represents the image height. The image is scaled by a scaling function to obtain a scaled image .

[0114]

[0115] Among them, Resize represents the scaling function. In the embodiment, the size of the scaled image is 224×224.

[0116] The scaled image then passes through the self-supervised learning model to generate block encoding features , , where: 196 represents the number of flattened image blocks, and 1536 represents the feature dimension.

[0117] The transformation function T rearranges the block encoding feature V into a block encoding feature map with a size of , .

[0118] The transformation function is expressed as:

[0119]

[0120] Among them: F represents the block encoding feature map, and V represents the block encoding feature.

[0121] Save the model that performs best on the test set, and use the saved optimal model to extract the block encoding features of the image dataset to obtain a block encoding feature map.

[0122] Compared with traditional image color features, the block-encoded feature map aggregates the detailed information of the image through a neural network and has better expressive ability. Compared with classical neural networks such as Inception v3 and VGG16, the block-encoded feature map retains the spatial information of the image, adds an attention mechanism inside the network, has stronger expressive ability and interpretability, and helps the training of the segmentation model.

[0123] For this step S103, the present invention improves the representation ability of pathological images through self-supervised learning, reduces the dependence on labeled data, and enhances the generalization ability of the model. The block-encoded features contain image position information, detailed information, and even rich information inside cells, which helps the training of the segmentation model. Self-supervised learning can learn useful feature representations from unlabeled data, which can help the model learn effective features even when there is only a small amount of labeled data.

[0124] Step S104 uses the block-encoded feature dataset to train the gland segmentation model.

[0125] Label the glands in the training set, and use the block-encoded feature maps of the training set to train the gland segmentation model. The gland segmentation model includes two core parts: downsampling (encoder) and upsampling (decoder).

[0126] The downsampling part is mainly composed of a convolutional neural network (CNN). Each convolutional layer is followed by an activation function (such as ReLU) and a batch normalization layer, which is responsible for extracting image features.

[0127] The convolutional layer can be expressed as:

[0128]

[0129] Where: is the input feature map, W is the weight of the convolutional kernel, b is the bias term, RELU represents the activation function, represents the normalization function, represents the convolution function.

[0130] The upsampling part adopts a transposed convolution structure with skip connections to retain image details.

[0131] The final output layer can be expressed as follows:

[0132]

[0133] Where: x is the output of the last layer of the decoder, W is the weight of the convolutional kernel, is the sigmoid activation function, which is used to convert the output into a probability value. Finally, the feature map is converted into an image probability of size 224×224, where the value of each pixel represents the probability of belonging to the gland category.

[0134] During the training process, the gradient is updated through DiceLoss, and the formula is as follows:

[0135]

[0136] Wherein, and respectively represent the label value and the predicted value of pixel i in the segmentation task, and N is the total number of pixels.

[0137] Evaluate on the test set to select the best gland segmentation model.

[0138] Use the optimal gland segmentation model to segment the glands in the image dataset to obtain gland segmentation images.

[0139] In view of the close connection between Gleason grading and glandular structure, the present invention proposes a gland segmentation model based on block coding features and a small labeled gland dataset. Through deep learning technology, this model can effectively separate glandular structures from pathological images. The gland segmentation model effectively solves the variability problem existing between different glandular regions in the same pathological image. Through accurate gland segmentation, it provides more accurate image features for subsequent grading prediction work, thereby improving the reliability of Gleason scoring and the accuracy of prediction.

[0140] In step S105, a morphological algorithm is used to perform secondary segmentation on each single gland in the gland segmentation image to obtain a number of single gland images, forming a new dataset, namely the single gland dataset.

[0141] The single gland dataset contains a total of 2678 single gland images, which are divided into: a training set and a test set. Among them: the training set has 2362 single gland images, and the test set has 316 single gland images.

[0142] As Figure 3 shown, step S106 includes:

[0143] Step S106.1: Sample any single gland image in the single gland dataset to obtain corresponding single gland images at different high resolutions;

[0144] Step S106.2: Extract features from the original single gland image to obtain gland global features;

[0145] Step S106.3: Divide the single gland image at high resolution into several image blocks, and extract the gland local features of each image block;

[0146] Step S106.4: Aggregate all the gland local features to obtain gland structure features;

[0147] Step S106.5: Fuse the global glandular features and the glandular structure features to obtain the fused glandular features;

[0148] Step S106.6: Repeat Step S106.1 - Step S106.5 until each single - gland image in the single - gland dataset is processed to obtain the corresponding fused glandular features.

[0149] Specifically, convert or map the coordinates of the single gland in the single - gland image to the coordinate system at high magnification. The coordinates of the gland in the low - magnification image are represented by and the magnification factor of the high - magnification image relative to the low - magnification image is represented by . The coordinates in the high - magnification image can be obtained by the following formula:

[0150]

[0151]

[0152] Where: , represent the corresponding coordinates in the high - magnification image respectively.

[0153] In this embodiment, single - gland images at 10× magnification and 20× magnification are obtained respectively. Then, each single - gland image at high magnification is divided into image patches, and the size of each image patch is defined as 224×224 pixels to ensure that the image patch contains enough cell - structure information.

[0154] For glands of different sizes, calculate the number of image patches that can be segmented at high magnification. Let the size of the gland be (at low magnification), the size of each image patch is pixels. The magnification factor is , and the size of the gland at high magnification becomes .

[0155] Then the number of image patches can be calculated by the following formula:

[0156]

[0157]

[0158] Where: represents rounding down, represents the number of horizontal image patches, represents the number of vertical image patches.

[0159] The total number of image patches is:

[0160]

[0161] In this embodiment, 55 image patches at 10× magnification and 220 image patches at 20× magnification are obtained from the single gland image.

[0162] Furthermore, feature extraction is performed on the original single gland image to obtain the global gland feature , , and the global gland feature is used to explain the overall gland structure.

[0163] Feature extraction is performed on each image patch of the single gland image at high magnification to obtain the local gland feature , , and the local gland feature is used to explain the local features of the gland cell structure. It should be noted that here, the self-supervised learning model such as in step S103 can be but is not limited to used to extract the corresponding image features.

[0164] The aggregation algorithm is used to aggregate the local gland features . The aggregation algorithm can include grouped statistics and feature engineering-based methods. In this embodiment, taking the mean algorithm as an example, the specific formula is as follows:

[0165]

[0166] Where: represents aggregating the k×196×1536 feature into the 196×1536-dimensional gland structure feature through the feature aggregation algorithm.

[0167] Finally, the global gland feature and the gland structure feature are fused to obtain the gland fusion feature , , which can be expressed as:

[0168]

[0169] Step S107 uses the fused feature dataset to train the gland scoring model.

[0170] The present invention classifies the growth patterns of prostate cancer tissues into 5 grades, specifically as follows:

[0171] 1) Gleason grade 1: The cancer cells are evenly distributed, showing a small gland structure, similar to normal prostate tissue.

[0172] 2) Gleason grade 2: The cancer cells still maintain the gland structure, but are more densely distributed and slightly uneven in size.

[0173] 3) Gleason grade 3: The cancer cells destroy the gland structure, are arranged scattered, and have irregular shapes.

[0174] 4) Gleason grade 4: Cancer cells aggregate into clusters, the glandular structure is significantly damaged, and there is obvious fusion between cells.

[0175] 5) Gleason grade 5: Cancer cells completely lose the glandular structure, grow diffusely, and the cells are extremely irregular.

[0176] Considering that the Gleason grading is not only related to the glandular structure but also associated with the cell morphology. Generally, the diameter of cancer cells is about 10um. It is difficult to capture the morphology between cells only on the glandular images with low resolution. Therefore, the invention trains a gland score model by combining the multi-resolution gland fusion features to predict the gland Gleason score.

[0177] The following are the training and prediction steps of the gland score model:

[0178] First, the gland fusion features are processed by max pooling to extract key features and obtain a feature vector .

[0179] Then, the attention mechanism will weight the feature vector after max pooling to strengthen the model's attention to the glandular area.

[0180] Let be a weight matrix, be a bias vector, and the weighted feature vector is calculated by the following formula:

[0181]

[0182] where: σ is an activation function, such as sigmoid or softmax, which is used to convert the weighted feature vector into a probability distribution representing the model's attention degree to each feature.

[0183] Finally, a multi-layer perceptron (MLP) is used to comprehensively analyze the weighted feature vector to achieve the prediction of the gland score.

[0184] During the training process, it is defined that grades 1 to 5 of the gland represent Gleason grade 1 to Gleason grade 5.

[0185] The training parameters include: the batch size is 32, the learning rate is 0.001, the optimizer is the Adam optimizer, the loss function is the cross-entropy loss function, and the number of training epochs is 100.

[0186]

[0187] Wherein: represents the cross-entropy loss function, represents the output of the hierarchical model, Y represents the true label, C represents the number of categories, and n represents the number of samples.

[0188] The present invention discloses a gland scoring model with a single gland as the basic unit for scoring glands in prostate pathological images. The Gleason grading method evaluates the invasiveness of prostate cancer based on the degree of differentiation of glandular regions and the degree of cell dispersion. Therefore, by combining multi-resolution images, specific analysis of glandular regions and even cell morphology can be achieved, more precisely reflecting the specific pathological features at the gland level and cell level, improving the interpretability and accuracy of scoring, and providing more consistent and reliable scoring results for pathologists.

[0189] In step S108, the single gland region with the largest gland area in the pathological image is selected as the main grading region, and the single gland region with the highest predicted Gleason score outside the main grading region is selected as the secondary grading region.

[0190] In this embodiment, the glandular regions of the pathological image are divided into three single gland regions, and the predicted grading scores according to the gland scoring model are 3, 3, and 4 respectively.

[0191] As Figure 4 shown, the yellow-covered part represents the region predicted as grade 3, and the red-covered part represents the region predicted as grade 4.

[0192] When determining the main grading region and the secondary grading region, the following criteria are followed:

[0193] Main grading region: Select the single gland region with the largest gland area as the main grading region. In Figure 4 the main grading region is the single gland region with a predicted score of 3, so the predicted Gleason score of the main grading region is 3.

[0194] Secondary grading region: Select the single gland region with the highest predicted Gleason score outside the main grading region as the secondary grading region. In Figure 4 the secondary grading region is the single gland region with a predicted score of 4, so the predicted Gleason score of the secondary grading region is 4.

[0195] In step S109, the predicted Gleason score of the main grading region and the predicted Gleason score of the secondary grading region are added together to obtain the final Gleason grading score.

[0196] In the above embodiment, the final Gleason grading score is 7 (i.e., 3 + 4 = 7), indicating that the cancer progresses relatively fast and more aggressive treatment is required.

[0197] Finally, a 96% accuracy rate was obtained by testing on the test set divided by Gleason2019. The confusion matrix of Gleason scores on the test set is as Figure 5 shown.

[0198] It should be noted that steps S101 - S109 are the training stage, and step S110 applies the trained models in the application stage, specifically including:

[0199] Step S110.1: Collect the pathological images of prostate cancer patients to be analyzed;

[0200] Step S110.2: Preprocess the pathological images to be analyzed;

[0201] Step S110.3: Use the trained self - supervised learning model to extract the block - encoded features of the preprocessed pathological images to be analyzed, forming a block - encoded feature dataset;

[0202] Step S110.4: Based on the block - encoded feature dataset obtained in step S110.3, use the trained gland segmentation model to segment the glands in the pathological images to be analyzed, obtaining gland segmentation images;

[0203] Step S110.5: Perform secondary segmentation on each single gland in the gland segmentation images obtained in step S110.4 to obtain several single - gland images, forming a single - gland dataset;

[0204] Step S110.6: Sample each single - gland image in the single - gland dataset obtained in step S110.5 to obtain single - gland images at different resolutions, and perform feature extraction and feature fusion on them to obtain several gland fusion features, forming a fusion feature dataset;

[0205] Step S110.7: Use the trained gland scoring model to predict the Gleason score of each single gland in the fusion feature dataset obtained in step S110.6;

[0206] Step S110.8: Screen the single - gland area with the largest gland area in the pathological images to be analyzed as the main grading area, and the single - gland area with the highest predicted Gleason score outside the main grading area as the secondary grading area;

[0207] Step S110.9: Add the predicted Gleason score of the main grading area and the predicted Gleason score of the secondary grading area to obtain the final Gleason grading score.

[0208] The present invention can effectively improve the accuracy and efficiency of Gleason grading, reduce the problem of strong subjectivity, and improve the stability and consistency of grading results. The automated image analysis and grading process significantly reduces the time and human resources required for manual evaluation, especially in areas with a shortage of pathologists, and can more effectively meet the growing demand for prostate cancer diagnosis.

[0209] In some other embodiments, as Figure 6 shown, the present invention discloses a Gleason grading scoring device, including:

[0210] A pathological image collection module 201 for collecting pathological images of prostate cancer patients and their corresponding annotation information;

[0211] An image preprocessing module 202 for preprocessing the collected pathological images to form an image data set;

[0212] A self-supervised learning model training module 203 for training a self-supervised learning model using the image data set. The self-supervised learning model is used to divide each pathological image in the image data set into several image patches and extract the patch coding features of each image patch to form a patch coding feature data set;

[0213] A gland segmentation model training module 204 for training a gland segmentation model using the patch coding feature data set. The gland segmentation model is used to segment the glands in the pathological image to obtain a gland segmentation image;

[0214] A gland secondary segmentation module 205 for performing secondary segmentation on each single gland in the gland segmentation image to obtain several single gland images and form a single gland data set;

[0215] A gland fusion feature acquisition module 206 for sampling each single gland image in the single gland data set to obtain single gland images at different resolutions, and performing feature extraction and feature fusion on them to obtain several gland fusion features and form a fusion feature data set;

[0216] A gland scoring model training module 207 for training a gland scoring model using the fusion feature data set. The gland scoring model is used to predict the Gleason score of each single gland;

[0217] A gland region division module 208 for screening the single gland region with the largest gland area in the pathological image as the main grading area, and the single gland region with the highest predicted Gleason score outside the main grading area as the secondary grading area;

[0218] The grading score calculation module 209 is used to add the predicted Gleason score of the main grading area and the predicted Gleason score of the secondary grading area to obtain the final Gleason grading score;

[0219] The model application module 210 is used to apply the trained model to the pathological image to be analyzed to obtain the Gleason grading score.

[0220] Furthermore, the preprocessing operations in the image preprocessing module include one or more of the following: removing image noise, removing image artifacts, and data augmentation.

[0221] Furthermore, the gland fusion feature acquisition module includes:

[0222] The image sampling unit is used to sample any single gland image in the single gland data set to obtain the corresponding single gland images at different high resolutions;

[0223] The gland global feature extraction unit is used to extract features from the original single gland image to obtain gland global features;

[0224] The gland local feature extraction unit is used to divide the single gland image at high resolution into several image blocks and extract the gland local features of each image block;

[0225] The gland structure feature acquisition unit is used to aggregate all the gland local features to obtain gland structure features;

[0226] The gland fusion feature acquisition unit is used to fuse the gland global features and the gland structure features to obtain gland fusion features;

[0227] The repeated execution unit is used to repeatedly execute the methods in the image sampling unit, the gland global feature extraction unit, the gland local feature extraction unit, the gland structure feature acquisition unit, and the gland fusion feature acquisition unit until each single gland image in the single gland data set is processed to obtain the corresponding gland fusion features.

[0228] Furthermore, the model application module includes:

[0229] The application collection unit is used to collect the pathological images of prostate cancer patients to be analyzed;

[0230] The application preprocessing unit is used to preprocess the pathological images to be analyzed;

[0231] The application feature extraction unit is used to extract the block coding features of the preprocessed pathological images to be analyzed by using the trained self-supervised learning model to form a block coding feature data set;

[0232] Apply a gland segmentation unit, which is used to segment the glands in the pathological image to be analyzed based on the block coding feature dataset obtained by the application feature extraction unit by using the trained gland segmentation model, so as to obtain a gland segmentation image;

[0233] Apply a secondary segmentation unit, which is used to perform secondary segmentation on each single gland in the gland segmentation image obtained by the application gland segmentation unit to obtain a number of single gland images, forming a single gland dataset;

[0234] Apply a feature fusion unit, which is used to sample each single gland image in the single gland dataset obtained by the application secondary segmentation unit to obtain single gland images at different resolutions, and perform feature extraction and feature fusion on them to obtain a number of gland fusion features, forming a fusion feature dataset;

[0235] Apply a gland scoring unit, which is used to predict the Gleason score of each single gland in the fusion feature dataset obtained by the application feature fusion unit by using the trained gland scoring model;

[0236] Apply a gland region division unit, which is used to screen the single gland region with the largest gland area in the pathological image to be analyzed as the main grading area, and the single gland region with the highest predicted Gleason score outside the main grading area as the secondary grading area;

[0237] Apply a grading score calculation unit, which is used to add the predicted Gleason score of the main grading area and the predicted Gleason score of the secondary grading area to obtain the final Gleason grading score.

[0238] Furthermore, it should be noted that: when the Gleason grading score device provided in the above embodiment performs Gleason grading score prediction, only the above division of each functional module is used for illustration. In actual application, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the Gleason grading score device is divided into different functional modules to complete all or part of the functions described above.

[0239] In addition, the Gleason grading score device provided in the above embodiment and the embodiment of the Gleason grading score method belong to the same concept. The specific implementation process can be seen in the method embodiment and will not be elaborated here.

[0240] In addition, in some other embodiments, as Figure 7 shown, the present invention also discloses a computing device, including:

[0241] One or more processors 301;

[0242] A memory 302;

[0243] and one or more programs, where the one or more programs are stored in the memory 302 and configured to be executed by the one or more processors 301, and the one or more programs include instructions of the Gleason grading scoring method disclosed in the above embodiments.

[0244] The processor 301 may include one or more processing cores, such as: a 4-core processor, an 8-core processor, etc. The processor 301 may be implemented in at least one of the following hardware forms: DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the wake state, also known as the CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 301 may be integrated with a GPU (Graphics Processing Unit), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 301 may further include an AI (Artificial Intelligence) processor, and the AI processor is used to process computational operations related to machine learning.

[0245] The memory 302 may include one or more computer-readable storage media, and the computer-readable storage media may be non-transitory. The memory 302 may further include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash storage devices. In some embodiments, the non-transitory computer-readable storage media in the memory 302 is used to store at least one instruction, and the at least one instruction is used to be executed by the processor 301 to implement the Gleason grading scoring method provided in the method embodiments of the present invention.

[0246] In addition, the computing device may optionally further include: a peripheral device interface and at least one peripheral device. The processor 301, the memory 302, and the peripheral device interface may be connected through a bus or signal lines. Each peripheral device may be connected to the peripheral device interface through a bus, signal lines, or a circuit board. Schematically, the peripheral devices include but are not limited to: a radio frequency circuit, a touch display screen, an audio circuit, and a power supply, etc.

[0247] Of course, the computing device may also include fewer or more components, and this embodiment does not limit this.

[0248] In addition, in some other embodiments, the present invention also discloses a storage medium storing one or more computer-readable programs, and the one or more programs include instructions adapted to be loaded and executed by the memory to perform the Gleason grading method disclosed in the above embodiments.

[0249] The present invention discloses a Gleason grading method, device, equipment and storage medium, which have the following beneficial effects:

[0250] First, the present invention uses a self-supervised learning model as the model encoder, which improves the representation ability of pathological images, reduces the dependence on labeled data, and enhances the generalization ability of the model.

[0251] Second, the present invention trains a gland segmentation model based on block-encoded features, which can effectively separate glandular structures from pathological images, can effectively solve the variability problem existing between different glandular regions in the same pathological image, and provides accurate image features for subsequent grading prediction work through precise gland segmentation, improving the reliability and prediction accuracy of Gleason grading.

[0252] Third, the present invention focuses on the global glandular features and local glandular features based on multi-resolution single-gland images, can capture the morphology between cells, increases the interpretability of subsequent Gleason grading, and significantly improves the prediction accuracy.

[0253] Fourth, the present invention trains a gland scoring model using gland fusion features, which more accurately reflects the specific pathological features of gland levels and cell levels, improving the interpretability and accuracy of scoring.

[0254] Fifth, the present invention determines the final Gleason grading by comprehensively considering the main grading region and the secondary grading region in the pathological image, which can comprehensively reflect the invasiveness and heterogeneity of the tumor, and provides more comprehensive and accurate pathological information for clinical decision-making.

[0255] In summary, the present invention can realize the automatic Gleason grading of pathological images, can greatly reduce the workload of pathologists, improve the work efficiency and detection ability of the pathology department, and has great clinical significance.

[0256] The foregoing has shown and described the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. What is described in the above embodiments and the specification is only to illustrate the principle of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements fall within the scope of the present invention claimed. The scope of the present invention claimed is defined by the appended claims and their equivalents.

Claims

1. The Gleason grading and scoring method, characterized in that, Including: Step S1: Collect the pathological images of prostate cancer patients and their corresponding annotation information; Step S2: Preprocess the collected pathological images to form an image dataset; Step S3: Use the image dataset to train a self-supervised learning model, which is used to divide each pathological image in the image dataset into several image patches and extract the patch encoding features of each image patch to form a patch encoding feature dataset; Step S4: Use the patch encoding feature dataset to train a gland segmentation model, which is used to segment the glands in the pathological image to obtain a gland segmentation image; Step S5: Perform secondary segmentation on each single gland in the gland segmentation image to obtain several single gland images, forming a single gland dataset; Step S6: Sample each single gland image in the single gland dataset to obtain single gland images at different resolutions, and perform feature extraction and feature fusion on them to obtain several gland fusion features, forming a fusion feature dataset; Step S7: Use the fusion feature dataset to train a gland scoring model, which is used to predict the Gleason score of each single gland; Step S8: Screen the single gland region with the largest gland area in the pathological image as the main grading area, and the single gland region with the highest predicted Gleason score outside the main grading area as the secondary grading area; Step S9: Add the predicted Gleason score of the main grading area and the predicted Gleason score of the secondary grading area to obtain the final Gleason grading score; Step S10: Apply the trained model to the pathological image to be analyzed to obtain the Gleason grading score; The step S6 includes: Step S6.1: Sample any single gland image in the single gland dataset to obtain the corresponding single gland images at different high resolutions; Step S6.2: Extract features from the original single gland image to obtain gland global features; Step S6.3: Divide the single gland image at high resolution into several image patches and extract the gland local features of each image patch; Step S6.4: Aggregate all the gland local features to obtain gland structure features; Step S6.5: Fuse the gland global features and the gland structure features to obtain gland fusion features; Step S6.6: Repeat steps S6.1 - S6.5 until each single gland image in the single gland dataset is processed to obtain the corresponding gland fusion features.

2. The Gleason grading and scoring method according to claim 1, wherein The preprocessing operations in the step S2 include one or more of the following: removing image noise, removing image artifacts, and data augmentation.

3. The Gleason grading and scoring method according to claim 1, wherein The step S10 includes: Step S10.1: Collect the pathological image of the prostate cancer patient to be analyzed; Step S10.2: Preprocess the pathological image to be analyzed; Step S10.3: Use the trained self-supervised learning model to extract the patch encoding features of the preprocessed pathological image to be analyzed, forming a patch encoding feature dataset; Step S10.4: Based on the block coding feature dataset obtained in Step S10.3, use the trained gland segmentation model to segment the glands in the pathological image to be analyzed, and obtain a gland segmentation image; Step S10.5: Perform secondary segmentation on each single gland in the gland segmentation image obtained in Step S10.4 to obtain a number of single gland images, and form a single gland dataset; Step S10.6: Sample each single gland image in the single gland dataset obtained in Step S10.5 to obtain single gland images at different resolutions, and perform feature extraction and feature fusion on them to obtain a number of gland fusion features, and form a fusion feature dataset; Step S10.7: Use the trained gland scoring model to predict the Gleason score of each single gland in the fusion feature dataset obtained in Step S10.6; Step S10.8: Screen the single gland region with the largest gland area in the pathological image to be analyzed as the main grading area, and the single gland region with the highest predicted Gleason score outside the main grading area as the secondary grading area; Step S10.9: Add the predicted Gleason score of the main grading area and the predicted Gleason score of the secondary grading area to obtain the final Gleason grading score.

4. Gleason grading and scoring device, characterized in that, Including: A pathological image collection module for collecting pathological images of prostate cancer patients and their corresponding annotation information; An image preprocessing module for preprocessing the collected pathological images to form an image dataset; A self-supervised learning model training module for training a self-supervised learning model using the image dataset. The self-supervised learning model is used to divide each pathological image in the image dataset into a number of image blocks and extract the block coding features of each image block to form a block coding feature dataset; A gland segmentation model training module for training a gland segmentation model using the block coding feature dataset. The gland segmentation model is used to segment the glands in the pathological image to obtain a gland segmentation image; A gland secondary segmentation module for performing secondary segmentation on each single gland in the gland segmentation image to obtain a number of single gland images and form a single gland dataset; A gland fusion feature acquisition module for sampling each single gland image in the single gland dataset to obtain single gland images at different resolutions, and performing feature extraction and feature fusion on them to obtain a number of gland fusion features and form a fusion feature dataset; A gland scoring model training module for training a gland scoring model using the fusion feature dataset. The gland scoring model is used to predict the Gleason score of each single gland; A gland region division module for screening the single gland region with the largest gland area in the pathological image as the main grading area, and the single gland region with the highest predicted Gleason score outside the main grading area as the secondary grading area; A grading score calculation module for adding the predicted Gleason score of the main grading area and the predicted Gleason score of the secondary grading area to obtain the final Gleason grading score; A model application module for applying the trained model to the pathological image to be analyzed to obtain a Gleason grading score; The gland fusion feature acquisition module includes: An image sampling unit for sampling any single gland image in the single gland data set to obtain corresponding single gland images at different high resolutions; A gland global feature extraction unit for extracting features from the original single gland image to obtain gland global features; A gland local feature extraction unit for dividing the single gland image at high resolution into several image blocks and extracting the gland local features of each image block; A gland structure feature acquisition unit for aggregating all the gland local features to obtain gland structure features; A gland fusion feature acquisition unit for fusing the gland global features and the gland structure features to obtain gland fusion features; A repeated execution unit for repeatedly executing the methods in the image sampling unit, the gland global feature extraction unit, the gland local feature extraction unit, the gland structure feature acquisition unit, and the gland fusion feature acquisition unit until each single gland image in the single gland data set is processed to obtain corresponding gland fusion features.

5. The Gleason grading and scoring device according to claim 4, wherein The preprocessing operations in the image preprocessing module include one or more of the following: removing image noise, removing image artifacts, and data augmentation.

6. The Gleason grading and scoring device according to claim 4, characterized in that, The model application module includes: An application collection unit for collecting the pathological images of prostate cancer patients to be analyzed; An application preprocessing unit for preprocessing the pathological images to be analyzed; An application feature extraction unit for using the trained self-supervised learning model to extract the block coding features of the preprocessed pathological images to be analyzed to form a block coding feature data set; An application gland segmentation unit for segmenting the glands in the pathological image to be analyzed based on the block coding feature data set obtained by the application feature extraction unit by using the trained gland segmentation model to obtain a gland segmentation image; An application secondary segmentation unit for performing secondary segmentation on each single gland in the gland segmentation image obtained by the application gland segmentation unit to obtain several single gland images to form a single gland data set; An application feature fusion unit for sampling each single gland image in the single gland data set obtained by the application secondary segmentation unit to obtain single gland images at different resolutions, and performing feature extraction and feature fusion on them to obtain several gland fusion features to form a fusion feature data set; An application gland scoring unit for using the trained gland scoring model to predict the Gleason score of each single gland in the fusion feature data set obtained by the application feature fusion unit; An application gland region division unit for screening the single gland region with the largest gland area in the pathological image to be analyzed as the main grading region, and the single gland region with the highest predicted Gleason score outside the main grading region as the secondary grading region; An application grading score calculation unit for adding the predicted Gleason score of the main grading region and the predicted Gleason score of the secondary grading region to obtain the final Gleason grading score.

7. A computing device, characterized in that, Including: One or more processors; A memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, and one or more of the programs include instructions for the Gleason grading scoring method according to any one of claims 1-3 above.

8. Storage medium, characterized in that, The storage medium stores one or more computer-readable programs, and one or more of the programs include instructions that are adapted to be loaded and executed by the memory for the Gleason grading scoring method according to any one of claims 1-3 above.

Images