Cancer grading prediction method and device, equipment and storage medium

Abstract

The invention discloses a cancer grading prediction method and device, equipment and a storage medium, and the method comprises the steps: collecting a pathological image of a patient and corresponding grading labeling information, carrying out the preprocessing, and dividing the image into a plurality of image blocks; training a self-supervised learning model by using the image block of each pathological image; clustering the image block coding vectors by using a clustering algorithm, and clustering the image block coding vectors of each pathological image into a plurality of category areas; carrying out feature sampling on the image block coding vectors of each category region, and carrying out feature fusion on the sampled image block coding vectors to obtain image fusion features of each pathological image; training a cancer grading model by using the image fusion features with the category region information of each pathological image, wherein the cancer grading model is used for predicting cancer grading; and applying the trained model to a to-be-analyzed pathological image, and predicting cancer classification. According to the method, cancers can be accurately graded, diseases can be found in time, and a treatment scheme can be formulated.

Description

Technical Field

[0001] The present invention belongs to the technical field of pathological image processing, and specifically relates to a cancer grading prediction method, device, equipment and storage medium. Background Art

[0002] Cervical tissue pathology diagnosis is the gold standard for confirming cervical cancer. Doctors observe the cell morphology and structure of pathological tissue sections under a microscope to determine the grade of cervical cancer. However, pathological diagnosis results are often affected by a variety of subjective factors, and the process of examining cell tissue morphology field by field is time-consuming and laborious, resulting in low efficiency and high reliance on human factors and subjective judgment. The application of artificial intelligence (AI) in cancer grade prediction is developing rapidly, mainly including data mining, machine learning, deep learning, etc. However, the existing artificial intelligence prediction methods have the following problems.

[0003] First, the existing pathological image segmentation method first uses manual annotation and Unet neural network to segment the tumor area in low-resolution pathological images. However, the tumor area of ​​cervical squamous cell carcinoma is difficult to capture in low-resolution images and cannot meet the accuracy requirements.

[0004] Second, the pixel level of fully digital pathology images ranges from tens of thousands to hundreds of thousands, making image processing difficult and time-consuming.

[0005] Third, tumor cells have acquired the ability to destroy the surrounding tissues through invasive growth. Therefore, the grade of the tumor is closely related to the degree of invasion of the tumor into the adjacent tissues. The existing grading methods do not take into account the connection between regions. Summary of the invention

[0006] In order to solve the above technical problems, the present invention proposes a cancer grading prediction method, device, equipment and storage medium.

[0007] In order to achieve the above object, the technical solution of the present invention is as follows: In a first aspect, the present invention discloses a method for predicting cancer grade, comprising: Step S1: Collect the patient's pathological images and their corresponding graded annotation information; Step S2: preprocessing each collected pathological image; Step S3: Divide each preprocessed pathological image into a number of image blocks; Step S4: using the image blocks of each pathological image obtained in step S3 to train a self-supervised learning model, the self-supervised learning model is used to divide each image block into a number of image slices, and extract image slice features of each image slice, and obtain an image block encoding vector of each image block; Step S5: clustering the image block coding vectors obtained in step S4 using a clustering algorithm, and clustering the image block coding vectors of each pathological image into a plurality of category regions; Step S6: performing feature sampling on the image block coding vectors of each category area, and performing feature fusion on the sampled image block coding vectors to obtain image fusion features of each pathological image; Step S7: using the image fusion features with category region information of each pathological image obtained in step S6 to train a cancer grading model, the cancer grading model is used to predict cancer grading; Step S8: Apply the trained model to the pathological image to be analyzed to predict the cancer grade.

[0008] Based on the above technical solution, the following improvements can be made: As a preferred solution, the preprocessing operation of step S2 includes: removing image noise, removing image artifacts, removing image background, and data augmentation.

[0009] As a preferred solution, step S6 includes: Step S6.1: feature sampling is performed on the image block coding vectors of each category region of each pathological image, and a representative image block coding vector in each category region is selected; Step S6.2: performing feature concatenation on all the image block coding vectors in each category region after sampling to obtain the regional fusion features of each category region; Step S6.3: Feature splicing of the regional fusion features of all category regions of each pathological image to obtain the image fusion features of each pathological image.

[0010] As a preferred solution, step S8 includes: Step S8.1: Collect the patient's pathological images to be analyzed; Step S8.2: preprocessing the collected pathological images to be analyzed; Step S8.3: Divide the preprocessed pathological image into a number of image blocks; Step S8.4: using the trained self-supervised learning model to divide the image block of the pathological image to be analyzed obtained in step S8.3 into a number of image slices, and extracting the image slice features of each image slice to obtain an image block encoding vector for each image block; Step S8.5: clustering the image block coding vectors obtained in step S8.4 using a clustering algorithm, and clustering the image block coding vectors of the pathological image to be analyzed into several category regions; Step S8.6: performing feature sampling on the image block coding vectors of each category area, and performing feature fusion on the sampled image block coding vectors to obtain image fusion features of the pathological image to be analyzed; Step S8.7: Based on the image fusion features of the pathological image to be analyzed obtained in step S8.6, the patient's cancer grade is predicted using the trained cancer grade model.

[0011] In a second aspect, the present invention further discloses a cancer grading prediction device, comprising: A collection module, used to collect the patient's pathological images and their corresponding graded annotation information; A preprocessing module, used for preprocessing each collected pathological image; A division module, used for dividing each preprocessed pathological image into a number of image blocks; A self-supervised learning model training module, used for training a self-supervised learning model using the image blocks of each pathological image obtained by the division module, wherein the self-supervised learning model is used for dividing each image block into a plurality of image slices, extracting image slice features of each image slice, and obtaining an image block coding vector of each image block; A clustering module, used for clustering the image block coding vectors obtained by the self-supervised learning model training module using a clustering algorithm, and clustering the image block coding vectors of each pathological image into a number of category regions; A sampling and fusion module is used to perform feature sampling on the image block coding vector of each category area, and perform feature fusion on the sampled image block coding vector to obtain image fusion features of each pathological image; A cancer grading model training module is used to train a cancer grading model using the image fusion features with category region information of each pathological image obtained by the sampling fusion module, and the cancer grading model is used to predict cancer grading; The application module is used to apply the trained model to the pathological images to be analyzed and predict the cancer grade.

[0012] As a preferred solution, the preprocessing operations in the preprocessing module include: removing image noise, removing image artifacts, removing image background, and data augmentation.

[0013] As a preferred solution, the sampling fusion module includes: A sampling unit, used for performing feature sampling on the image block coding vectors of each category region of each pathological image, and selecting representative image block coding vectors in each category region; A first fusion unit is used to perform feature splicing on all the image block coding vectors in each category area after sampling to obtain a regional fusion feature of each category area; The second fusion unit is used to perform feature splicing on the regional fusion features of all category regions of each pathological image to obtain the image fusion features of each pathological image.

[0014] As a preferred solution, the application module includes: An application collection unit is used to collect the patient's pathological images to be analyzed; An application preprocessing unit is used to preprocess the collected pathological images to be analyzed; Applying a division unit, for dividing the preprocessed pathological image into a plurality of image blocks; An application feature extraction unit is used to divide the image block of the pathological image to be analyzed obtained by the application division unit into a plurality of image slices by using the trained self-supervised learning model, and extract the image slice feature of each image slice to obtain an image block encoding vector of each image block; An application clustering unit is used to cluster the image block coding vectors obtained by the application feature extraction unit using a clustering algorithm, and cluster the image block coding vectors of the pathological image to be analyzed into a plurality of category regions; A sampling and fusion unit is used to perform feature sampling on the image block coding vector of each category area, and perform feature fusion on the sampled image block coding vector to obtain image fusion features of the pathological image to be analyzed; The application grading unit is used to predict the cancer grade of the patient by using the trained cancer grading model based on the image fusion features of the pathological image to be analyzed obtained by the application sampling fusion unit.

[0015] In a third aspect, the present invention further discloses a computing device, comprising: one or more processors; Memory; and one or more programs, wherein the one or more programs are stored in a memory and configured to be executed by one or more processors, and the one or more programs include instructions of any of the above cancer grade prediction methods.

[0016] In a fourth aspect, the present invention further discloses a storage medium storing one or more computer-readable programs, wherein the one or more programs include instructions suitable for being loaded by a memory and executing any of the above-mentioned cancer grading prediction methods.

[0017] The present invention discloses a cancer grading prediction method, device, equipment and storage medium, which have the following beneficial effects: First, the present invention utilizes self-supervised learning technology to enhance the model's ability to characterize pathological images and reduce dependence on labeled data, thereby improving the generalization ability of the model.

[0018] Second, the present invention distinguishes the main areas of the pathological image through the feature clustering algorithm, which can improve the subsequent operation speed and save time cost. The clustering algorithm can handle different staining styles or different cancer types according to the actual situation of the pathological image, which enhances the generalization performance of the algorithm.

[0019] Third, the present invention performs feature sampling and feature fusion on different features after clustering, which can avoid the situation where the prediction is biased due to the small proportion of tumor tissue.

[0020] Fourth, the present invention trains a cancer grading model by fusion features of images rich in information of different category regions. The grade of a tumor may be related to the degree of canceration of adjacent tissues. The cancer grading model based on category regions can capture subtle changes in pathological images, including cell morphology, structural heterogeneity, and characteristics of the tumor microenvironment, and takes into account the influence between different category regions and the influence within the category regions.

[0021] In summary, the present invention proposes a cancer grading prediction method based on self-supervised learning and feature clustering, which can accurately grade cancer, help to timely discover the disease and formulate a treatment plan. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 A flowchart of a cancer grade prediction method provided in an embodiment of the present invention.

[0024] Figure 2 A schematic diagram of the process of feature extraction of a self-supervised learning model provided in an embodiment of the present invention.

[0025] FIG3 (a) is an original pathological image provided by an embodiment of the present invention; FIG3( b ) is a diagram showing the clustering results provided by an embodiment of the present invention.

[0026] Figure 4 A schematic diagram of the process of cancer grading model prediction provided by an embodiment of the present invention.

[0027] Figure 5 The AUROC curve provided by the embodiment of the present invention.

[0028] Figure 6 A schematic diagram of the process flow of the application phase provided by an embodiment of the present invention.

[0029] Figure 7 A block diagram of a cancer grade prediction device provided in an embodiment of the present invention.

[0030] Figure 8 A block diagram of a computing device provided for an embodiment of the present invention.

[0031] Among them: 201-collection module, 202-preprocessing module, 203-division module, 204-self-supervised learning model training module, 205-clustering module, 206-sampling fusion module, 207-cancer grading model training module, 208-application module, 301-processor, 302-memory. DETAILED DESCRIPTION

[0032] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] Using ordinal numbers “first,” “second,” “third,” etc. to describe common objects merely indicates that different instances of similar objects are involved and is not intended to imply that the objects so described must have a given order in time, space, order, or in any other manner.

[0035] In addition, the expression of “comprising” an element is an “open” expression, which merely means that corresponding components or steps exist, and should not be interpreted as excluding additional components or steps.

[0036] In order to achieve the purpose of the present invention, in some embodiments of the cancer grade prediction method, cervical cancer is taken as an example. Figure 1 As shown, the cancer grade prediction method includes: Step S101: Collecting the patient's pathological images and their corresponding graded annotation information; Step S102: preprocessing each collected pathological image; Step S103: dividing each preprocessed pathological image into a number of image blocks; Step S104: using the image blocks of each pathological image obtained in step S103 to train a self-supervised learning model, the self-supervised learning model is used to divide each image block into a number of image slices, and extract image slice features of each image slice, to obtain an image block encoding vector of each image block; Step S105: clustering the image block coding vectors obtained in step S104 using a clustering algorithm, and clustering the image block coding vectors of each pathological image into a plurality of category regions; Step S106: performing feature sampling on the image block coding vectors of each category area, and performing feature fusion on the sampled image block coding vectors to obtain image fusion features of each pathological image; Step S107: using the image fusion features with category region information of each pathological image obtained in step S106 to train a cancer grading model, the cancer grading model is used to predict cancer grading; Step S108: Apply the trained model to the pathological image to be analyzed to predict the cancer grade.

[0037] Each of the above steps is explained in detail below.

[0038] Step S101 collects whole slide pathology images (WSI) of patients with cervical squamous cell carcinoma.

[0039] Specifically, in this embodiment, the cervical dysplasia dataset (IMP-cervix dataset) is used for experiments to construct the cervical squamous cell carcinoma dataset of the present invention. The cervical dysplasia dataset is a medical image dataset specifically used to study cervical lesions. The dataset contains 600 cervical LEEP (loop electrosurgical excision procedure) samples and surgical specimens, which are derived from the data archives of the Portuguese IMP Diagnostic Laboratory. The samples were digitized using two Leica GT450 whole slide image (WSI) scanners with a scanning magnification of 40 times and a pixel size of 0.26 microns. The samples can be specifically divided into the following four categories: 1) Non-neoplastic (NNeo, grade 0): normal cervical tissue without lesions.

[0040] 2) Low-grade squamous intraepithelial lesion (LSIL, grade 1): mild cervical lesions.

[0041] 3) High-grade squamous intraepithelial lesions (HSIL, grade 2): moderate to severe cervical lesions.

[0042] 4) Non-representative (others, category 3): includes other atypical samples.

[0043] In this example, the cervical squamous cell carcinoma dataset is divided into a training set and a test set in a ratio of 6:4, wherein the training set is used for model training, and the test set is used to evaluate the performance of the model.

[0044] like Figure 2As shown, step S102 preprocesses each collected pathological image, including but not limited to: removing image noise, removing image artifacts, removing image background, data augmentation, etc. By rotating, flipping, cropping, etc. the original pathological image, a series of new pathological images are generated, and the amount of training data for the subsequent model is increased, which can effectively expand the data set, make the model better adapt to different pathological image distributions, and improve the accuracy and stability of classification.

[0045] Define the size of layer 0 of the pathology image (high width), set the zoom ratio to B, then the size of the thumbnail is .

[0046] The threshold segmentation method is used to separate the tissue and background of the thumbnail to obtain a tissue mask image. The corresponding image blocks are subsequently exported based on the tissue mask image, which can effectively reduce redundant background image blocks.

[0047] The threshold segmentation formula is as follows:

[0048] in: Represents the tissue mask image, and R, G, and B represent the RGB three-channel values ​​of the thumbnail.

[0049] Of course, in some other embodiments, a deep learning model with a lightweight network structure, such as MobileNet or ShuffleNet, may be used for tissue segmentation.

[0050] Step S103 divides each preprocessed pathological image into a number of image tiles.

[0051] Defines the physical size of the image block ,in: is the length of the image block, is the width of the image block.

[0052] The conversion formula between physical size and pixel points is as follows:

[0053] Among them: mpp is the inherent attribute of the pathological image, Represents a pixel.

[0054] According to the tissue mask image and Export tissue image blocks. The number of image blocks exported for each pathology image varies depending on the physical size and tissue mask image.

[0055] Taking a pathological image as an example, define =256um, =256um, 1724 image blocks can be obtained, and the size of the image blocks is 512×512. express, , N represents the number of image blocks, represents the i-th image block.

[0056] Step S104 trains a self-supervised learning model.

[0057] Specifically, in this embodiment, the image blocks of the training set of the cervical squamous cell carcinoma dataset are used to train the self-supervised learning model, the Dinov2 model is used as the backbone network, and the optimal model is saved. The self-supervised learning model can efficiently extract multi-level pathological features from pathological images.

[0058] Feature extraction for self-supervised learning models includes: First, the image block is scaled by the scaling function to obtain the scaled image block , the scaled image block size is 224×224.

[0059]

[0060] Rescale represents a scaling function, which may be a method such as bilinear interpolation or nearest neighbor interpolation. In the example, bilinear interpolation is used for scaling.

[0061] Resized image blocks Input into the pre-trained model to extract features. The pre-trained model divides the image block into non-overlapping image patches, flattens each image patch into a vector, and generates the initial feature representation through mapping:

[0062] in: is the flattening function, Indicates The image slices, is the linear mapping matrix, is the bias term.

[0063] The pre-trained model considers the relative position between each image slice and the feature vector of each image slice. Perform position encoding:

[0064] in, is the position code and M is the number of image slices.

[0065] Feature vector after position encoding The input is sent to the multi-layer Transformer and the encoding vector E is obtained by aggregation in the multi-layer attention mechanism. .

[0066]

[0067] Where: E represents the image block encoding vector, L represents the feature dimension, MAS represents the multi-head attention mechanism, Indicates the number of layers of Transformer.

[0068] Self-supervised learning can extract useful feature representations from unlabeled data. Self-supervised learning models can learn effective features even with only a small amount of labeled data.

[0069] Step S105 clusters the image block encoding vectors obtained by using a clustering algorithm. The clustering algorithm may be, but is not limited to, K-Means, hierarchical clustering, DBSCAN, etc.

[0070] Specifically, in this embodiment, taking the K-Means algorithm as an example, the number of clusters k is set to 4, and for each image block encoding vector (where i=1,2,...,N), calculate its distance to each cluster center and assign it to the nearest cluster center. Use the following formula to calculate the distance:

[0071] Where: d represents the distance calculation formula, represents the j-th cluster center.

[0072] Through clustering, we can form a feature set .

[0073] in: It is a set of image block encoding vectors of 4 category areas, representing the 4 category areas of the pathological image.

[0074] Through the above clustering operation, the image blocks of the pathological image can be clustered into four category regions, each of which is represented by a set of image block encoding vectors, which can be used to further analyze and understand the structure and characteristics of the pathological image.

[0075] The present invention distinguishes the main areas of cervical squamous cell carcinoma pathological images through a feature clustering algorithm. The clustering algorithm can process different staining styles or different cancer types according to the actual situation of the pathological image, thereby enhancing the generalization performance of the algorithm.

[0076] The pixel level of all-digital pathology images is in the hundreds of thousands or even larger. Existing methods cannot meet the accuracy requirements for identifying regions of interest at low resolution. Feature clustering solves the problems of high processing difficulty and recognition accuracy of all-digital pathology images, speeds up calculations, and saves time and cost.

[0077] Figure 3 (a) is the original pathological image, and Figure 3 (b) shows the pathological images of the four category areas after clustering.

[0078] Step S106 performs feature sampling and feature fusion on the image block coding vector of each category area to obtain the image fusion feature of each pathological image. Specifically, it includes: Step S106.1: performing feature sampling on the image block coding vectors of each category region of each pathological image, and selecting representative image block coding vectors in each category region; Step S106.2: performing feature concatenation on all the image block coding vectors in each category region after sampling to obtain a regional fusion feature of each category region; Step S106.3: performing feature splicing on the regional fusion features of all category regions of each pathological image to obtain the image fusion features of each pathological image.

[0079] Specifically, in this embodiment, a random sampling strategy is adopted to obtain representative samples from the feature set F. The screening rules are set as follows:

[0080] in: represents the number of samples, Indicates the number of features in each category area, and min indicates the minimum function. Follow the above rules to obtain the sampled feature set }.

[0081] The vectors in the sampled feature set are concatenated to obtain the sampled image fusion feature F_concat. , S represents the number of features after concatenation.

[0082] The present invention randomly extracts different features after clustering to form new features, which can avoid the situation where the prediction is biased due to the small proportion of tumor tissue. Randomly extracting different regional features after clustering can ensure that the tumor tissue and other tissues are analyzed at the same quantity level, which also avoids the problem of tumor tissue screening.

[0083] The existing method extracts features from image blocks of the entire digital slice image and inputs them into the model, randomly extracts samples from different areas, and speeds up the training time of subsequent tasks. The extracted clustered sample features ensure the richness of the pathological image and avoid the problem of data omission.

[0084] like Figure 4 As shown, step S107 uses the obtained image fusion features to train a cancer grading model. The trained cancer grading model is a regression model based on different category regions, which is composed of an encoder, an attention mechanism, and a final scoring mechanism. The specific structure is as follows.

[0085] The encoder converts the image fusion feature F_concat into a low-dimensional vector:

[0086] in, represents the vector dot multiplication operation, represents the weight matrix, represents the bias term, Represents the activation function. It is the aggregate feature representation of each category area.

[0087] The attention mechanism is used to calculate the importance weights of the category regions in order to weight the instance features in the subsequent steps. It calculates the importance scores by mapping the features to smaller dimensions.

[0088] The calculation of the attention score involves three parts: linear transformation, activation function, and output layer. The linear transformation maps the input features to low dimensions, and after the linear transformation, the linear transformation is input to the activation function. The formula is as follows:

[0089] in: represents the activation function, is the linear transformation matrix, is the bias matrix, which transforms the output of the activation function Enter another linear change layer to get the final attention score .

[0090]

[0091] in: Represents the weight matrix. The attention mechanism performs masking according to the number (length) of instances to prevent weight calculation for empty instances.

[0092] Masked attention scores Normalized by the Softmax function to ensure that the sum of the weights is 1:

[0093] The weights obtained through the attention mechanism are weighted and summed up for the encoded features to generate a global weighted feature representation :

[0094] in: represents the i-th encoded feature, is the attention score corresponding to the i-th encoded feature. The output layer is used to give the final score to the weighted global weighted features to obtain the final score of cervical squamous cell carcinoma.

[0095] The output layer contains a linear layer, and the output calculation formula is:

[0096] In the above formula is the weight matrix of the output layer, is the bias term, It is the prediction score of the model, and reasonable classification rules are formulated based on the prediction score.

[0097] In this example, the score is set between 0 and 1 to represent level 0 (NNeo), between 1 and 2 to represent level 1 (LSIL), between 2 and 3 to represent level 3 (HSIL), and between 3 and 4 to represent non-representative. MSE is used as the loss function for gradient update, and the formula is as follows:

[0098] Where: n is the number of samples, Indicates The actual value of the samples, Indicates The predicted value of a sample.

[0099] Save the model after 25 iterations.

[0100] like Figure 5 As shown in the figure, in this example, AUROC is used to evaluate on the test set to select the optimal model for clinical application. AUROC (Area Under the Receiver Operating Characteristic Curve) is the area under the ROC curve. The cancer grading model finally predicted the AUROC of grade 0 to be 0.8998, the AUROC of grade 1 to be 0.8306, the AUROC of grade 2 to be 0.9066, and the AUROC of grade 3 to be 0.9766.

[0101] The present invention trains a cancer grading model by fusion features of images rich in information of different category regions. The grade of a tumor may be related to the degree of canceration of adjacent tissues. The cancer grading model based on category regions can capture subtle changes in pathological images, including cell morphology, structural heterogeneity, and characteristics of the tumor microenvironment, and takes into account the influence between different category regions and the influence within the category regions.

[0102] By using the regression algorithm to analyze and score each category of pathological images in detail, the cancer grading model can adapt to different pathological images and rating requirements, and flexibly respond to different types of pathological images by adjusting the parameters and rating rules of the regression algorithm. This adaptability makes the model not only suitable for a single pathological type, but also can be extended to a wider range of application scenarios.

[0103] like Figure 6 As shown, step S108 is an application step, including: Step S108.1: Collect the patient's pathological images to be analyzed; Step S108.2: pre-processing the collected pathological images to be analyzed; Step S108.3: Divide the preprocessed pathological image into a number of image blocks; Step S108.4: using the trained self-supervised learning model, the image block of the pathological image to be analyzed obtained in step S8.3 is divided into a number of image slices, and the image slice features of each image slice are extracted to obtain an image block encoding vector of each image block; Step S108.5: clustering the image block coding vectors obtained in step S8.4 using a clustering algorithm, and clustering the image block coding vectors of the pathological image to be analyzed into a number of category regions; Step S108.6: performing feature sampling on the image block coding vectors of each category area, and performing feature fusion on the sampled image block coding vectors to obtain image fusion features of the pathological image to be analyzed; Step S108.7: Based on the image fusion features of the pathological image to be analyzed obtained in step S108.6, the patient's cancer grade is predicted using the trained cancer grade model.

[0104] After the experiment, the patient's prediction score was 2.68 and the predicted grade was 3 (HSIL).

[0105] In addition, in some other embodiments, Figure 7 As shown, the present invention also discloses a cancer grading prediction device, comprising: A collection module 201 is used to collect the patient's pathological images and their corresponding graded annotation information; A preprocessing module 202, used for preprocessing each collected pathological image; A division module 203, used for dividing each preprocessed pathological image into a plurality of image blocks; A self-supervised learning model training module 204, used to train a self-supervised learning model using the image blocks of each pathological image obtained by the division module, wherein the self-supervised learning model is used to divide each image block into a plurality of image slices, and extract image slice features of each image slice to obtain an image block coding vector of each image block; A clustering module 205 is used to cluster the image block coding vectors obtained by the self-supervised learning model training module using a clustering algorithm, and cluster the image block coding vectors of each pathological image into a plurality of category regions; A sampling and fusion module 206 is used to perform feature sampling on the image block coding vector of each category area, and perform feature fusion on the sampled image block coding vector to obtain image fusion features of each pathological image; A cancer grading model training module 207 is used to train a cancer grading model using the image fusion features with category region information of each pathological image obtained by the sampling fusion module, and the cancer grading model is used to predict cancer grading; The application module 208 is used to apply the trained model to the pathological image to be analyzed to predict the cancer grade.

[0106] Furthermore, the preprocessing operations in the preprocessing module include: removing image noise, removing image artifacts, removing image background, and data augmentation.

[0107] Furthermore, the sampling fusion module includes: A sampling unit, used for performing feature sampling on the image block coding vectors of each category region of each pathological image, and selecting representative image block coding vectors in each category region; A first fusion unit is used to perform feature splicing on all the image block coding vectors in each category area after sampling to obtain a regional fusion feature of each category area; The second fusion unit is used to perform feature splicing on the regional fusion features of all category regions of each pathological image to obtain the image fusion features of each pathological image.

[0108] Furthermore, the application module includes: An application collection unit is used to collect the patient's pathological images to be analyzed; An application preprocessing unit is used to preprocess the collected pathological images to be analyzed; Applying a division unit, for dividing the preprocessed pathological image into a plurality of image blocks; An application feature extraction unit is used to divide the image block of the pathological image to be analyzed obtained by the application division unit into a plurality of image slices by using the trained self-supervised learning model, and extract the image slice feature of each image slice to obtain an image block encoding vector of each image block; An application clustering unit is used to cluster the image block coding vectors obtained by the application feature extraction unit using a clustering algorithm, and cluster the image block coding vectors of the pathological image to be analyzed into a plurality of category regions; A sampling and fusion unit is used to perform feature sampling on the image block coding vector of each category area, and perform feature fusion on the sampled image block coding vector to obtain image fusion features of the pathological image to be analyzed; The application grading unit is used to predict the cancer grade of the patient by using the trained cancer grading model based on the image fusion features of the pathological image to be analyzed obtained by the application sampling fusion unit.

[0109] Furthermore, it should be noted that the cancer grading prediction device provided in the above embodiment only uses the division of the above functional modules as an example when performing grading prediction. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the cancer grading prediction device can be divided into different functional modules to complete all or part of the functions described above.

[0110] In addition, the cancer grading prediction device and the cancer grading prediction method provided in the above embodiments belong to the same concept, and their specific implementation processes are detailed in the method embodiments, which will not be repeated here.

[0111] In addition, in some other embodiments, Figure 8 As shown, the present invention also discloses a computing device, including: One or more processors 301; Memory 302; and one or more programs, wherein the one or more programs are stored in the memory 302 and configured to be executed by the one or more processors 301, and the one or more programs include instructions of the cancer grade prediction method disclosed in the above embodiment.

[0112] The processor 301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 301 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 301 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 301 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0113] The memory 302 may include one or more computer-readable storage media, which may be non-transitory. The memory 302 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 302 is used to store at least one instruction, which is used to be executed by the processor 301 to implement the cancer grading prediction method provided by the method embodiment of the present invention.

[0114] In addition, the computing device may also optionally include: a peripheral device interface and at least one peripheral device. The processor 301, the memory 302 and the peripheral device interface may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface via a bus, a signal line or a circuit board. Schematically, the peripheral devices include but are not limited to: a radio frequency circuit, a touch display screen, an audio circuit, and a power supply.

[0115] Of course, the computing device may also include fewer or more components, which is not limited in this embodiment.

[0116] In addition, in some other embodiments, the present invention further discloses a storage medium, which stores one or more computer-readable programs, and the one or more programs include instructions, and the instructions are suitable for being loaded by a memory and executing the cancer grading prediction method disclosed in the above embodiment.

[0117] The present invention discloses a cancer grading prediction method, device, equipment and storage medium, which have the following beneficial effects: First, the present invention utilizes self-supervised learning technology to enhance the model's ability to characterize pathological images and reduce dependence on labeled data, thereby improving the generalization ability of the model.

[0118] Second, the present invention distinguishes the main areas of the pathological image through the feature clustering algorithm, which can improve the subsequent operation speed and save time cost. The clustering algorithm can handle different staining styles or different cancer types according to the actual situation of the pathological image, which enhances the generalization performance of the algorithm.

[0119] Third, the present invention performs feature sampling and feature fusion on different features after clustering, which can avoid the situation where the prediction is biased due to the small proportion of tumor tissue.

[0120] Fourth, the present invention trains a cancer grading model by fusion features of images rich in information of different category regions. The grade of a tumor may be related to the degree of canceration of adjacent tissues. The cancer grading model based on category regions can capture subtle changes in pathological images, including cell morphology, structural heterogeneity, and characteristics of the tumor microenvironment, and takes into account the influence between different category regions and the influence within the category regions.

[0121] In summary, the present invention proposes a cancer grading prediction method based on self-supervised learning and feature clustering, which can accurately grade cancer, help to timely discover the disease and formulate a treatment plan.

[0122] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for illustrating the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which shall fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for predicting cancer grade, characterized in that: include: Step S1: Collect the patient's pathological images and their corresponding graded annotation information; Step S2: preprocessing each collected pathological image; Step S3: Divide each preprocessed pathological image into a number of image blocks; Step S4: using the image blocks of each pathological image obtained in step S3 to train a self-supervised learning model, wherein the self-supervised learning model is used to divide each image block into a number of image slices, extract image slice features of each image slice, and obtain an image block encoding vector of each image block; Step S5: clustering the image block coding vectors obtained in step S4 using a clustering algorithm, and clustering the image block coding vectors of each pathological image into a plurality of category regions; Step S6: performing feature sampling on the image block coding vectors of each category area, and performing feature fusion on the sampled image block coding vectors to obtain image fusion features of each pathological image; Step S7: training a cancer grading model using the image fusion features with category region information of each pathological image obtained in step S6, wherein the cancer grading model is used to predict cancer grading; Step S8: Apply the trained model to the pathological image to be analyzed to predict the cancer grade.

2. The cancer grade prediction method according to claim 1, characterized in that: The preprocessing operations in step S2 include: removing image noise, removing image artifacts, removing image background, and data augmentation.

3. The cancer grade prediction method according to claim 1, characterized in that: The step S6 comprises: Step S6.1: feature sampling is performed on the image block coding vectors of each category region of each pathological image, and a representative image block coding vector in each category region is selected; Step S6.2: performing feature concatenation on all the image block coding vectors in each category region after sampling to obtain the regional fusion features of each category region; Step S6.3: Feature splicing of the regional fusion features of all category regions of each pathological image to obtain the image fusion features of each pathological image.

4. The cancer grade prediction method according to claim 1, characterized in that: The step S8 comprises: Step S8.1: Collect the patient's pathological images to be analyzed; Step S8.2: preprocessing the collected pathological images to be analyzed; Step S8.3: Divide the preprocessed pathological image into a number of image blocks; Step S8.4: using the trained self-supervised learning model to divide the image block of the pathological image to be analyzed obtained in step S8.3 into a number of image slices, and extracting the image slice features of each image slice to obtain an image block encoding vector for each image block; Step S8.5: clustering the image block coding vectors obtained in step S8.4 using a clustering algorithm, and clustering the image block coding vectors of the pathological image to be analyzed into several category regions; Step S8.6: performing feature sampling on the image block coding vectors of each category area, and performing feature fusion on the sampled image block coding vectors to obtain image fusion features of the pathological image to be analyzed; Step S8.7: Based on the image fusion features of the pathological image to be analyzed obtained in step S8.6, the patient's cancer grade is predicted using the trained cancer grade model.

5. A cancer grade prediction device, characterized in that: include: A collection module, used to collect the patient's pathological images and their corresponding graded annotation information; A preprocessing module, used for preprocessing each collected pathological image; A division module, used for dividing each preprocessed pathological image into a number of image blocks; A self-supervised learning model training module, used for training a self-supervised learning model using the image blocks of each pathological image obtained by the division module, wherein the self-supervised learning model is used for dividing each image block into a plurality of image slices, extracting image slice features of each image slice, and obtaining an image block coding vector of each image block; A clustering module, used for clustering the image block coding vectors obtained by the self-supervised learning model training module using a clustering algorithm, and clustering the image block coding vectors of each pathological image into a number of category regions; A sampling and fusion module is used to perform feature sampling on the image block coding vector of each category area, and perform feature fusion on the sampled image block coding vector to obtain image fusion features of each pathological image; A cancer grading model training module, used for training a cancer grading model using the image fusion features with category region information of each pathological image obtained by the sampling fusion module, wherein the cancer grading model is used for predicting cancer grading; The application module is used to apply the trained model to the pathological images to be analyzed and predict the cancer grade.

6. The cancer grade prediction device according to claim 5, characterized in that: The preprocessing operations in the preprocessing module include: removing image noise, removing image artifacts, removing image background, and data augmentation.

7. The cancer grade prediction device according to claim 5, characterized in that: The sampling fusion module comprises: A sampling unit, used for performing feature sampling on the image block coding vectors of each category region of each pathological image, and selecting representative image block coding vectors in each category region; A first fusion unit is used to perform feature splicing on all the image block coding vectors in each category area after sampling to obtain a regional fusion feature of each category area; The second fusion unit is used to perform feature splicing on the regional fusion features of all category regions of each pathological image to obtain the image fusion features of each pathological image.

8. The cancer grade prediction device according to claim 5, characterized in that: The application module includes: An application collection unit is used to collect the patient's pathological images to be analyzed; An application preprocessing unit is used to preprocess the collected pathological images to be analyzed; Applying a division unit, for dividing the preprocessed pathological image into a plurality of image blocks; An application feature extraction unit is used to divide the image block of the pathological image to be analyzed obtained by the application division unit into a plurality of image slices by using the trained self-supervised learning model, and extract the image slice feature of each image slice to obtain an image block encoding vector of each image block; An application clustering unit is used to cluster the image block coding vectors obtained by the application feature extraction unit using a clustering algorithm, and cluster the image block coding vectors of the pathological image to be analyzed into a plurality of category regions; A sampling and fusion unit is used to perform feature sampling on the image block coding vector of each category area, and perform feature fusion on the sampled image block coding vector to obtain image fusion features of the pathological image to be analyzed; The application grading unit is used to predict the cancer grade of the patient by using the trained cancer grading model based on the image fusion features of the pathological image to be analyzed obtained by the application sampling fusion unit.

9. A computing device, characterized in that include: one or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, and the one or more programs include instructions of the cancer grade prediction method described in any one of claims 1 to 4.

10. A storage medium, characterized in that The storage medium stores one or more computer-readable programs, wherein the one or more programs include instructions, and the instructions are suitable for being loaded by the memory and executing the cancer grading prediction method according to any one of claims 1 to 4.

Images