Digital pathological image grading system and method

Abstract

The invention provides a digital pathological image grading method and system, and the method comprises the steps: carrying out the cell recognition based on a Unet algorithm which carries out the coding, decoding and regression to the classification of pixel points with the same size as an original image, and extracting a cell part in a prostate pathological image; inputting the extracted cell part into a pre-training model and a classification algorithm based on a convolutional neural network (CNN), and detecting cancer cells contained in the cell part; and based on the detected relevant features of the cancer cells and the surrounding tissues thereof, predicting the pathological grading of the cancer cells through an algorithm of a network model CNN-RNN combining a pre-training model, a convolutional neural network and a recurrent neural network. The technical problems that the workload is large, the interpretation threshold is high, and errors and subjectivity exist in judgment of the proportion of cancer cells by naked eyes are solved.

Description

Technical Field

[0001] The present invention relates to the field of pathological slice image interpretation and diagnosis, and in particular to a digital pathological image grading method and system. Background Art

[0002] Prostate cancer is extremely harmful to men's health, and its incidence rate is increasing year by year. Pathological diagnosis is the "gold standard" in disease diagnosis. The interpretation of the pathological results of a single prostate puncture case generally requires a long time for senior pathology experts to interpret, which involves a large workload and a high threshold for interpretation. In addition, the naked eye's judgment of the proportion of cancer cells may result in certain errors and subjectivity, which may affect the prognosis.

[0003] Artificial Intelligence (AI) is a technology that simulates human thinking and judgment, and can analyze various problems and propose corresponding solutions. With the rapid development of chip technology, deep learning, and big data, AI is increasingly used in the medical field, especially in imaging, pathology, disease management, and drug development. Machine learning is a key technology for realizing artificial intelligence, among which deep learning is currently the most popular research direction. Deep learning combines low-level features (such as points, corners, and edges in images) through multi-layer processing to form higher-level abstract features (such as image labels), thereby revealing the intrinsic connection of data and performing target recognition and classification. In the field of deep learning, Convolutional Neural Network (CNN) is one of the most widely used models in image processing. It is mainly composed of convolutional layers, pooling layers, fully connected layers, and activation functions, and has demonstrated unique advantages in the field of image recognition and classification.

[0004] In the field of pathology, AI technology, especially deep learning, is of great value in improving the work efficiency of pathologists, reducing the misdiagnosis rate and improving the standardized quality of reports. The two-dimensional properties of digital pathology images provide an ideal application scenario for AI. Many studies have used deep learning to classify and identify pathology images. For example, one study used about 1,000 gastric biopsy samples in a collaborative database managed by the National Institute of Informatics (NII) of Japan to develop an artificial intelligence engine for detecting gastric cancer. These samples were used to train the model, and the results of the training were fed back to pathologists for review. Through continuous error correction and improvement, the accuracy of the diagnosis was enhanced. Researchers at Harvard University also successfully tracked cancers of unknown primary lesions by training convolutional neural network models. The clinical application of this achievement will greatly reduce the workload of oncologists and reduce the economic burden on patients.

[0005] Therefore, it is necessary to develop a method and system for detecting the quality of clinical tissue pathology sections of prostate cancer, so as to utilize the massive data possessed by artificial intelligence technology to solve the existing defects in clinical pathology work, such as large workload, high interpretation threshold, and certain errors and subjectivity in the naked eye's judgment of the proportion of cancer cells.

[0006] The existing invention patent application document "PET-MRI image prostate lesion three-dimensional visualization system and method" with publication number CN117457157A, the existing method includes: through the prostate full digital pathology section screening expert model, the prostate full digital pathology section data is screened; through the malignant tumor pathology recognition model, the malignant tumor is located by deep learning; the first group of prostate section data sets are labeled; and the preliminary Gleason grading model of prostate lesions is constructed through transfer learning; the preliminary Gleason grading model of prostate lesions is tested, trained and optimized, and Gleason grading is performed; according to the Gleason grading score two-dimensional pedigree map, the tumor infiltration depth information is incorporated, and a three-dimensional tumor pedigree map is established, and the three-dimensional tumor pedigree map is mapped to the prostate PET-MRI image, and the PET-MRI image prostate lesion three-dimensional visualization is accurately assisted to identify. However, the above-mentioned prior art does not directly use the prostate cancer pathology image data annotated by experts for the training and prediction of the deep learning model, and the transfer learning and pedigree map mapping in this scheme are prone to cascade errors.

[0007] The existing invention patent application document "A Gleason Grading System and Method for Digital Pathological Images of Prostate" with publication number CN116188407A, the existing method includes: obtaining digital pathological sections of prostate as images to be graded; identifying the foreground area in the image to be graded based on a preset algorithm extraction model; based on a preset sliding window, using a sliding window method to intercept the foreground area in the image to be graded, and obtaining a number of foreground blocks involving the foreground area; using a pre-trained semantic segmentation model to identify the tissue category and segment the area of ​​the foreground block to obtain a category mask block of the foreground block; splicing the category mask blocks of the foreground block to form a category mask map of the image to be graded; and obtaining the Gleason grading result according to the part of the foreground area in the category mask map. However, the existing technology does not fully learn expert knowledge, and it is difficult to fully and effectively utilize the edge module information of regional segmentation. At the same time, the semantic segmentation model used in the scheme is inaccurate, resulting in misjudgment of the result.

[0008] The existing invention patent application document "A Clinical Test Result Analysis Method Based on Medical Knowledge Graph" with publication number CN118538429A, the existing method includes: by establishing a centralized data warehouse, storing multi-source data in a unified manner, and using ETL tools to synchronize and integrate data to ensure data consistency and integrity, using convolutional neural network CNN and recurrent neural network RNN ​​deep learning technology to extract high-dimensional features from medical images and time series data, improve the accuracy and richness of feature extraction, and through data fusion technology, combine structured data and unstructured data to enhance the expressiveness and information integrity of data, and use graph neural network GNN to process and analyze the fused data to obtain the potential connection and interaction between different data modalities, use CNN and RNN to analyze time series data and image data, extract disease biomarkers, and predict the development of the disease. However, the aforementioned existing scheme has the problem of lack of training data, and at the same time, it is difficult for the scheme to avoid the problem of modal inconsistency and inability to fully align information when GNN processes the potential connection and interaction between different data modalities.

[0009] In the existing public literature, "A Review of the Current Research Status of Deep Learning Methods for Prostate Cancer Gleason", the Retina U-Net architecture combines Retina Net with U-Net and is designed specifically for processing medical images. In RetinaU-Net, lesion detection is independent of lesion segmentation (unlike other similar detection + segmentation architectures, such as MaskR-CNN). However, this existing technology can only perform lesion detection using expert-annotated data alone for lesion detection and segmentation, which restricts the lesion detection effect.

[0010] In the existing public literature "Construction and Application of Clinically Significant Prostate Cancer Diagnosis Prediction Model Based on FasterRCNN and mpMRI", the region-based convolutional neural network (RCNN) is the first deep learning network structure to be applied in the field of target detection. Candidate region selection, feature extraction, classification and boundary regression are its three major processes. The RCNN series based on this (RCNN, FastRCNN, FasterRCNN, etc.) is also one of the commonly used network structure series. Faster RCNN uses region proposal networks (RPN) to select candidate regions based on RCNN and Fast RCNN. Faster RCNN, after the development of RCNN and FastRCNN, integrates feature extraction, candidate box generation, ROI pooling, border regression and classification into the same network. The aforementioned existing solutions rely on CNN and RNN related algorithms for candidate region detection, and cannot guarantee the detection accuracy.

[0011] In summary, in the existing cancer cell detection and tumor grading technologies, few pre-trained models and CNN and RNN fusion training methods are used. The main reasons are as follows:

[0012] 1. Model complexity and high computational overhead: Before the emergence of high-performance GPU processors, whether it is a pre-trained model or fusion training of CNN and RNN, the model not only needs to process high-dimensional image data, but also needs to model complex time or sequence relationships. This complex network structure has very high requirements for computing resources and training time, especially when processing large-scale pathological image data, which requires the support of high-performance computing equipment. In recent years, with the continuous emergence of high-performance GPU processors, pre-trained models and fusion training of CNN and RNN have become feasible in terms of computing resources;

[0013] Lack of high-quality data and annotations: Whether it is a pre-trained model or CNN and RNN fusion training, the effectiveness of the method depends largely on a large amount of labeled data, and prostate cancer grade prediction usually requires high-quality pathological sections and accurate grade annotations. However, the acquisition of medical imaging data annotations is very expensive and time-consuming because it requires the participation of experienced professional pathologists. This lack of data annotation makes it difficult for pre-trained models to fully learn the characteristics of the target domain during the fine-tuning stage. In recent years, with the Prostate cancer grade Assessment (PANDA) competition providing approximately 11,000 digital HE-stained biopsy full-slice images annotated by experts and 500 self-annotated digital HE-stained biopsy full-slice images of the Pathology Department of the First Affiliated Hospital of Anhui Medical University, more and more high-quality labeled data has become available, making pre-trained models and CNN and RNN fusion training feasible in terms of effect.

[0014] Existing cancer cell detection and tumor grading technologies do not combine CNN and RNN and do not use pre-trained models, which are mainly affected by insufficient computing resources and lack of high-quality data. With the emergence of high-performance GPUs in recent years and the increase in high-quality data of prostate cancer pathology images (open source competitions + self-labeling), the application of these two technologies has become possible.

[0015] The existing technology has technical problems such as large workload, high threshold for interpretation, errors and subjectivity in judging the proportion of cancer cells by the naked eye. Summary of the invention

[0016] The technical problem to be solved by the present invention is: how to solve the technical problems in the prior art of large workload, high threshold for interpretation, and errors and subjectivity in the naked eye's judgment of the proportion of cancer cells.

[0017] The present invention adopts the following technical solutions to solve the above technical problems: A digital pathology image grading method comprises:

[0018] S1, performing pre-coding operation, re-decoding operation and regression operation on the prostate pathology image to obtain a regression image, using the pixel classification Unet algorithm to perform cell recognition on the regression image, and extracting a cell part image in the prostate pathology image;

[0019] S2. Input the cell part image into the pre-trained model to obtain image features; a classification algorithm based on a convolutional neural network (CNN) is used to detect cancer cell data and surrounding tissue related features from the cell part image;

[0020] S3. Based on the cancer cell data and the associated features of the surrounding tissues, the Gleason pathological grade information of the cancer cells is predicted through the pre-training model and the CNN-RNN algorithm model, wherein the CNN-RNN algorithm model includes: convolutional neural network and recursive neural network.

[0021] The present invention utilizes the characteristics of the pre-trained model, such as stronger migration ability, richer feature extraction, and stronger robustness, as well as the powerful image processing ability of the convolutional neural network (CNN), and can efficiently analyze and identify cancerous cells in pathological images. The present invention combines the advanced image analysis ability of the convolutional neural network (CNN) and the data sequence processing ability of the recurrent neural network (RNN), and can effectively identify and analyze cancerous cells in pathological images. By integrating the pre-trained model, convolutional neural network, and recurrent neural network, the present invention realizes automated and high-precision analysis from cell positioning to cancer cell area detection and then to pathological grading.

[0022] In a more specific technical solution, S1 includes:

[0023] S11, according to the high-level network demand information, reversely derive the calculation applicable input size of the input image;

[0024] S12, using 2×2 max pooling to halve the image size;

[0025] S13. Use transposed convolution to perform a re-decoding operation to complete the row and column dimension multiplication, restore the prostate case image to the original image size, and obtain a regression image.

[0026] In a more specific technical solution, in S12, no less than two convolutional layers are used, and a valid convolution method is adopted to reduce the image size to complete the image size halving operation.

[0027] In a more specific technical solution, the conv layers use the same mode to perform convolution operations.

[0028] The convolution operation of the convolution layer of the present invention selects the same mode, so there is no need to perform the crop operation, which not only avoids the potential size mismatch problem, but also simplifies the calculation process.

[0029] In a more specific technical solution, in S13, a 2×2 up-conv operation is used to achieve row and column dimension doubling.

[0030] In the decoding stage, the present invention is implemented by using transposed convolution. Specifically, a 2×2 up-conv operation is applied to achieve the doubling of row and column dimensions, thereby restoring the size of the original image. This method not only ensures the effective extraction of features and classification accuracy, but also optimizes the computational efficiency, making the entire cell recognition process more efficient and accurate.

[0031] In a more specific technical solution, S2 includes:

[0032] S21, extracting image feature vectors through the pre-trained model, passing the corresponding feature vectors to the convolution layer, extracting edge data and corner points through the filter in the convolution layer to generate corresponding feature maps;

[0033] S22, passing the corresponding feature maps to the ReLU activation function of the nonlinear transformation layer to perform nonlinear feature enhancement operations and capture pattern data;

[0034] S23, using the pooling layer, the corresponding feature maps are subjected to dimensionality reduction processing to obtain pooled and refined features;

[0035] S24, using a preset number of fully connected layers to process the pooled and refined features to obtain fully connected features;

[0036] S25. Based on the fully connected features, the Softmax activation function is used to predict the cancer cell area, and the cancer cell data and surrounding tissue correlation features are determined accordingly.

[0037] The present invention extracts image features through a pre-trained model and convolutional neural network, and processes them through multiple layers of convolutional layers, pooling layers, and activation layers, each of which focuses on capturing different features of the image, from basic edges and textures to more complex shapes and patterns. The trained model of the present invention can determine whether cancer cells are present in the image and accurately locate the specific location of the cancerous area.

[0038] Compared with the traditional scheme, in addition to lesion detection and segmentation, and candidate region detection using CNN and RNN related algorithms, the present invention also introduces a pre-trained model to provide medical information based on the large model, which is better than using expert-annotated data alone for lesion detection and has higher detection accuracy. The present invention directly uses expert-annotated prostate cancer pathology image data for training and prediction of deep learning models, reducing the cascade errors caused by transfer learning and pedigree graph mapping in traditional methods.

[0039] Based on the deep learning technology of convolutional neural network (CNN) and recurrent neural network (RNN), the present invention introduces a pre-trained model to provide medical information based on the large model, alleviates the problem of lack of training data in this method, and avoids the problem of modal inconsistency and inability to fully align information when GNN processes the potential connections and interactions between different data modalities.

[0040] In more specific technical solutions, S3 includes:

[0041] S31, extracting image feature vectors through the pre-trained model, transferring the corresponding feature vectors to the convolutional neural network, and processing cancer cell data and surrounding tissue related features layer by layer through the collaborative work of no less than 2 convolutional layers, pooling layers, and activation layers;

[0042] S32, inputting the image feature sequence into a recursive neural network to mine the spatial sequence relationship between the features, so as to obtain a recursive network processing result;

[0043] S33. The processing result of the recursive neural network will be passed through a preset number of fully connected layers to analyze the processing result of the recursive network and predict the Gleason pathological grading information of cancer cells.

[0044] The present invention first extracts image features through a pre-trained model and a convolutional neural network, and processes the image layer by layer through the collaborative work of multiple convolutional layers, pooling layers, and activation layers, where each layer is responsible for capturing different levels of image features, from basic edges and textures to more complex shapes and patterns. The extracted feature sequence is input into a recurrent neural network to mine the spatial sequence relationship between these features. The processing results of the recurrent neural network will be further analyzed through one or more fully connected layers, ultimately achieving accurate prediction of the pathological grade of cancer cells.

[0045] Compared with traditional technologies, the present invention uses a semantic segmentation model to identify prostate cancer image regions, and uses expert-annotated prostate cancer pathology image data to train a deep learning model to predict Gleason grading. It fully learns expert knowledge and can effectively utilize the edge module information of regional segmentation, reducing the misjudgment of results caused by the inaccurate semantic segmentation model in this method.

[0046] In a more specific technical solution, in S31, each convolution layer, pooling layer and activation layer captures edge data, texture data, shape data and pattern data, and processes the edge data, texture data, shape data and pattern data to obtain image features of different levels.

[0047] In a more specific technical solution, S31 also includes:

[0048] S311. Using the Unet network model, perform feature extraction on the input color two-dimensional image of the pathological section to obtain an image feature sequence, and determine the cell position in the pathological section based on the image feature sequence.

[0049] S312. Using the CNN-RNN algorithm model, perform deep detection on cells at each cell position to determine the cancer cell area.

[0050] In a more specific technical solution, a digital pathology image grading system includes:

[0051] The cell recognition module is used to perform pre-coding, re-decoding and regression operations on the prostate pathology image to obtain a regression image, and use the pixel point classification Unet algorithm to perform cell recognition on the regression image to extract the cell part image in the prostate pathology image;

[0052] The cancer cell detection module is used to input the cell part image into the pre-trained model to obtain image features, and then detect cancer cell data and surrounding tissue related features from the cell part image based on the classification algorithm of the convolutional neural network CNN. The cancer cell detection module is connected to the cell recognition module;

[0053] The cancer cell pathology grading prediction module is used to predict the Gleason pathology grading information of cancer cells based on the cancer cell data and the associated features of the surrounding tissues through a pre-trained model and a CNN-RNN algorithm model. The CNN-RNN algorithm model includes: a convolutional neural network and a recursive neural network. The cancer cell pathology grading prediction module is connected to the cancer cell detection module.

[0054] Compared with existing solutions, there are two major technical differences: First, the introduction of a pre-trained model for image encoding can solve the problems of weak generalization ability, weak robustness, and insufficient feature richness brought about by the direct use of CNN, RNN, GNN and related variant algorithms in existing solutions; Second, the use of a fusion algorithm of CNN and RNN to predict the Gleason pathological grading information of cancer cells, combining CNN's ability to capture image spatial features and RNN's ability to model sequence dependencies, enables the model to more comprehensively and accurately assess the cancer grade, especially when processing multiple slices or images at different time points, the advantages are more obvious.

[0055] Compared with the prior art, the present invention has the following advantages:

[0056] The present invention utilizes the characteristics of the pre-trained model, such as stronger migration ability, richer feature extraction, and stronger robustness, as well as the powerful image processing ability of the convolutional neural network (CNN), and can efficiently analyze and identify cancerous cells in pathological images. The present invention combines the advanced image analysis ability of the convolutional neural network (CNN) and the data sequence processing ability of the recurrent neural network (RNN), and can effectively identify and analyze cancerous cells in pathological images. By integrating the pre-trained model, convolutional neural network, and recurrent neural network, the present invention realizes automated and high-precision analysis from cell positioning to cancer cell area detection and then to pathological grading.

[0057] The convolution operation of the convolution layer of the present invention selects the same mode, so there is no need to perform the crop operation, which not only avoids the potential size mismatch problem, but also simplifies the calculation process.

[0058] In the decoding stage, the present invention is implemented by using transposed convolution. Specifically, a 2×2 up-conv operation is applied to achieve the doubling of row and column dimensions, thereby restoring the size of the original image. This method not only ensures the effective extraction of features and classification accuracy, but also optimizes the computational efficiency, making the entire cell recognition process more efficient and accurate.

[0059] The present invention extracts image features through convolutional neural networks and processes them through multiple layers of convolutional layers, pooling layers, and activation layers, each of which focuses on capturing different features of the image, from basic edges and textures to more complex shapes and patterns. The trained model of the present invention can determine whether cancer cells are present in the image and accurately locate the specific location of the cancerous area.

[0060] The present invention first extracts image features through a convolutional neural network, and processes the image layer by layer through the collaborative work of multiple convolutional layers, pooling layers, and activation layers, where each layer is responsible for capturing different levels of image features, from basic edges and textures to more complex shapes and patterns. The extracted feature sequence is input into a recurrent neural network to mine the spatial sequence relationship between these features. The processing results of the recurrent neural network will be further analyzed through one or more fully connected layers, ultimately achieving accurate prediction of the pathological grade of cancer cells.

[0061] The present invention solves the technical problems existing in the prior art of heavy workload, high threshold for interpretation, and errors and subjectivity in the naked eye's judgment of the proportion of cancer cells. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 A schematic diagram of basic steps of a digital pathology image grading method according to Embodiment 1 of the present invention;

[0063] Figure 2 This is a schematic diagram of data flow processing for detecting cancer cells in a prostate biopsy tissue pathology slice image according to Embodiment 2 of the present invention;

[0064] Figure 3 This is a diagram showing the effect of image processing of prostate puncture tissue pathology slices in Example 1 of the present invention;

[0065] Figure 4 This is a schematic diagram of the model architecture of the clinical tissue pathology section quality detection system according to Example 1 of the present invention; Figure 5 This is a schematic diagram of data flow processing of a clinical tissue pathology section quality detection system model according to Example 1 of the present invention. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] Example 1

[0068] like Figure 1 As shown, a digital pathology image grading method provided by the present invention includes the following basic steps:

[0069] S1. Cell recognition is performed based on the Unet algorithm that first encodes, then decodes, and regresses to the same size of pixels as the original image to extract the cell part in the prostate pathology image;

[0070] In this embodiment, it is necessary to determine the size of the input image, which is reversely calculated based on the needs of a higher level of the network to ensure that an appropriate and easy-to-calculate input size is selected.

[0071] In the encoding stage of this embodiment, the image size is halved by applying 2×2 maximum pooling. In this process, two convolution layers are introduced, and the valid convolution method is used to further reduce the image size. This change brings a challenge: when performing skip connection, due to the previous copy & crop operation (cropping to match the size), it may cause size inconsistency during connection. However, if the convolution operation selects the same mode, there is no need to perform the crop operation, which not only avoids potential size mismatch problems, but also simplifies the calculation process.

[0072] In the decoding stage of this embodiment, the transposed convolution is used for implementation, specifically, a 2×2 up-conv operation is applied to achieve doubling of the row and column dimensions, thereby restoring the size of the original image.

[0073] Through this method, not only the effective extraction of features and classification accuracy are guaranteed, but also the computational efficiency is optimized, making the entire cell recognition process more efficient and accurate.

[0074] S2, input the extracted cell part into the pre-trained model, and then detect the cancer cells contained therein based on the CNN (convolutional neural network) classification algorithm;

[0075] S3. Based on the relevant features of the detected cancer cells and their surrounding tissues, they are input into the pre-trained model, and then the pathological grade of cancer cells is predicted through the CNN-RNN (a combination of convolutional neural network and recurrent neural network) algorithm.

[0076] In this embodiment, image features are extracted through a pre-trained model and a convolutional neural network, and the image is processed layer by layer through the collaborative work of multiple convolutional layers, pooling layers, and activation layers, wherein each layer is responsible for capturing image features at different levels, from basic edges and textures to more complex shapes and patterns; the extracted feature sequence is input into a recurrent neural network to mine the spatial sequence relationship between these features; the processing results of the recurrent neural network will be further analyzed through one or more fully connected layers, ultimately achieving accurate prediction of the pathological grading of cancer cells.

[0077] In this embodiment, a Unet network model is used: the model is used to perform detailed feature extraction on the input color two-dimensional image of the pathological section, and accurately determine the position of the cells in the input pathological section according to the extracted features.

[0078] The cancer cell prediction convolutional neural network model used in this embodiment is used to perform in-depth detection of cells at various input locations to confirm the cancer cell area.

[0079] The convolutional neural network and recurrent neural network combined model used in this embodiment for judging the pathological grade of cancer cells: This composite model is used to accurately predict the Gleason pathological grade of cells that have been identified as cancer cells.

[0080] Example 2

[0081] This embodiment uses a database (PID-slim) of 4,000 clinical pathological tissue slice image data including normal and abnormal quality images collected by the high-resolution digital panoramic scanner pathological slice image acquisition system (WS-10) from the First Affiliated Hospital of Anhui Medical University as the training set and test set. The area under the accuracy and recall curves (AP) are used as the measurement criteria, and the Unet algorithm is used for cell recognition, the CNN algorithm is used for cancer cell detection, and the CNN-RNN joint algorithm is used for cancer cell pathology grading.

[0082] like Figure 2 As shown, this embodiment provides a method for detecting cancer cells in prostate puncture tissue pathology slice images based on the present invention, and a schematic diagram of the process of detecting cancer cells in prostate puncture tissue pathology slice images. The entire process of detecting cancer cells in prostate puncture tissue pathology slice images is divided into three sub-processes: cell recognition, cancer cell detection, and cancer cell pathology grade prediction.

[0083] In the cell recognition process of this embodiment, the appropriate and easy-to-calculate input image size is determined by reverse calculation based on the needs of the high-level network structure. In the encoding stage, 2×2 maximum pooling is used to halve the image size. In this process, two convolution layers are introduced, and the valid convolution method is used to further reduce the image size. This change brings a challenge: when performing skip connection, due to the previous copy&crop operation (cropping to match the size), it may cause size inconsistency during connection. However, if the convolution operation selects the same mode, there is no need to perform the crop operation, which not only avoids potential size mismatch problems, but also simplifies the calculation process. In the decoding stage, a 2×2 up-conv operation is applied to double the row and column dimensions and restore them to the size of the original image.

[0084] In the cancer cell detection process of this embodiment, the result of the cell recognition process output is used as input, first passed to the pre-trained model, and then a convolutional neural network (CNN) is constructed to perform the feature extraction task. The network extracts key features of the image, such as edges or corners, through the filters in the convolution layer. Each filter is specifically responsible for detecting specific image features from the original input data and generating corresponding feature maps. These feature maps are passed to the nonlinear transformation layer, i.e., the ReLU (Rectified Linear Unit) activation function, to enhance the nonlinear characteristics of the model so that the network can capture more complex patterns and relationships. The convolution layer is followed by the pooling layer, which is used to reduce the spatial dimension of the feature map, reduce the subsequent calculation amount, and extract the most critical feature information. After 3 convolution and pooling operations, it is further processed by 3 fully connected layers, and finally the Softmax activation function is used to predict the cancer cell area.

[0085] In the cancer cell pathological grading prediction process of the present embodiment, the cancer cell detection result is obtained as input, first passed to the pre-training model, and then the image processing capability of the convolutional neural network (CNN) and the data sequence processing advantage of the recursive neural network (RNN) are utilized to efficiently analyze and identify cancerous cells in the pathological image. The image is processed by the collaborative work of multiple convolutional layers, pooling layers, and activation layers, each of which is responsible for capturing different image features from simple to complex, such as edges, textures, and more complex shapes and patterns. The extracted feature sequence is input into the recursive neural network to mine the spatial sequence relationship between these features. The processing results of the recursive neural network will be further analyzed by three fully connected layers, and finally an accurate prediction of the pathological grading of cancer cells is achieved.

[0086] like Figure 3 As shown, the research object in this embodiment is 1200 cases of hematoxylin-eosin stained pathological tissue sections, of which 1080 cases are divided into training sets and 120 cases are divided into test sets. These 1200 pictures are preprocessed, and the defect areas and classification groups in the pictures are manually annotated. The preprocessed annotated files are divided into training files and test files. After model training is performed according to the training set and the test set, a trained model is obtained. Based on the trained model, the quality of clinical tissue pathology sections can be detected. Figure 2 As shown, a schematic diagram of the location and pathological grade of cancer cells detected in the pathological slice image of the prostate puncture tissue is given, and the location and pathological grade of cancer cells in the pathological slice image have been predicted.

[0087] Example 3

[0088] The above-mentioned embodiment 1 provides a digital pathology image grading method, and correspondingly, this embodiment provides a digital pathology image grading system. The system provided in this embodiment can implement the method for detecting cancer cells in prostate puncture tissue pathology section images of embodiment 1, and the system can be implemented by software, hardware, or a combination of software and hardware. For example, the system may include integrated or separate functional modules or functional units to perform the corresponding steps in each method of embodiment 1. Since the system of this embodiment is basically similar to the method embodiment, the process described in this embodiment is relatively simple, and the relevant parts can refer to the partial description of embodiment 1. The embodiment of the system provided in this embodiment is only schematic.

[0089] like Figure 4 and Figure 5 As shown, this embodiment provides a digital pathology image grading system, including:

[0090] The Unet algorithm is used for cell recognition, the pre-trained model is used for feature extraction, the CNN algorithm is used for cancer cell detection, and the CNN-RNN algorithm is used to predict the pathological grade of cancer cells.

[0091] Example 4

[0092] This embodiment provides a processing device corresponding to a digital pathology image grading method provided in this embodiment 1. The processing device may be a processing device for a client, such as a mobile phone, a laptop computer, a tablet computer, a desktop computer, etc., to execute the method of embodiment 1.

[0093] The processing device includes a processor, a memory, a communication interface and a bus, and the processor, the memory and the communication interface are connected through the bus to complete mutual communication. The memory stores a computer program that can be run on the processor, and the processor executes the method for detecting the quality of clinical tissue pathology sections provided in this embodiment 1 when running the computer program.

[0094] In some embodiments, the memory may be a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory.

[0095] In other embodiments, the processor may be a central processing unit (CPU), a digital signal processor (DSP), or other general-purpose processors of various types, which are not limited herein.

[0096] Example 5

[0097] A digital pathology image grading method of this embodiment 1 can be specifically implemented as a computer program product, which may include a computer-readable storage medium carrying computer-readable program instructions for executing the method for detecting cancer cells in prostate puncture tissue pathology section images described in this embodiment 1.

[0098] Computer readable storage media can be tangible devices that hold and store instructions used by instruction execution devices. Computer readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any combination thereof.

[0099] The solution used in this application is: collect a large amount of prostate cancer pathology slice data to train a pre-trained model, encode the prostate pathology image based on the pre-trained model, and then use the Unet algorithm to perform cell recognition on the regression image, the CNN algorithm to detect the range of cancer cells, and the CNN-RNN algorithm to predict the Gleason pathology grade information of cancer cells. Compared with the existing solutions, there are two major differences:

[0100] First, a pre-trained model is introduced to encode images. Compared with other methods that only use traditional deep learning solutions (such as CNN, RNN, GNN, etc.) to directly encode images, it has the following advantages:

[0101] 1. Stronger model migration capability: The pre-trained model has been trained on large-scale data sets and has good feature extraction capabilities, and can migrate these general features to specific tasks. In the quality detection of tissue pathology sections, the pre-trained model can use these rich feature representations to reduce dependence on a large amount of labeled data and improve the generalization ability of the model. When using CNN or RNN alone, only training with data in specific fields can easily lead to overfitting of the model or insufficient generalization ability.

[0102] 2. Richer feature extraction: The pre-trained model has a strong feature extraction capability, especially when processing high-dimensional and complex images, it can capture multi-level details and global context information. In contrast, the traditional CNN performs well in processing local features, but is weak in processing long-distance dependencies, while the RNN is good at processing sequence data, but cannot fully capture the spatial structure of image data. The pre-trained model can better integrate local and global information, thereby improving the accuracy of slice quality detection.

[0103] 3. More efficient model training: By using pre-trained models, higher performance can be achieved with less data and less training time. This is because the pre-trained model has learned useful representations on general datasets and can be quickly adapted to specific pathological image quality detection tasks through fine-tuning. Using CNN or RNN alone requires training from scratch, which often requires longer training time and more computing resources.

[0104] 4. Stronger robustness: The pre-trained model has been verified on a variety of tasks and data sets, and has strong robustness and stability. When dealing with problems such as noise and image quality differences that may exist in clinical pathology images, the system based on the pre-trained model can show stronger fault tolerance and reduce misjudgment. However, single CNN and RNN models are easily limited by the model structure and the quality of specific data sets, and may be unstable when facing different data distributions.

[0105] Second, the method of combining CNN and RNN is used to predict the grade of prostate cancer. Compared with other methods that only use CNN or RNN to predict the grade of prostate cancer, the method combines CNN's ability to capture image spatial features and RNN's ability to model sequence dependencies, enabling the model to more comprehensively and accurately evaluate cancer grades, especially when processing multiple slices or images at different time points. Specifically, there are the following advantages:

[0106] 1. Comprehensive utilization of spatial features and temporal / sequence features

[0107] CNN is good at processing image data, especially in capturing spatial local features in prostate cancer pathology sections. CNN extracts local features such as cell morphology, tissue structure, and tumor distribution through convolution operations, which are important for assessing the grade of cancer. RNN is good at processing sequence data and can capture the sequential dependencies between data. Although pathology images themselves are not sequence data, in the case of multiple slices (such as a continuous sequence of pathology slices), RNN can effectively process the association information between different slices, thereby providing context in the time or sequence dimension for cancer grade assessment.

[0108] By combining CNN and RNN, the model can simultaneously extract spatial local features of the image (CNN) and global dependencies across slices or sequences (RNN). This is more advantageous for dealing with the heterogeneity in prostate cancer pathology slices and the complexity in grade prediction, because the cancer grade depends not only on local features, but also on the overall tissue structure and its changing trend in multiple slices.

[0109] 2. Improved ability to capture global and local information

[0110] Using CNN alone can only extract local spatial features, which is insufficient for capturing global information across the entire image or across multiple slices. Although using RNN alone can model sequence information, its feature extraction ability is weak when processing high-dimensional image data.

[0111] The combination of CNN+RNN allows CNN to extract local features of the image, while RNN can further process the features output by CNN and establish the correlation and predict trends between previous and next slices by analyzing multiple pathological slices. This combination can more comprehensively capture the complex features required for cancer grade.

[0112] 3. Processing multi-scale information

[0113] In cancer grade prediction, pathological information at different scales may be of great significance at the same time. CNN is good at extracting image features of different scales from different receptive fields, while RNN can model the temporal or sequential dependencies between these multi-scale features. By combining CNN and RNN, the model can better handle features of different scales and capture potential cancer development patterns, thereby improving the accuracy of cancer grade prediction.

[0114] 4. Improve the generalization ability of the model

[0115] When using CNN or RNN alone, the model may show insufficient generalization ability when facing unknown data. For example, a simple CNN may lose some global information when encountering large heterogeneity, while a simple RNN may not be able to fully utilize the spatial details in the image. Combining CNN and RNN can enhance the robustness of the model in a complementary way, so that it can still make accurate predictions when facing different patients and different slice data.

[0116] 5. More effective feature compression and expression

[0117] CNN is responsible for extracting low-dimensional spatial features and compressing them into higher-level expressions, while RNN can further model the sequence dependencies between these high-level expressions. Therefore, the model combining CNN and RNN can reduce redundant information and improve the computational efficiency of the model by gradually compressing and abstracting features while maintaining predictive capabilities. In addition, this feature extraction process can also help the model better learn effective feature representations when the amount of data is small.

[0118] 6. Handling cross-time or cross-slice dependencies

[0119] If cancer grade prediction involves multiple examinations, images at different time points, or multiple slices, RNN can help the model capture dependencies between multiple time points or slices. For example, if a slice shows a mild abnormality, but subsequent slices show a more obvious deterioration trend, RNN can better judge the severity of the cancer by processing this sequential information. In contrast, using CNN alone may not be able to fully capture this dependency across time or slices.

[0120] The present invention introduces algorithms such as Unet, CNN, RNN, and pre-trained models into the detection of cancer cells in pathological slice images, and utilizes the massive data, powerful computing resources, and more advanced algorithms possessed by artificial intelligence technology to effectively solve the existing problems in clinical pathology work, such as large workload, high interpretation threshold, and certain errors and subjectivity in the naked eye's judgment of the proportion of cancer cells. Cancer cell detection in slices is performed through pathological artificial intelligence, thereby greatly improving the efficiency of cancer cell detection and evaluation.

[0121] In summary, the present invention utilizes the characteristics of the pre-trained model, such as stronger migration ability, richer feature extraction, and stronger robustness, as well as the powerful image processing ability of the convolutional neural network CNN, and can efficiently analyze and identify cancerous cells in pathological images. The present invention combines the advanced image analysis ability of the convolutional neural network (CNN) and the data sequence processing ability of the recurrent neural network (RNN), and can effectively identify and analyze cancerous cells in pathological images. By integrating convolutional neural networks and recurrent neural networks, the present invention realizes automated and high-precision analysis from cell positioning to cancer cell area detection and then to pathological grading.

[0122] The convolution operation of the convolution layer of the present invention selects the same mode, so there is no need to perform the crop operation, which not only avoids the potential size mismatch problem, but also simplifies the calculation process.

[0123] In the decoding stage, the present invention is implemented by using transposed convolution. Specifically, a 2×2 up-conv operation is applied to achieve the doubling of row and column dimensions, thereby restoring the size of the original image. This method not only ensures the effective extraction of features and classification accuracy, but also optimizes the computational efficiency, making the entire cell recognition process more efficient and accurate.

[0124] The present invention extracts image features through convolutional neural networks and processes them through multiple layers of convolutional layers, pooling layers, and activation layers, each of which focuses on capturing different features of the image, from basic edges and textures to more complex shapes and patterns. The trained model of the present invention can determine whether cancer cells are present in the image and accurately locate the specific location of the cancerous area.

[0125] The present invention first extracts image features through a convolutional neural network, and processes the image layer by layer through the collaborative work of multiple convolutional layers, pooling layers, and activation layers, where each layer is responsible for capturing different levels of image features, from basic edges and textures to more complex shapes and patterns. The extracted feature sequence is input into a recurrent neural network to mine the spatial sequence relationship between these features. The processing results of the recurrent neural network will be further analyzed through one or more fully connected layers, ultimately achieving accurate prediction of the pathological grade of cancer cells.

[0126] The present invention solves the technical problems existing in the prior art of heavy workload, high threshold for interpretation, and errors and subjectivity in the naked eye's judgment of the proportion of cancer cells.

[0127] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A digital pathology image grading method, characterized in that: The method comprises: S1, performing pre-coding operation, re-decoding operation and regression operation on the prostate pathology image to obtain a regression image, performing cell recognition on the regression image using a pixel point classification Unet algorithm, and extracting a cell part image in the prostate pathology image; S2, inputting the cell portion image into the pre-trained model to obtain image features; using a classification algorithm based on a convolutional neural network (CNN) to detect cancer cell data and surrounding tissue-related features from the cell portion image; S3. Based on the cancer cell data and the surrounding tissue association features, the Gleason pathological grade information of cancer cells is predicted through the pre-trained model and the CNN-RNN algorithm model, wherein the CNN-RNN algorithm model includes: a convolutional neural network and a recursive neural network.

2. A digital pathology image grading method according to claim 1, characterized in that: The S1 includes: S11, according to the high-level network demand information, reversely derive the calculation applicable input size of the input image; S12, using 2×2 max pooling to halve the image size; S13, using transposed convolution to implement the re-decoding operation, completing row and column dimension multiplication, restoring the prostate case image to the original image size, and obtaining the regression image.

3. A digital pathology image grading method according to claim 2, characterized in that: In the above 12, at least two convolutional layers are used and a valid convolution method is adopted to reduce the image size to complete the image size halving operation.

4. A digital pathology image grading method according to claim 3, characterized in that: The convlayers perform convolution operations in the same manner.

5. A digital pathology image grading method according to claim 2, characterized in that: In S13, a 2×2 up-conv operation is used to achieve the doubling of the row and column dimensions.

6. A digital pathology image grading method according to claim 1, characterized in that: The S2 includes: S21, extracting image feature vectors through the pre-trained model, passing the corresponding image feature vectors to the convolution layer, extracting edge data and corner points through the filter in the convolution layer to generate corresponding feature maps; S22, passing the corresponding feature maps to the ReLU activation function of the nonlinear transformation layer to perform a nonlinear characteristic enhancement operation to capture the pattern data; S23, using the pooling layer, performing dimensionality reduction processing on the corresponding feature maps to obtain pooled refined features; S24, using a preset number of fully connected layers to process the pooled refined features to obtain fully connected features; S25. According to the fully connected features, a Softmax activation function is used to predict the cancer cell region, and the cancer cell data and the surrounding tissue association features are determined accordingly.

7. A digital pathology image grading method according to claim 1, characterized in that: The S3 includes: S31, extracting image feature vectors through the pre-trained model, transferring the corresponding image feature vectors to the convolutional neural network CNN, and processing the cancer cell data and the surrounding tissue-related features layer by layer through the collaborative work of no less than two convolutional layers, pooling layers, and activation layers; S32, inputting the image feature sequence into the recursive neural network to mine the spatial sequence relationship between features, so as to obtain the recursive network processing result; S33, the processing result of the recursive neural network will be passed through a preset number of fully connected layers to analyze the processing result of the recursive network and predict the Gleason pathological grade information of the cancer cells.

8. A digital pathology image grading method according to claim 7, characterized in that: In the S31, the convolution layer, the pooling layer and the activation layer of each layer capture edge data, texture data, shape data and pattern data, and process the edge data, texture data, shape data and pattern data to obtain image features of different levels.

9. A digital pathology image grading method according to claim 7, characterized in that: The S31 further includes: S311, using the Unet network model, performing a feature extraction operation on the input color two-dimensional image of the pathological section to obtain the image feature sequence, and determining the cell position in the pathological section according to the image feature sequence. S312: Using the CNN-RNN algorithm model, perform depth detection on the cells at each of the cell positions to determine the cancer cell area.

10. A digital pathology image grading system, characterized in that: The system comprises: A cell recognition module is used to perform pre-coding operations, re-decoding operations and regression operations on the prostate pathology image to obtain a regression image, and use the pixel point classification Unet algorithm to perform cell recognition on the regression image to extract the cell part image in the prostate pathology image; A cancer cell detection module is used to input the cell portion image into a pre-trained model to obtain image features; based on a classification algorithm of a convolutional neural network (CNN), cancer cell data and surrounding tissue-related features are detected from the cell portion image, and the cancer cell detection module is connected to the cell recognition module; The cancer cell pathology grading prediction module is used to predict the Gleason pathology grading information of cancer cells based on the cancer cell data and the surrounding tissue association characteristics through the pre-trained model and the CNN-RNN algorithm model, wherein the CNN-RNN algorithm model includes: a convolutional neural network and a recursive neural network, and the cancer cell pathology grading prediction module is connected to the cancer cell detection module.

Images