Method and system for reading urine exfoliated cell liquid-based smear

Abstract

The present invention discloses a method and system for reading urine exfoliated cell liquid-based smears. The embodiment of the present invention respectively sets a cell detection model, a single cell classification model and a cluster cell classification model based on a convolutional neural network. After scanning and acquiring a digital image of the urine exfoliated cell liquid-based smear, the digital image is input into the set cell detection model, and the output is obtained by a rough classification digital image including single cells and a rough classification digital image including cluster cells. The rough classification digital image including single cells is input into the single cell classification model for processing to obtain a recognition result of single cells; the rough classification digital image including cluster cells is input into the cluster cell classification model for processing to obtain a recognition result of cluster cells; the recognition result of single cells and the recognition result of cluster cells are merged to obtain the recognition result of the urine exfoliated cell liquid-based smear. In this way, the embodiment of the present invention realizes the automatic reading of urine exfoliated cell smears.

Description

Technical Field

[0001] The invention relates to a medical cell image processing technology, and in particular to a method and system for reading urine exfoliated cell liquid-based smears. Background Art

[0002] Urothelial carcinoma is a multi-source malignant tumor originating from the urothelium. Its incidence ranks fifth among malignant tumors worldwide, and it poses a serious threat to human health. Urine exfoliated cytology is simple, non-invasive, and highly specific. It is the main examination method for the diagnosis of urinary system tumors and postoperative follow-up, and is helpful for the early diagnosis of tumors. Urine exfoliated cytology requires experienced pathologists to observe urine exfoliated cell liquid-based smears with the help of low-power and high-power microscopes to give diagnostic results. At present, there are certain limitations in the use of artificial microscopic reading to examine urine exfoliated cell liquid-based smears, which are mainly manifested in: a. low sensitivity; b. high false positive and misdiagnosis rate; c. shortage of experienced pathologists, and the reading volume, reading efficiency, and reading accuracy are greatly affected by human factors; d. Exfoliated cytology diagnosis is difficult, requiring repeated examination by experienced experts, and it is not easy to clearly classify tumor cells.

[0003] In recent years, artificial intelligence technology based on machine learning, deep learning and reinforcement learning has developed rapidly and has been widely used in many fields. The increasing workload and the imminent shortage of doctors have made the application of artificial intelligence in the medical field more and more urgent. With the rise of digital pathology and medical big data, automatic film reading technology based on artificial intelligence has gradually matured and begun to be applied in the fields of cervical cancer screening and blood cell analysis.

[0004] However, the use of artificial intelligence technology to read urine exfoliated cell liquid-based smears has not yet been applied. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a method for reading urine exfoliated cell liquid-based smears, which can realize automatic reading of urine exfoliated cell liquid-based smears.

[0006] The embodiment of the present invention also provides a urine exfoliated cell liquid-based smear reading system, which can realize automatic reading of urine exfoliated cell liquid-based smears.

[0007] The embodiment of the present invention is implemented as follows:

[0008] A method for reading urine exfoliated cell liquid-based smear, the method comprising:

[0009] Based on the convolutional neural network, the cell detection model, single cell classification model and cluster cell classification model were trained respectively;

[0010] Obtaining a digital image of a urine exfoliated cell liquid-based smear;

[0011] The digital image is input into the set cell detection model, and the coarse classification digital image including single cells and the coarse classification digital image including cluster cells are output;

[0012] Inputting the rough classification digital image including the single cell into the single cell classification model for processing to obtain the single cell recognition result;

[0013] Inputting the roughly classified digital image including the cluster cells into the cluster cell classification model for processing to obtain the recognition result of the cluster cells;

[0014] The obtained single cell identification result and the obtained cluster cell identification result are combined to obtain the identification result of the urine exfoliated cell liquid-based smear.

[0015] Preferably, the cell detection model, the single cell classification model and the cluster cell classification model respectively trained based on the convolutional neural network include:

[0016] imaging a plurality of urine exfoliated cell liquid-based smears to obtain digital images of the plurality of urine exfoliated cell liquid-based smears;

[0017] The obtained digital image is cut into blocks and the cell information is annotated on the block-based digital image, and a training data set and a test data set for the cell detection model, a training data set and a test data set for the single cell classification model, and a training data set and a test data set for the cluster cell classification model are generated respectively;

[0018] After the cell detection model is trained using the training data set of the cell detection model, the cell detection model is tested using the test data set of the cell detection model to obtain the cell detection model;

[0019] After the single-cell classification model is trained using the training data set of the single-cell classification model, the single-cell classification model is tested using the test data set of the single-cell classification model to obtain the single-cell classification model;

[0020] After the cluster cell classification model is trained using the training data set of the cluster cell classification model, and after the cluster cell classification model is tested using the test data set of the cluster cell classification model, the cluster cell classification model is obtained.

[0021] Preferably, the step of cutting and dividing the obtained digital image into blocks comprises:

[0022] Cutting the obtained digital image into repeated areas to obtain a block-based digital image;

[0023] The labeling of cell information on the divided digital image comprises:

[0024] The cell positions and cell types of the divided digital images are marked.

[0025] Preferably, the training to obtain the cell detection model includes:

[0026] A cell detection model is constructed based on a deformable convolutional neural network with an attention mechanism that does not require an anchor frame. The model extracts digital image features, uses a feature pyramid to perform three-layer feature fusion, and then makes a rough prediction of the cell position and category. The features of all cell regions are cross-compared using an attention mechanism, and then the category of each cell is predicted to obtain a coarsely classified digital image of a single cell and a coarsely classified digital image of a cluster of cells.

[0027] Preferably, the training to obtain a single cell classification model includes:

[0028] A single-cell classification model is constructed based on a convolutional neural network with weak supervision and lightweight cell nucleus detection methods. The model recognizes the cell nucleus area in the digital image through a lightweight detection method, extracts cell and cell nucleus features at the same time, and collaboratively analyzes the cell features and cell nucleus features. After classification, the single cell category is obtained as the single cell identification result.

[0029] Preferably, the cluster cell classification model obtained by training includes: constructing a cluster cell classification model based on a graph convolutional neural network, which uses a graph convolutional neural network to extract arrangement features of cluster cells in digital images, and uses a convolutional neural network to extract texture features and morphological features, and then splices the extracted features into feature vectors of cluster cells, and performs cluster cell classification using a multi-layer perceptron based on the feature vectors of cluster cells as the recognition result of cluster cells.

[0030] A urine exfoliated cell liquid-based smear reading system, the system comprises: a training unit, an acquisition unit and a reading unit, wherein:

[0031] A training unit, used for respectively training a cell detection model, a single cell classification model and a cluster cell classification model based on a convolutional neural network;

[0032] An acquisition unit, used for acquiring a digital image of a urine exfoliated cell liquid-based smear;

[0033] The image reading unit is used to input the digital image into the set cell detection model, and output a coarsely classified digital image including single cells and a coarsely classified digital image including cluster cells; input the coarsely classified digital image including single cells into the single cell classification model for processing to obtain the recognition result of single cells; input the coarsely classified digital image including cluster cells into the cluster cell classification model for processing to obtain the recognition result of cluster cells; merge the obtained single cell recognition result and the obtained cluster cell recognition result to obtain the recognition result of the urine exfoliated cell liquid-based smear.

[0034] Preferably, the training unit is also used to image multiple urine exfoliated cell liquid-based smears to obtain digital images of multiple urine exfoliated cell liquid-based smears; cut and block the obtained digital images and annotate the cell information of the blocked digital images to generate a training data set and a test data set for the cell detection model, a training data set and a test data set for the single cell classification model, and a training data set and a test data set for the cluster cell classification model respectively; after training the cell detection model with the training data set of the cell detection model, the cell detection model is tested and passed with the test data set of the cell detection model to obtain the cell detection model; after training the single cell classification model with the training data set of the single cell classification model, the single cell classification model is tested and passed with the test data set of the single cell classification model to obtain the single cell classification model; after training the cluster cell model with the training data set of the cluster cell classification model, the cluster cell classification model is tested and passed with the test data set of the cluster cell classification model to obtain the cluster cell classification model.

[0035] Preferably, the training unit is also used for the training to obtain a cell detection model including: constructing a cell detection model based on a deformable convolutional neural network with an attention mechanism that does not require an anchor frame, the model extracts digital image features, uses a feature pyramid to perform three-layer feature fusion, and then makes a rough prediction of the cell position and category; and uses the attention mechanism to perform feature cross-comparison on the features of all cell regions, and then predicts the category of each cell to obtain a coarsely classified digital image including a single cell and a coarsely classified digital image including clustered cells; and is also used for the training to obtain a single cell classification model including: constructing a single cell classification model based on a convolutional neural network with a weak supervision method and a lightweight cell nucleus detection method, the model recognizes the cell nucleus region in the digital image through a lightweight detection method, and simultaneously extracts cell and cell nucleus features, and after collaboratively analyzing the cell features and cell nucleus features, a single cell category is obtained through classification as the single cell recognition result.

[0036] The training unit is also used to train a cluster cell classification model, including: building a cluster cell classification model based on a graph convolutional neural network, the model uses a graph convolutional neural network to extract arrangement features of cluster cells in a digital image, and uses a convolutional neural network to extract texture features and morphological features, and then splices the extracted features into feature vectors of cluster cells, and performs cluster cell classification using a multi-layer perceptron based on the feature vectors of cluster cells as the recognition result of the cluster cells.

[0037] A urine exfoliated cell liquid-based smear reading device, comprising:

[0038] A memory; and a processor coupled to the memory, the processor being configured to execute any of the above-mentioned methods for reading urine exfoliated cell liquid-based smears based on instructions stored in the memory.

[0039] As can be seen above, the embodiment of the present invention respectively sets a cell detection model, a single cell classification model and a cluster cell classification model based on a convolutional neural network. After scanning and acquiring a digital image of a urine exfoliated cell liquid-based smear, the digital image is input into the set cell detection model, and the output is obtained by obtaining a coarsely classified digital image including single cells and a coarsely classified digital image including cluster cells. The coarsely classified digital image including single cells is input into the single cell classification model for processing to obtain the recognition result of single cells; the coarsely classified digital image including cluster cells is input into the cluster cell classification model for processing to obtain the recognition result of cluster cells; the recognition result of single cells obtained and the recognition result of cluster cells obtained are merged to obtain the recognition result of the urine exfoliated cell liquid-based smear, and output. In this way, the embodiment of the present invention realizes the automatic reading of urine exfoliated cell smears. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A flow chart of a method for reading urine exfoliated cell liquid-based smear provided in an embodiment of the present invention;

[0041] Figure 2 A flow chart of a method for obtaining a cell detection model, a single cell classification model, and a cluster cell classification model through training provided in an embodiment of the present invention;

[0042] Figure 3 A schematic diagram of the implementation process of the cell detection model provided in an embodiment of the present invention;

[0043] Figure 4 An overall implementation flow chart of the single cell classification model provided by an embodiment of the present invention;

[0044] Figure 5 The overall implementation flow chart of the cluster cell classification model provided by the embodiment of the present invention;

[0045] Figure 6A schematic diagram of the structure of a urine exfoliated cell liquid-based smear reading system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0047] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0048] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0049] As can be seen from the background technology, since the reading of urine exfoliated cell liquid-based smears is not realized by artificial intelligence technology, the automatic reading process of urine exfoliated cell liquid-based smears cannot be realized at present. How to realize the automatic reading of urine exfoliated cell liquid-based smears based on the characteristics of cells in urine exfoliated cell liquid-based smears by using artificial intelligence technology is a problem to be solved in the embodiment of the present invention. Therefore, the embodiment of the present invention respectively sets a cell detection model, a single cell classification model and a cluster cell classification model based on a convolutional neural network, scans and obtains a digital image of urine exfoliated cell liquid-based smears, inputs the digital image into the set cell detection model, outputs a rough classification digital image including a single cell and a rough classification digital image including cluster cells, inputs the rough classification digital image including a single cell into the single cell classification model for processing, and obtains a recognition result of a single cell; inputs the rough classification digital image including cluster cells into the cluster cell classification model for processing, and obtains a recognition result of cluster cells; merges the recognition result of the obtained single cell and the recognition result of the obtained cluster cells, and obtains the recognition result of the urine exfoliated cell liquid-based smear, and outputs it.

[0050] In this way, the embodiment of the present invention realizes the automatic reading process of urine exfoliated cell smear.

[0051] It can be seen that the embodiment of the present invention uses a convolutional neural network and introduces the clinical experience of pathologists to construct three models dedicated to the field of urine exfoliated cytology, thereby realizing automatic reading of urine exfoliated cytology. Compared with the manual reading provided by the background technology, the embodiment of the present invention effectively improves the reading efficiency and accuracy of urine exfoliated cytology liquid-based smears and reduces the occurrence of missed diagnosis.

[0052] The embodiment of the present invention actually integrates artificial intelligence technology and knowledge of urine exfoliated cell pathology to construct three models based on convolutional neural networks, thereby improving the weakness of using only one convolutional neural network in the identification of urine exfoliated cells, creating a dedicated recognition model, completing the automatic film reading process, and promoting the realization and application of automatic film reading of urine exfoliated cells.

[0053] Figure 1 A flow chart of a method for reading urine exfoliated cell liquid-based smear provided in an embodiment of the present invention, wherein the specific steps include:

[0054] Step 101: Based on the convolutional neural network, a cell detection model, a single cell classification model and a cluster cell classification model are respectively trained;

[0055] Step 102, obtaining a digital image of the urine exfoliated cell liquid-based smear;

[0056] Step 103, inputting the digital image into the set cell detection model, and outputting a coarsely classified digital image including single cells and a coarsely classified digital image including cluster cells;

[0057] Step 104, inputting the coarse classification digital image including the single cell into the single cell classification model for processing to obtain the recognition result of the single cell;

[0058] Step 105, inputting the roughly classified digital image including the cluster cells into the cluster cell classification model for processing to obtain the recognition result of the cluster cells;

[0059] Step 106: Combine the obtained single cell identification result and the obtained cluster cell identification result to obtain the identification result of the urine exfoliated cell liquid-based smear.

[0060] In this method, the cell detection model, the single cell classification model and the cluster cell classification model are respectively trained based on the convolutional neural network and include:

[0061] imaging a plurality of urine exfoliated cell liquid-based smears to obtain digital images of the plurality of urine exfoliated cell liquid-based smears;

[0062] The obtained digital image is cut into blocks and the cell information is annotated on the block-based digital image, and a training data set and a test data set for the cell detection model, a training data set and a test data set for the single cell classification model, and a training data set and a test data set for the cluster cell classification model are generated respectively;

[0063] After the cell detection model is trained using the training data set of the cell detection model, the cell detection model is tested using the test data set of the cell detection model to obtain the cell detection model;

[0064] After the single-cell classification model is trained using the training data set of the single-cell classification model, the single-cell classification model is tested using the test data set of the single-cell classification model to obtain the single-cell classification model;

[0065] After the cluster cell classification model is trained using the training data set of the cluster cell classification model, and after the cluster cell classification model is tested using the test data set of the cluster cell classification model, the cluster cell classification model is obtained.

[0066] Here, the obtained digital image is cut and divided into blocks including:

[0067] The obtained digital image is cut into repeated areas to obtain a block-based digital image.

[0068] Here, the labeling of cell information on the digital image after segmentation includes:

[0069] The cell positions and cell types of the divided digital images are marked.

[0070] In this method, the training to obtain a cell detection model includes:

[0071] A cell detection model is constructed based on a deformable convolutional neural network with an attention mechanism that does not require an anchor frame. The model extracts digital image features, uses a feature pyramid to perform three-layer feature fusion, and then makes a rough prediction of the cell position and category. The features of all cell regions are cross-compared using an attention mechanism, and then the category of each cell is predicted to obtain a coarsely classified digital image of a single cell and a coarsely classified digital image of a cluster of cells.

[0072] In this method, the training to obtain a single cell classification model includes:

[0073] A single-cell classification model is constructed based on a convolutional neural network with weak supervision and lightweight cell nucleus detection methods. The model recognizes the cell nucleus area in the digital image through a lightweight detection method, extracts cell and cell nucleus features at the same time, and collaboratively analyzes the cell features and cell nucleus features. After classification, the single cell category is obtained as the single cell identification result.

[0074] In this method, the training to obtain the cluster cell classification model includes:

[0075] A cluster cell classification model is constructed based on a graph convolutional neural network. The model uses a graph convolutional neural network to extract the arrangement features of cluster cells in digital images, and uses a convolutional neural network to extract texture features and morphological features. The extracted features are then concatenated into feature vectors of cluster cells. Cluster cell classification is performed using a multi-layer perceptron based on the feature vectors of cluster cells as the recognition result of cluster cells.

[0076] The embodiment of the present invention can construct an automatic film reading system based on the above method, thereby realizing automatic film reading of urine exfoliated cell liquid-based smears.

[0077] The following is a detailed description of the process of training a cell detection model, a single cell classification model, and a cluster cell classification model in accordance with the embodiment of the present invention with specific examples.

[0078] like Figure 2 As shown, Figure 2 A flowchart of a method for training a cell detection model, a single cell classification model, and a cluster cell classification model provided in an embodiment of the present invention, wherein the specific steps include:

[0079] Step 1: Obtain multiple urine exfoliated cell fluid-based smears, including negative and positive smears.

[0080] In this step, 2000 urine exfoliated cell liquid-based smears can be collected, including 1500 negative smears and 500 positive smears.

[0081] Step 2: obtaining digital images of multiple urine exfoliated cell liquid-based smears.

[0082] In this step, a digital scanner is used to scan and obtain digital images of a plurality of urine exfoliated cell liquid-based smears.

[0083] Step 3: Cut and divide the obtained digital image into blocks and annotate the cell information of the divided digital image to generate the training data set and test data set of the cell detection model, the training data set and test data set of the single cell classification model, and the training data set and test data set of the cluster cell classification model.

[0084] In this step, for each digital image of the urine exfoliated cell liquid-based smear, the repeated area is cut to obtain image blocks of 1024×1024 size.

[0085] In this step, after labeling all the obtained digital images after segmentation, such as cell location and cell category, the labeled digital image set is divided into a training data set and a test data set in a ratio of 8:2, which are used as the training data set and the test data set of the cell detection model. Here, the cell information labeling is performed by a professional pathologist who selects blocks with diagnostic significance to label the cell location and cell category.

[0086] In this step, the digital images after cell information annotation are classified according to cell categories into digital images with cluster cells and digital images with single cells, where the size of cluster cells is uniformly scaled to 448x448, and the size of single cells is uniformly scaled to 128x128.

[0087] The digital images with cluster cells are divided into a training data set and a test data set in a ratio of 8:2, which are used as the training data set and the test data set of the cluster cell classification model.

[0088] The digital images with single cells are divided into a training data set and a test data set in a ratio of 8:2 as the training data set and the test data set of the single cell classification model. Here, in order to realize the weakly supervised cell nucleus localization function in single cells, the cell nucleus regions of some single cells in the training set need to be labeled.

[0089] Step 4: Training of cell detection model.

[0090] In this step, the core of the automatic cytology film reading process is to achieve cell positioning and cell category recognition. The use of convolutional neural networks can achieve object positioning and category recognition, which can initially meet the needs of automatic identification of urine exfoliated cells. However, convolutional neural networks mainly focus on positioning, and their classification accuracy needs to be improved. Category recognition is particularly important in cell diagnosis. In order to improve the accuracy of urine exfoliated cell category recognition, the classification ability of the detection model is optimized, and an independent cell classification model is used to assist classification after cell detection. Because urine exfoliated cells have clusters and single cells, the cell morphology is quite different, and the diagnostic basis of cluster cells and single cells is different, different classification models are trained for single cells and cluster cells respectively.

[0091] In this step, the current mainstream detection model and medical expertise are combined to train a cell detection model suitable for urine exfoliated cell detection, and realize automatic positioning and category recognition of urine exfoliated cells. Doctors need to continuously cross-compare the morphology of different cells in the actual film reading process to determine whether the change in cell morphology is caused by the disease and determine the extent of the lesion. Here, based on the correlation between the doctor's diagnostic criteria and cell categories, a cell detection model for modeling cell relationships in a fusion field of view is proposed. Through the attention mechanism, a comparative relationship is established between all cells in the field of view, and cross-comparison of cell features is realized. The process of comparing cell morphology in the doctor's diagnosis is simulated, and a coarsely classified digital image including single cells and a coarsely classified digital image including clustered cells is obtained.

[0092] Step 5: Training of single-cell classification model.

[0093] In this step, a weakly supervised training method is used to target single cells that are commonly found in urine exfoliated cells. The traditional classification network has poor interpretability of the focus area in cell classification, is detached from medical knowledge, and relies entirely on network autonomous learning. This allows the model to focus on single cell classification tasks and classify the cell nucleus area.

[0094] This model is a lightweight cell nucleus detection model. It combines weak supervision technology to use the annotated cell nucleus positions in some cells to achieve rough positioning of the cell nucleus. After explicitly extracting the cell nucleus region features, the cell nucleus features are fused with the global cell features, and the single cell classification results are obtained through the classification layer.

[0095] Step 6: Training of cluster cell classification model.

[0096] In this step, cluster cells often appear in urine exfoliated cell liquid-based smears due to the existence of cell stacking. The identification of cluster cells is not only related to the characteristics of each cell, but also has a certain relationship with the cluster as a whole. Therefore, in order to achieve more accurate cluster cell classification and make the model more interpretable, a graph convolutional neural network is used here to train the cluster cell classification model. The model can extract the arrangement relationship in cluster cells, construct multi-dimensional information of cluster cell images, and improve the accuracy of cluster cell recognition.

[0097] Step 7: Build the film reading system.

[0098] In this step, based on the trained cell detection model, the trained single cell classification model and the trained cluster cell classification model, the three models are orderly combined to construct a film reading system. In the film reading system, the digital image is first processed into overlapping blocks, and each block is respectively subjected to the cell detection model for cell positioning and initial category judgment, and then the results of the detection model are respectively input into the cluster cell classification model and the single cell classification model for category refinement, and finally the recognition results of all cells are integrated, and the location and category of the diseased cells are automatically prompted for the digital image with diseased cells.

[0099] It can be seen that the core of the embodiment of the present invention is to construct a cell detection model, a single cell classification model and a cluster cell classification model. These three models provided by the embodiment of the present invention are described in detail.

[0100] Cellular detection model

[0101] Detection and positioning of cells with diagnostic significance is the basis of auxiliary diagnosis of urine exfoliated cells. Traditional deep learning-based detection models mainly include two branches, one is a single-step detection framework represented by YOLO and SSD, and the other is a two-step detection framework represented by Faster R-CNN and its improved model. Traditional detection models are based on preset anchor frames for detection. The setting of early anchor frames reduces the difficulty of training detection models and improves detection accuracy. However, anchor frames also have some problems, such as strong reliance on statistical information of training data sets, resulting in insufficient generalization, and difficulty in adapting to objects with large changes in size and aspect ratio. With the development of technology, detection methods that do not require anchor frames have emerged. Such detection methods can adapt to objects of different sizes and shapes, and are more suitable for the situation of variable cell morphology. Therefore, the embodiment of the present invention is based on the current detection model that does not require anchor frames, that is, the deformable convolutional neural network model (Reppoints), and introduces medical field expertise into the detection model through the attention module to form a convolutional neural network model suitable for urine exfoliated cell detection, so as to realize automatic positioning and initial category determination of urine exfoliated cells.

[0102] Based on the correlation between doctors' diagnostic criteria and cell categories, an embodiment of the present invention proposes a cell detection model that integrates attention mechanism and relationship modeling. The cell detection model uses a convolutional neural network to extract image features, and then uses a feature pyramid to perform three-layer feature fusion, and then performs a rough prediction of cell position and category, so as to screen out some areas with cells and reduce the amount of cross-comparison calculations. Then, the features of all cell areas are cross-compared using an attention mechanism to simulate the process of comparing different types of cells in a doctor's diagnosis, and then the category of each cell is predicted. The specific process is as follows: Figure 3 As shown, Figure 3A schematic diagram of the implementation process of the cell detection model provided in an embodiment of the present invention.

[0103] Single cell classification model

[0104] Traditional deep learning-based image classification methods use convolutional neural networks to extract image features for classification, relying purely on network autonomous learning, while cytological diagnosis has complete diagnostic specifications and judgment logic that is different from ordinary images. Therefore, the embodiment of the present invention integrates medical expertise into the image classification model to improve the interpretability and generalization of the model and achieve accurate classification of single cells exfoliated in urine.

[0105] In the process of cytological diagnosis, the cell nucleus provides important reference information. The embodiment of the present invention proposes a single-cell classification model that focuses on integrating cell nucleus features. By using weak supervision and lightweight cell nucleus detection methods, cell nucleus positioning and cell nucleus feature extraction are explicitly implemented in the cell classification network, highlighting the importance of cell nuclei in single-cell classification, which is in line with doctors' diagnostic habits and cytological diagnostic standards, and improves model classification accuracy and interpretability. The overall implementation process of the single-cell classification model is as follows: Figure 4 As shown, it includes: firstly, the cell nucleus region is recognized by a lightweight detection method, and the cell and cell nucleus features are extracted at the same time, and the cell features and cell nucleus features are analyzed collaboratively to obtain more comprehensive features for cell classification, and finally a single cell category is obtained through the classification layer.

[0106] Cluster cell classification model

[0107] Due to the existence of cell stacking, clustered cells often appear in urine exfoliated cell liquid-based smears. The identification of clustered cells is not only related to the characteristics of each cell, but also has a certain relationship with the cluster as a whole. Therefore, in order to achieve more accurate cluster cell classification and make the model more interpretable, the embodiment of the present invention uses a multi-channel feature extraction method that integrates a graph convolutional neural network to construct multi-dimensional information of clustered cell images.

[0108] The overall implementation process of the cluster cell classification model is as follows: Figure 5 As shown, the method includes: locating the key points of the nucleus of each cell in the cluster cell image through key point detection, then using the key points as nodes of the graph data structure to construct a graph structure of the cluster cell image, extracting the arrangement features of the cluster cell image through a graph convolutional network, and using a convolutional neural network to extract the texture, morphology and other features of the cluster image, and obtaining a more expressive feature vector of the cluster cell image by splicing the features extracted by the graph convolutional network and the convolutional neural network, and finally inputting the feature vector into a multi-layer perceptron to realize cluster cell classification.

[0109] Figure 6A schematic diagram of the structure of a urine exfoliated cell liquid-based smear reading system provided by an embodiment of the present invention includes: a training unit, an acquisition unit and a reading unit, wherein:

[0110] A training unit, used for respectively training a cell detection model, a single cell classification model and a cluster cell classification model based on a convolutional neural network;

[0111] An acquisition unit, used for acquiring a digital image of a urine exfoliated cell liquid-based smear;

[0112] The image reading unit is used to input the digital image into the set cell detection model, and output a coarsely classified digital image including single cells and a coarsely classified digital image including cluster cells; input the coarsely classified digital image including single cells into the single cell classification model for processing to obtain the recognition result of single cells; input the coarsely classified digital image including cluster cells into the cluster cell classification model for processing to obtain the recognition result of cluster cells; merge the obtained single cell recognition result and the obtained cluster cell recognition result to obtain the recognition result of the urine exfoliated cell liquid-based smear.

[0113] In the system, the training unit is also used to image multiple urine exfoliated cell liquid-based smears to obtain digital images of multiple urine exfoliated cell liquid-based smears; cut and block the obtained digital images and annotate the cell information of the blocked digital images to generate a training data set and a test data set for a cell detection model, a training data set and a test data set for a single cell classification model, and a training data set and a test data set for a cluster cell classification model respectively; after training the cell detection model with the training data set of the cell detection model, the cell detection model is tested with the test data set of the cell detection model to obtain the cell detection model; after training the single cell classification model with the training data set of the single cell classification model, the single cell classification model is tested with the test data set of the single cell classification model to obtain the single cell classification model; after training the cluster cell model with the training data set of the cluster cell classification model, the cluster cell classification model is tested with the test data set of the cluster cell classification model to obtain the cluster cell classification model.

[0114] In the system, the training unit is also used for the training to obtain the cell detection model, including:

[0115] A cell detection model is constructed based on a deformable convolutional neural network with an attention mechanism that does not require an anchor frame. The model extracts digital image features, uses a feature pyramid to perform three-layer feature fusion, and then makes a rough prediction of the cell position and category. The features of all cell regions are cross-compared using an attention mechanism, and then the category of each cell is predicted to obtain a coarsely classified digital image of a single cell and a coarsely classified digital image of a cluster of cells.

[0116] In the system, the training unit is also used for the training to obtain a single-cell classification model, including: constructing a single-cell classification model based on a convolutional neural network with a weak supervision method and a lightweight cell nucleus detection method, the model recognizes the cell nucleus area in the digital image through a lightweight detection method, and simultaneously extracts cell and cell nucleus features, after collaboratively analyzing the cell features and cell nucleus features, a single cell category is obtained through classification as a single cell identification result.

[0117] In the system, the training unit is also used in the method, wherein the training to obtain the cluster cell classification model includes: constructing a cluster cell classification model based on a graph convolutional neural network, the model using a graph convolutional neural network to extract arrangement features of cluster cells in a digital image, and using a convolutional neural network to extract texture features and morphological features, and then splicing the extracted features into feature vectors of cluster cells, and performing cluster cell classification using a multi-layer perceptron based on the feature vectors of cluster cells as the recognition result of the cluster cells.

[0118] It can be seen from the method and system provided in the embodiment of the present invention that the embodiment of the present invention constructs a detection and classification model of urine exfoliated cells that integrates professional knowledge in the medical field, and improves the professionalism and interpretability of convolutional neural networks in the field of urine exfoliated cell identification. The embodiment of the present invention fully realizes the digital image automatic reading system of urine exfoliated cell liquid-based smears, which changes the traditional single-microscope reading method and improves the work efficiency of doctors.

[0119] The flow chart and block diagram in the accompanying drawings of the present application show the possible architecture, function and operation of the system, method and computer program product according to the various embodiments disclosed in the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the box can also occur in the order of the standards in different figures. For example, the boxes represented by two connections can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0120] The embodiment of the present invention further provides an electronic device, in which the device for implementing the method of the embodiment of the present application can be integrated. Specifically:

[0121] The electronic device may include a processor with one or more processing cores, a memory with one or more computer-readable storage media, and a computer program stored in the memory and executable on the processor. When executing the program in the memory, the above-mentioned method for reading urine exfoliated cell liquid-based smears may be implemented.

[0122] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present application. In particular, without departing from the spirit and teachings of the present application, the features described in the various embodiments and / or claims of the present application may be combined and / or combined in a variety of ways, and all of these combinations and / or combinations fall within the scope disclosed in the present application.

[0123] Specific embodiments are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas, and is not used to limit the present application. For those skilled in the art, changes can be made in the specific implementation methods and application scopes according to the ideas, spirits and principles of the present invention, and any modifications, equivalent substitutions, improvements, etc. made therein should be included in the scope of protection of this application.

Claims

1. A method for reading urine exfoliated cell liquid-based smear, characterized in that: The method comprises: Based on the convolutional neural network, the cell detection model, single cell classification model and cluster cell classification model were trained respectively; Obtaining a digital image of a urine exfoliated cell liquid-based smear; The digital image is input into the set cell detection model, and the coarse classification digital image including single cells and the coarse classification digital image including cluster cells are output; Inputting the coarse classification digital image including the single cell into the single cell classification model for processing to obtain the single cell recognition result; wherein the single cell classification model realizes the recognition of the cell nucleus region by a lightweight detection method, extracts the cell and cell nucleus features at the same time, and collaboratively analyzes the cell features and cell nucleus features to obtain more comprehensive features for cell classification, and finally obtains the single cell category through the classification layer; Inputting the roughly classified digital image including the cluster cells into the cluster cell classification model for processing to obtain the recognition result of the cluster cells; The obtained single cell identification result and the obtained cluster cell identification result are combined to obtain the identification result of the urine exfoliated cell liquid-based smear; The cell detection model, the single cell classification model and the cluster cell classification model are respectively trained based on the convolutional neural network and include: imaging a plurality of urine exfoliated cell liquid-based smears to obtain digital images of the plurality of urine exfoliated cell liquid-based smears; The obtained digital image is cut into blocks and the cell information is annotated on the block-based digital image, and a training data set and a test data set for the cell detection model, a training data set and a test data set for the single cell classification model, and a training data set and a test data set for the cluster cell classification model are generated respectively; After the cell detection model is trained using the training data set of the cell detection model, the cell detection model is tested using the test data set of the cell detection model to obtain the cell detection model; After the single-cell classification model is trained using the training data set of the single-cell classification model, the single-cell classification model is tested using the test data set of the single-cell classification model to obtain the single-cell classification model; After the cluster cell classification model is trained using the training data set of the cluster cell classification model, and after the cluster cell classification model is tested using the test data set of the cluster cell classification model, the cluster cell classification model is obtained; Wherein, the cell detection model obtained by training includes: A cell detection model is constructed based on a deformable convolutional neural network with an attention mechanism that does not require an anchor frame. The model extracts digital image features, uses a feature pyramid to perform three-layer feature fusion, and then makes a rough prediction of the cell position and category. The features of all cell regions are cross-compared using an attention mechanism, and then the category of each cell is predicted to obtain a coarsely classified digital image of a single cell and a coarsely classified digital image of a cluster of cells.

2. The method according to claim 1, characterized in that The cutting and dividing of the obtained digital image comprises: Cutting the obtained digital image into repeated areas to obtain a block-based digital image; The labeling of cell information on the divided digital image comprises: The cell positions and cell types of the divided digital images are marked.

3. The method according to claim 1, characterized in that The training to obtain the single cell classification model includes: A single-cell classification model is constructed based on a convolutional neural network with weak supervision and lightweight cell nucleus detection methods. The model recognizes the cell nucleus area in the digital image through a lightweight detection method, extracts cell and cell nucleus features at the same time, and collaboratively analyzes the cell features and cell nucleus features. After classification, the single cell category is obtained as the single cell identification result.

4. The method according to claim 1, characterized in that The training to obtain a cluster cell classification model includes: constructing a cluster cell classification model based on a graph convolutional neural network, wherein the model uses a graph convolutional neural network to extract arrangement features of cluster cells in a digital image, and uses a convolutional neural network to extract texture features and morphological features, then concatenates the extracted features into feature vectors of cluster cells, and performs cluster cell classification using a multi-layer perceptron based on the feature vectors of cluster cells as a recognition result of the cluster cells.

5. A urine exfoliated cell liquid-based smear reading system, characterized in that: The system comprises: a training unit, an acquisition unit and an image reading unit, wherein: A training unit, used for respectively training a cell detection model, a single cell classification model and a cluster cell classification model based on a convolutional neural network; An acquisition unit, used for acquiring a digital image of a urine exfoliated cell liquid-based smear; The image reading unit is used to input the digital image into the set cell detection model, and output a coarse classification digital image including single cells and a coarse classification digital image including cluster cells; input the coarse classification digital image including single cells into the single cell classification model for processing to obtain the recognition result of single cells; input the coarse classification digital image including cluster cells into the cluster cell classification model for processing to obtain the recognition result of cluster cells; merge the obtained single cell recognition result and the obtained cluster cell recognition result to obtain the recognition result of the urine exfoliated cell liquid-based smear; wherein the single cell classification model realizes the recognition of the cell nucleus area through a lightweight detection method, extracts the cell and cell nucleus features at the same time, and collaboratively analyzes the cell features and the cell nucleus features to obtain more comprehensive features for cell classification, and finally obtains the single cell category through the classification layer; The training unit is also used to image multiple urine exfoliated cell liquid-based smears to obtain digital images of multiple urine exfoliated cell liquid-based smears; cut and block the obtained digital images and annotate the cell information of the blocked digital images to generate a training data set and a test data set for a cell detection model, a training data set and a test data set for a single cell classification model, and a training data set and a test data set for a cluster cell classification model respectively; after training the cell detection model with the training data set of the cell detection model, the cell detection model is tested and passed with the test data set of the cell detection model to obtain the cell detection model; after training the single cell classification model with the training data set of the single cell classification model, the single cell classification model is tested and passed with the test data set of the single cell classification model to obtain the single cell classification model; after training the cluster cell model with the training data set of the cluster cell classification model, the cluster cell classification model is tested and passed with the test data set of the cluster cell classification model to obtain the cluster cell classification model; Among them, the training unit is also used for the training to obtain a cell detection model, including: building a cell detection model based on a deformable convolutional neural network with an attention mechanism that does not require an anchor frame, the model extracts digital image features, uses a feature pyramid to perform three-layer feature fusion, and then makes a rough prediction of the cell position and category; and uses an attention mechanism to cross-compare the features of all cell regions, and then predicts the category of each cell to obtain a coarsely classified digital image including a single cell and a coarsely classified digital image including clustered cells.

6. The system according to claim 5, characterized in that The training unit is also used for the training to obtain a single cell classification model, including: constructing a single cell classification model based on a convolutional neural network with a weak supervision method and a lightweight cell nucleus detection method, the model identifies the cell nucleus area in the digital image through a lightweight detection method, and extracts cell and cell nucleus features at the same time. After collaborative analysis of the cell features and cell nucleus features, a single cell category is obtained through classification as a single cell recognition result; and is also used for the training to obtain a cluster cell classification model, including: constructing a cluster cell classification model based on a graph convolutional neural network, the model uses a graph convolutional neural network to extract arrangement features of cluster cells in digital images, and uses a convolutional neural network to extract texture features and morphological features, and then splicing the extracted features into feature vectors of cluster cells, and performing cluster cell classification by a multi-layer perceptron based on the feature vectors of cluster cells as the recognition result of cluster cells.

7. A urine exfoliated cell liquid-based smear reading device, characterized in that: include: Memory; And a processor coupled to the memory, the processor being configured to execute the method for reading urine exfoliated cell liquid-based smears as described in any one of claims 1 to 4 based on instructions stored in the memory.

Images