Automatic Interpretation System for Cytopathological Smears

Abstract

The present invention discloses an automatic interpretation system for cytopathological smears, comprising: an imaging module; an image acquisition module; an image storage and management module; an image preprocessing module; an intelligent interpretation module, which receives the image output by the image preprocessing module, predicts and classifies the image into normal samples and lesion samples, and further predicts and interprets the lesion type when the image is a lesion sample; and a report writing module, which automatically generates an interpretation text of the sample corresponding to the image. The present invention successfully applies the artificial intelligence-assisted diagnosis technology to rapid staining cytopathology, can significantly improve the accuracy and consistency of diagnosis, and reduce the workload of cytopathology doctors; by improving the existing convolutional neural network and utilizing the characteristics of the multi-channel attention mechanism, the present invention solves the uncertain factors introduced by manual sampling, can achieve high-accuracy full-scene and multi-classification tasks, and finally improves the accuracy of the classification and interpretation results.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence-assisted diagnosis and pathology, and particularly to an automatic interpretation system for cytopathological smears. Background Art

[0002] Lung cancer is a disease with the highest incidence and mortality rate worldwide, and it is also on the rise year by year in China, seriously endangering people's lives, health and safety. In 2017, the global incidence of lung cancer was 12.22 / 100,000, and the mortality rate was 19.88 / 100,000. Data from the World Cancer Foundation shows that the incidence rate of lung cancer in China in 2018 was 35.1 / 100,000, ranking 16th in the world. And data published by the National Cancer Center of China in 2019 shows that the incidence rate was 57.6 / 100,000 and the mortality rate was 45.87 / 100,000. Since there are no specific symptoms in the early stage of this disease, up to 2 / 3 of lung cancer patients are in the middle-late stage when they seek medical treatment, and the patients lose the indication for surgical treatment and the conditions for histopathological diagnosis of surgically resected specimens. Therefore, the pathological diagnosis of such patients often needs to be determined by fine needle aspiration of lung masses. Interventional treatment is the main treatment method for such patients, and pathological diagnosis will directly affect the choice of the patient's treatment plan. At present, the treatment of lung cancer is mainly based on individualized precision treatment. Using cytological specimens or core needle biopsy specimens for molecular pathological detection is the future diagnosis and treatment direction for middle-late stage lung cancer patients. And using rapid cytopathology technology (ROSE technology) for on-site determination is a diagnostic method that has been carried out internationally. It can provide information such as the adequacy of the puncture specimen, preliminary diagnosis and the next diagnosis and treatment direction for interventional doctors during an interventional diagnosis and treatment process, and can also provide the basis for related tumor interventional treatment at the same time. Diff-quik staining has become the preferred technology for on-site determination due to its rapidity and simplicity. Compared with the traditional method of specimen judgment by clinical interventional doctors based on experience, rapid cytopathology technology can effectively reduce the probability of repeated puncture or misdiagnosis of patients. Diff-quik staining reduces the number of undiagnosed cases by increasing the adequacy rate of specimens. According to data from the University of Pittsburgh School of Medicine in the United States (Guoping Cai, adebowale JAdeniran. Rapid on-site evaluation (ROSE)-a practical guide[M]. Springer 2019:8), the average undiagnosed ratio in cytopathological diagnosis without diff-quik staining is about 20%, while with diff-quik staining, the undiagnosed rate is 2-10%. It not only reduces the possibility of repeated puncture for material collection, but also there is no statistical difference in intraoperative complications compared with the control group without diff-quik technology.

[0003] The determination of rapid cytological diagnosis is generally completed by full-time cytopathologists. However, the cultivation of qualified cytologists takes a long time, and the vast majority of primary hospitals do not have full-time cytopathologists, which will greatly restrict the development and popularization of interventional medicine. In addition, domestic cytologists are not familiar with the image features of diff-quick staining, and the images of HE staining are quite different from those of diff-quick staining. Therefore, mastering this diagnostic technology will also cost a large amount of manpower and material resources. Due to the relatively high subjectivity of cytologist determination, there are differences in diagnosis among individuals, and it takes a long time. In large domestic public hospitals, the workload of cytopathologists far exceeds the standard of less than 100 cervical smears per day limited in European and American countries. Therefore, the on-site rapid cytological determination will further increase the workload of cytopathologists.

[0004] Artificial intelligence is a branch of computer science. An important application is to perform specific semantic segmentation or classification and identification of images by analyzing the acquired image data. The development of artificial intelligence has revolutionarily changed the future mode of morphological diagnosis, mainly manifested in the diagnosis of histopathology. With the development of artificial intelligence technology, auxiliary systems constructed using artificial intelligence algorithms have been applied to the pathological images of tissue sections and medical imaging images of multiple lesions. Based on objective and quantitative high-dimensional image features, they automatically and rapidly classify and diagnose images, which is a new type of clinical diagnosis and treatment technology with high feasibility.

[0005] Artificial intelligence abroad is mainly applied to thyroid fine needle aspiration specimens in non-gynecological cytology (Landau MS, Pantanowitz L. Artificial intelligence in cytopathology: a review of the literature and overview of commercial landscape[J]. J Am Soc Cytopathol, 2019;8(4):230-241. doi:10.1016 / j.jasc.2019.03.003. Epub 2019 Mar 25.). Urine specimens, breast fine needle aspiration specimens, hepatobiliary and pancreatic fine needle aspiration specimens, pleural and ascites specimens, and lung specimens are all in the research stage and have not been commercialized. Moreover, most of the images of these artificial intelligence are traditional Papanicolaou stains. In 2017, Teramoto (Teramoto A, Tsukamoto T, Kiriyama Y, et al. Automated Classification of Lung Cancer Types from Cytological Images Using Deep Convolutional Neural Networks[J]. Biomed Res Int, 2017;2017:4067832.) et al. used deep convolutional neural network (DCNN) technology to distinguish adenocarcinoma, squamous cell carcinoma, and small cell carcinoma in lung fine needle aspiration liquid-based cytology specimens. The accuracy rate reached 71.1%, and the accuracy rate for differentiating non-small cell carcinoma and small cell carcinoma reached 85.6%. However, the limitation of this study was that it could only distinguish the main types of lung cancer and could not distinguish the benign and malignant nature of the specimens.

[0006] Artificial intelligence in China is in its infancy in the field of lung cancer diagnosis. Wang Juan et al. (Wang Juan, Yang Zhen, Zhu Qiang, et al. Preliminary application of an artificial intelligence-based cytopathological diagnosis system in lung cancer diagnosis[J]. Acta Academiae Medicinae Militaris Tertiae, 2020, 41(9):897-900. DOI: 10.3969 / j.issn.2095-5227.2020.09.012.) were the first to use artificial intelligence (artificial neural network algorithm) in the application of cytopathological diagnosis of lung cancer. It mainly analyzes the ploidy of cancer cells based on the principle of DNA ploidy analysis. When the ploidy of cancer cells is greater than or equal to 5c, cancer cells are detected (2c represents normal cells). A total of 101 lung specimens were included, including bronchoscopic brushing, endoscopic needle aspiration, lung mass needle aspiration, and pleural and ascites, etc., and preliminary results were obtained: The coincidence rate between artificial intelligence cytopathological diagnosis and the gold standard was 66.3%, the sensitivity was 67%, and the specificity was 60%, showing a significant difference from the gold standard pathological method. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide an automatic interpretation system for cytopathological smears in view of the deficiencies in the above-mentioned prior art.

[0008] To solve the above technical problem, the technical solution adopted by the present invention is: an automatic interpretation system for cytopathological smears, comprising:

[0009] An imaging module;

[0010] An image acquisition module, which controls the imaging module to image the cytopathological smear to realize the image acquisition of the cytopathological smear;

[0011] An image storage and management module, which stores the images acquired by the image acquisition module and organizes them into an image sample library;

[0012] An image preprocessing module, which is used to preprocess the images output by the image sample library and output them after removing the unstained areas in the images;

[0013] An intelligent interpretation module, which receives the images output by the image preprocessing module, predicts and classifies the images into normal samples and lesion samples, and further predicts and interprets the lesion types when the images are lesion samples;

[0014] And a report writing module, which automatically generates an interpretation conclusion text of the sample corresponding to the image according to the result of the intelligent interpretation module.

[0015] Preferably, the imaging module includes a digital microscope, a motorized stage and a host computer; the motorized stage has X, Y, and Z axis moving functions, the cytopathological smear is placed on the motorized stage, and is imaged by the digital microscope, and the obtained images are transmitted to the host computer;

[0016] The automatic interpretation system for cytopathological smears further includes a motorized stage control module and an autofocus module, the motorized stage control module is used to control the movement of the motorized stage; the autofocus module is connected to the motorized stage control module and realizes autofocus by controlling the movement of the motorized stage.

[0017] Preferably, the image acquisition module is connected to the host computer to receive the images output by the host computer, and the working process of the image acquisition module is as follows:

[0018] First, control the motorized stage to move so that its center coincides with the central axis of the objective lens of the digital microscope, then complete autofocus through the focus control module, and then fix the focal length;

[0019] Control the movement of the electric stage to align the objective lens of the digital microscope with the upper left corner of the electric stage. Then, starting from the upper left corner, translate one unit distance in the X direction each time and take a photo to obtain an image. When reaching the right end point, translate one unit distance in the Y direction, and then continue to translate in the opposite direction of the X direction, taking a photo each time by translating one unit distance to perform shooting in a serpentine progression until all the cytopathological smears on the electric stage are photographed;

[0020] Obtain the images of the cytopathological smears from the host computer and output them to the image storage and management module.

[0021] Preferably, the image storage and management module obtains all the images of the cytopathological smears collected each time by the image acquisition module and numbers them. The number includes the serial number of the cytopathological smear corresponding to the image and the position number of the image, so as to obtain an image sample library composed of all the images of multiple cytopathological smears after multiple acquisitions.

[0022] Preferably, the processing method of the image preprocessing module includes the following steps:

[0023] For the images obtained from the image storage and management module, remove the unstained blood cell regions that are gray or white according to the color values of the RGB three channels, and retain the stained blood cell regions that are blue-violet;

[0024] Perform a binarization operation on the retained unstained regions to make the retained regions into a foreground mask and the removed regions into a background mask. First, perform a closing operation on the foreground mask part, and then perform edge dilation to obtain the final foreground mask. Use this foreground mask to crop the image obtained initially to obtain the preprocessed image.

[0025] The image preprocessing module can also classify and label the images, labeling the images with different labels, and the labels at least include positive labels, negative labels, and different pathological grading labels.

[0026] Preferably, the intelligent interpretation module includes a binary classification model for distinguishing normal samples from lesion samples and a multi-classification model for distinguishing lesion types. The binary classification model and the multi-classification model are obtained by training based on the same classification network through a first training set and a second training set respectively;

[0027] The first training set includes a number of normal sample images and a number of abnormal sample images with corresponding markings, and the second training set includes a number of abnormal sample images with markings of different lesion types.

[0028] Preferably, the classification network is a ResNet network, a ResNeXt network or a DenseNet network with an ecSK_Unit module added. The processing method of the ecSK_Unit module includes:

[0029] After the image matrix is input into the ecSK_Unit module, it first passes through a convolutional layer with convolutional kernels of 3×3 and 5×5 for feature extraction to obtain two feature matrices, denoted as feature matrix 1 and feature matrix 2. Then, through a connection layer, matrix splicing operations are performed on the two feature matrices in the channel dimension. The spliced feature matrix sequentially enters a global average pooling layer, two fully connected layers, and a Sigmoid layer to establish connections between channels. Then, in the screening and recombination layer, the importance of the image information contained in each channel is evaluated to obtain the information weights of each channel. According to the weight magnitudes, the channel feature maps of each channel are sorted, and the channel feature maps of the larger half of the weights are selected. The channel feature maps of this half are spliced in the channel dimension to obtain an information enhancement matrix. Finally, adaptive fusion is performed on the three feature matrix pairs of the obtained feature matrix 1, feature matrix 2, and information enhancement matrix. The three feature matrices are integrated by element-wise addition and then sequentially pass through a global average pooling layer, a fully connected layer, and a Softmax layer to obtain the corresponding weights of the three feature matrices. The three feature matrices are fused by weighted summation to obtain a new feature matrix as the output.

[0030] Preferably, the interpretation method of the intelligent interpretation module includes the following steps:

[0031] 1) Obtain all the images of the same cytopathological smear processed by the image preprocessing module;

[0032] 2) Use the binary classification model to classify and judge all the images into normal images and lesion images, and extract all the lesion images;

[0033] 3) Use the multi-classification model to classify and judge all the lesion images to obtain the lesion types of each lesion image;

[0034] 4) Statistically analyze all the images, form a first-level discrimination result output according to the proportion of lesion images in all the images, and then form a second-level discrimination result output according to the proportion of each lesion type in all the lesion images.

[0035] Preferably, in step 4), the first-level discrimination result and the second-level discrimination result are formed according to the following rules:

[0036] Ⅰ. For the first-level discrimination result P1, the following grading is performed according to the value range of the proportion K1 of lesion images in all the images:

[0037] K1 = 0, P1 is level 0; 0 < K1 ≤ 10, P1 is level 1; 10 < K1 ≤ 50, P1 is level 2; 50 < K1 ≤ 70, P1 is level 3; 70 < K1 ≤ 100, P1 is level 4;

[0038] The level of P1 represents the probability that the sample corresponding to the current cytopathological smear is diseased;

[0039] II. For the secondary discrimination result P2, the following grading is carried out according to the value range of the proportion K2 of each type of lesion image in all lesion images:

[0040] K2 = 0, P2 is level 0; 0 < K2 ≤ 10, P2 is level 1; 10 < K2 ≤ 50, P2 is level 2; 50 < K2 ≤ 70, P2 is level 3; 70 < K2 ≤ 100, P2 is level 4;

[0041] The level of P2 represents the probability that the case corresponding to the current cytopathological smear is a specific type of disease;

[0042] III. When the absolute value of the difference between P1 and P2 is greater than 2, the level of the secondary discrimination result P2 corresponding to the current lesion category is decreased by 1 and output as the final secondary discrimination result.

[0043] Preferably, the processing method of the report writing module includes the following steps:

[0044] S1. First, obtain the primary discrimination result P1. When P1 is level 0, output the interpretation result text with the meaning of "the current sample is healthy or the current sample has no disease" according to the result of P1; when P1 is not level 0, go to the next step;

[0045] S2. Obtain the secondary discrimination result P2, and correspond P2 being level 0, 1, 2, 3, 4 to the interpretation result texts with the following meanings in turn and output: "exclude the current type of disease or have no current type of disease", "low probability of the current type of disease", "suspected of the current type of disease", "high probability of the current type of disease", "confirmed as the current type of disease".

[0046] The beneficial effects of the present invention are as follows: The automatic cytopathological smear interpretation system provided by the present invention successfully applies the artificial intelligence-assisted diagnosis technology to rapid staining cytopathology, which can significantly improve the accuracy and consistency of diagnosis and reduce the workload of cytopathology doctors; in the intelligent interpretation module provided by the present invention, by improving the existing convolutional neural network and using characteristics such as multi-channel attention mechanism, the uncertain factors introduced by artificial sampling are solved, and high-accuracy full-scene and multi-classification tasks can be achieved, and finally the accuracy of the classification and interpretation results can be improved. Description of the Drawings

[0047] Figure 1 This is the principle block diagram of the automatic interpretation system for cytopathological smears of the present invention;

[0048] Figure 2 This is the workflow diagram of the automatic interpretation system for cytopathological smears of the present invention;

[0049] Figure 3 This is the structural schematic diagram of the ecSK_Unit module of the present invention;

[0050] Figure 4 This is the diffquick pathological staining image of lung adenocarcinoma cells in the embodiment of the present invention. Detailed implementation manners

[0051] The following further elaborates on the present invention in conjunction with embodiments, so that those skilled in the art can implement it with reference to the text of the specification.

[0052] It should be understood that terms such as "having", "comprising", and "including" as used herein do not exclude the presence or addition of one or more other elements or their combinations.

[0053] Referring to Figure 1 , an automatic interpretation system for cytopathological smears provided by the present invention includes:

[0054] An imaging module;

[0055] An image acquisition module, which controls the imaging module to image the cytopathological smear to achieve image acquisition of the cytopathological smear;

[0056] An image storage and management module, which stores the images acquired by the image acquisition module and organizes them to form an image sample library;

[0057] An image preprocessing module, which is used to preprocess the images output from the image sample library and output them after removing the unstained areas in the images;

[0058] An intelligent interpretation module, which receives the images output from the image preprocessing module, predicts and classifies the images into normal samples and lesion samples, and further predicts and interprets the lesion type when the images are lesion samples;

[0059] And a report writing module, which automatically generates the interpretation result text of the sample corresponding to the image according to the result of the intelligent interpretation module.

[0060] Among them, the imaging module includes a digital microscope, a motorized stage, and a host computer; both the digital microscope and the motorized stage are communicatively connected to the host computer. The motorized stage has X, Y, and Z axis movement functions. The cytopathological smear is placed on the motorized stage and imaged through the digital microscope, and the obtained images are transmitted to the host computer in real time;

[0061] Among them, the automatic interpretation system for cytopathological smears further includes an electric stage control module and an autofocus module. The electric stage control module is used to control the movement of the electric stage; the autofocus module is connected to the electric stage control module and realizes autofocus by controlling the movement of the electric stage according to factors such as image sharpness, ensuring that the transmitted image is in the clearest state.

[0062] Among them, the host computer is a computer or a smart phone or other intelligent terminal. In this embodiment, the host computer is a computer, and the image acquisition module, the image storage and management module, the image preprocessing module, the intelligent interpretation module, and the report writing module are connected in sequence and are all embedded in the computer.

[0063] In a preferred embodiment, the image acquisition module is connected to the host computer to receive the image output by the host computer. The working process of the image acquisition module is as follows:

[0064] First, control the electric stage to move until its center coincides with the central axis of the objective lens of the digital microscope, then complete autofocus through the focus control module, and then fix the focal length;

[0065] Control the electric stage to move so that the objective lens of the digital microscope is aligned with the upper left corner of the electric stage. Then, starting from the upper left corner, translate one unit distance in the X direction each time to take a photo to obtain an image. When reaching the right end point, translate one unit distance in the Y direction, and then continue to translate in the opposite direction of the X direction, taking a photo to obtain an image each time of translation. Shoot in a serpentine progression until all cytopathological smears on the electric stage are photographed;

[0066] Obtain the image of the cytopathological smear from the host computer and output it to the image storage and management module.

[0067] In a preferred embodiment, the cytopathological smear is a cytopathological smear stained with diffquick, or a cytopathological smear stained with other staining methods.

[0068] In a preferred embodiment, the image storage and management module obtains all the images of the cytopathological smear collected by the image acquisition module each time and numbers them. The number includes the serial number of the cytopathological smear corresponding to the image and the position number of the image, so as to obtain an image sample library composed of all the images of multiple cytopathological smears after multiple collections.

[0069] In a preferred embodiment, referring to Figure 2 , the processing method of the image preprocessing module includes the following steps:

[0070] For the images obtained from the image storage and management module, the gray or white unstained blood cell regions are removed according to the color values of the RGB three channels, while the stained blood cell regions in blue-violet are retained.

[0071] The retained unstained regions are binarized to make the retained regions the foreground mask, and the removed regions are the background mask. For the foreground mask part, first perform closing processing, and then perform edge dilation to obtain the final foreground mask. Use this foreground mask to crop the initially obtained image to obtain the preprocessed image.

[0072] In a further preferred embodiment, the image preprocessing module can also classify and label the images, labeling the images with different labels, and the labels at least include positive / abnormal labels, negative / normal labels, and different pathological grading labels. The images are labeled by the image preprocessing module for constructing training images.

[0073] In a preferred embodiment, the intelligent interpretation module includes a binary classification model for distinguishing normal samples from lesion samples and a multi-classification model for distinguishing lesion types. The binary classification model and the multi-classification model are obtained by training a same classification network based on a first training set and a second training set respectively; the first training set includes a number of normal sample images and a number of abnormal sample images with corresponding marks, and the second training set includes a number of abnormal sample images with marks of different lesion types.

[0074] First, the image preprocessing module labels a number of preprocessed images collected to construct a first training set including a number of normal sample images and a number of abnormal sample images, and a second training set including a number of abnormal sample images with marks of different lesion types; then use the first training set to train a classification network to obtain a binary classification model; use the second training set to train another classification network to obtain a multi-classification model.

[0075] In a further preferred embodiment, the classification network is a ResNet network, a ResNeXt network or a DenseNet network with an ecSK_Unit module added, and attention modules such as SENet and SKNet can also be added to the classification network.

[0076] In a further preferred embodiment, the structure of the ecSK_Unit module is as Figure 3 shown, and the processing method of the ecSK_Unit module includes:

[0077] After the image matrix is input into the ecSK_Unit module, it first passes through a convolutional layer with convolutional kernels of 3×3 and 5×5 for feature extraction to obtain two feature matrices, denoted as Feature Matrix 1 and Feature Matrix 2. Then, through a connection layer, matrix splicing operation is performed on the two feature matrices in the channel dimension. The spliced feature matrix then enters a global average pooling layer, two fully connected layers, and a Sigmoid layer in sequence to establish connections between channels. Then, in the screening and recombination layer, the importance of the image information contained in each channel is evaluated to obtain the information weights of each channel. According to the weight magnitudes, the feature maps of each channel are sorted, and half of the channel feature maps with larger weights are selected. These half-channel feature maps are spliced in the channel dimension to obtain an information-enhanced matrix. Finally, adaptive fusion is performed on the three feature matrix pairs of the obtained Feature Matrix 1, Feature Matrix 2, and information-enhanced matrix. The three feature matrices are integrated by element-wise addition and then pass through a global average pooling layer, a fully connected layer, and a Softmax layer in sequence to obtain the corresponding weights of the three feature matrices. The three feature matrices are fused by weighted summation to obtain a new feature matrix as the output.

[0078] In a preferred embodiment, the interpretation method of the intelligent interpretation module includes the following steps:

[0079] 1) Obtain all the images of the same cytopathological smear processed by the image preprocessing module;

[0080] 2) Use a binary classification model to classify and judge all the images into normal images and lesion images, and extract all the lesion images;

[0081] 3) Use a multi-classification model to classify and judge all the lesion images to obtain the lesion types of each lesion image;

[0082] 4) Statistically analyze all the images, form a first-level discrimination result output according to the proportion of lesion images in all the images, and then form a second-level discrimination result output according to the proportion of each lesion type in all the lesion images.

[0083] In a further embodiment, the first-level discrimination result and the second-level discrimination result are formed according to the following rules in step 4):

[0084] Ⅰ. For the first-level discrimination result P1, the following grading is performed according to the value range of the proportion K1 of lesion images in all the images:

[0085] K1 = 0, P1 is grade 0; 0 < K1 ≤ 10, P1 is grade 1; 10 < K1 ≤ 50, P1 is grade 2; 50 < K1 ≤ 70, P1 is grade 3; 70 < K1 ≤ 100, P1 is grade 4;

[0086] The grade of P1 represents the probability that the sample corresponding to the current cytopathological smear is diseased;

[0087] II. For the secondary discrimination result P2, the following grading is performed according to the value range of the proportion K2 of each type of lesion image in all lesion images:

[0088] When K2 = 0, P2 is at level 0; when 0 < K2 ≤ 10, P2 is at level 1; when 10 < K2 ≤ 50, P2 is at level 2; when 50 < K2 ≤ 70, P2 is at level 3; when 70 < K2 ≤ 100, P2 is at level 4;

[0089] The level of P2 represents the probability that the current cytopathological smear corresponds to a certain specific type of disease;

[0090] III. When the absolute value of the difference between P1 and P2 is greater than 2, the level of the secondary discrimination result P2 corresponding to the current lesion category is decreased by 1 and then output as the final secondary discrimination result.

[0091] In a further embodiment, the processing method of the report writing module includes the following steps:

[0092] S1. First, obtain the primary discrimination result P1. When P1 is at level 0, output the interpretation result text of "the current sample is healthy or the current sample has no disease" or the interpretation result text with the same meaning according to the result of P1, such as "the sample has no XX disease"; when P1 is not at level 0, proceed to the next step;

[0093] S2. Obtain the secondary discrimination result P2, and correspond P2 at level 0, level 1, level 2, level 3, and level 4 to the following interpretation result texts or the interpretation result texts including the following meanings in turn and output: "exclude the current type of disease or have no current type of disease", "low probability of the current type of disease", "suspected of the current type of disease", "high probability of the current type of disease", "confirmed as the current type of disease"; for example, "the sample has a high probability of XX disease, is suspected of XX disease, has a low probability of XX disease, excludes XX disease".

[0094] The above is the overall concept of the present invention. The following provides detailed embodiments on this basis to further illustrate the present invention.

[0095] Embodiment 1

[0096] In this embodiment, the digital microscope has a maximum optical zoom function of 100 times, and the host computer is a computer with an X86 architecture.

[0097] 1. Obtain the training set

[0098] 1-1. Collect healthy cells (negative, atypical), lung cancer cells and place them on a glass slide. Stain them using the diffquick technique to make a cytopathological smear. The lesion types are further divided into adenocarcinoma, squamous cell carcinoma, small cell carcinoma, and large cell neuroendocrine carcinoma. Refer to Figure 4 , and the arrow indicates adenocarcinoma cells (blue).

[0099] 1-2. Place the cytopathological smear on the motorized stage, and the imaging module and image acquisition module work:

[0100] First, control the motorized stage to move until its center coincides with the central axis of the objective lens of the digital microscope. Then, complete autofocus through the focus control module and fix the focal length. Control the motorized stage to move so that the objective lens of the digital microscope is aligned with the upper left corner of the motorized stage. The Canny operator-based edge detection algorithm and the template matching-based right angle detection algorithm can be used for corner detection. When the upper left corner edge appears in the field of view, the stage stops moving.

[0101] Starting from the upper left corner, translate 5 mm each time and take pictures in a serpentine progression until all cytopathological smears on the motorized stage are photographed. Obtain the images of the cytopathological smears from the host computer and output them to the image storage and management module. The image storage and management module obtains all the images of the cytopathological smears collected by the image acquisition module each time and numbers them. The number includes the serial number of the cytopathological smear corresponding to the image and the position number of the image, for example, in the form of "sample number - position number".

[0102] 1-3. Perform preprocessing through the image preprocessing module.

[0103] 1-4. Perform annotation through the image preprocessing module. Mark the images as two categories: normal (negative, atypical) and abnormal (tumor) to form the first training set. For the copies of abnormal samples, according to their respective lesion types, mark them as four categories: adenocarcinoma, squamous cell carcinoma, small cell carcinoma, and large cell neuroendocrine carcinoma to form the second training set. Use the first training set and the second training set respectively to train the ResNet network integrated with ecSK_Unit. The samples are amplified 8 times during training using orthogonal double flipping and histogram equalization. The loss function during training is cross-entropy to obtain a binary classification model and a multi-classification model.

[0104] II. Automatically interpret the test sample

[0105] 1. First, obtain the cytopathological smear images of the test sample according to the steps 1-1 to 1-3.

[0106] 2. Use a binary classification model to judge all images for normal and abnormal conditions. If it is judged as normal, the text "This sample is negative" will be output through the report writing module, and the process ends; if it is judged as abnormal, proceed to the next step;

[0107] 3. Use a multi-classification model to classify and judge all abnormal sample lesion images to obtain the specific classification of the samples. The four categories of adenocarcinoma, squamous cell carcinoma, small cell carcinoma, and large cell neuroendocrine carcinoma are respectively denoted as c1, c2, c3, and c4;

[0108] 4. Statistically calculate the proportions of c1 to c4 in all test samples. There are a total of 30 test samples, among which 15 are abnormal samples, accounting for 50%. That is, the primary discrimination result P1 is level 2;

[0109] 5. Statistically calculate the proportions of each of c1 to c4 in the abnormal samples, which are c1: 10%, and the corresponding secondary discrimination result P2 is level 1; c2: 35%, and the corresponding P2 is level 2; c3: 55%, and the corresponding P2 is level 3; c4: 0%, and the corresponding P2 is level 0;

[0110] According to the statistical results, P1 is level 2, and the secondary discrimination results of each abnormal class are respectively c1: level 1, c2: level 2, c3: level 3, c4: level 0. All secondary discrimination results do not need to be corrected;

[0111] 6. The report writing module obtains the discrimination results of the intelligent discrimination module and outputs the discrimination result text "This case is most likely small cell carcinoma, suspected squamous cell carcinoma, less likely adenocarcinoma, and large cell neuroendocrine carcinoma is excluded" according to the corresponding text library.

[0112] Although the embodiments of the present invention have been disclosed as above, it is not limited to only the applications listed in the specification and embodiments. It can be fully applied to various fields suitable for the present invention. For those familiar with the field, additional modifications can be easily made. Therefore, without departing from the general concept defined by the claims and the equivalent scope, the present invention is not limited to specific details.

Claims

1. An automatic interpretation system for cytopathological smears, characterized in that, Including: Imaging module; Image acquisition module, which controls the imaging module to image the cytopathological smear, and realizes the image acquisition of the cytopathological smear; Image storage and management module, which stores the images acquired by the image acquisition module and organizes them to form an image sample library; Image preprocessing module, which is used to preprocess the images output by the image sample library, and outputs the images after removing the unstained areas in the images; Intelligent interpretation module, which receives the images output by the image preprocessing module, predicts and classifies the images into normal samples and lesion samples, and when the image is a lesion sample, further predicts and interprets the lesion type; And a report writing module, which automatically generates the interpretation result text of the sample corresponding to the image according to the result of the intelligent interpretation module; The intelligent interpretation module includes a binary classification model for distinguishing normal samples and lesion samples and a multi-classification model for distinguishing lesion types; The interpretation method of the intelligent interpretation module includes the following steps: 1) Obtain all the images of the same cytopathological smear processed by the image preprocessing module; 2) Use the binary classification model to classify and judge all the images into normal images and lesion images, and extract all the lesion images; 3) Use the multi-classification model to classify and judge all the lesion images to obtain the lesion types of each lesion image; 4) Count all the images, form a first-level discrimination result output according to the proportion of the lesion images in all the images, and then form a second-level discrimination result output according to the proportion of each lesion type in all the lesion images; In step 4), the first-level discrimination result and the second-level discrimination result are formed according to the following rules: Ⅰ. For the first-level discrimination result P1, it is classified as follows according to the value range of the proportion K1 of the lesion images in all the images: K1 = 0, P1 is level 0; 0 < K1 ≤ 10, P1 is level 1; 10 < K1 ≤ 50, P1 is level 2; 50 < K1 ≤ 70, P1 is level 3; 70 < K1 ≤ 100, P1 is level 4; The level of P1 represents the probability that the sample corresponding to the current cytopathological smear is diseased; Ⅱ. For the second-level discrimination result P2, it is classified as follows according to the value range of the proportion K2 of each type of lesion image in all the lesion images: K2 = 0, P2 is level 0; 0 < K2 ≤ 10, P2 is level 1; 10 < K2 ≤ 50, P2 is level 2; 50 < K2 ≤ 70, P2 is level 3; 70 < K2 ≤ 100, P2 is level 4; The level of P2 represents the probability that the case corresponding to the current cytopathological smear is a certain specific type of disease; Ⅲ. When the absolute value of the difference between P1 and P2 is greater than 2, the level of the second-level discrimination result P2 corresponding to the current lesion category is reduced by 1 and then output as the final second-level discrimination result.

2. The automated cytopathological smear interpretation system according to claim 1, wherein The imaging module includes a digital microscope, an electric stage and a host computer; the electric stage has the functions of moving in the X, Y, and Z axes, the cytopathological smear is placed on the electric stage, imaged by the digital microscope, and the obtained images are transmitted to the host computer; The automatic cytopathological smear interpretation system further includes an electric stage control module and an autofocus module. The electric stage control module is used to control the movement of the electric stage; the autofocus module is connected to the electric stage control module and realizes autofocus by controlling the movement of the electric stage.

3. The automated cytopathological smear interpretation system according to claim 2, wherein The image acquisition module is connected to the host computer to receive the images output by the host computer. The working process of the image acquisition module is as follows: First, control the electric stage to move until its center coincides with the central axis of the objective lens of the digital microscope, then complete autofocus through the autofocus module, and then fix the focal length; Control the movement of the electric stage to align the objective lens of the digital microscope with the upper left corner of the electric stage. Then, starting from the upper left corner, translate one unit distance in the X direction each time to take a photo to obtain an image. When reaching the right end point, translate one unit distance in the Y direction, and then continue to translate in the opposite direction of the X direction, taking a photo to obtain an image each time a unit distance is translated. Shoot in a serpentine progression until all cytopathological smears on the electric stage are photographed; Obtain the images of the cytopathological smears from the host computer and output them to the image storage and management module.

4. The automated cytopathological smear interpretation system according to claim 3, wherein, The image storage and management module acquires all the images of the cytopathological smears collected each time by the image acquisition module and numbers them. The number includes the serial number of the cytopathological smear corresponding to the image and the position number of the image, so as to obtain an image sample library composed of all the images of multiple cytopathological smears after multiple acquisitions.

5. The automatic cytopathological smear interpretation system according to claim 4, characterized in that, The processing method of the image preprocessing module includes the following steps: For the images obtained from the image storage and management module, remove the unstained blood cell regions that are gray or white according to the color values of the RGB three channels, and retain the stained blood cell regions that are blue-violet; Perform a binarization operation on the retained unstained regions to make the retained regions the foreground mask and the removed regions the background mask. First, perform a closing operation on the foreground mask part, and then perform edge dilation to obtain the final foreground mask. Use this foreground mask to crop the image obtained initially to obtain the preprocessed image.

6. The automatic cytopathological smear interpretation system according to claim 5, wherein The binary classification model and the multi-classification model are obtained by training based on the same classification network through the first training set and the second training set respectively; The first training set includes a number of normal sample images and a number of abnormal sample images with corresponding labels, and the second training set includes a number of abnormal sample images with labels of different lesion types.

7. The automatic cytopathological smear interpretation system according to claim 6, wherein The classification network is a ResNet network, a ResNeXt network or a DenseNet network with an ecSK_Unit module added. The processing method of the ecSK_Unit module includes: After the image matrix is input into the ecSK_Unit module, it first passes through a convolutional layer with convolutional kernels of 3×3 and 5×5 for feature extraction to obtain two feature matrices, denoted as Feature Matrix 1 and Feature Matrix 2. Then, through the connection layer, matrix splicing operations are performed on the two feature matrices in the channel dimension. The spliced feature matrix then enters the global average pooling layer, two fully connected layers, and the Sigmoid layer in sequence to establish connections between channels. Then, in the screening and recombination layer, the importance of the image information contained in each channel is evaluated to obtain the information weights of each channel. According to the weight magnitudes, the channel feature maps of each channel are sorted, and the half of the channel feature maps with larger weights are selected. Splicing is performed on these half of the channel feature maps in the channel dimension to obtain an information enhancement matrix. Finally, adaptive fusion is performed on the three feature matrix pairs of the obtained Feature Matrix 1, Feature Matrix 2, and information enhancement matrix. The three feature matrices are integrated by element-wise addition and then pass through the global average pooling layer, fully connected layer, and Softmax layer in sequence to obtain the corresponding weights of the three feature matrices. The three feature matrices are fused by weighted summation to obtain a new feature matrix as the output.

8. The automatic cytopathological smear interpretation system according to claim 1, wherein The processing method of the said report writing module includes the following steps: S1. First, obtain the primary discrimination result P1. When P1 is at level 0, output the interpretation result text with the meaning of "the current sample is healthy or the current sample has no disease" according to the result of P1. When P1 is not at level 0, proceed to the next step; S2. Obtain the secondary discrimination result P2. Correspond the cases where P2 is at level 0, 1, 2, 3, and 4 to the interpretation result texts with the following meanings and output them: "exclude the disease of the current type or have no disease of the current type", "low probability of the disease of the current type", "suspected of the disease of the current type", "high probability of the disease of the current type", "confirmed as the disease of the current type".

Images