Hierarchical deep neural network-based antinuclear antibody pattern classification method and device

Abstract

The present invention relates to a hierarchical deep neural network-based ANA pattern classification method and device and, particularly, to a hierarchical deep neural network-based ANA pattern classification method and device for classifying HEp-2 cell patterns through deep learning technology in order to provide a path for more accurate and reliable ANA pattern recognition.

Description

Hierarchical deep neural network-based antinuclear antibody pattern classification method and device

[0001] The present invention relates to a hierarchical deep neural network-based antinuclear antibody (ANA) pattern classification method and device, and more particularly, to a hierarchical deep neural network-based ANA pattern classification method and device that classifies HEp-2 cell patterns using deep learning technology to provide a path for more accurate and reliable ANA pattern recognition.

[0002]

[0003] Autoimmune diseases can cause a variety of patterns of inflammation and organ dysfunction. The prevalence of autoimmune diseases is increasing due to various factors, including increased exposure to hazardous substances due to industrialization, an aging population, and the rise in concurrent chronic diseases and polypharmacy. Autoimmune diseases can affect essentially any organ system and affect individuals of all ages.

[0004] A major sign of this disease is the production of ANA.

[0005] ANA is a collective term for antibodies against various antigens present in cells, and is one of the most important serological tests for diagnosing various autoimmune diseases, including rheumatic diseases. Although various ANA testing methods have been developed, the indirect immunofluorescence assay (IIFA) using the HEp-2 (Human Epithelial Cell Tumor Line) cell line as a stromal cell is still considered the standard screening test. The ANA IIFA test classifies the staining pattern according to the distribution of fluorescent substances observed under a fluorescence microscope, and diseases associated with specific autoantibodies corresponding to each classification can be diagnosed or predicted based on the ANA pattern. The ANA IIFA test presents several challenges because the stromal cells, fixatives, and other reagents used in the test are not yet standardized, and the testing process is often performed manually. In addition, even if the biological or technical variations of the examination process itself are excluded, the proportion of variations depending on the reader is large, and it has been reported that the agreement rate of the reading results tends to be low, ranging from 37.6% to 72.3% depending on the skill or training period of the reader.

[0006] To overcome these limitations, automated systems for ANA IIFA screening and interpretation are being developed. However, these automated screening and interpretation systems still have limitations: they are closed systems that only analyze images obtained during the screening process, and the number of ANA patterns that can be classified using these systems is very limited, at 5 to 7.

[0007] Recently, the International Consensus on ANA Patterns (ICAP) committee, which promotes the standardization of ANA pattern nomenclature, has proposed an agreed-upon standard nomenclature for ANA patterns from AC (Anti-Cell) 0 to 29 and recommends that classifications be interpreted and reported at this level whenever possible.

[0008] Timely detection of ANA is crucial for preventing disease progression, preventing disease exacerbation, and minimizing complications. In other words, monitoring ANA can prevent the spread of the disease and enable early detection and treatment to alleviate its severity.

[0009] However, classifying the 30 ANA patterns recommended by ICAP using existing technologies is still difficult in terms of cost and execution.

[0010] One object of the present invention is to provide a hierarchical deep neural network-based ANA pattern classification method and device for classifying HEp-2 cell images through deep learning technology to provide a path for more accurate and reliable ANA pattern recognition in order to solve the above-mentioned problems.

[0011] One object of the present invention is to provide a hierarchical deep neural network-based ANA pattern classification method and device capable of classifying ANA patterns by hierarchically applying an artificial intelligence model to HEp-2 image data in four stages in order to solve the aforementioned problem.

[0012] The purpose of the present invention is not limited to the tasks mentioned above. Other objects and advantages of the present invention not mentioned above can be understood through the following description and will be more clearly understood through embodiments of the present invention. Furthermore, it will be appreciated that the objects and advantages of the present invention can be realized by the means and combinations thereof set forth in the claims.

[0013] According to an embodiment, the hierarchical deep neural network-based ANA pattern classification method of the present invention may include a step of acquiring a HEp-2 cell image, a first classification step of distinguishing a positive pattern and a negative pattern from the HEp-2 cell image based on a learned first artificial intelligence model, a second classification step of inputting the positive pattern into a learned second artificial intelligence model and distinguishing it into at least one pattern among a cell nucleus, a cytoplasm, and a mitosis pattern, a third classification step of inputting the cell nucleus pattern, the cytoplasm pattern, and the mitosis pattern output from the second classification step into a learned third artificial intelligence model and distinguishing each of them into at least one pattern, and a fourth classification step of inputting the patterns output from the third classification step into a learned fourth artificial intelligence model and distinguishing them into a plurality of unique AC patterns.

[0014] According to an embodiment, the hierarchical deep neural network-based ANA pattern classification device of the present invention may include a photographing device that acquires a HEp-2 image, and a server that executes a first classification step of acquiring a HEp-2 cell image and distinguishing a positive pattern and a negative pattern from the HEp-2 cell image based on a learned first artificial intelligence model, executes a second classification step of inputting the positive pattern into a learned second artificial intelligence model and distinguishing it into at least one pattern among a cell nucleus, a cytoplasm, and a mitosis pattern, executes a third classification step of inputting the cell nucleus pattern, the cytoplasm pattern, and the mitosis pattern output from the second classification step into a learned third artificial intelligence model and distinguishes each of them into at least one pattern, and executes a fourth classification step of inputting the patterns output from the third classification step into a learned fourth artificial intelligence model and distinguishing them into a plurality of unique AC patterns.

[0015]

[0016] According to an embodiment of the present invention, early diagnosis and customized treatment for autoimmune diseases are enabled through hierarchical deep neural network-based ANA pattern classification.

[0017] Additionally, embodiments of the present invention have the potential to improve clinical decision-making by assisting clinicians in identifying cellular patterns through hierarchical deep neural network-based ANA pattern classification.

[0018] The effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0019] FIG. 1 is a block diagram illustrating a hierarchical deep neural network-based ANA pattern classification system according to an embodiment of the present invention.

[0020] FIG. 2 is a conceptual diagram illustrating the concept of a hierarchical deep neural network according to one embodiment of the present invention.

[0021] FIG. 3 is a conceptual diagram illustrating the concept of a hierarchical deep neural network according to one embodiment of the present invention.

[0022] FIG. 4 is a conceptual diagram illustrating the structure of a hierarchical deep neural network according to one embodiment of the present invention.

[0023] Figure 5 illustrates an example of a data set according to one embodiment of the present invention.

[0024] FIG. 6 is a conceptual diagram illustrating a hierarchical deep neural network-based ANA pattern classification concept according to one embodiment of the present invention.

[0025] FIG. 7 is a flowchart for explaining a hierarchical deep neural network-based ANA pattern classification method according to one embodiment of the present invention.

[0026]

[0027] The present invention can be implemented in various different forms and is not limited to the embodiments described herein. In the following embodiments, portions not directly related to the description are omitted for clarity. However, this does not mean that such omitted components are unnecessary when implementing a device or system to which the present invention is applied. Furthermore, the same reference numbers are used throughout the specification to refer to identical or similar components.

[0028] In the following description, terms such as "first," "second," etc. may be used to describe various components; however, these components should not be limited by these terms, and these terms are used only to distinguish one component from another. Furthermore, in the following description, singular expressions include plural expressions unless the context clearly indicates otherwise.

[0029] In the following description, it should be understood that terms such as “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0030] FIG. 1 is a block diagram illustrating a hierarchical deep neural network-based ANA pattern classification system according to an embodiment of the present invention.

[0031] As illustrated in FIG. 1, the hierarchical deep neural network-based ANA pattern classification system may include a server (100), a photographing device (200), and a display device (300).

[0032] According to one embodiment of the present invention, examples of a network for sharing information between a server (100), a photographing device (200), and a display device (300) may include, but are not limited to, a 3GPP (3rd Generation Partnership Project) network, an LTE (Long Term Evolution) network, a 5G network, a WIMAX (World Interoperability for Microwave Access) network, a wired / wireless Internet, a LAN (Local Area Network), a Wireless LAN (Wireless Local Area Network), a WAN (Wide Area Network), a PAN (Personal Area Network), a Bluetooth network, a Wifi network, an NFC (Near Field Communication) network, a satellite broadcasting network, an analog broadcasting network, a DMB (Digital Multimedia Broadcasting) network, etc.

[0033] The server (100) is configured to acquire an image from a photographing device (200) and classify the presence or absence of ANA and the ANA pattern. The server (100) according to one embodiment of the present invention may include an image acquisition unit (110), a data generation unit (120), a preprocessing unit (130), a learning unit (140), and an ANA pattern classification unit (150), as illustrated in FIG. 2. However, the configuration of the server (100) is not limited to those disclosed above. For example, the server (100) may further include a database for storing information.

[0034] The image acquisition unit (110) can acquire multiple HEp-2 cell images. The image acquisition unit (110) can receive HEp-2 cell images from a photographing device (200). The photographing device (200) may be, but is not limited to, a fluorescence microscope. For example, a digital image of a HEp-2 cell slide can be captured at 200x magnification using a fluorescence microscope camera integrated into an automatic analyzer.

[0035] The image acquisition unit (110) can collect pathologically confirmed HEp-2 cell images. In addition, the image acquisition unit (110) can receive multiple HEp-2 cell images from image storage devices and database systems of multiple hospitals. The image storage devices of multiple hospitals may be devices that store HEp-2 cell images confirmed by specialists at multiple hospitals.

[0036] In addition, the image acquisition unit (110) can acquire an image (image) by photographing a first area of ​​the test object arbitrarily set by changing any one of angle, direction, and distance. The image acquisition unit (110) can acquire a HEp-2 cell image in JPEG format.

[0037] On the other hand, the image acquisition unit (110) may exclude images that are of low quality or low resolution, such as images out of focus, artifacts, or negative areas, during the image acquisition process. In other words, the image acquisition unit (110) may exclude images that are not applicable to the deep learning algorithm.

[0038] The data generation unit (120) can generate a training data set and a verification data set for applying a deep learning algorithm. The data sets can be generated by classifying the data sets into a training data set required for artificial neural network training and a verification data set for verifying the progress information of artificial neural network training. For example, the data generation unit (120) can randomly classify images to be used in the training data set and images to be used in the verification data set from among the HEp-2 cell images acquired from the image acquisition unit (110). In addition, the data generation unit (120) can use the remaining data sets after selecting the verification data set as the training data set. The verification data set can be randomly selected. The ratio of the verification data set and the training data set can be determined by a preset reference value. In this case, the preset reference value can be set to 10% for the verification data set and 90% for the training data set, but is not limited thereto.

[0039] The data generation unit (120) can generate data sets by distinguishing between a training data set and a validation data set to prevent overfitting. For example, since the learning characteristics of the neural network structure can lead to overfitting in the training data set, the data generation unit (120) can prevent overfitting of the artificial neural network by utilizing the validation data set.

[0040] At this time, the validation dataset may be a dataset that does not overlap with the training dataset. Since the validation data is not used to build the artificial neural network, it is the first data encountered by the artificial neural network during validation. Therefore, the validation dataset can be an appropriate dataset for evaluating the performance of the artificial neural network when new images (new images not used in training) are input.

[0041] Data set preparation follows a hierarchical structure to facilitate subsequent experiments.

[0042] The dataset can be prepared by first classifying positive and negative patterns from HEp-2 cell images at the highest level, then performing cell component classification within the positive pattern classification to classify positive patterns into nuclear, cytoplasmic, and mitotic patterns, then performing pattern classification to classify at least one nuclear pattern, cytoplasmic pattern, and mitotic pattern per pattern, and then classifying the patterns into unique AC patterns.

[0043] Figure 5 illustrates an example of a data set according to one embodiment of the present invention.

[0044] As shown in Fig. 5, if the data set is composed of 13,816 images in total, Level 1 consists of 6,654 positive pattern data and 7,162 negative pattern data, Level 2 consists of 4,975 positive nuclear patterns, 1,280 cytoplasmic patterns, and 399 mitotic patterns, and Level 3 consists of data of Competent Level reporting patterns, including nuclear patterns AC-1 (n = 1,121), AC-2 (n = 444), AC-3 (n = 675), AC-4 or AC-5 (n = 1,727), AC-6 or AC-7 (n = 125), AC-8 or AC-9 or AC-10 (n = 548), AC-11 or AC-12 (n = 125), AC-13 or AC-14 (n = 147); cytoplasmic pattern AC-15 or AC-16 or AC-17 (n = 180), AC-19 or AC-20 (n = 211), AC-21 (n = 597), AC-22 (n = 190), AC-23 (n = 3); The data of the Expert Level reporting patterns at Level 4 consisted of mitotic patterns (n ​​= 399), nuclear patterns AC-1 (n = 1,121), AC-2 (n = 444), AC-3 (n = 675), AC-4 (n = 765), AC-5 (n = 962), AC-6 (n = 50), AC-7 (n = 55), AC-8 (n = 180), AC-9 (n = 307), AC-10 (n = 61), AC-11 (n = 72), AC-12 (n = 53), AC-13 (n = 48), AC-14 (n = 99), AC-29 (n = 83);Cytoplasmic patterns AC-15 (n = 59), AC-16 (n = 82), AC-17 (n = 39), AC-18 (n = 99), AC-19 (n = 143), AC-20 (n = 68), AC-21 (n = 597), AC-22 (n = 190), AC-23 (n = 3); Mitotic patterns AC-24 (n = 51), AC-25 (n = 67), AC-26 (n = 175), AC-27 (n = 53), AC-28 (n = 53), and at level 4, a problem arises due to the difference in the number of available samples across the patterns, resulting in severe data imbalance.;

[0045] Some patterns are commonly observed, while others are rarely seen over the entire data collection period.

[0046] The ultimate goal here is to efficiently classify patterns from AC-1 to AC-29 while addressing issues related to data availability and feature variability.

[0047] The preprocessing unit (130) can preprocess a data set so that it can be applied to a deep learning algorithm. The deep learning algorithm can be composed of two parts: a convolutional neural network (CNN) structure and a fully-connected deep neural network (FCNN) structure.

[0048] The learning unit (140) can build an artificial neural network through learning that inputs a data set that has undergone a preprocessing process and outputs items related to the ANA pattern classification results.

[0049] According to one embodiment of the present invention, the learning unit (140) can apply a deep learning algorithm consisting of two parts, a convolutional neural network (CNN) structure and a fully connected deep neural network (FCNN) structure, as each artificial intelligence model to output a classification result corresponding to the input.

[0050] A fully connected deep neural network (FCNN) is a neural network characterized by two-dimensional horizontal and vertical connections between nodes, no connections between nodes in the same layer, and connections only between nodes in immediately adjacent layers.

[0051] The learning unit (140) can build a training model through learning using a convolutional neural network (CNN) that uses a learning data set that has gone through a preprocessing process as input and the output of the convolutional neural network (CNN) as input to a fully connected deep neural network (FCNN).

[0052] According to one embodiment of the present invention, a convolutional neural network (CNN) can extract multiple specific feature patterns that analyze an input image. The extracted specific feature patterns can then be used for final classification in a fully connected deep neural network (FCNN). Specifically, according to one embodiment of the present invention, as illustrated in FIG. 4, the convolutional neural network (CNN) can be configured with multiple hierarchically executed convolutional neural networks (CNNs).

[0053] Convolutional neural networks (CNNs) are a type of neural network primarily used in speech and image recognition. They are designed to process multidimensional array data, making them specialized for processing multidimensional arrays such as color images. Therefore, most deep learning techniques in image recognition are based on convolutional neural networks (CNNs).

[0054] Convolutional neural networks (CNNs) process images by dividing them into multiple data segments rather than processing them as a single piece of data. This allows them to extract partial features of the image, even if the image is distorted, resulting in accurate performance.

[0055] A convolutional neural network (CNN) can be structured with multiple layers. Each layer can consist of a convolution layer, an activation function, a max pooling layer, an activation function, and a dropout layer. The convolution layer acts as a filter called a kernel, which partially processes the entire image (or a newly generated feature pattern) to extract a new feature pattern of the same size as the image. The convolution layer can adjust the values ​​of the feature pattern through an activation function to facilitate processing. The max pooling layer samples and resizes a portion of the input image to reduce the image size. While the size of the feature pattern is reduced through the convolution and max pooling layers in a CNN, multiple feature patterns can be extracted through the use of multiple kernels. The dropout layer is a method for intentionally omitting some weights during training of the CNN weights for efficient training. However, the dropout layer may not be applied when conducting actual testing using the trained model.

[0056] Multiple feature patterns extracted from a convolutional neural network (CNN) can be passed to the next stage, a fully-connected deep neural network (FCNN), and used for classification. The number of layers in a CNN can be adjusted. By adjusting the number of layers according to the amount of training data for model training, a more stable model can be built.

[0057] In addition, the learning unit (140) can build a judgment (training) model through learning by using a learning data set that has gone through a preprocessing process as an input to a convolutional neural network (CNN) and using the output of the convolutional neural network (CNN) as an input to a fully connected deep neural network (FCNN). In other words, the learning unit (140) can allow image data that has gone through a preprocessing process to first enter a convolutional neural network (CNN) and allow the output from the convolutional neural network (CNN) to enter a fully connected deep neural network (FCNN).

[0058] The learning unit (140) can learn by feeding back the results through a backpropagation algorithm that gradually changes the weights of the neural network structure in a direction that reduces the error by comparing the error between the results derived by applying the training data to the deep learning algorithm structure (a structure formed from FCNN via CNN) and the actual results. The backpropagation algorithm may adjust the weights that are connected from each node to the next node in order to reduce the error of the results (the difference between the actual value and the result value). The learning unit (140) can learn the neural network using a learning data set and a verification data set to obtain weight parameters and derive a final judgment model.

[0059] The ANA pattern classification unit (150) can classify ANA patterns through a learned hierarchical deep neural network after a new data set has gone through a preprocessing process.

[0060] In other words, the ANA pattern classification unit (150) can derive a pattern classification for new data using the final hierarchical deep neural network model derived from the learning unit (140) described above. The new data may be data including HEp-2 images that the user wishes to judge.

[0061] The new data set can be preprocessed to a state applicable to a deep learning algorithm through a preprocessing process in the preprocessing unit (130). The preprocessed new data set is then input to the learning unit (140), where ANA patterns can be classified based on learning parameters.

[0062] According to one embodiment of the present invention, the ANA pattern classification unit (150) can classify the HEp-2 image into AC-1 to AC-14 and AC-29 as cell nucleus patterns, AC-15 to AC-23 as cytoplasmic patterns, and AC-24 to AC-28 as mitotic patterns.

[0063] The learned artificial intelligence model according to an embodiment of the present invention includes four artificial intelligence models that are performed hierarchically.

[0064] FIG. 3 is a conceptual diagram illustrating the concept of a hierarchical deep neural network according to one embodiment of the present invention, and FIG. 6 is a conceptual diagram illustrating the concept of ANA pattern classification based on a hierarchical deep neural network according to one embodiment of the present invention.

[0065] As illustrated in FIGS. 3 and 6, the learned artificial intelligence model according to the present invention may include a first artificial intelligence model (CNN Level 1) that distinguishes between positive patterns and negative patterns from HEp-2 cell images, a second artificial intelligence model (CNN Level 2) that distinguishes between positive patterns and at least one pattern among nucleus, cytoplasm, and mitosis, a third artificial intelligence model (CNN Level 3) that distinguishes between at least one pattern among nucleus patterns, cytoplasm patterns, and mitosis patterns, and a fourth artificial intelligence model (CNN Level 4) that distinguishes between output patterns and a plurality of unique AC patterns.

[0066] These first, second, third, and fourth artificial intelligence models can be hierarchically operated with HEp-2 cell images as input to ultimately classify 30 ANA fluorescence patterns (1 negative pattern and 29 positive patterns).

[0067] Specifically, the learned first artificial intelligence model may include a first convolutional neural network that extracts texture information or boundary information as first feature information, and a first fully connected deep neural network that classifies HEp-2 cell images into positive and negative patterns using the first feature information as input. The learned first artificial intelligence model may be referred to as a positive-negative level.

[0068] The learned second AI model may include a second convolutional neural network that extracts cellular components as second feature information, and a second fully-connected deep neural network that uses the second feature information as input to classify at least one pattern among the nucleus, cytoplasm, and mitosis. The learned second AI model may be referred to as the Nuclear-Cytoplasmic-Mitotic Group Level.

[0069] The learned third AI model may include a third convolutional neural network that extracts the first patterns as third feature information, and a third fully-connected deep neural network that classifies the third feature information into multiple classes. The learned third AI model can be referred to as a competency level.

[0070] Here, the first patterns refer to patterns extracted as third feature information by the third convolutional neural network.

[0071] The learned fourth artificial intelligence model may include a fourth convolutional neural network that extracts second patterns as fourth feature information, and a fourth fully-connected deep neural network that classifies the fourth feature information into AC patterns. The learned fourth artificial intelligence model can be considered expert-level.

[0072] Here, the second patterns refer to patterns extracted as fourth feature information by the fourth convolutional neural network.

[0073] Here, the AC pattern may include AC-1 to AC-14, AC-29 as a nuclear pattern, AC-15 to AC23 as a cytoplasmic pattern, and AC-24 to AC-28 as a mitotic pattern.

[0074] FIG. 7 is a flowchart for explaining a hierarchical deep neural network-based ANA pattern classification method according to one embodiment of the present invention.

[0075] Referring to FIG. 7, a hierarchical deep neural network-based ANA pattern classification method according to one embodiment of the present invention is described, but overlapping content with the technology described with reference to FIGS. 1 to 6 is omitted.

[0076] Referring to FIG. 7, a hierarchical deep neural network-based ANA pattern classification method according to an embodiment of the present invention may include a step (S110) of first obtaining a HEp-2 cell image, a first classification step (S120) of distinguishing a positive pattern and a negative pattern from the HEp-2 cell image based on a learned first artificial intelligence model, a second classification step (S130) of inputting a positive pattern into a learned second artificial intelligence model and distinguishing it into at least one pattern among a cell nucleus, a cytoplasm, and a mitosis pattern, a third classification step (S140) of inputting the cell nucleus pattern, the cytoplasm pattern, and the mitosis pattern output from the second classification step into a learned third artificial intelligence model and distinguishing each of them into at least one pattern, and a fourth classification step (S150) of inputting the patterns output from the third classification step into a learned fourth artificial intelligence model and distinguishing them into a plurality of unique AC patterns.

[0077] The learned first artificial intelligence model may include a first convolutional neural network that extracts texture information or boundary information as first feature information, and a first fully connected deep neural network that classifies positive patterns and negative patterns using the first feature information as input.

[0078] The learned second artificial intelligence model may include a second convolutional neural network that extracts cellular components as second feature information, and a second fully connected deep neural network that classifies at least one pattern among a cell nucleus, cytoplasm, and mitosis by using the second feature information as input.

[0079] The learned third artificial intelligence model may include a third convolutional neural network that extracts first patterns as third feature information, and a third fully connected deep neural network that classifies the third feature information into multiple classes as input.

[0080] Here, the first patterns refer to patterns extracted as third feature information by the third convolutional neural network.

[0081] The learned fourth artificial intelligence model may include a fourth convolutional neural network that extracts second patterns as fourth feature information, and a fourth fully connected deep neural network that classifies the fourth feature information as AC patterns.

[0082] Here, the second patterns refer to patterns extracted as fourth feature information by the fourth convolutional neural network.

[0083] The AC pattern may include AC-1 to AC-14, AC-29 as a nuclear pattern, AC-15 to AC23 as a cytoplasmic pattern, and AC-24 to AC-28 as a mitotic pattern.

[0084] The method according to one embodiment of the present invention described above can be implemented as computer-readable code on a medium having a program recorded thereon. The computer-readable medium includes all types of recording devices that store data that can be read by a computer system. Examples of computer-readable media include hard disk drives (HDDs), solid-state disk drives (SSDs), silicon disk drives (SDDs), read-only memory (ROM), random-access memory (RAM), CD-ROMs, magnetic tapes, floppy disks, and optical data storage devices.

[0085] The description of the embodiments of the present invention described above is for illustrative purposes only, and those skilled in the art will understand that the present invention can be easily modified into other specific forms without changing the technical spirit or essential characteristics of the present invention. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single component may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

[0086] The scope of the present invention is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention.

Claims

1. Step of acquiring HEp-2 cell images; A first classification step for distinguishing positive and negative patterns from the HEp-2 cell image based on the learned first artificial intelligence model; A second classification step of inputting the above positive pattern into a learned second artificial intelligence model to distinguish it into at least one pattern among the cell nucleus, cytoplasm, and mitosis; A third classification step of inputting the cell nuclear pattern, cytoplasmic pattern, and mitotic pattern output in the second classification step into a learned third artificial intelligence model and classifying each of them by at least one pattern; and A hierarchical deep neural network-based ANA pattern classification method, including a fourth classification step of inputting the patterns output in the third classification step into a learned fourth artificial intelligence model and classifying them into multiple unique AC patterns.

2. In paragraph 1, The first artificial intelligence model learned above is: A hierarchical deep neural network-based ANA pattern classification method, comprising a first convolutional neural network that extracts texture information or boundary information as first feature information, and a first fully connected deep neural network that classifies the positive pattern and the negative pattern using the first feature information as input.

3. In paragraph 1, The second artificial intelligence model learned above is, A hierarchical deep neural network-based ANA pattern classification method, comprising a second convolutional neural network that extracts cell components as second feature information, and a second fully connected deep neural network that classifies at least one pattern among the cell nucleus, cytoplasm, and mitosis using the second feature information as input.

4. In paragraph 1, The third artificial intelligence model learned above is: A hierarchical deep neural network-based ANA pattern classification method, comprising a third convolutional neural network that extracts first patterns as third feature information, and a third fully connected deep neural network that classifies the third feature information into multiple classes as input.

5. In paragraph 1, The above-mentioned fourth artificial intelligence model is, A hierarchical deep neural network-based ANA pattern classification method, comprising a fourth convolutional neural network that extracts second patterns as fourth feature information, and a fourth fully connected deep neural network that classifies the fourth feature information as an AC pattern.

6. In paragraph 5, The above AC pattern is, A hierarchical deep neural network-based ANA pattern classification method, including AC-1 to AC-14, AC-29 as nuclear patterns, AC-15 to AC23 as cytoplasmic patterns, and AC-24 to AC-28 as mitotic patterns.

7. A photographing device for acquiring HEp-2 images; and A first classification step is executed to distinguish between positive and negative patterns from the HEp-2 cell image based on the learned first artificial intelligence model, The above positive pattern is input into the learned second artificial intelligence model and a second classification step is executed to classify it into at least one pattern among the cell nucleus, cytoplasm, and mitosis. The cell nuclear pattern, cytoplasmic pattern, and mitotic pattern output in the second classification step are input into the learned third artificial intelligence model, and a third classification step is executed to classify each of them by at least one pattern. A hierarchical deep neural network-based ANA pattern classification device, comprising a server that executes a fourth classification step of inputting the patterns output in the third classification step into a learned fourth artificial intelligence model and classifying them into multiple unique AC patterns.

8. In paragraph 7, The first artificial intelligence model learned above is: An ANA pattern classification device based on a hierarchical deep neural network, comprising a first convolutional neural network that extracts texture information or boundary information as first feature information, and a first fully connected deep neural network that classifies the positive pattern and the negative pattern using the first feature information as input.

9. In paragraph 7, The second artificial intelligence model learned above is, A hierarchical deep neural network-based ANA pattern classification device, comprising a second convolutional neural network that extracts cell components as second feature information, and a second fully connected deep neural network that classifies at least one pattern among the cell nucleus, cytoplasm, and mitosis using the second feature information as input.

10. In paragraph 7, The third artificial intelligence model learned above is: An ANA pattern classification device based on a hierarchical deep neural network, comprising a third convolutional neural network that extracts first patterns as third feature information, and a third fully connected deep neural network that classifies the third feature information into multiple classes as input.

11. In paragraph 7, The above-mentioned fourth artificial intelligence model is, An ANA pattern classification device based on a hierarchical deep neural network, comprising a fourth convolutional neural network that extracts second patterns as fourth feature information, and a fourth fully connected deep neural network that classifies the fourth feature information as an AC pattern.

12. In paragraph 11, The above AC pattern is, A hierarchical deep neural network-based ANA pattern classifier, comprising AC-1 to AC-14, AC-29 as nuclear patterns, AC-15 to AC23 as cytoplasmic patterns, and AC-24 to AC-28 as mitotic patterns.

Images