Medical diagnosis device for classifying cell types based on ensemble image classification model and cell type classification method using same

Abstract

A medical diagnosis device for classifying cell types according to the present disclosure may comprise: a memory for storing at least one instruction; and at least one processor for executing the at least one instruction. The at least one processor may receive peripheral blood slide image data, perform preprocessing on the peripheral blood slide image data, and classify at least one cell type among an AML group, an ALL group, an IG group, and a NORMAL group from the peripheral blood slide image data by using an ensemble image classification model. The ensemble image classification model may include at least one ConvNext model and at least one Transformer model.

Description

Medical diagnostic device for classifying cell types based on an ensemble image classification model and a cell type classification method using the same

[0001] The present disclosure relates to a cell type classification technique, and more particularly, to a medical diagnostic device that classifies cell types based on an ensemble image classification model and a cell type classification method using the same.

[0002] In recent years, the medical field has been actively developing diagnostic systems utilizing artificial intelligence (AI) and deep learning technologies. In particular, advancements in image analysis technology have led to the emergence of AI models that diagnose and predict various diseases from pathological image data. These technologies assist medical professionals in their diagnosis by quickly and accurately analyzing pathological images, contributing to increased diagnostic efficiency. In particular, deep learning-based image classification models demonstrate high accuracy and reliability, playing a crucial role in the early diagnosis and prediction of various diseases.

[0003] Acute leukemia (AL) is a malignant disease that originates in the bone marrow and blood, and is divided into acute myeloid leukemia (AML) and acute lymphoblastic leukemia (ALL). Diagnosing these diseases plays a crucial role in patient treatment, and accurate cell type classification is essential. AML and ALL are characterized by specific cell types, myeloblasts and lymphoblasts, respectively. Distinguishing them from immature granulocytes (IG) and normal cells (NORMAL) is also crucial.

[0004] Accurate diagnosis of acute leukemia can determine the success of treatment. However, improving the accuracy of cell type classification remains a significant challenge. Existing analysis methods are time-consuming, costly, and heavily dependent on expert experience. Therefore, an AI-based automated diagnostic system is needed to improve the accuracy of cell type classification and provide rapid, efficient diagnosis.

[0005] One purpose of the present disclosure is to provide a medical diagnostic device that classifies cell types such as AML, ALL, IG, and NORMAL using an ensemble image classification model that combines a ConvNext model and a Transformer model, and a cell type classification method using the same.

[0006] Another object of the present disclosure is to provide a medical diagnostic device and a cell type classification method using the same, which efficiently processes image data through preprocessing techniques such as image normalization, noise removal, and data augmentation, and optimizes cell type classification by adjusting the weights of an ensemble model according to the cell type being targeted.

[0007] However, the problems to be solved by the present disclosure are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0008] A medical diagnostic device for classifying cell types according to the present disclosure for achieving the above-described technical task includes a memory storing at least one command; and at least one processor executing the at least one command, wherein the at least one processor receives peripheral blood slide image data, performs preprocessing on the peripheral blood slide image data, and classifies at least one cell type among AML group, ALL group, IG group, and NORMAL group from the peripheral blood slide image data using an ensemble image classification model, and the ensemble image classification model may include at least one ConvNext model and at least one Transformer model.

[0009] At this time, the ensemble image classification model may include two ConvNext models, two DEIT models, and one CAIT model, and the ConvNext model may include at least one of ConvNext 224 and ConvNext 352, the DEIT model may include at least one of DEIT 224 and DEIT 384, and the CAIT model may include CAIT 224.

[0010] Additionally, the at least one processor can classify the preprocessed peripheral blood slide image data into at least one of Neutrophil, Band neutrophil, Eosinophil, Basophil, Monocyte, Lymphocyte, Variant lymphocyte, Myeloblast, Lymphoblast, Myelocyte, Metamyelocyte, and Promyelocyte.

[0011] In addition, the at least one processor may divide the peripheral blood slide image data into five subsets (folds) with the AML group, the ALL group, the IG group, and the NORMAL group in equal proportions, and train the ensemble image classification model using 5-stratified k-fold cross-validation for the five subsets.

[0012] Additionally, the at least one processor may combine the final outputs in a soft voting manner when classifying the at least one cell type from the peripheral blood slide image data using the ensemble image classification model.

[0013] Additionally, the at least one processor can target classification of a specific cell type among the at least one cell type by differently assigning a prior weight to the ensemble image classification model based on a user's setting.

[0014] In addition, the at least one processor may provide a visualization of the at least one cell type classified from the peripheral blood slide image data to a user, calculate a performance index including at least one of accuracy, precision, recall, and F1 score for the at least one cell type classified from the peripheral blood slide image data, and provide the calculated performance index to the user.

[0015] In addition, a method for classifying cell types performed by a processor of a device according to the present disclosure for achieving the above-described technical task includes the steps of: receiving peripheral blood slide image data; performing preprocessing on the peripheral blood slide image data; and classifying at least one cell type among an AML group, an ALL group, an IG group, and a NORMAL group from the peripheral blood slide image data using an ensemble image classification model, wherein the ensemble image classification model may include at least one ConvNext model and at least one Transformer model.

[0016] In addition, a computer program stored in a computer-readable recording medium for implementing the present disclosure may be further provided.

[0017] In addition, a computer-readable recording medium recording a computer program for implementing the present disclosure may be further provided.

[0018] According to the aforementioned problem solving means of the present disclosure, the medical diagnostic device of the present disclosure and the cell type classification method using the same can provide a highly accurate diagnosis result by classifying cell types such as AML, ALL, IG, and NORMAL using an ensemble image classification model that combines a ConvNext model and a Transformer model.

[0019] In addition, the medical diagnostic device of the present disclosure and the cell type classification method using the same can efficiently process image data through preprocessing techniques such as image normalization, noise removal, and data augmentation, and optimize cell type classification by adjusting the weights of the ensemble model according to the cell type being targeted.

[0020] The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0021] Figure 1 is a drawing showing a block configuration of a medical diagnostic device of the present disclosure.

[0022] Figure 2 is a conceptual diagram showing the operation of the medical diagnostic device of the present disclosure.

[0023] Figure 3 is a flowchart showing the operation of the medical diagnostic device of the present disclosure.

[0024] FIG. 4 is a diagram showing examples of cell types and learning sets classified by the medical diagnostic device of the present disclosure.

[0025] FIGS. 5A to 5L are exemplary images of the first to twelfth cell types classified by the medical diagnostic device of the present disclosure.

[0026] Fig. 6 is a diagram showing the configuration and operation of the ensemble image classification model of the present disclosure.

[0027] Figure 7 is a diagram showing the performance of the ensemble image classification model of the present disclosure.

[0028] Figure 8 is a diagram showing the cell classification results of the ensemble image classification model of the present disclosure.

[0029] Figure 9 is a diagram showing the cell classification performance for the entire cell population of the medical diagnostic device of the present disclosure.

[0030] Figure 10 is a diagram showing the cell classification performance of the medical diagnostic device of the present disclosure for the AML group.

[0031] Figure 11 is a diagram showing the cell classification performance for the ALL group of the medical diagnostic device of the present disclosure.

[0032] FIG. 12 is a diagram showing the cell classification performance for the IG group of the medical diagnostic device of the present disclosure.

[0033] Throughout this disclosure, the same reference numerals denote the same components. This disclosure does not describe all elements of the embodiments, and any content that is common in the technical field to which this disclosure pertains or that overlaps between embodiments is omitted. The terms "part, module, element, block" used in the specification may be implemented in software or hardware, and depending on the embodiments, multiple "parts, modules, elements, blocks" may be implemented as a single component, or a single "part, module, element, block" may include multiple components.

[0034] Throughout the specification, when a part is said to be "connected" to another part, this includes not only direct connection but also indirect connection, and indirect connection includes connection via a wireless communication network.

[0035] Additionally, when a part is said to "include" a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise specifically stated.

[0036] Throughout the specification, when we say that an element is "on" another element, this includes not only cases where the element is in contact with the other element, but also cases where another element exists between the two elements.

[0037] The terms first, second, etc. are used to distinguish one component from another, and the components are not limited by the aforementioned terms.

[0038] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0039] The identification codes for each step are used for convenience of explanation and do not describe the order of each step. Each step may be performed in a different order than specified unless the context clearly indicates a specific order.

[0040] The operating principle and embodiments of the present disclosure are described below with reference to the attached drawings.

[0041] As used herein, the term "device according to the present disclosure" encompasses a variety of devices capable of performing computational processing and providing results to a user. For example, the device according to the present disclosure may include a computer, a server device, and a portable terminal, or may be any one of them.

[0042] Here, the computer may include, for example, a notebook, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser.

[0043] The above server device is a server that processes information by communicating with an external device, and may include an application server, a computing server, a database server, a file server, a game server, a mail server, a proxy server, and a web server.

[0044] The above portable terminal may include, for example, a wireless communication device that ensures portability and mobility, and may include all kinds of handheld-based wireless communication devices such as a PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminal, a smart phone, and a wearable device such as a watch, a ring, a bracelet, an anklet, a necklace, glasses, contact lenses, or a head-mounted device (HMD).

[0045] The artificial intelligence-related functions according to the present disclosure are operated via a processor and memory. The processor may be comprised of one or more processors. In this case, one or more processors may be a general-purpose processor such as a CPU, an AP, a Digital Signal Processor (DSP), a graphics-only processor such as a GPU or a Vision Processing Unit (VPU), or an artificial intelligence-only processor such as an NPU. One or more processors control the processing of input data according to predefined operating rules or artificial intelligence models stored in memory. Alternatively, if one or more processors are artificial intelligence-only processors, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0046] The predefined operation rules or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that the basic artificial intelligence model is trained using a learning algorithm using a plurality of learning data, thereby creating a predefined operation rules or artificial intelligence model set to perform a desired characteristic (or purpose). This learning may be performed on the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and / or system. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0047] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values, and performs neural network operations through operations between the operation results of the previous layer and the multiple weights. The multiple weights of the multiple neural network layers may be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained from the artificial intelligence model is reduced or minimized during the learning process. The artificial neural network may include a deep neural network (DNN), and examples thereof include, but are not limited to, a convolutional neural network (CNN), a deep neural network (DNN), a RecuREnt neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional RecuREnt deep neural network (BRDNN), or deep Q-networks.

[0048] According to an exemplary embodiment of the present disclosure, a processor can implement artificial intelligence. Artificial intelligence refers to a machine learning method based on an artificial neural network that imitates human neurons (biological neurons) to enable machines to learn. Artificial intelligence methodologies can be categorized into supervised learning, in which input data and output data are provided together as training data depending on the learning method, so that the solution (output data) to the problem (input data) is determined; unsupervised learning, in which only input data is provided without output data, so that the solution (output data) to the problem (input data) is not determined; and reinforcement learning, in which a reward (Reward) is provided from an external environment whenever an action (Action) is taken in the current state (State), and learning is performed in a direction to maximize this reward. In addition, artificial intelligence methodologies can be categorized according to the architecture, which is the structure of the learning model. The architectures of widely used deep learning technologies can be categorized into convolutional neural networks (CNNs), recurrent neural networks (RNNs), transformers, and generative adversarial networks (GANs).

[0049] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0050] FIG. 1 is a drawing showing a block configuration of a medical diagnostic device (100) of the present disclosure, and FIG. 2 is a conceptual diagram (200) showing the operation of the medical diagnostic device (100) of the present disclosure.

[0051] Referring to FIG. 1, a medical diagnostic device (100) according to one embodiment may include a processor (120), a memory (110), and a communication unit (not shown). The configuration illustrated in FIG. 1 illustrates a configuration related to an explanation of an embodiment according to the present disclosure, and other configurations (e.g., a communication unit) for performing functions of an electronic device may be added in addition to the illustrated configuration, and the illustrated configurations may be omitted.

[0052] According to one embodiment, the memory (110) is a storage medium used by the medical diagnostic device (100), and can store data such as at least one command or setting information corresponding to at least one program. The program can include an operating system (OS) program and various application programs.

[0053] In one embodiment, the memory (110) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., an SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read only memory (ROM), an electrically erasable programmable ROM (EEPROM), a programmable ROM (PROM), a magnetic memory, a magnetic disk, and an optical disk.

[0054] In one embodiment, the communication unit may include a wireless communication unit (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication unit. The communication unit may communicate with an external electronic device via a network (e.g., a long-distance communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a local area network or a wide area network)).

[0055] In one embodiment, the communication unit may support 5G networks and next-generation communication technologies, such as new radio access technology (NR). The NR access technology may support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimizing terminal power and connecting multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The communication unit may support high-frequency bands (e.g., mmWave bands) to achieve high data transmission rates, for example. The communication unit may support various technologies to secure performance in high-frequency bands, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antennas, analog beamforming, or large scale antennas.

[0056] According to one embodiment, the processor (120) may, for example, execute software (e.g., a program) to control at least one other component (e.g., a hardware or software component) of a medical diagnostic device (100) connected to the processor (120) and perform various data processing or operations.

[0057] In one embodiment, the processor (120) may store commands or data received from another component (e.g., a communication unit) in volatile memory, process the commands or data stored in the volatile memory, and store resulting data in non-volatile memory as at least part of data processing or calculation.

[0058] In one embodiment, the processor (120) may include a main processor (e.g., a central processing unit or an application processor) or a secondary processor (e.g., a graphics processing unit, a neural processing unit (NPU), or a communication processor) that can operate independently or in conjunction with the main processor. For example, the processor (120) may include at least one of a central processing unit (CPU), an application processor (AP), or a microprocessor.

[0059] A processor (120) according to one embodiment may perform operations of a medical diagnostic device (100) described through the drawings below. For example, as shown in the conceptual diagram (200) of FIG. 2, the processor (120) may receive peripheral blood slide image data. The processor (120) may perform preprocessing on the peripheral blood slide image data. The processor (120) may classify at least one cell type among the AML group, the ALL group, the IG group, and the NORMAL group from the peripheral blood slide image data using an ensemble image classification model. Here, the ensemble image classification model may include at least one ConvNext model and at least one Transformer model.

[0060] In this way, the processor (120) can provide highly accurate diagnostic results by classifying cell types using an ensemble image classification model that combines a ConvNext model and a Transformer model. In particular, the processor (120) can quickly and accurately classify important cell types such as AML, ALL, IG, and NORMAL.

[0061] Additionally, the processor (120) can efficiently process peripheral blood slide image data through preprocessing techniques such as image normalization, noise removal, and data augmentation to maximize the performance of the ensemble image classification model. This ensures consistent diagnostic performance in various environments.

[0062] In addition, the processor (120) can supplement the weaknesses of individual models and increase the stability and reliability of predictions by combining the prediction results of multiple models using a soft voting method.

[0063] Additionally, the processor (120) can improve the predictive power of a specific cell type by providing the ability to adjust the weights of the ensemble image classification model according to the cell type targeted by the user. In other words, the processor (120) can flexibly optimize cell type classification according to the user's needs.

[0064] Additionally, the processor (120) visualizes the classified cell types and provides them to the user, thereby facilitating easy understanding of the diagnostic results. Furthermore, the processor (120) can evaluate the performance of the ensemble image classification model by providing performance indicators such as accuracy, precision, recall, and F1 score.

[0065] FIG. 3 is a flowchart illustrating the operation of the medical diagnostic device (100) of the present disclosure. FIG. 4 is a diagram illustrating examples (400) of cell types and learning sets classified by the medical diagnostic device (100) of the present disclosure. FIGS. 5A to 5L are example images (500A to 500I) of the first to twelfth cell types classified by the medical diagnostic device (100) of the present disclosure. FIG. 6 is a diagram illustrating the configuration and operation (600) of the ensemble image classification model of the present disclosure.

[0066] Referring to FIGS. 3 to 6, a medical diagnostic device (100) according to the present disclosure may receive peripheral blood slide image data (operation 310), perform preprocessing on the peripheral blood slide image data (operation 320), and classify at least one cell type among AML group, ALL group, IG group, and NORMAL group from the peripheral blood slide image data using an ensemble image classification model (operation 330). For example, the ensemble image classification model may include at least one ConvNext model and at least one Transformer model.

[0067] In one example, in operation 310, the medical diagnostic device (100) may receive peripheral blood slide image data. For example, the peripheral blood slide image data may be received from an external image acquisition device. For example, the external image acquisition device may receive the peripheral blood slide image data from at least one of a microscope, a digital slide scanner, and an image data server.

[0068] Meanwhile, according to an embodiment, the medical diagnostic device (100) may perform at least one of integrity check, image quality check, metadata processing, and security processing on the received peripheral blood slide image data.

[0069] A medical diagnostic device (100) can learn cell types from peripheral blood slide image data and classify cell types. The peripheral blood slide image data may include at least one cell type from among the AML group, the ALL group, the IG group, and the NORMAL group. For example, the peripheral blood slide image data may include image data for at least one of Neutrophil, Band neutrophil, Eosinophil, Basophil, Monocyte, Lymphocyte, Variant lymphocyte, Myeloblast, Lymphoblast, Myelocyte, Metamyelocyte, and Promyelocyte.

[0070] As shown in Figure 4, the peripheral blood slide image data may include 33,869 training sets. For example, the peripheral blood slide image data may include 13,220 training sets for Neutrophils. For example, the peripheral blood slide image data may include 521 training sets for Band Neutrophils. For example, the peripheral blood slide image data may include 295 training sets for Eosinophils. For example, the peripheral blood slide image data may include 219 training sets for Basophils. For example, the peripheral blood slide image data may include 1,849 training sets for Monocytes. For example, the peripheral blood slide image data may include 9,899 training sets for Lymphocytes. For example, the peripheral blood slide image data may include 777 training sets for Variant Lymphocytes. For example, peripheral blood slide image data may include 3,510 training sets for myeloblasts. For example, peripheral blood slide image data may include 1,628 training sets for lymphoblasts. For example, peripheral blood slide image data may include 1,056 training sets for myelocytes. For example, peripheral blood slide image data may include 582 training sets for metamyelocytes. For example, peripheral blood slide image data may include 313 training sets for promyelocytes.

[0071] The primary cell may be a neutrophil. For example, referring to Figure 5a, the primary cell may be a neutrophil (500a) with a segmented nucleus and granules. For example, the primary cell is an important cell of the innate immune system that detects and destroys pathogens such as bacteria. It can primarily induce an inflammatory response at the site of infection and engulf the pathogens.

[0072] The secondary cell may be a band neutrophil. For example, referring to Figure 5b, the secondary cell may be a rod-shaped neutrophil (500b) with a band-shaped nucleus and a non-segmented form. For example, the secondary cell may be an immature neutrophil that can transform into a mature neutrophil during infection or an inflammatory response.

[0073] The third cell may be an eosinophil. For example, referring to Figure 5c, the third cell may be an eosinophil (500c), which has a two-lobed nucleus and large eosinophilic granules. For example, the third cell is involved in defense against parasitic infections and allergic responses, and can modulate inflammatory responses by secreting compounds such as histamine.

[0074] The fourth cell may be a basophil. For example, referring to Figure 5d, the fourth cell may be a basophil (500d) with a multilobated nucleus and large alkaline granules. For example, the fourth cell is involved in allergic and inflammatory responses, and can release histamine and heparin to regulate inflammation and prevent blood clotting.

[0075] Fifth-cells may be monocytes. For example, referring to Figure 5e, fifth-cells may be monocytes (500e) with a large, single nucleus, often with a bean-shaped nucleus. For example, fifth-cells can migrate to tissues and differentiate into macrophages or dendritic cells, engulfing pathogens and regulating immune responses.

[0076] The sixth cell may be a lymphocyte. For example, referring to Figure 5f, the sixth cell may be a lymphocyte (500f) with a large, round nucleus and a relatively small amount of cytoplasm. For example, the sixth cell is an important cell of the adaptive immune system, capable of differentiating into B cells and T cells, producing antibodies and carrying out cell-mediated immune responses.

[0077] Type VII cells may be variant lymphocytes. For example, referring to Figure 5g, type VII cells may be variant lymphocytes (500g) with abnormally large cytoplasm and nuclei of various shapes. For example, type VII cells are atypical lymphocytes that appear in response to viral infection or other stimuli and can enhance the immune response.

[0078] The 8th cell may be a myeloblast. For example, referring to Figure 5h, the 8th cell may be an immature cell (500h) with a large nucleus and a small amount of cytoplasm. For example, the 8th cell is an immature white blood cell precursor that originates in the bone marrow and can differentiate into various types of myeloid white blood cells.

[0079] The ninth cell may be a lymphoblast. For example, referring to Figure 5i, the ninth cell may be an immature cell (500i) with a large nucleus and a small amount of cytoplasm. For example, the ninth cell may be a precursor of a lymphocyte and may differentiate into a B lymphocyte or a T lymphocyte.

[0080] The 10th cell may be a myelocyte. For example, referring to Figure 5j, the 10th cell may be a myelocyte (500j) with a relatively large cytoplasm and a highly developed nucleus. For example, the 10th cell may be an intermediate-stage immature myeloid leukocyte that can differentiate into more mature granulocytes (e.g., neutrophils, eosinophils, basophils).

[0081] The stage 11 cell may be a metamyelocyte. For example, referring to Figure 5k, the stage 11 cell may be a metamyelocyte (500k), which has a bean-shaped nucleus and more cytoplasm. For example, the stage 11 cell may be a cell in the final stage of differentiation into a mature granulocyte in the bone marrow.

[0082] The 12th cell may be a promyelocyte. For example, referring to Figure 5l, the 12th cell may be a promyelocyte (500l) with a large cytoplasm and prominent granules. For example, the 12th cell may be derived from a myeloblast and may be a cell in the stage before differentiating into a myelocyte.

[0083] According to an example, in operation 320, the medical diagnosis device (100) may perform preprocessing on the peripheral blood slide image data. For example, the medical diagnosis device (100) may perform at least one of a first preprocessing that performs image normalization, a second preprocessing that performs noise reduction, and a third preprocessing that performs data augmentation.

[0084] In one embodiment, the medical diagnostic device (100) may perform a first preprocessing step, which involves image normalization. Image normalization may be a preprocessing step that adjusts peripheral blood slide image data to a certain range to enhance learning efficiency and balance each channel. For example, image normalization allows an ensemble image classification model to consistently process peripheral blood slide image data, increase learning speed, and prevent overfitting.

[0085] For example, the medical diagnosis device (100) can divide each pixel value of the peripheral blood slide image data by 255 and adjust it to a range between 0 and 1. For example, the medical diagnosis device (100) can perform normalization by calculating the mean and standard deviation of each channel (e.g., RGB), subtracting the mean from each pixel value, and dividing by the standard deviation.

[0086] Image normalization can stabilize model learning by adjusting the range of data and balance each channel to prevent ensemble image classification models from being biased toward a specific channel.

[0087] In one embodiment, the medical diagnostic device (100) may perform a second preprocessing step that removes noise. Denoising may be a preprocessing step that removes unnecessary noise from an image, thereby improving the signal-to-noise ratio (SNR) and better preserving important features. For example, noise removal may increase the clarity and quality of the image, and enhance the performance of an ensemble image classification model.

[0088] For example, the medical diagnosis device (100) can replace the current pixel value by using the average value of surrounding pixels in an average filter method. For example, the medical diagnosis device (100) can remove noise by using a Gaussian filter method and calculating a weighted average of each pixel of the image with surrounding values ​​using a kernel that follows a Gaussian distribution. For example, the medical diagnosis device (100) can remove noise by using the median value of surrounding pixel values ​​centered on the current pixel in a median filter method. For example, the medical diagnosis device (100) can remove noise while preserving the boundary portion of the image in an anisotropic diffusion method.

[0089] In one embodiment, the medical diagnostic device (100) may perform a third preprocessing step that involves data augmentation. Data augmentation may be a preprocessing step that increases the amount of data by modifying existing image data to generate new learning data. For example, data augmentation may improve the generalization performance of an ensemble image classification model and prevent overfitting.

[0090] For example, the medical diagnosis device (100) can rotate the image by a certain angle. For example, the medical diagnosis device (100) can rotate the image by 90 degrees, 180 degrees, 270 degrees, or by any angle to enable learning from various angles.

[0091] For example, the medical diagnostic device (100) can flip the image left and right or up and down. For example, the medical diagnostic device (100) can reduce the directional bias of the model by diversifying the directionality of the image.

[0092] For example, the medical diagnostic device (100) can facilitate learning of objects of various sizes by enlarging or reducing an image.

[0093] For example, a medical diagnostic device (100) can facilitate object recognition learning at various locations by cropping and using a portion of an image.

[0094] Additionally, the medical diagnostic device (100) can perform at least one of brightness and contrast adjustment, addition of noise, and color jittering.

[0095] This data augmentation can increase the amount of data by transforming peripheral blood slide images in various ways and improve the generalization performance of ensemble image classification models.

[0096] According to an example, in operation 330, the medical diagnostic device (100) can classify at least one cell type among the AML group, the ALL group, the IG group, and the NORMAL group from the peripheral blood slide image data using an ensemble image classification model.

[0097] Referring to FIG. 6, the medical diagnostic device (100) can classify the preprocessed peripheral blood slide image (610) into at least one of the AML group, the ALL group, the IG group, and the NORMAL group using an ensemble image classification model.

[0098] AML may be composed primarily of immature cells of the myeloid lineage. For example, AML may include myeloblasts, promyelocytes, myelocytes, and metamyelocytes.

[0099] ALL may be composed of immature cells of the lymphoid lineage. For example, ALL may include lymphoblasts.

[0100] The IG group may be composed of immature granulocytes. For example, the IG group may include myelocytes, metamyelocytes, and promyelocytes.

[0101] The NORMAL group may include normal white blood cell types. For example, the NORMAL group may include Neutrophils, Band Neutrophils, Eosinophils, Basophils, Monocytes, Lymphocytes, and Variant Lymphocytes.

[0102] The medical diagnostic device (100) can accurately classify various cell types from peripheral blood slide image data using a high-performance ensemble image classification model. For example, the medical diagnostic device (100) can rapidly and accurately diagnose various diseases, including acute leukemia (AL).

[0103] Meanwhile, the ensemble image classification model used by the medical diagnostic device (100) for cell type classification may be a single artificial intelligence model or may be implemented as multiple artificial intelligence models. The artificial intelligence model may be composed of a neural network (or artificial neural network) and may include statistical learning algorithms that mimic biological neurons in machine learning and cognitive science. A neural network may refer to a model in general that has problem-solving capabilities by changing the binding strength of synapses through learning, in which artificial neurons (nodes) that form a network by combining synapses. The neurons of the neural network may include a combination of weights or biases. The neural network may include one or more layers composed of one or more neurons or nodes. For example, the device may include an input layer, a hidden layer, and an output layer. The neural network constituting the device can infer a desired result (output) from an arbitrary input (input) by changing the weights of neurons through learning.

[0104] At least one processor included in the medical diagnosis device (100) can generate a neural network, train a neural network, perform a calculation based on received input data, generate an information signal based on the calculation result, or retrain a neural network. The models of the neural network may include various types of models such as CNN (Convolution Neural Network) such as GoogleNet, AlexNet, VGG Network, R-CNN (Region with Convolution Neural Network), RPN (Region Proposal Network), RNN (RecuREnt Neural Network), S-DNN (Stacking-based deep Neural Network), S-SDNN (State-Space Dynamic Neural Network), Deconvolution Network, DBN (Deep Belief Network), RBM (Restrcted Boltzman Machine), Fully Convolutional Network, LSTM (Long Short-Term Memory) Network, Classification Network, etc., but are not limited thereto. The processor may include one or more processors for performing calculations according to the models of the neural network. It may include a processor. For example, the neural network may include a deep neural network.

[0105] Neural networks include CNN (Convolutional Neural Network), RNN (RecuREnt Neural Network), perceptron, multilayer perceptron, FF (Feed Forward), RBF (Radial Basis Network), DFF (Deep Feed Forward), LSTM (Long Short Term Memory), GRU (Gated RecuREnt Unit), AE (Auto Encoder), Variational Auto Encoder (VAE), Denoising Auto Encoder (DAE), Sparse Auto Encoder (SAE), Markov Chain (MC), Hopfield Network (HN), Boltzmann Machine (BM), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Deep Convolutional Network (DCN), Deconvolutional Network (DN), Deep Convolutional Inverse Graphics (DCIGN) Network), GAN (Generative Adversarial Network), LSM (Liquid State Machine), ELM (Extreme Learning It will be understood by those skilled in the art that any neural network may be included, including but not limited to, a Machine (Echo State Network), an ESN (Echo State Network), a DRN (Deep Residual Network), a DNC (Differentiable Neural Computer), an NTM (Neural Turning Machine), a CN (Capsule Network), a KN (Kohonen Network), and an AN (Attention Network).

[0106] According to an exemplary embodiment of the present disclosure, at least one processor included in the medical diagnosis device (100) is a CNN (Convolution Neural Network) such as GoogleNet, AlexNet, VGG Network, etc., R-CNN (Region with Convolution Neural Network), RPN (Region Proposal Network), RNN (RecuREnt Neural Network), S-DNN (Stacking-based deep Neural Network), S-SDNN (State-Space Dynamic Neural Network), Deconvolution Network, DBN (Deep Belief Network), RBM (Restrcted Boltzman Machine), Fully Convolutional Network, LSTM (Long Short-Term Memory) Network, Classification Network, Generative Modeling, eXplainable AI, Continual AI, Representation Learning, AI for Material Design, BERT for natural language processing, SP-BERT, MRC / QA, Text Analysis, Dialog System, GPT-3, GPT-4, Visual Analytics for vision processing, Visual Understanding, Video Synthesis, ResNet Anomaly for data intelligence Various artificial intelligence structures and algorithms can be used, including but not limited to Detection, Prediction, Time-Series Forecasting, Optimization, Recommendation, and Data Creation.

[0107] As shown in Fig. 6, the medical diagnosis device (100) may include an ensemble image classification model composed of a total of five models. The ensemble image classification model may include at least one ConvNext model (620) and at least one Transformer model (630).

[0108] Specifically, the ensemble image classification model may include two ConvNext models, two DEIT models, and one CAIT model. For example, the ConvNext model may include at least one of ConvNext 224 and ConvNext 352. For example, the DEIT model may include at least one of DEIT 224 and DEIT 384. For example, the CAIT model may include CAIT 224.

[0109] Preprocessed peripheral blood slide image data can be input to five individual models. For example, two ConvNext models, two DEIT models, and one CAIT model can each independently make predictions on the input peripheral blood slide image, outputting probability values ​​for the cell types AML, ALL, IG, and NORMAL.

[0110] In one embodiment, the medical diagnosis device (100) can train the ensemble image classification model using 5-stratified k-fold cross-validation. For example, the medical diagnosis device (100) can divide the peripheral blood slide image data into five subsets (folds) with equal proportions of the AML group, the ALL group, the IG group, and the NORMAL group, and train the ensemble image classification model using 5-stratified k-fold cross-validation for the five subsets.

[0111] Specifically, 5-stratified k-fold cross-validation can maintain the same proportion of AML, ALL, IG, and NORMAL groups within each subset to address the data imbalance problem and improve the generalization performance of the ensemble image classification model.

[0112] For example, the medical diagnostic device (100) can divide the collected image data into five subsets (folds). The medical diagnostic device (100) can be set so that the ratio of cell types in each subset remains the same.

[0113] The medical diagnosis device (100) can train an ensemble image classification model through cross-validation on five subsets. For example, the medical diagnosis device (100) can train an ensemble image classification model by using one of the five subsets as validation data and the remaining four subsets as training data. Each of the five subsets can be sequentially used as validation data. That is, the medical diagnosis device (100) can repeat training of the ensemble image classification model five times so that each subset is used as validation data once.

[0114] A medical diagnostic device (100) can reliably classify at least one cell type among the AML group, the ALL group, the IG group, and the NORMAL group based on peripheral blood slide image data through a 5-stratified k-fold cross-validation process.

[0115] In one embodiment, the medical diagnostic device (100) may combine (640) the final output using a soft voting method when classifying at least one cell type from the peripheral blood slide image data using the ensemble image classification model. For example, the soft voting method may be a method of making a final decision by combining the predicted probabilities of each individual model.

[0116] The medical diagnosis device (100) may include an ensemble image classification model composed of five individual models. For example, the medical diagnosis device (100) may include two ConvNext models (e.g., ConvNext 224, ConvNext 352), two DEIT models (e.g., DEIT 224, DEIT 384), and one CAIT model (e.g., CAIT 224). Each individual model may input peripheral blood slide image data and independently output a probability value for a cell type. For example, the ConvNext 224 model may predict that a specific image has a probability of AML of 0.7, a probability of ALL of 0.1, a probability of IG of 0.1, and a probability of NORMAL of 0.1. The medical diagnosis device (100) may collect predicted probability values ​​for each cell type from the five individual models.

[0117] The medical diagnosis device (100) can calculate a final probability value by weighting the predicted probability values ​​of each individual model. The weighted average can be calculated by assigning a prior weight according to the importance of each individual model. For example, the medical diagnosis device (100) can assign a prior weight of 0.2 to the first ConvNext model, 0.1 to the second ConvNext model, 0.3 to the first DEIT model, 0.3 to the second DEIT model, and 0.1 to the first CAIT model. The medical diagnosis device (100) can determine the cell type with the highest probability among the calculated final probability values ​​as the final prediction, and classify the peripheral blood slide image data into the final predicted cell type. For example, if the final probability values ​​are 0.62 for AML, 0.14 for ALL, 0.14 for IG, and 0.1 for NORMAL, the final predicted cell type can be determined as AML.

[0118] In one embodiment, the medical diagnostic device (100) can target classification of a specific cell type among the at least one cell type by differently assigning a prior weight to the ensemble image classification model based on a user's settings.

[0119] For example, the medical diagnostic device (100) may provide a function to optimize the classification performance of a specific cell type by assigning different pre-weights to the ensemble image classification model according to the user's settings. The function to change the pre-weights according to the targeting is useful when the user wishes to strengthen the diagnosis of a specific cell type, and by flexibly adjusting the weights, the predictive performance of the ensemble image classification model can be adjusted to suit the user's needs.

[0120] For example, a medical diagnostic device (100) can enhance diagnostic flexibility and accuracy by adjusting the predictive performance of a model according to a specific diagnostic purpose. For example, the medical diagnostic device (100) can derive optimal prediction results suited to various clinical situations by allowing the user to adjust the weights of individual models as needed.

[0121] In one embodiment, the medical diagnostic device (100) can visualize and provide to a user at least one cell type classified from the peripheral blood slide image data.

[0122] For example, a medical diagnostic device (100) can visualize classified cell types through a user interface (UI). The UI supports high-resolution graphics so that the user can clearly display the classification results.

[0123] For example, the medical diagnostic device (100) may display classified cell types along with the original image, allowing the user to easily compare diagnostic results. For example, each cell type may be clearly distinguished through color coding, labels, or other visual indicators.

[0124] For example, the medical diagnostic device (100) may provide detailed information about each classified cell along with a visual image. For example, the detailed information about the cell may include cell type, classification probability, model reliability, etc.

[0125] In one embodiment, the medical diagnostic device (100) can calculate a performance index including at least one of accuracy, precision, recall, and F1 score for the at least one cell type classified from the peripheral blood slide image data, and provide the performance index to a user.

[0126] For example, accuracy can represent the proportion of correctly classified cells among all predictions. Precision, for example, can represent the proportion of true positive cells among positive predictions. Recall, for example, can represent the proportion of true positive cells that were correctly predicted. For example, the F1 score, which is the harmonic mean of precision and recall, can be used to assess the balance between the two metrics.

[0127] Additionally, depending on the embodiment, the medical diagnostic device (100) can generate a report including the performance indicators. For example, the medical diagnostic device (100) can provide the report in various formats, such as PDF or Excel, thereby documenting and effectively sharing the diagnostic results.

[0128] FIG. 7 is a diagram (700) showing the performance of the ensemble image classification model of the present disclosure, FIG. 8 is a diagram (800) showing the cell classification results of the ensemble image classification model of the present disclosure, and FIG. 9 is a diagram showing the cell classification performance (900) for the entire cell population of the medical diagnostic device (100) of the present disclosure.

[0129] Referring to FIGS. 7 and 8, the ensemble image classification model combined the final results using soft voting. The ensemble image classification model of the medical diagnostic device (100) of the present disclosure can achieve excellent performance by appropriately ensembling a ConvNext model, which is effective in distinguishing between lymphoblasts and myeloblasts, and a Transformer model, which is effective in distinguishing between lymphoblasts and lymphocytes.

[0130] For example, an ensemble image classification model can combine a ConvNext model at a ratio of 30% and a Transformer model at a ratio of 70%. Specifically, an ensemble image classification model can combine ConvNext 352 at a ratio of 10% and ConvNext 224 at a ratio of 20%. Additionally, an ensemble image classification model can combine Deit 224 at a ratio of 25%, Deit 384 at a ratio of 20%, and Cait 224 at a ratio of 25%.

[0131] As shown in FIG. 7, it can be confirmed that the ensemble image classification model of the medical diagnosis device (100) of the present disclosure shows relatively superior diagnostic results compared to other artificial intelligence models. In addition, as shown in FIG. 8, it can be confirmed that the ensemble image classification model of the medical diagnosis device (100) of the present disclosure shows superior AUROC, precision, recall, and F1 score for all 12 cell types. In addition, as shown in FIG. 9, it can be seen that the ensemble image classification model of the medical diagnosis device (100) of the present disclosure shows higher scores than individual models for each of the 12 cell types, demonstrating superior performance.

[0132] FIG. 10 is a diagram showing the cell classification performance (1000) of the medical diagnostic device (100) of the present disclosure for the AML group.

[0133] Referring to FIG. 10, it can be seen that the ensemble image classification model of the medical diagnosis device (100) of the present disclosure predicts a total of 3,140 myeloblasts out of a total of 3,510 myeloblasts in the True label, showing an accuracy of 89.5%. In addition, it can be seen that the ensemble image classification model of the medical diagnosis device (100) of the present disclosure incorrectly recognized 95 Lymphoblasts as part of the ALL series, which is 2.7%, and the error level is very low.

[0134] FIG. 11 is a diagram showing the cell classification performance (1100) for the ALL group of the medical diagnostic device (100) of the present disclosure.

[0135] Referring to FIG. 11, the ensemble image classification model of the medical diagnosis device (100) of the present disclosure predicted 1080 out of 1628 Lymphoblasts as myeloblasts in the true label, showing an accuracy of 66.3%. In addition, even when the ensemble image classification model of the medical diagnosis device (100) of the present disclosure incorrectly predicted Lymphoblasts, the error rate was 299 lymphocytes and 18.4%, which is similar to the error rate based on expert judgment. In addition, it can be seen that the ensemble image classification model of the medical diagnosis device (100) of the present disclosure incorrectly recognized Lymphoblasts as AML series at a low rate of 194 lymphocytes and 11.9%.

[0136] FIG. 12 is a diagram showing the cell classification performance (1200) for the IG group of the medical diagnostic device (100) of the present disclosure.

[0137] Referring to FIG. 12, it can be seen that the ensemble image classification model of the medical diagnosis device (100) of the present disclosure has a very high prediction rate for IG (e.g., promyelocyte, myelocyte, metamyelocyte), and in particular, has a high accuracy for promyelocyte. In addition, the ensemble image classification model of the medical diagnosis device (100) of the present disclosure showed a tendency to assign myelocytes and metamyelocytes to somewhat diverse cells, but overall, it hardly recognized them as lymphoblasts (1), and actually judged myelocytes, promyelocytes, and metamyelocytes as myeloblasts at a very low frequency.

[0138] In this way, the medical diagnostic device (100) of the present disclosure and the cell type classification method using the same can provide a highly accurate diagnostic result by classifying cell types such as AML, ALL, IG, and NORMAL using an ensemble image classification model that combines a ConvNext model and a Transformer model.

[0139] In addition, the medical diagnostic device (100) of the present disclosure and the cell type classification method using the same can efficiently process image data through preprocessing techniques such as image normalization, noise removal, and data augmentation, and optimize cell type classification by adjusting the weights of the ensemble model according to the cell type being targeted.

[0140] However, since this has been described above, a duplicate explanation will be omitted.

[0141] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0142] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of the program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

[0143] Although the embodiments described above have been described with limited drawings, those skilled in the art will recognize that various modifications and variations are possible based on the above teachings. For example, appropriate results can be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents. Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the following claims.

Claims

1. In a medical diagnostic device for classifying cell types, a memory storing at least one instruction; and comprising at least one processor for executing at least one instruction; At least one processor, Receive peripheral blood slide image data, Preprocessing is performed on the above peripheral blood slide image data, Using an ensemble image classification model, classifying at least one cell type among AML group, ALL group, IG group, and NORMAL group from the peripheral blood slide image data, The above ensemble image classification model is, Containing at least one ConvNext model and at least one Transformer model, Medical diagnostic devices.

2. In paragraph 1, The above ensemble image classification model includes two ConvNext models, two DEIT models, and one CAIT model, The above ConvNext model includes at least one of ConvNext 224 and ConvNext 352, The above DEIT model includes at least one of DEIT 224 and DEIT 384, The above CAIT model includes CAIT 224. Medical diagnostic devices.

3. In paragraph 1, At least one processor, Classifying the above preprocessed peripheral blood slide image data into at least one of Neutrophil, Band neutrophil, Eosinophil, Basophil, Monocyte, Lymphocyte, Variant lymphocyte, Myeloblast, Lymphoblast, Myelocyte, Metamyelocyte, and Promyelocyte. Medical diagnostic devices.

4. In paragraph 1, At least one processor, The peripheral blood slide image data is divided into five subsets (folds) with the same proportion of the AML group, the ALL group, the IG group, and the NORMAL group, Train the ensemble image classification model using 5-stratified k-fold cross-validation on the above five subsets. Medical diagnostic devices.

5. In paragraph 1, At least one processor, When classifying at least one cell type from the peripheral blood slide image data using the above ensemble image classification model, the final output is combined using a soft voting method. Medical diagnostic devices.

6. In paragraph 1, At least one processor, By differently assigning a prior weight to the ensemble image classification model based on the user's settings, targeting the classification of a specific cell type among the at least one cell type, Medical diagnostic devices.

7. In paragraph 1, At least one processor, Visualizing at least one cell type classified from the peripheral blood slide image data and providing it to the user, Calculating a performance index including at least one of accuracy, precision, recall, and F1 score for at least one cell type classified from the peripheral blood slide image data, Providing the above-mentioned performance indicators to the user, Medical diagnostic devices.

8. In a method for classifying cell types performed by a processor of a device, A step of receiving peripheral blood slide image data; A step of performing preprocessing on the peripheral blood slide image data; and A step of classifying at least one cell type among AML group, ALL group, IG group, and NORMAL group from the peripheral blood slide image data using an ensemble image classification model; The above ensemble image classification model is, Containing at least one ConvNext model and at least one Transformer model, Cell type classification method.

9. In paragraph 8, The above ensemble image classification model includes two ConvNext models, two DEIT models, and one CAIT model, The above ConvNext model includes at least one of ConvNext 224 and ConvNext 352, The above DEIT model includes at least one of DEIT 224 and DEIT 384, The above CAIT model includes CAIT 224. Cell type classification method.

10. In paragraph 8, The above classification steps are: Classifying the above preprocessed peripheral blood slide image data into at least one of Neutrophil, Band neutrophil, Eosinophil, Basophil, Monocyte, Lymphocyte, Variant lymphocyte, Myeloblast, Lymphoblast, Myelocyte, Metamyelocyte, and Promyelocyte. Cell type classification method.

11. In paragraph 8, The above classification steps are: The peripheral blood slide image data is divided into five subsets (folds) with the same proportion of the AML group, the ALL group, the IG group, and the NORMAL group, Train the ensemble image classification model using 5-stratified k-fold cross-validation on the above five subsets. Cell type classification method.

12. In paragraph 8, The above classification steps are: When classifying at least one cell type from the peripheral blood slide image data using the above ensemble image classification model, the final output is combined using a soft voting method. Cell type classification method.

13. In paragraph 8, The above classification steps are: By differently assigning a prior weight to the ensemble image classification model based on the user's settings, targeting the classification of a specific cell type among the at least one cell type, Cell type classification method.

14. In paragraph 8, A step of providing a user with visualization of at least one cell type classified from the peripheral blood slide image data; and Further comprising a step of calculating a performance index including at least one of accuracy, precision, recall, and F1 score for the at least one cell type classified from the peripheral blood slide image data, and providing the calculated performance index to a user. Cell type classification method.

15. A computer-readable recording medium storing a program for implementing a cell type classification method performed by a processor of a device described in any one of claims 8 to 14.

Images