AI MODS-based Tuberculosis Diagnosis System And Operation Method Thereof

Abstract

The present invention discloses an AI-based MODS tuberculosis diagnostic system and a method of operation thereof, which can provide diagnostic information necessary for confirmation and prescription when high-risk tuberculosis bacteria are found in a patient's body by analyzing tuberculosis-related image data measured through MODS-based optical equipment using AI reading technology to detect tuberculosis bacteria and test drug susceptibility of tuberculosis bacteria. The present invention comprises: an optic module for capturing an image of cultured tuberculosis bacteria; a launcher module for viewing the image captured by the optic module and storing the captured image; and an AI module for learning and analyzing the image stored in the launcher module by artificial intelligence; wherein the AI ​​module comprises a learning unit for learning artificial intelligence on the image of cultured tuberculosis bacteria and an analysis unit for analyzing the image stored in the launcher module based on the content learned by the learning unit.

Description

Technology Field

[0001] The present invention relates to an AI-based MODS-based tuberculosis diagnostic system and a method of operating the same. More specifically, it relates to an AI-based MODS-based tuberculosis diagnostic system and a method of operating the same that can provide diagnostic information necessary for confirmation and prescription when high-risk tuberculosis bacteria are found in a patient's body by analyzing tuberculosis disease-related image data measured through MODS-based optic equipment using AI reading technology to detect tuberculosis bacteria and test for drug susceptibility of tuberculosis bacteria. Background Technology

[0002] Tuberculosis is a major zoonotic infectious disease caused by Mycobacterium tuberculosis. While it primarily infects the lungs, it is a contagious disease that can spread to other organs. Since tuberculosis is often transmitted through the air, symptoms frequently manifest in the lungs; however, it can also infect the central nervous system, lymphatic tissue, bloodstream, genitourinary system, bones, and joints, meaning symptoms can appear throughout the entire body. Tuberculosis infecting the lungs is called pulmonary tuberculosis, while tuberculosis infecting other organs is called extrapulmonary tuberculosis. Tuberculosis bacteria parasitize cells within various organs, and because the immune system attacks them in conjunction with host cells, extensive tissue destruction occurs. If left untreated, this leads to a critical condition and, with a high frequency, death. Having claimed approximately 1.25 million lives worldwide in 2023 alone, tuberculosis is currently the deadliest infectious disease in the world, surpassing COVID-19 to become the leading cause of death. The fact that tuberculosis continues to record such figures even with the current advanced infectious disease response system suggests that structural limitations exist in the existing diagnosis and treatment system.

[0003] Tuberculosis is one of the three major diseases of humanity that is attracting international attention due to its high incidence and mortality rates, but it remains an unresolved global health problem because early detection is difficult due to the high cost and time consumption of existing tests.

[0004] Existing tuberculosis diagnostic methods are classified into solid culture, liquid culture, and molecular diagnostics, each of which has distinct limitations in terms of time, cost, and accuracy. Solid culture is inexpensive but takes several weeks or more to reach a diagnosis, while liquid culture is fast but involves a heavy burden of operating costs due to its reliance on expensive equipment. Molecular diagnostics (PCR-based) is very fast but is limited to target genes, has poor response to mutations, and carries the possibility of false negatives.

[0005] In contrast, the MODS (Microscopic-observation drugs-susceptibility) method, which allows for the simultaneous performance of microbiological diagnosis and drug susceptibility assessment based on Mycobacterium tuberculosis culture images, is recommended by WHO guidelines as the most accurate testing method. Prior art literature

[0006] Republic of Korea Published Patent Application No. 10-2017-0138732 Republic of Korea Registered Patent Application No. 10-1900200 Republic of Korea Registered Patent Application No. 10-2630064 The problem to be solved

[0007] The present invention aims to solve the above-mentioned problems by providing an AI-based MODS tuberculosis diagnostic system and a method of operation thereof, which can detect tuberculosis bacteria and test drug susceptibility of tuberculosis bacteria by analyzing tuberculosis-related image data measured through MODS-based optical equipment using AI interpretation technology, and provide diagnostic information necessary for confirmation and prescription when high-risk tuberculosis bacteria are found in a patient's body. means of solving the problem

[0008] An AI-based MODS tuberculosis diagnostic system according to one embodiment of the present invention for achieving the above-mentioned purpose comprises: an optic module in which an image of cultured tuberculosis bacteria is captured; a launcher module in which the image captured by the optic module is verified and the captured image is stored; and an AI module in which the image stored in the launcher module is learned and analyzed by artificial intelligence.

[0009] The AI ​​module includes a learning unit in which artificial intelligence learning is performed on an image of cultured tuberculosis bacteria, and an analysis unit in which an image stored in the launcher module is analyzed based on the content learned by the learning unit.

[0010] The above learning unit includes a learning data annotation unit in which annotation is performed on the learning data, a learning data augmentation unit in which the annotated learning data is augmented, and a learning data learning unit in which the augmented learning data is learned.

[0011] The analysis unit comprises a data collection unit in which an image stored in the launcher module is collected, a data preprocessing unit in which the image collected by the data collection unit is preprocessed, a data analysis unit in which the image preprocessed by the data preprocessing unit is analyzed to analyze at least one of tuberculosis bacteria and the drug susceptibility of the tuberculosis bacteria, and a report generation unit in which the content analyzed by the data analysis unit is generated as a report.

[0012] The above optic module includes a plate mounting part on which a well plate in which tuberculosis bacteria are cultured is mounted, an LED light source part located above the plate mounting part and irradiating an observation light source, a microscope part having a plurality of lenses located below the plate mounting part and magnifying an image of tuberculosis bacteria, and a reflector located below the plurality of lenses and reflecting the image magnified by the plurality of lenses in a direction perpendicular to the direction of irradiation of the light source, a camera part that captures an image of tuberculosis bacteria culture magnified through the lens of the image reflected by the reflector, and a barcode recognition part that automatically recognizes a QR code or barcode of the well plate mounted on the plate mounting part and stores sample information.

[0013] Additionally, the optic module further includes a UV lamp unit positioned close to the side of the LED light source unit to irradiate ultraviolet rays to the plate mounting unit, a determination unit to determine whether the well plate is loaded to the plate mounting unit, and an ultraviolet irradiation control unit that receives an unloading signal from the determination unit and irradiates ultraviolet rays from the UV lamp unit for a preset time when the well plate is unloaded from the plate mounting unit after the well plate has been loaded and a tuberculosis bacteria culture image has been captured.

[0014] Here, the above determination unit can, for example, determine whether there is a weight mounted on the plate mounting part, or determine based on an image of the well plate mounted on the plate mounting part.

[0015] Therefore, if the UV irradiation control unit irradiates ultraviolet light from the UV lamp unit for a certain period of time after unloading the well plate from the plate mounting unit, the operator does not need to perform the unnecessary action of manually turning on the UV lamp unit after unloading from the mounting unit and turning off the UV lamp unit after a certain period of time has elapsed.

[0016] In addition, the optic module further includes a shooting unit located above the plate mounting unit and capturing the upper part of the plate mounting unit, a verification unit that checks for foreign matter in the plate mounting unit in the image captured by the shooting unit after unloading the well plate from the upper part of the plate mounting unit, and a UV irradiation amount control unit that controls the UV element control unit to irradiate UV light for a longer time than the preset time when foreign matter is confirmed by the verification unit.

[0017] Accordingly, when no foreign matter is present on the upper surface of the plate mounting portion, ultraviolet light is irradiated for a preset time, and when foreign matter is present, ultraviolet light is irradiated sufficiently for a time longer than the preset time to sufficiently sterilize the upper surface of the plate mounting portion with ultraviolet light, thereby increasing the sterilization efficiency.

[0018] In addition, the above ultraviolet irradiation dose control unit increases the ultraviolet irradiation time in proportion to the amount of foreign matter confirmed by the above confirmation unit.

[0019] Therefore, when foreign matter is present on the upper surface of the plate mounting portion, instead of uniformly increasing the ultraviolet irradiation time, the ultraviolet irradiation period is increased in proportion to the amount of foreign matter, thereby enabling ultraviolet sterilization corresponding to the amount of foreign matter.

[0020] The AI ​​module further includes a 3D conversion unit that converts the tuberculosis culture image into a 3D image by estimating depth values ​​in 2D image data for the tuberculosis culture image captured by the camera unit and generating spatial numerical values.

[0021] The AI ​​module further includes a selective 3D conversion unit that selectively converts a 2D image of the tuberculosis bacteria culture image into a 3D image, and when the tuberculosis bacteria culture image consists of a background image and a tuberculosis bacteria image, the selective 3D conversion unit maintains the background image as a 2D image and converts the tuberculosis bacteria image into a 3D image.

[0022] Therefore, since the background image does not need to be analyzed as a 3D image, converting only the tuberculosis bacteria image into a 3D image allows for the analysis of the tuberculosis bacteria culture image by generating spatial numerical values ​​while reducing memory capacity and analysis time.

[0023] The AI ​​module further includes a resolution-separating 3D conversion unit that separates the area of ​​a 2D image of the tuberculosis bacteria culture image and generates a 3D image with a different resolution, and when the tuberculosis bacteria culture image consists of a background image and a tuberculosis bacteria image, the resolution-separating 3D conversion unit sets the resolution of the 3D image generated from the tuberculosis bacteria image to be higher than the resolution of the 3D image generated from the background image.

[0024] Therefore, since the background image is a part that does not need to be analyzed as a 3D image, a 3D image is generated at a low resolution, and since the tuberculosis bacteria image is a part that needs to be analyzed accurately, a 3D image is generated at a high resolution. This allows for the analysis of tuberculosis bacteria culture images by generating spatial numerical values ​​while reducing memory capacity and analysis time.

[0025] In the training data annotation section of the above AI module, annotation is performed by selecting parts of tuberculosis bacteria with cord formation and clumping from images of cultured tuberculosis bacteria, changing the color of the selected parts, and verifying them.

[0026] In the data analysis section of the above AI module, a sensitivity specialized model using Equation 1 below and a specificity specialized model using Equation 2 below are implemented as an ensemble model architecture based on a hierarchical structure using Equation 3 below.

[0027] (Equation 1) Sensitivity = TP / (TP+FN)

[0028] (Equation 2) Specificity = TN / (TN+FP)

[0029] (Equation 3) Hierarchical structure = TP+TN / (TP+TN+FP+FN)

[0030] In the above equations 1 to 3, TP (True Positive) represents the case where the actual result is positive and the prediction is also positive, FN (False Negative) represents the case where the actual result is positive and the prediction is negative, FP (False Positive) represents the case where the actual result is negative and the prediction is positive, and TN (True Negative) represents the case where the actual result is negative and the prediction is also negative.

[0031] A method of operation of an AI-based MODS tuberculosis diagnostic system according to an embodiment of the present invention for achieving the above-mentioned purpose comprises: a first step in which a well plate in which tuberculosis bacteria are cultured is mounted on an optic module and an image is captured; a second step in which the captured image is collected and preprocessed; a third step in which a depth value is added to the preprocessed image; and a fourth step in which the image with the added depth value is analyzed to detect tuberculosis bacteria and analyze the drug susceptibility of the tuberculosis bacteria to generate a report.

[0032] A method of operation for an AI-based MODS-based tuberculosis diagnostic system that implements edge computing by lightweighting the learning unit through the analysis of an image in the fourth step above, by selecting one or more of pruning, quantization, knowledge distillation, and lightweight network design. It includes a fourth step in which susceptibility is analyzed and a report is generated. Effects of the invention

[0033] As described above, the present invention has the advantage of reducing the time required for simultaneous tuberculosis screening and drug susceptibility testing from an average of 15 to 30 days for conventional tuberculosis screening to 3 to 7 days.

[0034] In addition, the present invention can reduce the time required for conventional microscope analysis work from 1.5 hours to 15 minutes through AI automatic reading. Brief explanation of the drawing

[0035] Figure 1 is a conceptual diagram of the AI-based MODS tuberculosis diagnostic system of the present invention. FIG. 2 is a side view of the optic module of the present invention. Figure 3(a) is a basic layout of a well plate, and Figure 3(b) is a layout of a well plate capable of detecting tuberculosis in 5 patients. Figure 4 is a configuration diagram of the learning unit of the present invention. Figure 5(a) shows the cord formation portion of Mycobacterium tuberculosis, and Figure 5(b) shows the clumping portion of Mycobacterium tuberculosis. Figure 6 illustrates the annotation process performed in the learning data annotation unit of the present invention. Figure 7(a) shows the annotation of the shape of tuberculosis bacteria for learning, and Figure 7(b) shows the annotation of tuberculosis bacteria in the form of polygons. Figure 8 is a configuration diagram of the analysis unit of the present invention. FIG. 9 is an example of a report generated by the report generation unit of the present invention. FIG. 10 is an example of a 2D image converted into a 3D image by the 3D conversion unit of the present invention. Figure 11 shows the confusion matrix. FIG. 12 is a Backbone-Neck-Head structure diagram of the present invention. Figure 13 is a flowchart of the operation method of the MODS-based tuberculosis diagnostic system by AI of the present invention. Specific details for implementing the invention

[0036] Expressions such as “comprising” or “may comprise” that may be used in various embodiments of the present disclosure indicate the presence of the disclosed corresponding function, operation, or component, etc., and do not limit one or more additional functions, operations, or components, etc. Furthermore, in various embodiments of the present disclosure, terms such as “comprising” or “having” are intended to specify the presence of the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0037] In various embodiments of the present disclosure, expressions such as “or” include any and all combinations of the words listed together. For example, “A or B” may include A, may include B, or may include both A and B.

[0038] Expressions such as "first," "second," "first," or "second" used in various embodiments of the present disclosure may modify various components of the various embodiments, but do not limit such components. For example, such expressions do not limit the order and / or importance of such components. Such expressions may be used to distinguish one component from another. For example, the first learner device and the second learner device are both learner devices and represent different learner devices. For example, without departing from the scope of the various embodiments of the present disclosure, the first component may be named the second component, and similarly, the second component may be named the first component.

[0039] When it is stated that a component is "connected" or "connected" to another component, it should be understood that the component may be directly connected or connected to the other component, but that a new component may exist between the component and the other component. On the other hand, when it is stated that a component is "directly connected" or "directly connected" to another component, it should be understood that no new component exists between the component and the other component.

[0040] In embodiments of the present disclosure, terms such as "module," "unit," "part," etc. are used to refer to a component that performs at least one function or operation, and such component may be implemented in hardware or software, or a combination of hardware and software. Additionally, a plurality of "modules," "units," "parts," etc. may be integrated into at least one module or chip and implemented as at least one processor, except where each needs to be implemented in specific individual hardware.

[0041] The terms used in the various embodiments of this disclosure are used merely to describe specific embodiments and are not intended to limit the various embodiments of this disclosure. The singular expression includes the plural expression unless the context clearly indicates otherwise.

[0042] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the various embodiments of this disclosure pertain.

[0043] Terms such as those defined in commonly used dictionaries should be interpreted as having meanings consistent with their meanings in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in the various embodiments of the present disclosure.

[0045] Hereinafter, an AI-based MODS tuberculosis diagnostic system according to an embodiment of the present invention will be described with reference to the drawings. Figure 1 is a conceptual diagram of the AI-based MODS tuberculosis diagnostic system of the present invention. Referring to Figure 1, the AI-based MODS tuberculosis diagnostic system of the present invention includes an optic module (100) and an AI module (300). Here, MODS (Microscopic-observation drugs-susceptibility) is a method of measuring drug resistance while directly observing cultured tuberculosis bacteria through a microscope, and is a culture testing method proven to have higher sensitivity and faster speed than conventional culture-based tuberculosis testing. The optic module (100) and the launcher module (200) are connected via a wired or wireless network, and the launcher module (200) and the AI ​​module (300) can be connected via a wired or wireless network; however, to prevent external hacking or data falsification, the AI ​​module (300) is not connected to an external cloud server or a third-party communication network.

[0046] The optic module (100) captures an image of cultured tuberculosis bacteria. The optic module (100) automatically captures an image of tuberculosis bacteria cultured in a MODS culture environment. FIG. 2 is a side view of the optic module (100) of the present invention. Referring to FIG. 2, the optic module (100) includes a plate mounting part (110) on which a well plate (W) in which tuberculosis bacteria are cultured is mounted, an LED light source part (120) located above the plate mounting part (110) to supply a light source to the plate, a microscope part (130) having a plurality of lenses (131) located below the plate mounting part (110) to magnify the tuberculosis bacteria and a reflector (132) that reflects the magnified image to the plurality of lenses (131) from below the plurality of lenses (131), and a camera part (140) positioned horizontally on the reflector (132) to capture the magnified image through the lenses (131). A well plate (W) having 24 wells can be mounted on the plate mounting section (110), the microscope section (130) is equipped with multiple high-magnification lenses (131), and the camera section (140) captures images of tuberculosis bacteria culture through automatic focus adjustment. Additionally, the optic module (100) is equipped with a barcode recognition section to automatically recognize the QR code or barcode of the well plate (W) mounted on the plate mounting section (110) and store sample information. Conventionally, due to the process in which a skilled user directly moves the sample, focuses through the microscope section (130), and visually inspects the sample, the physical distance between the sample and the examiner is very close, and the exposure to the risk of infection is high. Therefore, the camera section (140) can reduce the time required for the inspection and ensure user safety through a miniaturized automatic focus adjustment device that automatically sets the focus when inspecting the sample. Additionally, the camera unit (140) is equipped with a driving stage so that the camera unit (140) moves along the XYZ axes and can take images without moving the well plate (W).In addition, the LED light source (120) is equipped with a UV lamp so that after the inspection is finished, ultraviolet light can be irradiated onto the plate mounting part (110) to disinfect the plate mounting part (110).

[0047] The well plate (W) of the present invention is equipped with 24 wells and can test specimens from 1 to 5 patients. However, it is not limited to 24 wells, and if necessary, a well plate (W) equipped with 1, 6, 12, 48, or 96 wells can be used. FIG. 3(a) is a basic layout of the well plate (W), and FIG. 3(b) is a layout of the well plate (W) capable of detecting tuberculosis in 5 patients. As shown in FIG. 3(a), the basic layout allows for the detection of tuberculosis bacteria and drug susceptibility testing (INH, RIF, MFX, LFX) using a specimen from one patient at two concentrations per drug, Critical concentration (CC) and Clinical breakpoint (CB). This is a layout that allows for the detection of tuberculosis bacteria, the presence of multidrug-resistant tuberculosis, and redundant testing of the FQ drugs MFX and LFX in a single patient. In addition, for quality control (QC), it is possible to use both common tuberculosis bacteria and multidrug-resistant (MDR) bacteria. Since the layout is designed for a single patient, it has the advantage of increasing the accuracy of results by allowing for duplicate testing. Furthermore, in addition to the basic 24-well plate (W) layout, it can be changed to a layout configuration like Fig. 3(b), which allows testing of five patient specimens at once, in cases where the incidence of drug-resistant tuberculosis patients is low and the incidence of tuberculosis is expected to be high. By setting up multiple layouts, there is an advantage in being able to apply a layout suitable for treatment according to the environment of the hospital or institution (whether there are many or few tuberculosis patients) and the patient's symptoms.

[0048] The launcher module (200) can view images captured by the optic module (100) and stores the captured images. The launcher module (200) of the present invention may include one or more processors, one or more memories, one or more storage, and one or more communication interfaces, and these may be connected to each other via a bus. An image of cultured tuberculosis bacteria captured by the camera unit (140) of the optic module (100) is transmitted to the launcher module (200), which is connected to the optic module (100) via a wired or wireless network. The launcher module (200) is equipped with an output unit that can view images captured by the optic module (100) and an image storage unit that stores the captured images. The launcher module (200) includes an input unit that allows a user to change settings or input text, and an output unit that allows viewing images or reports. Additionally, the launcher module (200) is equipped with a setting section that can change the settings of the optic module (100). The optic module (100) is equipped with its own software and can be operated and controlled independently by default, but the user can change most of the settings of the optic module (100) through the setting section. Additionally, the user can check the report generated by the report generation section (324) through the output section of the launcher module (200).

[0049] The AI ​​module (300) performs learning and analysis of images stored in the launcher module (200) by artificial intelligence and may be configured to include a learning unit (310) and an analysis unit (320). The AI ​​module (300) of the present invention may include one or more processors, one or more memories, one or more storage, and one or more communication interfaces, and these may be connected to each other via a bus.

[0050] The learning unit (310) performs artificial intelligence learning on images of cultured tuberculosis bacteria. FIG. 4 is a configuration diagram of the learning unit (310) of the present invention. Referring to FIG. 4, the learning unit (310) of the present invention may be configured to include a learning data annotation unit (311), a learning data augmentation unit (312), and a learning data learning unit (313). Learning of artificial intelligence must precede the detection of tuberculosis bacteria and the drug susceptibility test of tuberculosis bacteria. To this end, a learning DB must be constructed, and for this purpose, a learning DB reflecting two morphological characteristics of high-risk tuberculosis bacteria (positive tuberculosis bacteria), namely cord formation and clumping, must be constructed. FIG. 5(a) shows the cord formation portion of tuberculosis bacteria, and FIG. 5(b) shows the clumping portion of tuberculosis bacteria. Tuberculosis bacteria do not have a uniform shape depending on the culture period, and the length, size, and shape of cord formation and clumping are all different. Furthermore, the two characteristics mentioned above show somewhat different forms depending on the clinical Mycobacterium tuberculosis isolated from the patient and the presence or absence of resistance to tuberculosis drugs. For this reason, Mycobacterium tuberculosis is classified as typical unstructured data; compared to other general unstructured data, it is difficult to acquire, starting from the initial stages, as it requires not only specialized knowledge but also limited accessibility. In the case of the training database, rather than training the entire image through the annotation stage, it is necessary to target objects that are easy to train and annotate the corresponding regions for training.

[0051] In training the training unit (310), the training:verification:test ratio in the training DB can be divided into 8:1:1 or 7:2:1 to perform training, verification, and testing, and retraining can be repeated after feedback on the training DB. Since the issue of overfitting cannot be excluded when training with a small amount of data that only emphasizes the model's performance, a procedure robust against overfitting must be performed within the framework. Therefore, for the training unit (310), it is desirable to train the training:verification:test ratio in the entire training DB at a ratio of 8:1:1 or 7:2:1. If the training of the training data is lowered to less than 7, sufficient training of the training unit (310) cannot be achieved due to a lack of training data, and if the training of the training data is raised to more than 8, verification or testing of the training data may be insufficient. Additionally, if the verification of the training data is lowered to less than 1, sufficient verification of the training data cannot be achieved, and if the verification of the training data is raised to more than 2, the reliability of the training data may be lowered due to excessive verification. In addition, if the test of the training data is lowered to 1 or less, sufficient testing of the training data cannot be performed, and if the test of the training data is raised to more than 1, testing of the training data is performed frequently, which may reduce the practical benefits of testing. In the case of the training unit (310), if the ratio of training:verification:test in the entire training DB is set to 8:1:1 or 7:2:1, there is an advantage that training, verification, and testing can be performed in an optimal combination. Verification and testing, which are feedback to the training DB, are continuously performed, and the data with updated feedback is converted back into training data, allowing the training unit (310) to learn by repeating this. Through this repeated learning, errors in the training data can be reduced.

[0052] The training data annotation unit (311) performs annotation on the training data. Since strict accuracy is required due to the nature of medical AI, the annotation on the training data is first performed by an expert. FIG. 6 illustrates the annotation process performed in the training data annotation unit (311) of the present invention. Referring to FIG. 6, the parts of the tuberculosis bacteria with cord formation and clumping are selected from images of cultured tuberculosis bacteria, the color of the selected parts is changed, and the annotation is performed by finally verifying this. Verification by an expert is required at every step, and detailed annotation work is performed so that the artificial intelligence can accurately identify the features of the target object while deep learning is in progress, and finally, a training DB is constructed. FIG. 7(a) shows the annotation of the shape of the tuberculosis bacteria for training, and FIG. 7(b) shows the annotation of the tuberculosis bacteria in a polygonal shape. Initially, annotation is performed by an expert, but once a certain period of learning is performed by the learning data learning unit (313), the learning data annotation unit (311) can automatically select and automatically annotate tuberculosis bacteria objects in the image without the help of an expert.

[0053] The training data augmentation unit (312) augments the annotated training data. Data augmentation is a procedure performed to increase robustness while securing data, which is the most common method among methods to improve model performance. Since the growth of tuberculosis bacteria is irregular, data augmentation is performed to maximize training efficiency with a small amount of data. The training data augmentation unit (312) of the present invention may use techniques such as left-right flipping, rotation, cropping, and size adjustment for data augmentation. The training data augmentation unit (312) may perform data augmentation using the Generative Adversarial Networks (GAN) technique of a vision model. GAN is a powerful generative AI model that excels at generating new synthetic data that mimics a given real data distribution. GAN consists of a generator and a discriminator. The generator generates fake data by using random noise as input and attempting to convert it into a sample that appears to be taken from the original training data. The discriminator distinguishes between real data and fake data generated by the generator, outputting a probability indicating how likely the input sample is to be real. During the learning process, the generator strives to continuously improve its ability to deceive the discriminator, while the discriminator strives to improve its ability to identify fakes. This adversarial process, driven by backpropagation, continues until the samples generated by the generator are so convincing that the discriminator cannot distinguish them from real data. In this way, the training data augmentation unit (312) can augment the training data.

[0054] The training data learning unit (313) learns augmented training data. In training a model suitable for the purpose, the training data learning unit (313) needs to select an appropriate architecture that detects a relatively small and irregularly shaped Region of Interest (ROI) present in the image, and for this purpose, a YOLO-based architecture can be used.

[0055] The learning data storage unit (314) stores augmented learning data learned in the learning data learning unit (313). The learning data storage unit (314) is equipped with a memory capable of storing the learned augmented learning data. The memory may include at least one type of storage medium among a flash memory type, a hard disk type, an SSD type (Solid State Disk type), an SSD type (Silicon Disk Drive type), a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), RAM (random access memory; RAM), SRAM (static random access memory), ROM (read-only memory; ROM), EEPROM (electrically erasable programmable read-only memory), PROM (programmable read-only memory), magnetic memory, a magnetic disk, and an optical disk.

[0056] The analysis unit (320) analyzes the image stored in the launcher module (200) based on the content learned by the learning unit (310). FIG. 8 is a configuration diagram of the analysis unit (320) of the present invention. Referring to FIG. 8, the analysis unit (320) of the present invention may be configured to include a data collection unit (321), a data preprocessing unit (322), a data analysis unit (323), and a report generation unit (324).

[0057] The data collection unit (321) collects images stored in the launcher module (200). The data collection unit (321) collects images captured by the camera unit (140) of the optic module (100) through the launcher module (200) and transmits them to the data preprocessing unit (322).

[0058] The data preprocessing unit (322) preprocesses the images collected from the data collection unit (321). The data preprocessing unit (322) preprocesses the images of cultured tuberculosis bacteria by adjusting brightness, size, position, etc., so that the images can be easily analyzed by artificial intelligence.

[0059] The data analysis unit (323) analyzes the images preprocessed by the data preprocessing unit (322) to analyze tuberculosis bacteria and the drug susceptibility of the tuberculosis bacteria. The data analysis unit (323) analyzes the images preprocessed by the patterns of tuberculosis bacteria learned by the training data learning unit (313) to determine whether the tuberculosis bacteria cultured in the well plate (W) are positive or negative. For positive tuberculosis bacteria, the drug susceptibility of the well into which the tuberculosis bacteria drug was injected is analyzed. Drug susceptibility can be performed for INH, RIF, MFX, and LFX drugs.

[0060] The report generation unit (324) generates a report based on the content analyzed by the data analysis unit (323). FIG. 9 is an example of a report generated by the report generation unit (324) of the present invention. Referring to FIG. 9, the report generated by the report generation unit (324) basically includes the subject's ID number, subject's name, sample number, specimen type, culture date, plate type, plate option, plate setting date, etc., and includes whether tuberculosis bacteria are positive and the results of the tuberculosis bacteria's drug susceptibility. The report generated by the report generation unit (324) is transmitted to the launcher module (200), and the report can be checked in the output unit.

[0061] The AI ​​module (300) further includes a 3D conversion unit that converts the shape of tuberculosis bacteria into 3D by generating spatial numerical values ​​through a generative model that estimates depth values ​​in 2D image data. Since the present invention is based on liquid culture in which tuberculosis bacteria are cultured in a liquid medium, the tuberculosis bacteria in the culture medium exhibit actual 3D structural characteristics; however, due to the nature of the 2D image obtained by a microscope—that is, because only cross-sections of the tuberculosis bacteria are captured during the image acquisition process—spatial information may be lost. To compensate for this disadvantage, the 3D conversion unit of the present invention can reconstruct the shape of the tuberculosis bacteria into 3D by generating spatial numerical values ​​that were lost through a generative model that estimates depth values ​​in 2D image data. This can be applied as a feature to effectively distinguish each object when the cord formation structure of the tuberculosis bacteria is formed in a complex manner, overlapping each other in space, through the depth value corresponding to the Z-axis on the spatial plane that was lost. Through this, existing image-based pixel values ​​and voxel values ​​extracted through the above technique are used to appropriately learn both 2D and 3D features, thereby achieving the effect of making the learning model more robust. In particular, since 3D features allow for the multidimensional identification of the characteristics of tuberculosis bacteria, they can significantly improve not only the detection of tuberculosis bacteria but also the accuracy of not detecting foreign substances that are not tuberculosis bacteria. FIG. 10 is an example diagram showing the conversion of a 2D image into a 3D image by the 3D conversion unit of the present invention. As shown in FIG. 10, the tuberculosis bacteria culture image can be converted into a 3D image by estimating depth values ​​from the 2D image data of the tuberculosis bacteria culture image captured by the camera unit to generate spatial numerical values.

[0062] Here, the 2D image of the tuberculosis culture image can be selectively converted into a 3D image. When the tuberculosis culture image consists of a background image (R1) and a tuberculosis image (R2), the background image (R1) is maintained as a 2D image while the tuberculosis image (R2) is converted into a 3D image. Since the background image (R1) is a part that does not need to be analyzed as a 3D image, converting only the tuberculosis image (R2) into a 3D image allows the tuberculosis culture image to be analyzed by generating spatial numerical values ​​while reducing memory capacity and analysis time.

[0063] Meanwhile, a 3D image can be generated by distinguishing the area of ​​the 2D image of the tuberculosis bacteria culture image and varying the resolution. When the tuberculosis bacteria culture image is composed of a background image (R1) and a tuberculosis bacteria image (R2), the resolution of the 3D image generated from the tuberculosis bacteria image (R2) is set higher than the resolution of the 3D image generated from the background image (R1). For example, if the ratio of the resolution of the 3D image generated from the tuberculosis bacteria image (R2) to the resolution of the 3D image generated from the background image (R1) is set to 1:0.1 to 0.3, the background image (R1) is a part that does not need to be analyzed as a 3D image, so a 3D image is generated at a low resolution, and the tuberculosis bacteria image (R2) is a part that needs to be analyzed accurately, so a 3D image is generated at a high resolution, thereby allowing the tuberculosis bacteria culture image to be analyzed by generating spatial numerical values ​​while reducing memory capacity or analysis time.

[0064] The AI ​​module (300) can implement a sensitivity specialized model and a specificity specialized model as an ensemble model architecture based on a hierarchical structure. Sensitivity (true positive rate) is the probability that a person who is actually positive is determined to be positive in the test, and specificity (true negative rate) is the probability that a person who is actually negative is determined to be negative in the test. FIG. 11 shows a confusion matrix. Here, TP (True Positive) is the case where the person is actually positive and the prediction is also positive, FN (False Negative) is the case where the person is actually positive and the prediction is negative, FP (False Positive) is the case where the person is actually negative and the prediction is positive, and TN (True Negative) is the case where the person is actually negative and the prediction is also negative. Referring to FIG. 11, sensitivity is the accuracy calculated based on the actual class in the confusion matrix, and is calculated by Equation 1 below.

[0065] (Equation 1) Sensitivity = TP / (TP+FN)

[0066] Specificity is calculated by Equation 2 below.

[0067] (Equation 2) Specificity = TN / (TN+FP)

[0068] In the case of this invention, since it is particularly difficult to achieve both sensitivity and specificity simultaneously, there is potential for improvement in all aspects, including not only enhancing the quality of the training data but also technical advancement of the model and the complementarity of the model architecture. If an ensemble model is implemented based on a hierarchical structure using existing sensitivity-specialized and specificity-specialized models, the priority of specificity becomes more critical; therefore, it is necessary to place the specificity model as the primary model and supplement its results with the sensitivity model. It is somewhat difficult to construct a perfect AI training model using a mono-model approach. In particular, the training data used in this invention is characterized by the fact that tuberculosis bacteria exhibit an atypical form, and since the images are measured based on MODS, the globally shared data is limited. Consequently, considering not only the atypical structure of the data but also potential modulation in various environments, even a well-trained AI model using a mono-model often suffers from reduced robustness when the environment changes even slightly. Accordingly, it is highly likely that various specialized models will be derived during the development process, such as models specialized for sensitivity, models specialized for specificity, or models for specific purposes. In such cases, it is necessary to construct a complex model structure through an ensemble architecture to leverage the strengths of all models. The ensemble architecture of the present invention may be selected from a parallel structure or a hierarchical structure.

[0069] The AI ​​module (300) can implement edge computing by selecting one or more of pruning, quantization, knowledge distillation, and lightweight network design to lighten the learning unit (310). If a server is established within the hospital, the AI ​​module (300) can be implemented through the server. However, in a closed environment where a server is not established within the hospital, it is necessary to implement edge computing that does not require a server. In this invention, rather than lightening in terms of speed, which requires real-time response in units of 0.01 seconds like autonomous driving, robust optimization in terms of storing continuously accumulated data and data transmission between HW and SW is given higher priority. In this invention, the learning unit (310) can be lightened through pruning, quantization, knowledge distillation, and lightweight network design. Pruning is a method of iterative learning that removes weights or neurons below a reference value. Quantization is a method of reducing storage space by quantizing the decimal representation of parameters such as weights, activation values, and gradient values ​​into integers. Knowledge distillation is a technique designed to train a smaller model by ensembling several previously trained models. Compact network design is a method of modifying the neural network structure to improve computational efficiency. Through edge computing, AI diagnosis of tuberculosis bacteria is performed in the AI ​​module (300) and does not pass through an external cloud server or a third-party communication network. Therefore, since there is no external leakage path, there is structurally no possibility of hacking, eavesdropping, or data falsification. In addition, the patient's sputum images are generated and diagnosed only within the hospital's internal system, and are processed in a de-identified state that does not contain identification information when stored or transmitted, thus satisfying all major security standards under the Personal Information Protection Act and the Medical Service Act.Since this invention applies a closed operating environment and an administrator mode accessible only to authenticated operating accounts, it can be safely operated independently without connecting to the hospital computer network.

[0070] The AI ​​module (300) can use a YOLO-based architecture, and YOLO is composed of a Backbone, Neck, and Head. FIG. 12 is a Backbone-Neck-Head structure diagram of the present invention. Referring to FIG. 12, the Backbone is a core component of the YOLO architecture and plays the role of extracting features of various scales from an input image. This process includes stacking convolutional layers and special blocks to generate feature maps of various resolutions. In the Backbone, depth values ​​are added to the 2D image to reconstruct it into 3D. The Neck combines features of different scales and transmits them to the Head for prediction. This process generally includes upsampling and concatenating feature maps of multiple levels, through which the model can effectively capture multi-scale information. The Head plays the role of generating a final prediction in terms of object detection and classification. It processes the feature map transmitted from the Neck to finally output bounding boxes and class labels for objects within the image. That is, the Head detects tuberculosis bacteria through the 3D reconstructed image.

[0072] Hereinafter, an operation method of an AI-based MODS-based tuberculosis diagnostic system according to an embodiment of the present invention will be described with reference to the drawings. FIG. 13 is a flowchart of the operation method of an AI-based MODS-based tuberculosis diagnostic system according to the present invention. Referring to FIG. 13, the operation method of an AI-based MODS-based tuberculosis diagnostic system according to the present invention may be composed of the following four steps.

[0073] Step 1 (S10) is a step in which a well plate (W) containing cultured tuberculosis bacteria is mounted on an optic module (100) and an image is captured. A specimen from a subject is collected and cultured using a culture kit, and the well plate (W) containing the cultured specimen is mounted on the plate mounting part (110) of the optic module (100) for image capture. The QR code or barcode of the mounted well plate (W) is recognized by the barcode recognition part, and image capture is initiated. After mounting the well plate (W), the LED light source part (120) is activated, and the magnified image through a plurality of lenses (131) and a reflector (132) is captured by the camera part (140). All processes after the well plate (W) is mounted on the optic module (100) are automatically performed by the optic module (100). The user can view the captured image through the launcher module (200).

[0074] The second step (S20) is a step in which captured images are collected and preprocessed. In the first step (S10), the images captured by the optic module (100) are transmitted to and stored in the launcher module (200), and the images stored in the launcher module (200) are collected by the data collection unit (321), and preprocessing is performed to facilitate image analysis by artificial intelligence. The data preprocessing unit (322) preprocesses the images by adjusting brightness, size, position, etc.

[0075] Step 3 (S30) is a step in which depth values ​​are added to the preprocessed image. In this step, the shape of the tuberculosis bacteria is reconstructed in 3D by generating spatial numerical values ​​that were lost through a generative model that estimates depth values ​​from 2D image data. The depth values ​​corresponding to the Z-axis on the spatial plane that were lost in the 2D image can be effectively used to distinguish each object when the cord formation structure of the tuberculosis bacteria is formed in a complex manner by overlapping each other in space during culture.

[0076] Step 4 (S40) is a step in which an image with added depth values ​​is analyzed to detect tuberculosis bacteria, the drug susceptibility of tuberculosis bacteria is analyzed, and a report is generated. The data analysis unit (323) analyzes the image based on the content learned by the learning data learning unit (313) to identify the cord formation and clumping parts within the image, thereby detecting tuberculosis bacteria and confirming the drug susceptibility of tuberculosis bacteria. Once tuberculosis bacteria and drug susceptibility are confirmed, a report is generated based on the analyzed results. The user can view the report through the output unit of the launching module (200).

[0078] The system described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the system and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0079] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0080] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0081] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0082] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below. Explanation of the symbols

[0083] 100 : Optic Module 110 : Plate part 120 : LED light source 130 : Microscope section 131 : Lens 132 : Reflector 140 : Camera section 200 : Launcher Module 300 : AI Module 310: Learning Unit 311: Training Data Annotation Section 312: Training Data Augmentation Section 313 : Training data training section 314 : Training data storage 320: Analysis Unit 321 : Data Collection Unit 322 : Data Preprocessing Section 323 : Data Analysis Department 324 : Report Generation Section W : Well plate

Claims

Claim 1 An optic module for capturing an image of cultured tuberculosis bacteria; a launcher module for verifying the image captured by the optic module and storing the captured image; and an AI module in which images stored in the launcher module are learned and analyzed by artificial intelligence; wherein the AI ​​module includes a learning unit in which artificial intelligence learning is performed on images of cultured tuberculosis bacteria and an analysis unit in which images stored in the launcher module are analyzed based on the content learned by the learning unit; wherein the learning unit includes a learning data annotation unit in which annotation is performed on learning data, a learning data augmentation unit in which annotated learning data is augmented, and a learning data learning unit in which augmented learning data is learned; wherein the analysis unit includes a data collection unit in which images stored in the launcher module are collected, a data preprocessing unit in which images collected from the data collection unit are preprocessed, a data analysis unit in which images preprocessed from the data preprocessing unit are analyzed to analyze tuberculosis bacteria and the drug susceptibility of said tuberculosis bacteria, and a report generation unit in which the content analyzed by the data analysis unit is generated as a report; and wherein the optic module includes a plate mounting unit in which a well plate in which tuberculosis bacteria are cultured is mounted, an LED light source unit located above the plate mounting unit to irradiate an observation light source, and below the plate mounting unit The microscope unit comprises a plurality of lenses positioned to magnify an image of tuberculosis bacteria and a reflector positioned below the plurality of lenses to reflect the magnified image to the plurality of lenses; a camera unit that captures a culture image of tuberculosis bacteria magnified through a lens from the image reflected by the reflector; and a barcode recognition unit that automatically recognizes a QR code or barcode of a well plate mounted on the plate mounting unit and stores sample information. Additionally, the optic module includes a UV lamp unit positioned close to the side of the LED light source unit to irradiate ultraviolet light onto the plate mounting unit, and a determination unit that determines whether the well plate is loaded onto the plate mounting unit.The apparatus further includes an ultraviolet irradiation control unit that receives an unloading signal from the discrimination unit and irradiates ultraviolet light from the UV lamp unit for a preset time after the well plate is loaded and a tuberculosis bacteria culture image is captured and the well plate is unloaded from the plate mounting unit; additionally, the optic module further includes a shooting unit located above the plate mounting unit and capturing the upper part of the plate mounting unit; a verification unit that checks for foreign substances in the plate mounting unit from the image captured by the shooting unit after the well plate is unloaded from the upper part of the plate mounting unit; and an ultraviolet irradiation dose control unit that controls the ultraviolet light to be irradiated for a time longer than the preset time in the ultraviolet irradiation control unit when foreign substances are confirmed by the verification unit; the ultraviolet irradiation dose control unit increases the ultraviolet irradiation time in proportion to the amount of foreign substances confirmed by the verification unit; and the AI ​​module further includes a 3D conversion unit that converts the tuberculosis bacteria culture image into a 3D image by estimating depth values ​​in 2D image data for the tuberculosis bacteria culture image captured by the camera unit and generating spatial numerical values. The above AI module further includes a resolution-separated 3D conversion unit that distinguishes regions of a 2D image for the tuberculosis bacterium culture image and generates a 3D image with different resolutions, and when the tuberculosis bacterium culture image consists of a background image and a tuberculosis bacterium image, the resolution-separated 3D conversion unit sets the resolution of the 3D image generated from the tuberculosis bacterium image to be higher than the resolution of the 3D image generated from the background image, and in the training data annotation section of the AI ​​module, annotation is performed by selecting parts of the tuberculosis bacterium with cord formation and clumping among the images of the cultured tuberculosis bacteria, changing the color of the selected parts, and verifying, and in the data analysis section of the AI ​​module,An AI-based MODS-based tuberculosis diagnostic system implementing a sensitivity-specialized model using Equation 1 below and a specificity-specialized model using Equation 2 below as an ensemble model architecture based on a hierarchical structure using Equation 3 below. (Equation 1) Sensitivity = TP / (TP+FN) (Equation 2) Specificity = TN / (TN+FP) (Equation 3) Hierarchical structure = TP+TN / (TP+TN+FP+FN) In Equations 1 to 3 above, TP (True Positive) represents the case where the actual value is positive and the prediction is also positive, FN (False Negative) represents the case where the actual value is positive and the prediction is negative, FP (False Positive) represents the case where the actual value is negative and the prediction is positive, and TN (True Negative) represents the case where the actual value is negative and the prediction is also negative. Claim 2 delete Claim 3 delete Claim 4 delete Claim 5 delete Claim 6 delete

Images