Method for analyzing lesions in medical images

Abstract

According to an embodiment of the present disclosure, a method for analyzing lesions in medical images, executed in a computing device, is disclosed, which may include using a pre-processing module to generate an input image for a pre-trained detection module from a medical image including a breast region, using the detection module to generate a probability value for the presence of a nodule in at least one region of interest based on the input image and first position information of the at least one region of interest, and using a post-processing module to determine second position information for a suspected nodule present in the medical image from the first position information based on the probability value for the presence of the nodule.

Description

Technical Field

[0001] The present invention relates to a method for processing medical images, and more specifically, to a method for detecting and evaluating lesions related to a specific disease existing in medical images using artificial intelligence.

Background Art

[0002] Medical images are materials that enable understanding of the physical states of various organs of the human body. Medical images include digital radiography images (X-ray), computed tomography (CT), magnetic resonance imaging (MRI), etc.

[0003] Research and technological development related to methods for automating the detection of lesions of specific diseases using medical images have been continuously carried out. However, the conventional technology only focuses on identifying the lesions existing in medical images, and does not generate and process the information related to the lesions in a form suitable for the diagnosis of specific diseases. That is, from the perspective of providing the necessary information suitable for the diagnosis of specific diseases, the conventional technology is in a situation where it cannot provide the performance that meets the purpose of detecting and evaluating lesions.

[0004] U.S. Registered Patent No. 7305111 (December 4, 2007) discloses an automated method for detecting pulmonary nodules for lung cancer screening.

Summary of the Invention

Problems to be Solved by the Invention

[0005] The present disclosure has been devised in view of the foregoing background art, and an object thereof is to provide a method for detecting and evaluating lesions for diagnosing a specific disease existing in medical images.

Means for Solving the Problems

[0006] Based on an embodiment of the present disclosure for realizing the foregoing problems, a method for lesion analysis in a medical image executed by a computing device is disclosed. The method includes: using a preprocessing module to generate an input image for a pre-learned detection module from a medical image including a chest region; using the detection module to generate a probability value related to the presence of nodules in at least one region of interest and first position information of the at least one region of interest based on the input image; and using a postprocessing module to determine second position information related to suspicious nodules present in the medical image from the first position information based on the probability value related to the presence of the nodules.

[0007] In an alternative embodiment, the step of generating the input image for the detection module includes: using the preprocessing module to calculate values of Hounsfield units based on a three-dimensional medical image including the chest region; and using the preprocessing module to generate a plurality of two-dimensional medical images from the three-dimensional medical image for which the values of the Hounsfield units have been calculated.

[0008] In an alternative embodiment, the step of generating the probability value related to the presence of the nodules and the first position information includes: using a first sub-detection module included in the detection module to generate a first probability value and first position information related to the at least one region of interest based on the plurality of two-dimensional medical images; and using a second sub-detection module included in the detection module to estimate a second probability value related to the at least one region of interest based on the three-dimensional medical image and the first position information.

[0009] In an alternative embodiment, the step of generating the first probability value and the first position information includes: using a first neural network module included in the first sub-detection module to generate a plurality of first feature maps having a plurality of sizes based on the plurality of two-dimensional medical images; using a second neural network module included in the first sub-detection module to concatenate at least a part of the plurality of first feature maps based on the sizes of the plurality of first feature maps to generate a plurality of second feature maps; and using a third neural network module included in the first sub-detection module to match the plurality of second feature maps with a predetermined anchor box to generate the first probability value and the first position information related to the at least one region of interest.

[0010] In an alternative embodiment, the step of generating the first probability value and the first position information includes: using the first sub-detection module to cluster at least a part of the plurality of regions of interest based on the ratio of the overlapping regions between the plurality of regions of interest when there are a plurality of regions of interest; and using the first sub-detection module to correct the coordinate system included in the first position information.

[0011] In an alternative embodiment, the step of calculating the second probability value includes: using a fourth neural circuit network module included in the second sub-detection module, based on the position information, performing encoding on a patch extracted from the three-dimensional medical image to generate at least one third feature map; using a fifth neural circuit network module included in the second sub-detection module, based on the third feature map, performing decoding to generate at least one fourth feature map; and using a sixth neural circuit network module included in the second sub-detection module, based on the feature map generated by integrating the third feature map and the fourth feature map, generating the second probability value related to at least one region of interest.

[0012] In an alternative embodiment, the second sub-detection module can be pre-trained by performing a first operation of training a neural circuit network based on randomly sampled training images and a second operation of training a neural circuit network based on training images selected based on recall and precision.

[0013] In an alternative embodiment, the step of obtaining the second position information related to the suspicious nodule includes: using the post-processing module to compare a probability value related to the existence of the nodule generated by a weighted sum of the first probability value and the second probability value with a threshold; and using the post-processing module to determine, as the second position information related to the suspicious nodule, the first position information of at least one region of interest corresponding to the probability value related to the existence of the nodule selected as a result of the comparison.

[0014] In an alternative embodiment, the method further includes generating a mask related to the suspicious nodule based on a patch of the medical image corresponding to the second position information using a pre-trained measurement module; and generating numerical information including at least one of a diameter and a volume of the suspicious nodule based on the mask related to the suspicious nodule.

[0015] In an alternative embodiment, the mask related to the suspicious nodule may include a first mask related to the entire region of the suspicious nodule generated based on a 3D patch corresponding to the second position information; and a second mask related to a region representing a specific attribute of the suspicious nodule generated based on the 3D patch corresponding to the second position information.

[0016] In an alternative embodiment, the method further includes classifying a class related to a state of the suspicious nodule based on the patch of the medical image and the mask related to the suspicious nodule using a pre-trained classification module.

[0017] In an alternative embodiment, the step of classifying a class related to a state of the suspicious nodule may include determining at least one of a type related to an attribute of the suspicious nodule, presence or absence of spiculation, and presence or absence of calcification based on the patch and the mask using different sub-modules included in the classification module.

[0018] In an alternative embodiment, the method further includes calculating an evaluation score of the suspicious nodule based on an auxiliary index for lung cancer diagnosis and based on the class and numerical information related to the state of the suspicious nodule; and when the subject of the input image corresponds to the subject of the analyzed image, using a pre-trained tracking module to correct the evaluation score of the medical image or the evaluation score of the analyzed image based on the imaging time points of the input image and the analyzed image.

[0019] In an alternative embodiment, the method may further include generating a user interface based on at least one of the second position information, mask, class, numerical information, or evaluation score related to the suspicious nodule.

[0020] In an alternative embodiment, the method may further include estimating the malignancy of the suspicious nodule using a pre-trained malignancy prediction module based on the second position information, class related to the state, and numerical information of the suspicious nodule.

[0021] In an alternative embodiment, the method may further include estimating the malignancy of the suspicious nodule using a pre-trained malignancy prediction module based on the patch of the medical image and the mask related to the suspicious nodule.

[0022] In an alternative embodiment, the method may further include generating a user interface based on at least one of the second position information, mask, class, numerical information, or malignancy related to the suspicious nodule.

[0023] Based on another embodiment of the present disclosure for realizing the foregoing problems, a method for analyzing lesions in medical images executed by a computing device is disclosed. The method includes: generating an input patch of a pre-trained evaluation module based on position information of a suspicious nodule existing in a medical image including a chest region; and generating a mask related to the suspicious nodule based on at least one input patch corresponding to the position information by using the pre-trained evaluation module.

[0024] In an alternative embodiment, the step of generating a mask related to the suspicious nodule includes: generating a first mask related to the entire region of the suspicious nodule based on the at least one input patch by using a first sub-evaluation module included in the evaluation module; and generating a second mask related to a region representing a specific attribute of the suspicious nodule based on the at least one input patch by using a second sub-evaluation module included in the evaluation module.

[0025] In an alternative embodiment, when a plurality of input patches having a plurality of sizes related to one suspicious nodule are input to the evaluation module in the step of generating the first mask, the step includes: generating a plurality of first sub-masks related to the one suspicious nodule from each of the plurality of input patches by using the first sub-evaluation module; and generating a first mask related to the entire region of the one suspicious nodule based on a combination of the plurality of first sub-masks by using the first sub-evaluation module and based on a result of the combination.

[0026] In an alternative embodiment, the step of generating the second mask includes: generating a second sub-mask related to a candidate region representing a specific attribute of the suspicious nodule based on the at least one input patch by using the second sub-evaluation module; identifying an overlapping region between the first mask and the second sub-mask by using the second sub-evaluation module; and generating a second mask related to a region representing a specific attribute of the suspicious nodule based on the identified overlapping region.

[0027] Based on another embodiment of the present disclosure for realizing the foregoing problems, a method for analyzing lesions in a medical image executed by a computing device is disclosed. The method includes receiving a patch generated based on position information of a suspicious nodule existing in a medical image including a chest region and a mask related to the suspicious nodule; and classifying a class related to the state of the suspicious nodule based on the patch and the mask using a pre-trained classification module.

[0028] In an alternative embodiment, the step of classifying a class related to the state of the suspicious nodule may include determining a type related to an attribute of the suspicious nodule based on the patch and the mask using a first sub-classification module included in the classification module; determining the presence or absence of spicules of the suspicious nodule based on the patch and the mask using a second sub-classification module included in the classification module; or determining the presence or absence of calcification of the suspicious nodule based on the patch and the mask using a third sub-classification module included in the classification module, including at least one of the steps.

[0029] In an alternative embodiment, the step of determining a type related to an attribute of the suspicious nodule may further include determining a first type related to a solid attribute of the suspicious nodule based on the patch and the mask using a first attribute classification module included in the first sub-classification module; determining a second type related to the solid attribute of the suspicious nodule based on the patch and the mask using a second attribute classification module included in the first sub-classification module; and finally determining a type related to the solid attribute of the suspicious nodule based on a result of comparing the first type and the second type using a third attribute classification module included in the first sub-classification module. In this case, the first attribute classification module can be pre-trained through a neural network.

[0030] In an alternative embodiment, the step of determining the second type may include: using the second attribute classification module to calculate, from the patch, the ratio of a plurality of voxels whose Hounsfield unit values are higher than a predetermined Hounsfield unit value among the plurality of voxels included in the mask; using the second attribute classification module to compare the ratio of the plurality of voxels with a threshold value; and based on the result of the comparison, determining a second type related to the solid attribute of the suspected nodule present in the patch.

[0031] In an alternative embodiment, the step of finally determining the type related to the solid attribute of the suspected nodule may include: when the first type is a type not included in the second type, using the third attribute classification module to determine the first type as the final type related to the solid attribute of the suspected nodule; and when the first type is a type included in the second type, using the third attribute classification module to determine the second type as the final type related to the solid attribute of the suspected nodule.

[0032] In an alternative embodiment, the first type may include solid, part-solid or non-solid. And the second type may include solid or non-solid.

[0033] In an alternative embodiment, the second sub-classification module may also be pre-trained through a neural network.

[0034] In an alternative embodiment, the step of determining the presence or absence of calcification of the suspicious nodule may include: using the third sub-classification module to calculate, from the patch, the ratio of a plurality of voxels whose Hounsfield unit value is higher than a predetermined Hounsfield unit value among the plurality of voxels included in the mask; using the third sub-classification module to compare the ratio of the plurality of voxels with a threshold value; and based on the result of the comparison, determining the presence or absence of calcification of the suspicious nodule present in the patch.

[0035] In an alternative embodiment, the third sub-classification module may also be pre-trained through a neural network.

[0036] Based on one embodiment of the present disclosure for realizing the above problems, a computer program stored in a computer-readable storage medium is disclosed. When the computer program is executed on one or more processors, the following operations for analyzing lesions in medical images are executed, and the operations include: using a preprocessing module to generate an input image for a pre-trained detection module from a medical image including a chest region; using the detection module to generate, based on the input image, a probability value related to the presence of nodules in at least one region of interest and first position information of the at least one region of interest; and using a post-processing module to determine, based on the probability value related to the presence of the nodules, second position information related to suspicious nodules present in the medical image from the first position information.

[0037] Based on an embodiment of the present disclosure for solving the foregoing problems, a computing device for analyzing lesions in medical images is disclosed. The device includes a processor including at least one core; a memory including a plurality of program codes executable by the processor; and a network unit for receiving a medical image including a chest region. The processor uses a preprocessing module to generate an input image of a pre-trained detection module from the medical image including the chest region, uses the detection module to generate a probability value related to the presence of nodules in at least one region of interest and first position information of the at least one region of interest based on the input image, and can use a postprocessing module to determine second position information related to suspected nodules existing in the medical image from the first position information based on the probability value related to the presence of the nodules.

[0038] Based on an embodiment of the present disclosure for realizing the foregoing problems, a user terminal for providing a user interface is disclosed. The user terminal can include a processor including at least one core; a memory; a network unit for receiving a user interface generated based on analysis information of lesions included in a medical image from a computing device; and an output unit for providing the user interface. In this case, the analysis information of the lesions can include at least one of position information of suspected nodules, a mask related to the suspected nodules, a class related to the state of the suspected nodules, numerical information of the suspected nodules, evaluation information related to the suspected nodules, or malignancy of the suspected nodules.

Advantages of the Invention

[0039] The present disclosure can provide a method for detecting and evaluating lesions for diagnosing a specific disease existing in a medical image.

Brief Description of the Drawings

[0040]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

[0041] Various embodiments will be described below with reference to the drawings. In this specification, various explanations are presented to facilitate the understanding of the present disclosure. However, it is obvious that such embodiments can be implemented without such specific explanations.

[0042] As used herein, terms such as "component", "module", "system", etc. refer to computer-related entities, hardware, firmware, software, combinations of software and hardware, or the execution of software. For example, a component can be a processing procedure executed on a processor, a processor, an object, an execution thread, a program, and / or a computer, but is not limited thereto. For example, an application executed on a computing device and the computing device can both be components. One or more components can reside within a processor and / or an execution thread. One component can be localized within one computer. One component can be distributed among two or more computers. Also, such components can be executed in various computer-readable media having various data structures stored therein. A component can communicate through local and / or remote processing, etc., using, for example, signals that include one or more data packets (e.g., data and / or signals from one component interacting with other components in a local system or a distributed system, and data transmitted via a network such as the Internet to other systems).

[0043] Note that the term "or" is used with the intention of meaning an inclusive "or" rather than an exclusive "or". That is, when not specifically specified and not clear from the context, "X uses A or B" is meant to mean one of the natural inclusive substitutions. That is, it is possible that "X uses A or B" applies to any of the following: X uses A; X uses B; or X uses both A and B. Also, the term "and / or" herein is to be understood to refer to all possible combinations of one or more of the recited related items.

[0044] Also, the term "comprising (including)" as a predicate and / or "comprising (including)" as a modifier should be understood to mean that the feature and / or component exists. However, the term "comprising (including)" as a predicate and / or "comprising (including)" as a modifier should be understood not to exclude the existence or addition of one or more other additional features, components, and / or groups thereof. Also, when the number is not particularly specified, or when it is not clear from the context that the singular form is indicated, the singular should generally be interpreted to mean "one or more" in this specification and the claims.

[0045] And the term "at least one of A or B" should be interpreted to mean "the case of including only A", "the case of including only B", and "the case of a combination of A and B".

[0046] Those skilled in the art should further recognize that the various exemplary logical blocks, configurations, modules, circuits, means, logics, and algorithm steps described as being related to the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability between hardware and software, the various exemplary components, blocks, configurations, means, logics, modules, circuits, and steps have been generally described in terms of their functionality. Whether such functionality is implemented as hardware or as software depends on the specific application and design constraints of the overall system. A skilled technician can implement the functionality described in various ways for an individual specific application. However, such a determination regarding the implementation should not be construed as departing from the scope of the present disclosure.

[0047] The description of the embodiments shown herein is provided so that those of ordinary skill in the art of the present disclosure can make use of or practice the present invention. Various modifications to such embodiments will be readily apparent to those of ordinary skill in the art of the present disclosure. The general principles defined herein can be applied to other embodiments without departing from the scope of the present disclosure. Accordingly, the present invention is not limited to the embodiments shown herein. The present invention should be construed in the broadest sense consistent with the principles and novel features shown herein.

[0048] In the present disclosure, a network function, an artificial neural circuit network, and a neural network can be used interchangeably.

[0049] On the other hand, the terms "image" or "image data" used in the detailed description and claims of the present invention refer to multi-dimensional data composed of discrete image elements (e.g., pixels in a two-dimensional image), in other words, an object that can be visually recognized (e.g., displayed on a video screen), or a digital representation of that object (e.g., a file corresponding to the pixel output of a CT or MRI device, etc.).

[0050] For example, "image" or "picture" can be a medical image of a subject collected by computed tomography (CT), magnetic resonance imaging (MRI), ultrasound, or any other known medical imaging system in the technical field of the present invention. The image does not necessarily have to be provided for medical purposes and can also be provided for non-medical purposes, such as X-ray imaging for security inspection.

[0051] In the detailed description and claims of the present invention, the "DICOM (Digital Imaging and Communications in Medicine)" standard is a general term for all standard specifications used for digital image representation and communication in medical devices. The DICOM standard is published by a joint committee composed of the American College of Radiology (ACR) and the National Electrical Manufacturers Association (NEMA).

[0052] Also, in the detailed description and claims of the present invention, the "Picture Archiving and Communication System (PACS)" is a term referring to a system that stores, processes, and transmits medical images in accordance with the DICOM standard. It is possible to store the data of medical images obtained using digital medical imaging devices such as X-rays, CTs, and MRIs in the DICOM format and transmit them to in-hospital or external terminals via a network. It is also possible to add reading results and medical records to the data.

[0053] FIG. 1 is a block configuration diagram of a computing device for analyzing lesions in a medical image according to an embodiment of the present disclosure.

[0054] The configuration of the computing device (100) illustrated in FIG. 1 is merely an exemplary simplified illustration. In an embodiment of the present disclosure, the computing device (100) may include other configurations for implementing the computing environment of the computing device (100), and it is also possible to configure the computing device (100) with only a part of the disclosed configurations.

[0055] The computing device (100) can include a processor (110), a memory (130), and a network unit (150).

[0056] In one embodiment of the present disclosure, the processor (100) can be composed of one or more cores, and can include a central processing unit (CPU), a general-purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), etc., for data analysis and deep learning processors. The processor (110) can read a computer program stored in the memory (130) and execute data processing for machine learning in one embodiment of the present disclosure. Based on one embodiment of the present disclosure, the processor (110) can perform operations for neural network learning. In deep learning (DL), the processor (110) can execute calculations for neural network learning, such as processing input data for learning, extracting features from input data, calculating errors, and updating the weights of the neural network using backpropagation. At least one of the CPU, GPGPU, and TPU of the processor (110) can process network function learning. For example, both the CPU and GPGPU can perform network function learning and data classification using network functions. Note that in one embodiment of the present disclosure, the processors of multiple computing devices can be used together to perform network function learning and data classification using network functions. Also, the computer program executed in the computing device in one embodiment of the present disclosure can be a program executable by the CPU, GPGPU, or TPU.

[0057] In one embodiment of the present disclosure, the processor (110) can read and interpret lesions related to a specific disease based on a medical image using at least one pre-trained machine learning module. The processor (110) can input the medical image into the first machine learning module to identify the location information of the lesions present in the medical image. The processor (110) can generate a patch corresponding to a portion of the medical image based on the location information of the lesions. The processor (110) can input the location information of the lesions and the corresponding patch into the second machine learning module to generate a mask related to the lesions. In this case, the mask can mean an aggregate of data including information related to the area where the lesions exist in the medical image. The processor (110) can input the patch and the mask into the third machine learning module to generate class information representing the state of the lesions. Through such operations, the processor (110) can generate information related to the lesions (e.g., the location and state of the lesions in the body) that serve as diagnostic criteria for a specific disease based on the medical image.

[0058] For example, the processor (110) can input a medical image including a chest region input to the network unit (150) into a pre-trained detection module. In this case, the medical image including the chest region can include a three-dimensional CT image including at least one lung tissue, etc. The processor (110) can input the medical image including the chest region into the detection module to obtain the location information of suspicious nodules present in the medical image. The location information of the suspicious nodules can include the central coordinate system of the region identified as a suspicious nodule in the medical image. When the medical image is a three-dimensional CT image, the location information of the suspicious nodules can include the coordinate values of the center (X, Y, Z) of the region determined to be a suspicious nodule.

[0059] The processor (110) can extract a patch corresponding to the position information from a medical image including the chest region based on the position information of the suspicious nodule obtained using the detection module. The processor (110) can input the patch corresponding to the position information of the suspicious nodule into a pre-trained evaluation module. The processor (110) can input the patch generated from the medical image into the evaluation module to generate a mask related to the suspicious nodule. In other words, the processor (110) can use the evaluation module to extract information related to the region where the suspicious nodule exists within the patch.

[0060] The processor (110) can input the patch extracted previously together with the mask related to the suspicious nodule generated using the evaluation module into a pre-trained classification module. The processor (110) can input the mask and the patch related to the suspicious nodule together into the classification module to classify the class related to the state of the suspicious nodule. In this case, the state of the suspicious nodule can include features, attributes, etc. of the suspicious nodule that form the basis for the judgment of lung diseases. In other words, the processor (110) can use the classification module to identify the state of the suspicious nodule in the patch of the medical image in order to obtain lesion information for the diagnosis of lung diseases.

[0061] In one embodiment of the present disclosure, the processor (110) can evaluate a lesion read from a medical image based on an auxiliary index for diagnosing a specific disease. The processor (110) can calculate a numerical value related to the region where the lesion exists based on a mask generated by the second machine learning module. The processor (110) can calculate an evaluation score of the lesion based on the numerical information including the numerical value of the lesion and the class information related to the state of the lesion based on an auxiliary index for diagnosing a specific disease. Further, the processor (110) can also predict the malignancy of the lesion using a pre-trained machine learning module. The processor (110) can estimate the malignancy of the lesion based on the position information of the lesion, the class information related to the state of the lesion, and the numerical information of the lesion through the fourth machine learning module. Through such operations, the processor (110) can generate evaluation information for a lesion that can be utilized as a diagnostic index for a specific disease based on a medical image.

[0062] For example, the processor (110) can generate numerical information related to the region where a suspicious nodule exists in a medical image based on the mask of the suspicious nodule generated using the evaluation module. In this case, the numerical information can include numerical values related to the diameter or volume of the suspicious nodule. The processor (110) can calculate an evaluation score for the suspicious nodule based on the numerical information of the suspicious nodule and the class related to the state of the suspicious nodule classified using the classification module, based on the auxiliary indicators for lung disease diagnosis stored in the memory (130). In this case, the auxiliary indicators for lung disease diagnosis can include classification indicators based on Lung-RADS (Lung CT Screening Reporting and Data System). In other words, the processor (110) can utilize both the structural information and the attribute information of the suspicious nodule, and determine the evaluation score of the suspicious nodule according to the criteria determined based on the auxiliary indicators for lung disease diagnosis. The evaluation score determined by the processor (110) can be utilized for the diagnosis and prognosis prediction of lung diseases for the subject of the medical image.

[0063] The processor (110) can input the position information of a suspicious nodule generated by the detection module, the class information related to the state of the suspicious nodule generated by the classification module, and the numerical information of the suspicious nodule generated based on the mask into a pre-trained malignancy prediction module. The processor (110) can input the position information, class information, and numerical information of the suspicious nodule into the malignancy prediction module to estimate the malignancy of the suspicious nodule. Also, the processor (110) can input the patch extracted from the medical image and the mask generated by the evaluation module into the malignancy prediction module to estimate the malignancy of the suspicious nodule. In other words, the processor (110) can utilize the quantitative information of the suspicious nodule to estimate the malignancy of the suspicious nodule, and can also utilize the image information related to the suspicious nodule to estimate the malignancy of the suspicious nodule. The processor (110) can predict the malignancy of the suspicious nodule affecting the lung disease by considering together the position information, structural information, and attribute information of the suspicious nodule existing in the medical image including the chest region through the malignancy prediction module. The malignancy predicted by the processor (110) can be utilized for the diagnosis of lung disease and prognosis prediction of the subject of the medical image, etc.

[0064] In one embodiment of the present disclosure, the processor (110) can correct the evaluation score related to the lesion based on a plurality of medical images related to a specific subject in a time-series relationship using a pre-trained machine learning module. When a medical image related to the same subject as the image analyzed by the processor (110) is input to the computing device (100), the processor (110) can use the fifth machine learning module to match the analyzed image with the lesion existing in the subsequently input medical image, so as to grasp the changed information. Then, the processor (110) can reflect the changed information in the evaluation score related to the lesion and correct the evaluation score. When there is no changed information, the processor (110) can maintain the existing evaluation score without correcting the evaluation score related to the lesion.

[0065] For example, based on the medical image received via the network unit (150), the processor (110) can perform the aforementioned operations to calculate the evaluation score of a suspicious nodule, and save the calculated evaluation score in the memory (130). When a new medical image is received via the network unit (150), the processor (110) can determine whether the subject of the new medical image corresponds to the subject of the analyzed medical image. That is, the processor (110) can determine whether the identifier (ID) used to identify the subject of the new medical image matches one of the identifiers of the plurality of analyzed medical images. When the identification ID identifier of the new medical image matches one of the identifiers of the plurality of analyzed medical images, the processor (110) can use the pre-trained tracking module to match the same suspicious nodule existing in the existing image and the new image. The processor (110) can identify the change information of the suspicious nodule matched using the tracking module, and based on the change information, can modify the evaluation score related to the suspicious nodule. Through such operations, the processor (110) can effectively track the changes in the lesions of a specific subject and improve the accuracy of the information necessary for the judgment related to the prognosis of lung diseases.

[0066] In one embodiment of the present disclosure, the memory (130) can store any form of information generated or determined by the processor (110) and any form of information received by the network unit (150).

[0067] In one embodiment of the present disclosure, the memory (130) can include a storage medium of at least one type among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), Random Access Memory (RAM), Static Random Access Memory (SRAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Programmable Read-Only Memory (PROM), magnetic memory, magnetic disk, and optical disk. The computing device (100) can also operate in cooperation with a web storage that executes the storage function of the memory (130) on the internet. The description of the foregoing memory is merely exemplary, and the present disclosure is not limited thereto.

[0068] The network unit (150) in one embodiment of the present disclosure can operate in cooperation with a known wired or wireless communication system having any form.

[0069] The network unit (150) can receive a medical image in which a human organ is depicted from a medical imaging system. For example, a medical image in which a human organ is represented can be learning data or inference data for a machine learning module learned with two-dimensional features or three-dimensional features. A medical image in which a human organ is depicted can be a three-dimensional CT region including at least one lung region. A medical image in which a human organ is depicted is not limited to the foregoing examples, and can include all images related to human organs acquired by imaging, such as X-ray images, MR images, etc.

[0070] In addition, the network unit (150) can transmit and receive information processed by the processor (110), user interfaces, etc. via communication with other terminals. For example, the network unit (150) can provide a user interface generated by the processor (110) to a client (e.g., a user terminal). Also, the network unit (150) can receive an external input made by a user to the client and transfer it to the processor (110). In this case, the processor (110) can process operations such as output, correction, change, addition, etc. of the information provided via the user interface based on the external input of the user received from the network unit (150).

[0071] On the other hand, the computing device (100) in one embodiment of the present disclosure can include a server as a computing system that transmits and receives information via communication with a client. In this case, the client can be any form of terminal that can access the server. For example, the computing device (100) that is a server can receive a medical image from a medical imaging system, analyze the lesion, and provide a user interface including the analysis result to the user terminal. In this case, the user terminal can output the user interface received from the computing device (100) that is a server and receive or process information input through interaction with the user.

[0072] The user terminal can display a user interface provided to provide analysis information of a lesion (e.g., a suspicious nodule, etc.) included in the medical image sent from the computing device (100) that is a server. Although illustration is omitted, the user terminal can include a network unit that receives a user interface from the computing device (100), a processor including at least one core, a memory, an output unit that provides a user interface, and an input unit that receives an external input made by the user.

[0073] In an additional embodiment, the computing device (100) can also include any form of terminal that receives data resources generated at any server and performs additional information processing.

[0074] FIG. 2 is a schematic diagram showing a network function in one embodiment of the present disclosure.

[0075] A machine learning model for chronic disease prediction or a deep learning model for preprocessing electrocardiogram signals according to an embodiment of the present disclosure can include a neural circuit network. Throughout this specification, an arithmetic model, a neural circuit network, a network function, and a neural network can be used in the same meaning. A neural circuit network generally consists of a set of interconnected computing units commonly called nodes. Such nodes can also be referred to as neurons. A neural circuit network is configured to include at least one or more nodes. The nodes (or neurons) constituting the neural circuit network can be interconnected by one or more links.

[0076] In a neural circuit network, one or more nodes connected via a link can form a relationship of relatively input nodes and output nodes. The concepts of input nodes and output nodes are relative. Any node that becomes an output node for a certain node can become an input node in the relationship with other nodes, and vice versa. As described above, the relationship between the input node and the output node can be established centering on the link. One input node can be connected via a link to one or more output nodes, and vice versa.

[0077] In the relationship between an input node and an output node connected via a single link, the value of the data at the output node can be determined based on the data input to the input node. Here, the node interconnecting the input node and the output node can have a weight value. The weight value can be variable, but can be changed by a user or an algorithm for the neural network to perform a desired function. For example, when one or more input nodes are interconnected to one output node by respective links, the output node can determine the value of the output node based on the values input to the input nodes connected to the output node and the weight values set for the links corresponding to the respective input nodes.

[0078] As described above, a neural network is formed by one or more nodes interconnected via one or more links, forming the relationship between an input node and an output node within the neural network. In a neural network, the characteristics of the neural network can be determined by the number of nodes and links, the correlation between the nodes and links, and the values of the weight values assigned to the respective links. For example, when there are two neural networks with the same number of nodes and links, but with different values of the link weight values, the two neural networks can be recognized as different.

[0079] A neural network can be composed of a set of one or more nodes. A subset of the nodes constituting the neural network can form a layer. Among the plurality of nodes constituting the neural network, some can form one layer based on the distance from the first input node. For example, a set of nodes with a distance of n from the first input node can form the nth layer. The distance from the first input node can be defined based on the minimum number of links that must be traversed to reach the node from the first input node. However, such a definition of a layer is arbitrarily cited for explanation, and the configuration of layers in a neural network can be defined in a manner different from the above description. For example, the layer of nodes can also be defined based on the distance from the final output node.

[0080] The first input node can mean one or more nodes in the neural circuit network where data is directly input without passing through a link in relation to other nodes. Or, in the relationship between nodes based on links in the neural circuit network, it can mean a node that does not have other input nodes connected via a link. Similarly, the final output node can mean one or more nodes in the neural circuit network that do not have an output node in relation to other nodes. Also, the hidden node can mean a node that is not the first input node or the final output node and constitutes the neural circuit network.

[0081] The neural circuit network according to an embodiment of the present disclosure can be a neural circuit network in which the number of nodes in the input layer is the same as the number of nodes in the output layer, and as it progresses from the input layer to the hidden layer, the number of nodes first decreases and then increases again. The neural circuit network according to an embodiment of the present disclosure can be a neural circuit network in which the number of nodes in the input layer is less than the number of nodes in the output layer, and the number of nodes decreases as it progresses from the input layer to the hidden layer. Also, the neural circuit network according to another embodiment of the present disclosure can be a neural circuit network in which the number of nodes in the input layer is more than the number of nodes in the output layer, and the number of nodes increases as it progresses from the input layer to the hidden layer. The neural circuit network in another embodiment of the present disclosure can be a neural circuit network that combines the above-described neural circuit networks.

[0082] A deep neural network (DNN) can be meant to refer to a neural network that includes a plurality of hidden layers in addition to an input layer and an output layer. By using a deep neural network, it is possible to grasp the latent structures of data. That is, it is possible to grasp the latent structures of photos, articles, videos, voices, music (for example, whether something is shown in a photo, what the content and sentiment of an article are, what the content and sentiment of a voice are, etc.). Deep neural networks can include convolutional neural networks (CNNs), recurrent neural networks (RNNs), auto encoders, GANs (Generative Adversarial Networks), restricted boltzmann machines (RBMs), deep belief networks (DBNs), Q networks, U networks, Sham networks, Generative Adversarial Networks (GANs), etc. The above-mentioned deep neural networks are merely illustrative and the present disclosure is not limited thereto.

[0083] In one embodiment of the present disclosure, the network function can also include an autoencoder. An autoencoder can be a type of artificial neural network circuit for outputting output data similar to the input data. The autoencoder can include at least one hidden layer, and an odd number of hidden layers can be arranged between the input and output layers. The number of nodes in each layer can decrease from the number of nodes in the input layer towards the intermediate layer called the bottleneck layer (encoding), and can also expand in a form symmetric to the reduction from the bottleneck layer towards the output layer (symmetric to the input layer). The autoencoder can perform non-linear dimensionality reduction. The number of input and output layers can correspond to the dimensions after preprocessing of the input data. In the autoencoder structure, the number of nodes in the hidden layers included in the encoder can have a structure that decreases as it gets farther from the input data. If the number of nodes in the bottleneck layer (the layer with the fewest number of nodes located between the encoder and the decoder) is too small, there is a possibility that a sufficient amount of information may not be transmitted, so it may be maintained at a certain number or more (for example, more than half of the input layer, etc.).

[0084] The neural network can be trained in at least one of supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. The learning of the neural network can be a process of providing the neural network with knowledge for the neural network to perform a specific operation.

[0085] A neural network can be trained in a direction to minimize the error of the output. In the training of a neural network, training data is repeatedly input into the neural network, the error between the output of the neural network regarding the training data and the target is calculated, and the error of the neural network is backpropagated from the output layer of the neural network to the input layer in the direction to reduce the error, and the weight values of each node of the neural network are updated. In the case of supervised learning, training data with the correct answer labeled for each individual training data is used (that is, labeled training data), and in the case of unsupervised learning, there may be cases where the correct answer is not labeled for each individual training data. That is, for example, the training data in supervised learning regarding data classification can be data where each category is labeled for each training data. By inputting the labeled training data into the neural network and comparing the output (category) of the neural network with the label of the training data, it is possible to calculate the error. As another example, in the case of unsupervised learning regarding data classification, it is possible to calculate the error by comparing the input training data with the output of the neural network. The calculated error is backpropagated in the reverse direction (that is, from the output layer to the input layer direction) in the neural network, and it is possible to update the connection weight values of each node of each layer of the neural network through backpropagation. The amount of change in the connection weight value of each node to be updated can be determined by the learning rate. The calculation of the neural network for the input data and the backpropagation of the error can constitute the learning cycle (epoch). The application method of the learning rate can change according to the number of repetitions of the learning cycle of the neural network. For example, in the initial stage of the training of the neural network, the learning rate can be increased to improve the efficiency by enabling the neural network to quickly ensure a certain level of performance, and in the latter half of the training, the learning rate can be decreased to increase the accuracy.

[0086] In the learning of neural networks, generally, the training data can be a subset of the actual data (i.e., the data to be processed using the learned neural network). Therefore, there may be a learning cycle in which the error related to the training data decreases, but the error related to the actual data increases. Overfitting is a phenomenon in which the error increases in the actual data because the training data is overlearned in this way. For example, a neural network that has learned cats by seeing yellow cats may not be able to recognize a cat when it sees a cat of a color other than yellow, which can be a type of overfitting. Overfitting can cause an increase in the error of machine learning algorithms. To prevent such overfitting, various optimization methods can be applied. To prevent overfitting, methods such as increasing the training data, regularization, dropout (deactivating some of the nodes of the network during the learning process), and utilizing a batch normalization layer can be applied.

[0087] FIG. 3 and FIG. 4 are block diagrams showing the operation process and structure of the detection module included in the computing device in one embodiment of the present disclosure.

[0088] Referring to FIG. 3, in one embodiment of the present disclosure, a computing device (100) can include a detection module (200) that extracts information related to suspicious nodules present in a medical image (11). The detection module (200) includes a preprocessing module (210) that generates an input image for the detection module (220) from a medical image (11) including a chest region transmitted to the computing device (100), a detection module (220) that derives information related to at least one region of interest based on the input image generated by the preprocessing module (210), and a postprocessing module (230) that derives position information related to suspicious nodules based on the information related to the region of interest derived through the detection module (220). In this case, the detection module (220) included in the detection module (200) can include at least one neural network. The detection module (220) can perform the above-described operations through a pre-trained neural network. The preprocessing module (210) and the postprocessing module (230) can also include at least one pre-trained neural network and can perform the above-described operations through the neural network.

[0089] Referring to FIG. 4, the preprocessing module (210) can receive an input of a CT image in which lung tissue, which is a three-dimensional medical image including a chest region, is photographed. The preprocessing module (210) can classify the input CT image in units of image groups based on the standard DICOM format. In this case, the classifier generally adopts values indicating the same series among the DICOM type-1 attributes, but actually, it may depend on the environment in which the computing device (100) is used. The preprocessing module (210) can classify the CT image in units of image groups by itself as described above, but can also receive an input of a CT image that has already been classified.

[0090] The preprocessing module (210) can calculate the value of the Hounsfield unit (HU) based on the image group classified by the classifier. For example, the preprocessing module (210) can sequentially calculate the value of the Hounsfield unit from the image group using the attribute values based on the standard DICOM format. In this case, the attribute values based on the standard DICOM format can be the Rescale Intercept Attribute (0028,1052) and the Rescale Slope Attribute (0028,1053). Instead of using the calculated value of the Hounsfield unit as it is, the preprocessing module (210) can clip it as a first range such as [-1000,600], and then linearly transform it into a second range such as [0,1]. Subsequently, the preprocessing module (210) can perform interpolation such as cubic B-spline interpolation, and convert the image so that the voxel spacing related to the three axes of the axial plane, coronal plane, and sagittal plane respectively satisfies predetermined numerical values. For example, the predetermined numerical values can be 1.0, 0.67, 0.67 for each axis respectively. However, the above-mentioned numerical values and descriptions are only examples and do not limit the interpretation of the present invention, and can be changed within the scope understandable by those skilled in the art.

[0091] The preprocessing module (210) can generate a plurality of two-dimensional images from a CT image in which the value of the Hounsfield unit is calculated. That is, since the detection module (220) includes a 2.5-dimensional neural network module that uses a two-dimensional image as an input, the preprocessing module (210) can perform an operation of processing the image into a form suitable for the input of the detection module (220). For example, the detection module (220) can include a 2.5-dimensional neural network module that uses an input of (axial, coronal, sagittal) = (7, 540, 540). However, the above-described numerical values and descriptions are merely examples and do not limit the interpretation of the present invention, and can be changed within a range understandable to those skilled in the art. In accordance with the size of the input of such a 2.5-dimensional neural network module, the preprocessing module (210) can process a three-dimensional CT image and generate a plurality of two-dimensional images. The plurality of two-dimensional images generated by the preprocessing module (210) can be used as an input to the first sub-detection module (221) of the detection module (220) described later.

[0092] Referring to FIG. 4, the detection module (220) can include a first sub-detection module (221) that generates a first probability value and first position information related to at least one region of interest based on the input image generated by the preprocessing module (210), and a second sub-detection module (222) that estimates a second probability value related to at least one region of interest based on the input image generated by the preprocessing module (210) and the first position information. In this case, the first probability value and the second probability value can be numerical values representing the probability that a suspicious nodule is included in each region of interest identified by each module (221, 222). For example, the CT image that has passed through the preprocessing module (210) can be input to the first sub-detection module (221) and the second sub-detection module (222) respectively, and can be used to extract information related to the region of interest determined to contain nodules. The first sub-detection module (221) can receive the input of the preprocessed CT image and calculate a first probability value and coordinate values related to at least one region of interest. In this case, the coordinate values calculated by the first sub-detection module (221) can be used as the input of the second sub-detection module (222). The second sub-detection module (222) can extract a three-dimensional patch from the preprocessed CT image based on the coordinate values calculated by the first sub-detection module (221). The second sub-detection module (222) can re-estimate the second probability value indicating that the region of interest identified by the first sub-detection module (221) actually contains nodules based on the three-dimensional patch. The specific operation process and structure of the first sub-detection module (221) and the second sub-detection module (222) will be described in detail later with reference to FIGS. 5 and 6.

[0093] The post - processing module (230) can derive information related to a suspicious nodule, which is the final output value of the detection module (200), based on the information related to the region of interest derived through the detection module (220). The post - processing module (230) can generate a third probability value related to the presence of a nodule through the weighted sum of the first probability value derived through the first sub - detection module (221) and the second probability value derived through the second sub - detection module (222). The post - processing module (230) can compare the third probability value with a threshold value and select a region corresponding to a suspicious nodule from among a plurality of regions of interest. In other words, the post - processing module (230) can determine the position information of the region of interest corresponding to the probability value selected as a result of comparing the third probability value with the threshold value as the position information of the suspicious nodule. Through the above - mentioned process, the post - processing module (230) can finally output second position information (15) including the central coordinate value of the suspicious nodule.

[0094] For example, assume that the first probability value of the first sub - detection module (221) is p1, the second probability value of the second sub - detection module (222) is p2, and the diameter of the region of interest identified by the first sub - detection module (221) is d. The post - processing module (230) can determine the p of the third probability value for each region of interest in the following manner using the above three input values.

[0095] (1) When p1 < 0.05, p = 0

[0096] (2) When d ≧ 7 [mm] and p1 < 0.97, p = 0

[0097] (3) When d ≦ 4 [mm] and p1 < 0.97, p = 0

[0098] (4) In other cases, p = 0.2×p1 + 0.8×p2

[0099] When the p of the third probability value is greater than or equal to a predetermined threshold value, the post-processing module (230) can use the central coordinate value of the region of interest corresponding to the probability value greater than or equal to the threshold value as the final output. The basic value of the predetermined threshold value can be 0.82, and it can be changed to 0.78 or 0.91 depending on the sensitivity. However, the above numerical values and descriptions are only examples and do not limit the interpretation of the present invention, and can be changed within the scope understandable to those skilled in the art.

[0100] FIG. 5 is a block configuration diagram showing the structure of the first sub-detection module included in the detection module in one embodiment of the present disclosure.

[0101] Referring to FIG. 5, in one embodiment of the present disclosure, the first sub-detection module (221) can include a first neural network module (240) that receives the input of a plurality of two-dimensional images generated from the pre-processing module (210) and generates a plurality of first feature maps having a plurality of sizes. The first sub-detection module (221) can include a second neural network module (250) that combines at least a part of the plurality of first feature maps based on the sizes of the plurality of first feature maps to generate a plurality of second feature maps. Further, the first sub-detection module (221) can include a third neural network module (260) that matches the plurality of second feature maps with a predetermined anchor box to generate a first probability value and first position information related to at least one region of interest.

[0102] For example, the first sub-detection module (221) can have the structure of a neural network that receives the input of a 2.5-dimensional image. As described above, the first sub-detection module (221) can receive an image in the form of [7, 540, 540] as an input and output a probability value and a coordinate value related to a series of regions of interest determined to include nodules. Referring to FIG. 5, the neural network of the first sub-detection module (221) can have a backbone-neck-head structure.

[0103] The backbone structure can include a first neural network module (240) in a form where a bottleneck block (241) including a skip-connection is repeated after a stem-cell block including a pooling layer. The bottleneck block (241) can expand or reduce the size of the feature map through a stride value. In the backbone structure, each of the first neural network modules (240) can generate a plurality of first feature maps having a plurality of sizes for detecting nodules of different sizes. Specifically, a plurality of first feature maps having sizes of [68,68], [68,68], [34,34], [34,34], [34,34] can be the output values of the first neural network module (240). The plurality of first feature maps generated by the first neural network module (240) can be used as the input to the neck structure. However, the numerical values related to the above sizes are only examples and do not limit the interpretation of the present invention, and can be changed within a range understandable to those skilled in the art.

[0104] If a series of processes for generating output values from the backbone structure is regarded as a process of encoding feature maps, the neck structure can be interpreted as a process of decoding feature maps into a form suitable for appropriately combining a plurality of feature maps generated from the backbone structure and executing the final detection process. The neck structure can include a second neural network module (250) that combines a plurality of first feature maps generated by the first neural network module (240) of the Backbone structure according to size to generate a plurality of second feature maps. The second neural network module (250) can include at least one intermediate block (251) that combines a plurality of first feature maps with similar sizes to generate a plurality of second feature maps at the next level. Also, the second neural network module (250) can combine all the first feature maps given as input through at least one intermediate block (251) to generate a plurality of second feature maps. Specifically, one of the second neural network modules (250) can combine a plurality of first feature maps corresponding to a size of [68, 68] to generate a plurality of second feature maps. Another one of the second neural network modules (25) can combine a plurality of first feature maps with a size of [34, 34] to generate a plurality of second feature maps. Also, the second neural network module (250) can combine all the first feature maps to generate a plurality of second feature maps. The plurality of second feature maps generated by the second neural network module (250) can be used as the input of the head structure.

[0105] The head structure can include a third neural network module (260) including a plurality of detection blocks (261) that individually receive the inputs of a plurality of second feature maps generated by the second neural network module (250) of the neck structure. That is, in the head structure, the third neural network module (260) can generate the presence probability value and position information of the nodules in the region of interest corresponding to the output value of the first sub-detection module (221) based on the output of the second neural network module (250). Specifically, the detection block (261) of the third neural network module (260) matches the second feature map to a pre-defined anchor box, and outputs the probability value that the region of interest corresponding to the second feature map contains nodules, and the size and position information of the nodules existing in the region of interest corresponding to the offset between the anchor box and the actual output value.

[0106] On the other hand, although not shown in FIG. 5, when there are a plurality of regions of interest identified through the third neural circuit network module (260), the first sub-detection module (221) can cluster a plurality of regions of interest determined to contain the same nodule and integrate them into one nodule. Further, the first sub-detection module (221) can perform a correction operation on the first position information of the plurality of regions of interest through the third neural circuit network module (260). For example, the first sub-detection module (221) can project any plurality of regions of interest onto the same plane and cluster at least a part of the plurality of regions of interest based on the ratio of the overlapping regions between the plurality of regions of interest. Specifically, the first sub-detection module (221) calculates the IOU (intersection over union) between any plurality of regions of interest, and if the calculated value exceeds a threshold value that is inversely proportional to the Euclidean distance between the plurality of regions of interest, it can be determined that the same nodule is included in the plurality of regions of interest and clustering can be performed. In the case of the clustered regions of interest, the first sub-detection module (221) can calculate the position information and the diameter value of the region with the highest probability value of the nodule presence as its representative values. The first sub-detection module (221) can convert the plurality of first position information of the plurality of regions of interest obtained through various inputs into the same coordinate system and finally generate the input value of the second sub-detection module (222).

[0107] FIG. 6 is a block configuration diagram showing the structure of the second sub-detection module included in the detection module in one embodiment of the present disclosure.

[0108] Referring to FIG. 6, in one embodiment of the present disclosure, the second sub-detection module (222) can include a fourth neural network module (270) that performs encoding based on a 3D patch extracted from an image output from the preprocessing module (210) with reference to the first position information to generate at least one third feature map. In this case, the second sub-detection module (222) can extract, from the preprocessed 3D image, a region corresponding to the first position information as a patch having a predetermined size in order to generate an input patch of the fourth neural network module (270). The second sub-detection module (222) can include a fifth neural network module (280) that generates at least one fourth feature map by executing decoding based on the third feature map. Although not shown in FIG. 6, the second sub-detection module (222) can include a sixth neural network module that generates a second probability value related to at least one region of interest based on a feature map generated by integrating the third feature map and the fourth feature map.

[0109] For example, the second sub-detection module (222) uses the central coordinate value of the region of interest given as the output value of the first sub-detection module (221), and has as its input value a three-dimensional patch extracted from the image processed by the preprocessing module (210). Specifically, the size of the three-dimensional patch can be [72, 72, 72]. The second sub-detection module (222) can receive the input of the three-dimensional patch and output one real value indicating whether or not the patch contains nodules. The second sub-detection module (222) can include a neural network with an encoder-decoder structure as shown in FIG. 6. The second sub-detection module (222) can include a fourth neural network module (270) including one or more encoder blocks (271) for adjusting the size of the three-dimensional patch, and a fifth neural network module (280) including one or more decoder blocks (281). The three-dimensional patch that has passed through the fourth neural network module (270) can be compressed to a size of [3, 3, 3]. The compressed patch can be restored to a size of [18, 18, 18] while passing through the fifth neural network module (280). In such a restoration process, a combination (element-wise sum) with the third feature map of the fourth neural network module (270) having the same size can be performed, and a more complex fourth feature map can be generated. The second sub-detection module (222) can output a second probability value related to the presence of nodules in the region of interest based on the fourth feature map using a sixth neural network module that performs three-dimensional convolution.

[0110] On the one hand, the second sub-detection module (222) can be learned by performing a first operation of training a neural network based on randomly sampled training images and a second operation of training the neural network based on training images selected based on recall and precision. For example, the second sub-detection module (222) can receive the input of a plurality of three-dimensional patches for random sampling training and perform a primary training of the neural network with three-dimensional features. When the training is completed based on a plurality of randomly sampled three-dimensional patches for training, the second sub-detection module (222) can perform a secondary training of the neural network based on a plurality of three-dimensional patches for training that are relatively difficult to predict for the plurality of three-dimensional patches for primary training. In this case, the plurality of three-dimensional patches for training that are relatively difficult to predict can be a plurality of three-dimensional patches with a higher recall rate and a lower precision rate compared to the plurality of three-dimensional patches for primary training. Through such a kind of curriculum learning, it is possible to significantly improve the probability value estimation function of the second sub-detection module (222).

[0111] FIG. 7 is a flowchart showing the process of the operation of the detection module in one embodiment of the present disclosure.

[0112] Referring to FIG. 7, in step S110, a computing device (100) according to an embodiment of the present disclosure can receive a medical image for lesion analysis from a medical image management system. The medical image for lesion analysis can also be a three-dimensional CT image including a chest region. When the computing device (100) receives a three-dimensional CT image as a medical image, the computing device (100) can generate an input image for the detection module by processing the medical image using a preprocessing module. In this case, the preprocessing module can perform operations such as extraction of the value of the Hounsfield unit from the medical image, adjustment of the size through linear transformation and padding of the medical image, and generation of a two-dimensional image through segmentation of the medical image.

[0113] In step S120, the computing device (100) can generate a probability value related to the presence of nodules in at least one region of interest and first position information of the region of interest based on the image processed through the preprocessing module using the detection module. In this case, the region of interest can be understood as a region where nodules are predicted to exist. Therefore, the first position information of the region of interest can be understood as a group of candidates for the position information for the suspicious nodules that will be finally determined in step S130 described later.

[0114] In step S130, the computing device (100) can determine second position information related to the suspicious nodules present in the medical image based on the output value of the detection module using the postprocessing module. The computing device (100) can determine the second position information related to the suspicious nodules present in the medical image from the first position information based on the probability value related to the presence of nodules using the postprocessing module. For example, when there are a plurality of regions of interest, the computing device (100) can compare the probability values of the plurality of regions of interest with a predefined threshold value. The computing device (100) can determine the region of interest corresponding to the probability value equal to or greater than the predefined threshold value as a suspicious nodule. In other words, the computing device (100) can determine the position information (i.e., the first position information) of the region of interest corresponding to the probability value equal to or greater than the threshold value as the position information (i.e., the second position information) related to the suspicious nodule. Through such a process, the computing device (100) can accurately detect the nodules present in the medical image.

[0115] FIG. 8 is a block configuration diagram showing the process of reading and evaluating the lesions of the computing device in an embodiment of the present disclosure.

[0116] Referring to FIG. 8, in one embodiment of the present disclosure, a processor (110) of a computing device (100) can cause a detection module (200) to input a medical image (21) including at least one lung region, and generate position information (22) of a suspected nodule present in the lung region. Based on the position information (22) of the suspected nodule, the processor (110) can cause an evaluation module (300) to input a three-dimensional patch (23) extracted from the medical image (21), and generate a plurality of masks (24, 25). In this case, the first mask (24) can be a mask including information related to the entire region of the suspected nodule. The second mask (25) can be a mask including information related to a region in which the suspected nodule represents a specific attribute (e.g., solid, etc.) among the entire region of the suspected nodule. The processor (110) can cause the three-dimensional patch (23) and the plurality of masks (24, 25) to be input to a classification module (400), and generate class information (27) representing the type of attribute of the suspected nodule, the presence or absence of spiculation, the presence or absence of calcification, etc.

[0117] On the one hand, the processor (110) can generate numerical information (26) including at least one of the diameter and volume of a suspicious nodule based on a plurality of masks (24, 25) related to the suspicious nodule. In this case, the numerical information (26) includes the first numerical information generated based on the first mask (24), and when the suspicious nodule is classified into a specific class, it can further include the second numerical information generated based on the second mask (25). The first numerical information can represent at least one of the diameter and volume related to the entire region of the suspicious nodule existing in the medical image (21). The second numerical information can include a numerical value representing at least one of the diameter and volume related to the region representing a specific attribute of the suspicious nodule existing in the medical image (21). The processor (110) can calculate structural numerical values related to the shape, size, etc. of the region corresponding to the suspicious nodule in the three-dimensional patch (23) based on the information included in the first mask (24). However, when the class related to the state of the suspicious nodule corresponds to a predetermined type related to a specific attribute of the suspicious nodule (e.g., part-solid), the processor (110) can calculate the aforementioned numerical values based on the information included in the second mask (25) together with the information included in the first mask (24).

[0118] The processor (110) can calculate an evaluation score (28) of a suspicious nodule based on numerical information (26) and class information (27) based on an auxiliary index (30) for diagnosing lung diseases. For example, the processor (110) examines the numerical information (26) and class information (27) of a suspicious nodule detected from a medical image (21) based on an auxiliary index (30) for diagnosing lung cancer, and can evaluate the suspicious nodule with one of a plurality of scores defined by the auxiliary index (30). Specifically, the processor (110) uses numerical values related to at least one of the total area of the suspicious nodule and the diameter, volume, etc. of the area representing the solid attribute included in the numerical information (26), and information related to the type, presence or absence of spicules, and presence or absence of calcification related to the solid attribute included in the class information (27), and can determine which of a plurality of categories in the Lung-RADS classification the suspicious nodule belongs to. Based on the class information (27), when the type related to the solid attribute of the suspicious nodule is solid or non-solid, the processor (110) can use the first numerical information included in the numerical information (26) to determine which of a plurality of categories in the Lung-RADS classification the suspicious nodule belongs to. Based on the class information (27), when the type related to the solid attribute of the suspicious nodule is partially solid, the processor (110) can use both the first numerical information and the second numerical information included in the numerical information (26) to determine which of a plurality of categories in the Lung-RADS classification the suspicious nodule belongs to. The processor (110) can determine one of a plurality of categories in the Lung-RADS classification as the evaluation score (28) of the suspicious nodule based on the result of such determination.

[0119] The processor (110) can predict the degree to which a suspicious nodule suspected to be the cause of lung disease affects the lungs based on a plurality of pieces of information related to the suspicious nodule output through the detection module (200), the evaluation module (300), and the classification module (400). The processor (110) can estimate the malignancy (29) of a suspicious nodule based on the position information (22), numerical information (26), and class information (27) of the suspicious nodule using a pre-trained malignancy prediction module (600). For example, the processor (110) inputs information related to the central coordinate value of the nodule included in the position information (22), the value of the size of the nodule included in the numerical information (26), the type of solid attribute included in the class information (27), the presence or absence of spiculation, the presence or absence of calcification, etc. into the malignancy prediction module (600) to calculate the malignancy (29) of the suspicious nodule.

[0120] Although not shown in FIG. 8, the processor (110) can also estimate the malignancy (29) of a suspicious nodule based on the 3D patch (23) extracted from the medical image (21) and the masks (24, 25) generated by the evaluation module (300) using a pre-trained malignancy prediction module (600). That is, the processor (110) can directly input a plurality of quantitative information (22, 26, 27) related to the suspicious nodule extracted from the medical image (21) into the malignancy prediction module (600) to estimate the malignancy (29), and can also input a plurality of image information (23, 24, 25) generated by processing the medical image (21) into the malignancy prediction module (600) to estimate the malignancy (29).

[0121] FIG. 9 is a block configuration diagram showing a process of correcting an evaluation result of a lesion of a computing device in an embodiment of the present disclosure.

[0122] Referring to FIG. 9, the processor (110) of the computing device (100) in one embodiment of the present disclosure can modify the evaluation score for a suspicious nodule based on a plurality of medical images taken of the same subject over time. The processor (110) can modify the evaluation score for a suspicious nodule based on each imaging time point of a plurality of medical images taken of the same subject over time. When a medical image is sequentially input to the computing device (100), the processor (110) can determine whether there is a history of an image targeting the same subject as the medical image (41) being input and analyzed. If it is determined that there is no history related to an image targeting the same subject as the medical image (41), the processor (110) recognizes the medical image (41) as a medical image related to a new subject, and through step A (50), it is possible to calculate an evaluation score (45) related to a suspicious nodule. In this case, step A (50) can be regarded as corresponding to the calculation process of the evaluation score (28) shown in FIG. 8.

[0123] When it is determined that there is an analyzed image of the same subject as the medical image (41), the processor (110) can perform registration between the analyzed image and the medical image (41) using a pre-trained tracking module (500). Here, registration means an operation of aligning the relative positional relationship between the analyzed image with a time difference and the medical image (41). The processor (110) can use the tracking module (500) to match a suspicious nodule existing in the analyzed image after registration with a suspicious nodule existing in the medical image (41). Although not shown in FIG. 9, the processor (110) can perform step A (50) on the medical image (41) and identify change information between the matched suspicious nodules. In this case, step A (50) can be regarded as corresponding to the calculation process of the evaluation score (28) shown in FIG. 8. The processor (110) can modify the evaluation score of the medical image (41) or the evaluation score of the analyzed image based on the change information. When the medical image (41) is taken before the time when the analyzed image is taken, the processor (110) can modify the evaluation score of the analyzed image based on the change information. Conversely, when the medical image (41) is taken after the time when the analyzed image is taken, the processor (110) can modify the evaluation score of the medical image based on the change information. That is, the processor (110) can compare the imaging times of the medical image (41) and the analyzed image and modify the evaluation score of the image taken relatively recently. That is, the processor (110) can modify the evaluation score for the image taken relatively recently in order to effectively track the temporal change of the suspicious nodule. Through such a process, the processor (110) can finally generate a corrected evaluation score (49) for the suspicious nodule for a specific subject. Such an operation of modifying the evaluation score can be repeatedly executed every time the medical image (41) is input to the computing device (100).

[0124] FIG. 10 is a flowchart relating to a method for analyzing a lesion in a medical image according to an embodiment of the present disclosure.

[0125] Referring to FIG. 10, at step S210, when a medical image including a chest region is input, a computing device (100) according to an embodiment of the present disclosure can determine whether there is an image having the same identifier as the input image among the analyzed images. In this case, the identifier refers to identification information for the imaging object of the image. For example, the computing device (100) can determine whether the subject of the input CT image corresponds to the subject of the analyzed CT image. If the subject of the input CT image corresponds to the subject of the analyzed CT image, the computing device (100) can perform a series of operations to correct the calculated and stored evaluation score of the suspicious nodule. If the subject of the input CT image does not correspond to the subject of the analyzed CT image, the computing device (100) can consider that a medical image related to a new subject has been input, and based on the input CT image, perform a series of operations to read and evaluate the suspicious nodule.

[0126] The following describes approximately the process for correcting the evaluation score of a suspicious nodule that is executed when it is determined that the subject of the input image corresponds to the subject of the existing image.

[0127] At step S221, the computing device (100) can obtain the central position information of the suspicious nodule existing in the lung tissue based on the input image. For example, the computing device (100) can generate information related to at least one candidate region of the suspicious nodule based on the input image using a pre-trained detection module. The computing device (100) can generate the central position information of the suspicious nodule based on the information related to the candidate region using the detection module.

[0128] At the stage of S222, the computing device (100) can extract an image patch based on the central position information of a suspicious nodule from the input image. The computing device (100) can generate a mask of the suspicious nodule based on the image patch extracted from the input image. The computing device (100) can generate numerical information including the structural numerical values of the suspicious nodule based on the mask of the suspicious nodule. For example, the computing device (100) can generate a plurality of masks related to the suspicious nodule based on the image patch using a pre-trained evaluation module. The computing device (100) can calculate numerical values related to the diameter, volume, etc. of the suspicious nodule based on the information contained in the plurality of masks and generate numerical information. In this case, the numerical information can include at least one of first numerical information including structural information related to the entire region of the suspicious nodule and second numerical information including structural information related to a region representing a specific attribute (e.g., solid, etc.) of the suspicious nodule.

[0129] At stage S223, the computing device (100) can perform class classification related to the state of a suspicious nodule based on the image patch and mask generated at stage S222. The classes related to the state of a suspicious nodule can include a first class representing the type related to the solid attribute of the suspicious nodule, a second class representing the presence or absence of spicules of the suspicious nodule, or a third class indicating the presence or absence of calcification of the suspicious nodule. The computing device (100) can classify the state of the suspicious nodule into at least one of the first class, the second class, and the third class. For example, the computing device (100) can classify the type related to the solid attribute of the suspicious nodule as solid, partially solid, or non-solid based on the image patch and multiple masks using the first sub-classification module of the classification module. The computing device (100) can classify the suspicious nodule as spiculated or non-spiculated based on the image patch and multiple masks using the second sub-classification module of the classification module. The computing device (100) can classify the suspicious nodule as calcified or non-calcified based on the image patch and multiple masks using the third sub-classification module of the classification module.

[0130] At stage S224, the computing device (100) can calculate the evaluation score and malignancy of the suspicious nodule based on the central position information of the suspicious nodule generated at stage S221, the numerical information generated at stage S222, and the class information generated at stage S223. For example, the computing device (100) can calculate the evaluation score of the suspicious nodule by performing an operation on the numerical information and class information in accordance with the criteria defined in a predetermined diagnostic assistance index. The computing device (100) can estimate the malignancy of the suspicious nodule based on the central position information, numerical information, and class information of the suspicious nodule using the malignancy prediction module. Also, the computing device (100) can estimate the malignancy of the suspicious nodule based on the image patch and mask generated at stage S222 using the malignancy prediction module.

[0131] In stage S225, the computing device (100) can perform alignment to match the relative positions of the input image and the existing image. The computing device (100) can match a suspicious nodule in the input image read in the previous stage with a suspicious nodule in the existing image that has been read and saved, and can grasp the change of the suspicious nodule. For example, the computing device (100) can perform alignment between the input image and the existing image using a pre-trained tracking module. The computing device (100) can match at least one suspicious nodule existing in each of the two images for which the alignment has been completed, and can determine whether a change has occurred between the matched suspicious nodules.

[0132] In step S226, when it is determined that a change has occurred in a suspicious nodule matched between the input image and the existing image, the computing device (100) can calculate a corrected evaluation score by reflecting the evaluation score of the suspicious nodule derived from the input image or the evaluation score of the suspicious nodule derived from the existing image in the mutual evaluation score. In this case, it is possible to determine the image for which the evaluation score is corrected based on the imaging time point of the image. For example, when the input image is an image of a specific subject taken in 2009 and the existing image is an image of the specific subject taken in 2015, the computing device (100) can also correct the evaluation score of the existing image by reflecting the evaluation score of the input image in the evaluation score of the existing image. Conversely, when the input image is an image of a specific subject taken in 2015 and the existing image is an image of the specific subject taken in 2009, the computing device (100) can also correct the evaluation score of the input image by reflecting the evaluation score of the existing image in the evaluation score of the input image. When it is determined that there is no change in the matched suspicious nodule between the input image and the existing image, the computing device (100) can maintain the existing evaluation score without correction.

[0133] On the other hand, steps S231 to S234 related to the process of reading and evaluating suspicious nodules performed when it is determined that the subjects in the input image and the existing image do not correspond are corresponding to the above-mentioned steps S221 to S224, so specific descriptions are omitted.

[0134] FIG. 11 is a flowchart related to a method for analyzing a lesion in a medical image in an alternative embodiment of the present disclosure.

[0135] Referring to FIG. 11, when a medical image including a chest region is input, the computing device (100) in an alternative embodiment of the present disclosure can preferentially perform radiological reading and evaluation on suspicious nodules. Different from FIG. 10, in FIG. 11, after performing radiological reading and evaluation on suspicious nodules, it is determined whether the input image is an image of the same subject as the existing image. That is, it can be understood that there are differences in the context of the determination for correcting the evaluation score of suspicious nodules between the method shown in FIG. 10 and the method shown in FIG. 11. Therefore, regarding the detailed content of each step (steps S310 to S370) in FIG. 11, the corresponding content in FIG. 10 will be omitted from the description.

[0136] FIGS. 12 and 13 are block diagrams showing the operation process and structure of an evaluation module included in a computing device in an embodiment of the present disclosure.

[0137] Referring to FIG. 12, the computing device (100) in an embodiment of the present disclosure can include an evaluation module (300) that generates a mask (65) of an area suspected to be a nodule from an input patch (61) of a medical image. The evaluation module (300) can receive the input of the patch (61) generated from a medical image including a chest region and generate a mask (65) related to a suspicious nodule. In this case, the computing device (100) can extract the input patch (61) of the evaluation module (300) from a medical image including a chest region based on the position information of the suspicious nodule. That is, the input patch (61) can be a predetermined image unit including the area corresponding to the position information of the suspicious nodule. The computing device (100) can generate the position information of the suspicious nodule used for generating the input patch (61) by using the detection module (200) described through FIG. 3. The computing device (100) can also receive the position information itself of the suspicious nodule used for generating the input patch (61) through a user terminal, an external medical information system, etc.

[0138] The evaluation module (300) can include a first sub-evaluation module (310) that generates a first mask related to the entire region of a suspicious nodule based on at least one input patch (61), and a second sub-evaluation module (320) that generates a second mask related to a region representing a specific attribute of the suspicious nodule based on at least one input patch (61). The first sub-evaluation module (310) and the second sub-evaluation module (320) can perform the above-described operations through a pre-trained neural network. In this case, the neural network of each module (310, 320) can include a convolutional neural network capable of performing segmentation regardless of the size related to the input image.

[0139] Referring to FIG. 13, the first sub-evaluation module (310) can receive an input of a three-dimensional patch generated based on the position information of a suspicious nodule and generate a first mask representing the entire region of the suspicious nodule. For example, the first sub-evaluation module (310) can receive an input of a three-dimensional patch generated based on the position information of a suspicious nodule generated by the detection module (200) in FIG. 3 and generate a first mask representing the region suspected of being a nodule in the patch. The first sub-evaluation module (310) can also receive an input of a three-dimensional patch generated based on the position information of a suspicious nodule received from an external medical information system and generate a first mask representing the region suspected of being a nodule in the patch. In this case, the size of the three-dimensional patch can be [32, 32, 32], but is not limited thereto.

[0140] In addition, the first sub-evaluation module (310) can include a neural network having a fully convolutional network (FCN) structure. Therefore, the first sub-evaluation module (310) can receive an input patch regardless of the size of the input. That is, the first sub-evaluation module (310) can receive inputs of input patches of various sizes related to one suspicious nodule, combine various outputs, and generate a first mask related to the suspicious nodule. For example, when a plurality of input patches having various sizes related to one suspicious nodule are input to the evaluation module (300), the first sub-evaluation module (310) can generate a plurality of first sub-masks related to one suspicious nodule from each of the plurality of input patches. The first sub-evaluation module (310) can combine the plurality of first sub-masks and generate a first mask related to the entire region of one suspicious nodule based on the result of the combination. In this case, various ensemble algorithms can be applied to the combination method.

[0141] The second sub-evaluation module (320) can receive an input of a three-dimensional patch generated based on the position information of a suspicious nodule and generate a second mask related to the region representing the solid component of the suspicious nodule. For example, the second sub-evaluation module (320) can receive an input of a three-dimensional patch generated based on the position information of a suspicious nodule generated by the detection module (200) in FIG. 3 and generate a second mask related to the region representing the solid component in the region suspected of being a nodule in the patch. The first sub-evaluation module (310) can also receive an input of a three-dimensional patch generated based on the position information of a suspicious nodule received from an external medical information system and generate a second mask related to the region representing the solid component in the region suspected of being a nodule in the patch. In this case, the size of the three-dimensional patch can be [32, 32, 32], but is not limited thereto.

[0142] Referring to FIG. 13, since the second mask related to the region representing the solid element of the suspicious nodule is included in the first mask related to the entire region of the suspicious nodule, the second sub - evaluation module (320) can utilize the output result of the first sub - evaluation module (310) to generate the second mask. For example, the second sub - evaluation module (320) can generate a second sub - mask related to the candidate region representing the solid element of the suspicious nodule based on at least one input patch. The second sub - evaluation module (320) can identify the overlapping region between the first mask generated by the first sub - evaluation module (310) and the second sub - mask. The second sub - evaluation module (320) can generate a second mask representing the solid element of the suspicious nodule based on the identified region. In this case, the neural network structure for generating the second mask of the second sub - evaluation module (320) can correspond to the neural network structure of the aforementioned first sub - evaluation module (310).

[0143] On the other hand, the evaluation module (300) can execute the learning of the neural network by automatically selecting relatively difficult - to - predict learning data using OHEM (online hard example mining). By being learned through OHEM, the evaluation module (300) can improve the performance of identifying and extracting suspicious nodules for mask generation.

[0144] FIG. 14 is a flowchart showing the operation process of the evaluation module in an embodiment of the present disclosure.

[0145] Referring to FIG. 14, in step S410, a computing device (100) according to an embodiment of the present disclosure can receive a medical image for lesion analysis from a medical image management system. The medical image for lesion analysis can also be a three-dimensional CT image including a chest region. The computing device (100) can use a detection module to extract position information of suspicious nodules from the medical image through a process similar to that in FIG. 7. The computing device (100) can generate an input patch for an evaluation module from the medical image based on the position information of the suspicious nodules. In other words, the computing device (100) can identify the position information of the suspicious nodules through an analysis of the three-dimensional CT image and extract a three-dimensional patch of a predetermined size including the nodules and the regions suspected to be nodules from the three-dimensional CT image. On the other hand, as described above, the computing device (100) can directly extract the position information of the suspicious nodules using the detection module, but it can also receive and use the position information of the suspicious nodules through an external system.

[0146] In step S420, the computing device (100) can use a first sub-evaluation module to generate a first mask related to the entire region of the suspicious nodules based on the input patch. For example, the computing device (100) can input the three-dimensional patch generated in step S410 into the first sub-evaluation module. The computing device (100) can generate a first mask representing all the regions determined to be suspicious nodules in the patch through the first sub-evaluation module that has received the input of the three-dimensional patch. In this case, the first mask can be understood as an aggregate of data including meta-information such as the position and size related to the entire region of the suspicious nodules.

[0147] In step S430, the computing device (100) can generate a second sub-mask related to a candidate region representing a specific attribute of a suspicious nodule based on the input patch using the second sub-evaluation module. For example, the computing device (100) can input the three-dimensional patch generated in step S410 into the second sub-evaluation module. The computing device (100) can generate a second sub-mask related to a candidate region representing a solid attribute among all regions determined to be suspicious nodules in the patch through the second sub-evaluation module that has received the input of the three-dimensional patch.

[0148] In step S440, the computing device (100) can generate a second mask related to a region representing a specific attribute of a suspicious nodule based on the overlapping region between the first mask generated in step S420 and the second sub-mask generated in step S430 using the second sub-evaluation module. For example, the computing device (100) can identify the overlapping region between the first sub-mask and the second sub-mask through the second sub-evaluation module. The computing device (100) can finally determine that the overlapping region is a region representing the solid attribute of the suspicious nodule, and can generate a second mask representing the overlapping region through the second sub-evaluation module. In this case, the second mask can be understood as an aggregate of data including meta-information such as the position and size of the region representing the solid attribute of the suspicious nodule.

[0149] FIG. 15 and FIG. 16 are block diagrams showing the operation process and structure of the classification module included in the computing device in an embodiment of the present disclosure.

[0150] Referring to FIG. 15, a computing device (100) in an embodiment of the present disclosure can include a classification module (400) that generates class information (79) related to the state of a region suspected of being a nodule based on an input patch (71) of a medical image and a mask (75) related to the suspected nodule. The classification module (400) can receive an input of the mask (75) representing the suspected nodule together with the patch (71) generated from the medical image including the chest region, and classify the class related to the state of the suspected nodule. In this case, the computing device (100) can extract the input patch (71) of the classification module (400) from the medical image including the chest region based on the position information of the suspected nodule. That is, the input patch (71) can be a predetermined image unit including the region corresponding to the position information of the suspected nodule. The computing device (100) can generate the position information of the suspected nodule used for generating the input patch (71) by using the detection module (200) described with reference to FIG. 3. The computing device (100) can also receive the position information itself of the suspected nodule used for generating the input patch (71) through a user terminal, an external medical information system, or the like. Further, the computing device (100) can generate a mask (75) including at least one of the entire region of the suspected nodule and the region representing a specific attribute from the input patch (71) by using the evaluation module (300) described with reference to FIG. 12.

[0151] The classification module (400) can include a first sub-classification module (410) that determines the type related to the attributes of the suspicious nodule based on the input patch (71) generated from the medical image and the mask (75) related to the suspicious nodule. The classification module (400) can include a second sub-classification module (420) that determines the presence or absence of spicules of the suspicious nodule based on the input patch (71) and the mask (75). Also, the classification module (400) can include a third sub-classification module (430) that determines the presence or absence of calcification of the suspicious nodule based on the input patch (71) and the mask (75). The classification module (400) can determine at least one of the type related to the attributes of the suspicious nodule, the presence or absence of spicules of the suspicious nodule, and the presence or absence of calcification of the suspicious nodule through the plurality of different sub-classification modules (410, 420, 430) described above, and output class information (79). In this case, the second sub-classification module (420) can execute the above-described operation through a pre-trained neural network.

[0152] Referring to FIG. 16, the first sub-classification module (410) receives an input of a three-dimensional patch generated based on the position information of a suspicious nodule and a mask of the suspicious nodule generated from the three-dimensional patch, and can classify the type related to the attribute of the suspicious nodule. In this case, the type related to the attribute of the suspicious nodule can include solid, partially solid, or non-solid. That is, the first sub-classification module (410) can determine how much solid elements the suspicious nodule identified from the input image contains. For example, the first sub-classification module (410) can classify the type related to the attribute of the suspicious nodule into solid, partially solid, or non-solid based on the three-dimensional patch and the mask using a deep learning algorithm. Also, the first sub-classification module (410) can classify the type related to the attribute of the suspicious nodule into solid or non-solid based on the three-dimensional patch and the mask according to a predetermined rule. The first sub-classification module (410) can finally determine the type related to the attribute of the suspicious nodule by integrating the classification result based on the deep learning algorithm and the classification result based on the rule.

[0153] The second sub-classification module (420) receives an input of a three-dimensional patch generated based on the position information of a suspicious nodule and a mask of the suspicious nodule generated from the three-dimensional patch based on a pre-trained neural network, and can determine the presence or absence of spicules of the suspicious nodule. The neural network included in the second sub-classification module (420) can include a convolutional neural network trained with three-dimensional features. When using the three-dimensional patch and the mask related to the suspicious nodule generated therefrom together as in the second sub-classification module (420), it is possible to improve the classification performance related to the attribute of the suspicious nodule compared to using only the three-dimensional patch. The information related to the spicules of the suspicious nodule generated by the second sub-classification module (420) can be used for the evaluation of the suspicious nodule (e.g., calculation of Lung-RADS score, calculation of malignancy, etc.).

[0154] The third sub-classification module (430) can receive an input of a three-dimensional patch generated based on the position information of a suspicious nodule and a mask of the suspicious nodule generated from the three-dimensional patch, and can determine the presence or absence of calcification of the suspicious nodule. Specifically, the third sub-classification module (430) can calculate, from the three-dimensional patch, the ratio of a plurality of voxels whose values of the Hounsfield unit of the plurality of voxels included in the mask are higher than the value of a predetermined Hounsfield unit. The third sub-classification module (430) can compare the aforementioned ratio with a threshold value, and can determine the presence or absence of calcification of the suspicious nodule present in the patch based on the result of the comparison. For example, the third sub-classification module (430) can calculate, based on the three-dimensional patch, the number of voxels having a value equal to or higher than the value of a predetermined Hounsfield unit with respect to the total number of voxels included in the mask. If the ratio of the voxels calculated based on the value of the Hounsfield unit is higher than a specific threshold value, the third sub-classification module (430) can determine that calcification has progressed in the suspicious nodule included in the three-dimensional patch. If the ratio of the voxels calculated based on the value of the Hounsfield unit is lower than a specific threshold value, the third sub-classification module (430) can determine that calcification has not progressed in the suspicious nodule included in the three-dimensional patch. The calcification information of the suspicious nodule generated by the third sub-classification module (430) can be used for the evaluation of the suspicious nodule (e.g., calculation of the Lung-RADS score, calculation of malignancy, etc.).

[0155] On the other hand, the third sub-classification module (430) can also determine the presence or absence of calcification of the suspicious nodule based on a pre-trained neural network. For example, the third sub-classification module (430) can receive an input of the three-dimensional patch and the mask, and can determine the progress of calcification of the suspicious nodule through the pre-trained neural network. Also, the third sub-classification module (430) can receive an input of the three-dimensional patch and the mask, and can classify the suspicious nodule into calcified or non-calcified through the pre-trained neural network.

[0156] FIG. 17 is a block configuration diagram showing the structure of a first sub-classification module included in a classification module in one embodiment of the present disclosure.

[0157] Referring to FIG. 17, the first sub-classification module (410) can receive an input of a three-dimensional patch generated based on the position information of a suspicious nodule and a mask of the suspicious nodule generated from the three-dimensional patch, and include a first attribute classification module (411) that determines a first type related to the solid attribute of the suspicious nodule. The first sub-classification module (410) can receive an input of the three-dimensional patch and the mask of the suspicious nodule, and include a second attribute classification module (412) that determines a second type related to the solid attribute of the suspicious nodule. Further, the first sub-classification module (410) can include a third attribute classification module (413) that finally determines the type related to the solid attribute of the suspicious nodule based on the result of comparing the outputs of the first attribute classification module (411) and the second attribute classification module (412). In this case, the first attribute classification module (411) can perform the above-described operation through a pre-trained neural network.

[0158] The first attribute classification module (411) can receive an input of a three-dimensional patch generated based on the position information of a suspicious nodule and a mask of the suspicious nodule generated from the three-dimensional patch based on a pre-trained neural network, and determine a first type related to the solid attribute of the suspicious nodule. In this case, the first type can include one of solid, partially solid, and non-solid. The neural network included in the first attribute classification module (411) can include a convolutional neural network learned with three-dimensional features. When using the three-dimensional patch and the mask related to the suspicious nodule generated based thereon together with the first attribute classification module (411), it is possible to improve the classification performance related to the solid attribute of the suspicious nodule compared to using only the three-dimensional patch. The first type of the suspicious nodule generated by the first attribute classification module (411) can be used to finally determine the type related to the solid attribute of the suspicious nodule.

[0159] The second attribute classification module (412) can receive an input of a three-dimensional patch generated based on the position information of a suspicious nodule and a mask of the suspicious nodule generated from the three-dimensional patch, and determine a second type related to the solid attribute of the suspicious nodule. In this case, the second type can include one of solid and non-solid. Specifically, the second attribute classification module (412) can calculate, from the three-dimensional patch, the ratio of a plurality of voxels whose Hounsfield unit value of the plurality of voxels included in the mask is higher than a predetermined Hounsfield unit value. The second attribute classification module (412) can compare the aforementioned ratio with a threshold value and determine the second type of the suspicious nodule existing in the patch based on the result of the comparison. For example, the second attribute classification module (412) can calculate, based on the three-dimensional patch, the number of voxels having a value equal to or higher than a predetermined Hounsfield unit value with respect to the total number of voxels included in the mask. If the ratio of the voxels calculated based on the Hounsfield unit value is higher than a specific threshold value, the second attribute classification module (412) can determine that the suspicious nodule included in the three-dimensional patch corresponds to a solid. If the ratio of the voxels calculated based on the Hounsfield unit value is lower than a specific threshold value, the second attribute classification module (412) can determine that the suspicious nodule included in the three-dimensional patch corresponds to a non-solid. The second type of the suspicious nodule generated by the second attribute classification module (412) can be used to finally determine the type related to the solid attribute of the suspicious nodule.

[0160] The third attribute classification module (413) can compare the first type, which is the output of the first attribute classification module (411), with the second type, which is the output of the second attribute classification module (412), and finally determine the type related to the solid attribute of the suspicious nodule. Specifically, the third attribute classification module (413) can determine whether the first type is a type included in the second type. When the first type is not a type included in the second type, the third attribute classification module (413) can determine the first type as the final type related to the solid attribute of the suspicious nodule. When the first type is a type included in the second type, the third attribute classification module (413) can determine the second type as the final type related to the solid attribute of the suspicious nodule. For example, when the first type is determined to be partially solid, since the first type is not a type included in the second type, which is one of solid and non-solid, the third attribute classification module (413) can finally determine the first type, which is partially solid, as the type related to the solid attribute of the suspicious nodule. When the first type is determined to be one of solid and non-solid, since the first type is a type included in the second type, which is one of solid and non-solid, the third attribute classification module (413) can finally determine the second type as the type related to the solid attribute of the suspicious nodule.

[0161] FIG. 18 is a block configuration diagram showing the operation process of the classification module in an embodiment of the present disclosure.

[0162] Referring to FIG. 18, in step S510, a computing device (100) in an embodiment of the present disclosure can receive a patch generated from a medical image for lesion analysis and a mask related to the lesion from an external image analysis system. The medical image for lesion analysis can also be a three-dimensional CT image including a chest region. The patch can be a three-dimensional patch extracted from the three-dimensional CT image based on the position information of the suspicious nodule. The mask related to the lesion can be a mask related to at least one of the entire region of the suspicious nodule generated based on the three-dimensional patch or the region representing the solid attribute.

[0163] In step S510, the computing device (100) can also receive a medical image for analyzing a lesion and generate a patch generated from the medical image and a mask related to the lesion by itself. The computing device (100) can use a detection module to extract position information of a suspicious nodule from the medical image through a process similar to that in FIG. 7. The computing device (100) can generate an input patch of an evaluation module from the medical image based on the position information of the suspicious nodule. In other words, the computing device (100) can identify the position information of the suspicious nodule through analysis of the three-dimensional CT image and extract a three-dimensional patch of a predetermined size including the nodule and the suspected region from the three-dimensional CT image. On the other hand, as described above, the computing device (100) can directly extract the position information of the suspicious nodule using the detection module, but can also receive and use the position information of the suspicious nodule through an external system. In addition, the computing device (100) can use an evaluation module to generate a mask related to the suspicious nodule from the three-dimensional patch through a process as shown in FIG. 14.

[0164] Through step S510, the computing device (100) distinguishes steps S520 to S540 for determining the type related to the solid attribute of the suspicious nodule, step S550 for determining the presence or absence of spicules of the suspicious nodule, or step S560 for determining the presence or absence of calcification of the suspicious nodule based on the received or generated three-dimensional patch and mask, and can execute them individually as necessary. The computing device (100) can execute each step individually and generate class information used to evaluate the suspicious nodule.

[0165] In step S520, the computing device (100) can determine a first type related to the solid attribute of a suspicious nodule based on a patch and a mask using a first attribute classification module pre-trained with three-dimensional features. For example, in step S510, the computing device (100) can input a mask generated from a three-dimensional patch together with the received or generated three-dimensional patch into the first attribute classification module based on a neural circuit network. The computing device (100) can classify the attribute of the region determined to be a nodule in the patch as solid, partially solid, or non-solid through the first attribute classification module that has received the input of the three-dimensional patch and the mask. Therefore, the first type can be determined as one of solid, partially solid, and non-solid.

[0166] In step S530, the computing device (100) can determine a second type related to the solid attribute of a suspicious nodule based on a patch and a mask using a second attribute classification module. For example, in step S510, the computing device (100) can input a mask generated from a three-dimensional patch together with the received or generated three-dimensional patch into the second attribute classification module based on rules. The computing device (100) can classify the attribute of the region determined to be a nodule in the patch as solid or non-solid through the second attribute classification module that has received the input of the three-dimensional patch and the mask. In this case, the second attribute classification module can classify the attribute of the region determined to be a nodule in the patch as solid or non-solid based on the value of the Hounsfield unit of the voxels included in the mask. Therefore, the second type can be determined as one of solid and non-solid. Step S530 can be executed in parallel with step S510.

[0167] In step S540, the computing device (100) can use the third attribute classification module to determine the final type related to the solid attribute of the suspicious nodule based on the first type determined in step S520 and the second type determined in step S530. For example, the computing device (100) can use the third attribute classification module to compare the first type and the second type. If the first type is a unique type not included in one of the second types, the computing device (100) can use the third attribute classification module to determine the first type as the final type related to the solid attribute of the suspicious nodule. If the first type is a type included in one of the second types, the computing device (100) can use the third attribute classification module to determine the second type as the final type related to the solid attribute of the suspicious nodule.

[0168] In step S550, the computing device (100) can use the second sub-classification module pre-trained with 3D features to determine the presence or absence of spicules in the suspicious nodule based on the patch and the mask. For example, in step S510, the computing device (100) can input the mask generated from the 3D patch together with the received or generated 3D patch into the second sub-classification module based on the neural circuit network. The computing device (100) can determine the presence or absence of spicules in the region determined as a nodule in the patch through the second sub-classification module that has received the input of the 3D patch and the mask. In this case, the neural circuit network structure included in the second sub-classification module can correspond to the neural circuit network structure included in the first attribute classification module.

[0169] In step S560, the computing device (100) can determine the presence or absence of calcification of a suspicious nodule based on a patch and a mask using a third sub-classification module. For example, in step S510, the computing device (100) can input a mask generated from the 3D patch together with the received or generated 3D patch into the third sub-classification module based on rules. The computing device (100) can determine the presence or absence of calcification in the region determined to be a nodule in the patch through the third sub-classification module that has received the input of the 3D patch and the mask. In this case, the third sub-classification module can determine whether calcification has progressed in the region determined to be a nodule in the patch based on the value of the Hounsfield unit of the voxels included in the mask. Also, in step S510, the computing device (100) can input a mask generated from the 3D patch together with the received or generated 3D patch into the third sub-classification module based on a deep learning algorithm. The computing device (100) can determine the presence or absence of calcification in the region determined to be a nodule in the patch through the pre-trained third sub-classification module that has received the input of the 3D patch and the mask.

[0170] Based on an embodiment of the present disclosure, a computer-readable storage medium storing a data structure is disclosed.

[0171] A data structure can mean the organization, management, and storage of data that enables efficient access to and modification of the data. A data structure can mean the organization of data to solve a specific problem (e.g., data search in the shortest time, data storage, data modification). A data structure can also be defined as the physical or logical relationship between data elements designed to support a specific data processing function. The logical relationship between data elements can include the concatenation relationship between data elements as considered by the user. The physical relationship between data elements can include the actual relationship between data elements physically stored on a computer-readable storage medium (e.g., hard disk). A data structure can specifically include a set of data, the relationships between the data, and functions or commands applicable to the data. With an effectively designed data structure, a computing device can perform calculations while minimizing the use of the resources of the computing device. Specifically, the computing device can enhance the efficiency of operations, reading, insertion, deletion, comparison, exchange, and search through an effectively designed data structure.

[0172] Data structures can be classified into linear data structures and non-linear data structures according to their forms. A linear data structure may be a structure in which only one data is connected after another data. Linear data structures can include lists, stacks, queues, and deques. A list can mean a series of data sets with an internal order. A list can include a linked list. A linked list can be a data structure in which data is connected in such a way that each data has a pointer and is connected in a column. In a linked list, the pointer can include connection information with the next or previous data. A linked list can be represented as a singly linked list, a doubly linked list, or a circular linked list according to its form. A stack may be a data list structure with restricted access to data. A stack may be a linear data structure that can process (e.g., insert or delete) data only at one end of the data structure. The data stored in the stack may be a data structure (LIFO - Last in First Out) where the later it enters, the earlier it comes out. A queue is a data arrangement structure with restricted access to data and, unlike a stack, can be a data structure (FIFO - First in First Out) where the later the data is stored, the later it comes out. A deque can be a data structure that can process data at both ends of the data structure.

[0173] A non-linear data structure may be a structure in which multiple data are connected after one data. Non-linear data structures can include graph data structures. A graph data structure can be defined by vertices and edges, and an edge can include a line connecting two different vertices. A graph data structure can include a tree data structure. A tree data structure can be a data structure formed by a path connecting two different vertices among the multiple vertices included in the tree. That is, it can be a data structure that does not form a loop in the graph data structure.

[0174] In this specification, an arithmetic model, a neural circuit network, a network function, and a neural network can be used interchangeably (hereinafter, they will be uniformly described using the term "neural network"). A data structure can include a neural network. And a data structure including a neural network can be stored in a computer-readable storage medium. A data structure including a neural network can also include data input to the neural network, weight values of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions associated with each node or layer of the neural network, and a loss function for training the neural network. A data structure including a neural network can include any of the components of the above-disclosed configurations. That is, a data structure including a neural network can be configured to include all or any combination of data input to the neural network, weight values of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions associated with each node or layer of the neural network, a loss function for training the neural network, etc. In addition to the above-described configurations, a data structure including a neural network can include any other information that determines the characteristics of the neural network. Also, the data structure can include all forms of data used or generated in the arithmetic process of the neural network and is not limited to the above-mentioned matters. A computer-readable storage medium can include a computer-readable recording medium and / or a computer-readable transmission medium. A neural network can generally be composed of a set of interconnected computing units called nodes. Such nodes can be called neurons. A neural network is composed of at least one or more nodes.

[0175] A data structure can include data input to a neural network. A data structure that includes data input to a neural network can be stored in a computer-readable storage medium. The data input to the neural network can include training data input during the training process of the neural network and / or input data input to the neural network after training is completed. The data input to the neural network can include pre-processed data and / or data to be pre-processed. The pre-processing can include a data processing process for inputting the data to the neural network. Therefore, the data structure can include data to be pre-processed and data generated by the pre-processing. The foregoing data structure is merely illustrative and the present disclosure is not limited thereto.

[0176] A data structure can include the weights of a neural network (in this specification, weights and parameters can be used interchangeably). And a data structure that includes the weights of a neural network can be stored in a computer-readable storage medium. A neural network can include a plurality of weights. The weights are variable and can be varied by a user or an algorithm to perform the function desired by the neural network. For example, when one or more input nodes are interconnected by respective links to one output node, the output node can determine the output node value based on the values input to the input nodes connected to the output node and the parameters set for the respective links corresponding to the input nodes. The foregoing data structure is merely illustrative and the present disclosure is not limited thereto.

[0177] By way of example and not limitation, the weighted values can include weighted values that vary during the neural network learning process and / or weighted values after the neural network learning is completed. The weighted values that are varied during the neural network learning process can include weighted values at the start of the learning cycle and / or weighted values that are varied during the learning cycle. The weighted values after the neural network learning is completed can include weighted values after the learning cycle is completed. Accordingly, a data structure including the weighted values of the neural network can include a data structure including weighted values that vary during the neural network learning process and / or weighted values after the neural network learning is completed. Accordingly, the above-described weighted values and / or combinations of each weighted value shall be included in a data structure including the weighted values of the neural network. The foregoing data structure is merely illustrative and the present disclosure is not limited thereto.

[0178] A data structure including the weighted values of the neural network can be stored in a computer-readable storage medium (e.g., memory, hard disk) after going through a serialization process. Serialization can be a process of converting a data structure into a form that can be stored in the same or another computing device and later reconstructed and used. A computing device can serialize a data structure and send and receive data via a network. A data structure including the weighted values of the serialized neural network can be reconstructed on the same computing device or another computing device through deserialization. A data structure including the weighted values of the neural network is not limited to serialization. Further, a data structure including the weighted values of the neural network can include a data structure (e.g., non-linear data structures such as B-Tree, Trie, m-way search tree, AVL tree, Red-Black Tree) for enhancing the efficiency of operations while minimizing the use of resources of the computing device. The foregoing matters are merely illustrative and the present disclosure is not limited thereto.

[0179] The data structure can include hyper-parameters of the neural network. And the data structure including the hyper-parameters of the neural network can be stored in a computer-readable storage medium. The hyper-parameters can be variables that can be varied by the user. The hyper-parameters can include, for example, a learning rate, a cost function, the number of learning cycle iterations, weight initialization (e.g., setting the range of weights to be initialized), the number of hidden units (e.g., the number of hidden layers, the number of nodes in the hidden layer). The foregoing data structure is merely illustrative and the present disclosure is not limited thereto.

[0180] FIG. 19 is a simplified and general schematic diagram related to an exemplary computing environment in which an embodiment of the present disclosure can be implemented.

[0181] Although it has been described above that the present disclosure can generally be implemented by a computing device, those skilled in the art will well understand that the present disclosure can be implemented in combination with computer-executable instructions that can be executed on one or more computers and / or other program modules and / or as a combination of hardware and software.

[0182] Generally, a module in this specification includes routines, programs, components, data structures, and the like that perform a particular task or implement a particular abstract data type. Also, those skilled in the art will readily understand that the methods of the present disclosure can be implemented by other computer system configurations, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based platforms, or programmable household appliances, etc. (each of which can operate in conjunction with one or more associated devices).

[0183] The embodiments described in the present disclosure can further be implemented in a distributed computing environment where a certain task is executed by a remote processing device connected through a communication network. In a distributed computing environment, program modules can be located in both local and remote memory storage devices.

[0184] The computer includes a variety of computer-readable media. Any media accessible by the computer can be a computer-readable media, and such computer-readable media includes volatile and non-volatile media, transitory and non-transitory media, removable and non-removable media. By way of example and not limitation, computer-readable media can include computer-readable storage media and computer-readable transmission media. Computer-readable storage media is volatile and non-volatile media, transitory and non-transitory media, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer-readable storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD (digital video disk) or other optical disk storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other media that can be accessed by a computer and used to store information.

[0185] Computer-readable transmission media typically implements computer-readable instructions, data structures, program modules or other data, etc. in a modulated data signal such as a carrier wave or other transport mechanism, and includes all information transmission media. The term modulated data signal means a signal that sets or changes one or more of the characteristics of the signal so as to encode information in the signal. By way of example and not limitation, computer-readable transmission media includes wired media such as a wired network or a direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Any combination by any of the foregoing media is also considered to be within the scope of computer-readable transmission media.

[0186] An exemplary environment (1100) is shown that implements various aspects of the present disclosure, including a computer (1102), which includes a processing device (1104), a system memory (1106), and a system bus (1108). The system bus (1108) connects system components, including but not limited to the system memory (1106), to the processing device (1104). The processing device (1104) can be any of a variety of commercial processors. Dual processors and other multiprocessor architectures can also be utilized as the processing device (1104).

[0187] The system bus (1108) can be any of a plurality of types of bus structures that can be further interconnected to a local bus that uses any of a memory bus, a peripheral device bus, and various commercial bus architectures. The system memory (1106) includes a read-only memory (ROM) (1110) and a random access memory (RAM) (1112). The basic input / output system (BIOS) is stored in non-volatile memory (1110) such as ROM, EPROM, EEPROM, etc., and this BIOS includes basic routines that support the exchange of information between a plurality of components in the computer (1102) during startup and the like. The RAM (1112) can also include high-speed RAM such as static RAM for caching data.

[0188] In the computer (1102), there is also a built-in hard disk drive (HDD) (1114) (e.g., EIDE, SATA) - this built-in hard disk drive (1114) can also be configured for external use within a suitable chassis (not shown) -, a magnetic floppy disk drive (FDD) (1116) (e.g., for reading from and writing to a removable diskette (1118)) and an optical disk drive (1120) (e.g., for reading from a CD-ROM disk (1122), reading from and writing to other high-capacity optical media such as DVDs). The hard disk drive (1114), magnetic disk drive (1116) and optical disk drive (1120) can each be connected to the system bus (1108) by a hard disk drive interface (1124), a magnetic disk drive interface (1126) and an optical drive interface (1128), respectively. The interface (1124) for the implementation of an external drive includes, for example, at least one or both of USB (Universal Serial Bus) and IEEE1394 interface technologies.

[0189] These drives and the computer-readable media associated therewith provide non-volatile storage of data, data structures, computer-executable instructions, and the like. In the case of the computer (1102), the drives and media correspond to storing any data in a suitable digital format. Although the foregoing description of computer-readable storage media refers to HDDs, removable magnetic disks, and removable optical media such as CDs or DVDs, those skilled in the art will recognize that other types of storage media readable by a computer, such as zip drives, magnetic cassettes, flash memory cards, cartridges, and the like, can also be used in exemplary operating environments, and furthermore, it will be well understood that any of such media can contain computer-executable instructions for performing the methods of this disclosure.

[0190] A number of program modules, including an operating system (1130), one or more application programs (1132), other program modules (1134), and program data (1136), can be stored in a drive and RAM (1112). All or part of the operating system, applications, modules, and / or data can also be cached in RAM (1112). It will be well understood that the present disclosure can be implemented by various commercially available operating systems or combinations of multiple operating systems.

[0191] A user can input commands and information into the computer (1102) through one or more wired or wireless input devices, such as a keyboard (1138) and a pointing device such as a mouse (1140). Other input devices (not shown) may include a microphone, an IR remote control, a joystick, a game pad, a stylus pen, a touch screen, and others. These and other input devices may be connected to the processing device (1104) through an input device interface (1142) that is well connected to the system bus (1108), but can also be connected by other interfaces such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, and others.

[0192] A monitor (1144) or other type of display device is also connected to the system bus (1108) through an interface such as a video adapter (1146). In addition to the monitor (1144), a computer generally includes other peripheral output devices such as speakers, printers, and others (not shown).

[0193] The computer (1102) can operate in a networked environment using logical connections to one or more remote computers, such as (a plurality of) remote computers (1148) via wired and / or wireless communication. The (a plurality of) remote computers (1148) can be workstations, server computers, routers, personal computers, portable computers, microprocessor-based entertainment devices, peer devices, or other common network nodes, and generally include many or all of the components described for the computer (1102), but for simplicity, only the memory storage device (1150) is illustrated. The illustrated logical connections include wired and wireless connections in a local area network (LAN) (1152) and / or a larger network, such as a wide area network (WAN) (1154). Such LAN and WAN networking environments are common in offices and companies, facilitating enterprise-wide computer networks such as intranets, all of which can be connected to computer networks throughout the world, such as the Internet.

[0194] When used in a LAN networking environment, the computer (1102) is connected to a local network (1152) through a wired and / or wireless communication network interface or adapter (1156). The adapter (1156) can facilitate wired or wireless communication to the LAN (1152), which also includes a wireless access point installed thereon for communicating with the wireless adapter (1156). When used in a WAN networking environment, the computer (1102) can include a modem (1158), connect to a communication server on the WAN (1154), or have other means of establishing communication through the WAN (1154), such as through the Internet. The modem (1158), which can be internal or external and can be a wired or wireless device, is connected to the system bus (1108) through a serial port interface (1142). In a networked environment, program modules or portions thereof described for the computer (1102) can be stored in a remote memory / storage device (1150). It is readily understood that the network connections shown are exemplary and that other means of establishing communication links between multiple computers can be used.

[0195] The computer (1102) operates to communicate with any wireless device or unit arranged and operating in wireless communication, such as a printer, scanner, desktop and / or portable computer, PDA (portable data assistant), communication satellite, any equipment or location related to a wirelessly detectable tag, and a telephone. This includes at least Wi-Fi and Bluetooth (registered trademark) wireless technologies. Thus, the communication can be in a predefined structure like a conventional network or simply an ad hoc communication between at least two devices.

[0196] Wi-Fi (Wireless Fidelity) enables connection to the Internet and the like without being wired. Wi-Fi is a wireless technology such as a cell phone that allows such devices, for example, computers to send and receive data indoors and outdoors, that is, from anywhere within the coverage area of a base station. Wi-Fi networks use wireless technologies such as IEEE802.11 (a, b, g, etc.) to provide a secure, reliable, and high-speed wireless connection. Wi-Fi can be used to connect computers to each other and to the Internet and wired networks (using IEEE802.3 or Ethernet). Wi-Fi networks can operate at data rates such as 11 Mbps (802.11a) or 54 Mbps (802.11b) in unlicensed 2.4 and 5 GHz wireless bands, or can operate in products that include both bands (dual band).

[0197] Those with ordinary knowledge in the technical field of the present disclosure can understand that information and signals can be represented using any of a variety of different technologies and techniques. For example, the data, instructions, commands, information, signals, bits, symbols, and chips that can be referenced in the above description can be represented by voltage, current, electromagnetic waves, magnetic fields, etc., or particles, optical fields, etc., or particles, or any combination thereof.

[0198] Those of ordinary skill in the art of the present disclosure will appreciate that the various exemplary logical blocks, modules, processors, means, circuits, algorithm steps recited in the description of the embodiments disclosed herein can be implemented in electronic hardware, various forms of program or design code (referred to herein as "software" for convenience), or any combination of these. To clearly illustrate such interchangeability of hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above in terms of their functions. Whether such functions are implemented in hardware or software depends upon the design constraints imposed on a particular application and the overall system. Those of ordinary skill in the art of the present disclosure can implement the functions described in various ways for each particular application, but such implementation decisions should not be construed as departing from the scope of the present disclosure.

[0199] The various embodiments shown herein can be implemented by a method, apparatus, or article of manufacture using standard programming and / or engineering techniques. The term "article of manufacture" includes any computer program accessible from any computer-readable device, carrier, or medium. For example, computer-readable storage media include magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical disks (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, cards, sticks, key drives, etc.), but are not limited thereto. Also, the various storage media shown herein include one or more devices for storing information and / or other machine-readable media.

[0200] It should be understood that the specific order or hierarchical structure of the multiple steps in the presented process is an example of an exemplary approach. Based on design priorities, it should be understood that within the scope of the present disclosure, the specific order or hierarchical structure of the steps in the process can be rearranged. The appended method claims provide elements of various steps in a sample order, but are not meant to be limited to the specific order or hierarchical structure shown.

[0201] The description of the presented embodiments is provided so that those of ordinary skill in any art of the present disclosure can make use of or practice the present disclosure. Various modifications to such embodiments will be readily apparent to those of ordinary skill in the art of the present disclosure, and the general principles defined herein can be applied to other embodiments without departing from the scope of the present disclosure. Accordingly, the present disclosure is not limited by the embodiments shown herein, but should be construed in the broadest scope consistent with the principles and novel features shown herein.

Claims

1. A method for lesion analysis in medical images executed by a computing device including at least one processor, comprising: generating, using a preprocessing module, an input image for a pre-trained detection module from the medical image; generating, using the detection module, a probability value related to the presence of nodules in at least one region of interest and first position information of the at least one region of interest based on the input image; and determining, using a postprocessing module, second position information related to suspicious nodules present in the medical image from the first position information based on the probability value related to the presence of the nodules; wherein the step of generating the probability value related to the presence of the nodules and the first position information includes generating, using a first sub-detection module included in the detection module, a first probability value and the first position information related to the at least one region of interest based on a plurality of two-dimensional medical images; and estimating, using a second sub-detection module included in the detection module, a second probability value related to the at least one region of interest based on a three-dimensional medical image and the first position information; A method.

2. In claim 1, the step of generating the input image of the detection module includes calculating, using the preprocessing module, a value of a Hounsfield unit based on a three-dimensional medical image; and generating, using the preprocessing module, a plurality of two-dimensional medical images from the three-dimensional medical image in which the value of the Hounsfield unit is calculated; wherein is a method.

3. In claim 1, the step of generating the first probability value and the first position information includes generating, using a first neural network module included in the first sub-detection module, a plurality of first feature maps having a plurality of sizes based on the plurality of two-dimensional medical images; generating, using a second neural network module included in the first sub-detection module, a plurality of second feature maps by concatenating at least a part of the plurality of first feature maps based on the sizes of the plurality of first feature maps; and Using the third neural circuit network module included in the first sub-detection module, matching the plurality of second feature maps to a predetermined anchor box to generate the first probability value and the first position information related to the at least one region of interest; comprising method.

4. In claim 3, The step of generating the first probability value and the first position information is When there are a plurality of regions of interest, using the first sub-detection module to cluster at least a part of the plurality of regions of interest based on the ratio of the overlapping regions between the plurality of regions of interest; and Using the first sub-detection module to correct the coordinate system included in the first position information; further comprising method.

5. In claim 1, The step of estimating the second probability value is Using the fourth neural circuit network module included in the second sub-detection module, encoding based on the patch extracted from the three-dimensional medical image based on the first position information to generate at least one third feature map; Using the fifth neural circuit network module included in the second sub-detection module to perform decoding based on the third feature map to generate at least one fourth feature map; and Using the sixth neural circuit network module included in the second sub-detection module to generate the second probability value related to the at least one region of interest based on the feature map generated by integrating the third feature map and the fourth feature map; comprising method.

6. In claim 1, The second sub-detection module A first operation of training a neural circuit network based on a randomly sampled training image; and A second operation of training a neural circuit network based on a training image selected based on recall and precision; By executing, it is pre-trained method.

7. In claim 1, The step of determining the second position information related to the suspicious nodule is Using the post-processing module to compare the probability value related to the existence of the nodule generated by the weighted sum of the first probability value and the second probability value with a threshold value; and Determining, using the post-processing module, the first position information of the at least one region of interest corresponding to a probability value related to the presence of a nodule selected as a result of the comparison, as the second position information related to the suspicious nodule; comprising a method.

8. In claim 1, generating, using a pre-trained measurement module, a mask related to the suspicious nodule based on a patch of a medical image corresponding to the second position information; and generating numerical information including at least one of a diameter and a volume of the suspicious nodule based on the mask related to the suspicious nodule; further comprising a method.

9. In claim 8, the mask related to the suspicious nodule is a first mask related to the entire region of the suspicious nodule generated based on a 3D patch corresponding to the second position information; and a second mask related to a region representing a specific attribute of the suspicious nodule generated based on a 3D patch corresponding to the second position information; comprising a method.

10. In claim 8, classifying, using a pre-trained classification module, a class related to the state of the suspicious nodule based on the patch of the medical image and the mask related to the suspicious nodule; further comprising a method.

11. In claim 10, the step of classifying the class related to the state of the suspicious nodule is outputting, using a first sub-classification module of the classification module, a first class indicating a type related to an attribute of the suspicious nodule based on the patch and the mask; outputting, using a second sub-classification module of the classification module, a second class indicating whether the suspicious nodule has spiculation based on the patch and the mask; outputting, using a third sub-classification module of the classification module, a third class indicating whether the suspicious nodule is calcified based on the patch and the mask; including at least one of a method.

12. In claim 10, calculating an evaluation score of the suspicious nodule based on a class related to the state of the suspicious nodule and the numerical information based on an auxiliary index for lung cancer diagnosis; and When the subject of the input image corresponds to the subject of the analyzed image, using a pre-trained tracking module, based on the shooting time points of the input image and the analyzed image, modifying the evaluation score of the medical image or the evaluation score of the analyzed image; further comprising; method.

13. In claim 12, generating a user interface based on at least one of the second position information, the mask, the class, the numerical information, or the evaluation score related to the suspicious nodule; further comprising; method.

14. In claim 10, using a pre-trained malignancy prediction module, by inputting the second position information of the suspicious nodule, the class and the numerical information related to the state, calculating the malignancy of the suspicious nodule; further comprising; method.

15. In claim 10, using a pre-trained malignancy prediction module, by inputting the patch of the medical image and the mask related to the suspicious nodule, calculating the malignancy of the suspicious nodule; further comprising; method.

16. In claim 14 or claim 15, generating a user interface based on at least one of the second position information, the mask, the class, the numerical information, or the malignancy related to the suspicious nodule; further comprising; method.

17. A computer program stored in a computer-readable storage medium, the computer program, when executed by one or more processors, causes the following operations to be performed for analyzing lesions in a medical image, the operations being using a preprocessing module to generate an input image for a pre-trained detection module from the medical image; using the detection module to generate a probability value related to the presence of nodules in at least one region of interest and first position information of the at least one region of interest based on the input image; and using a postprocessing module to determine second position information related to suspicious nodules present in the medical image from the first position information based on the probability value related to the presence of the nodules; comprising; The operation of generating the probability value related to the presence of the nodules and the first position information is An operation of generating a first probability value and the first position information related to the at least one region of interest based on a plurality of two-dimensional medical images by using a first sub-detection module included in the detection module; and An operation of estimating a second probability value related to the at least one region of interest based on a three-dimensional medical image and the first position information by using a second sub-detection module included in the detection module; Including A computer program stored in a computer-readable storage medium.

18. A computing device for analyzing lesions in medical images, comprising A processor including at least one core; A memory including a plurality of program codes executable in the processor; and A network unit for receiving medical images; Including The processor Generates an input image of a pre-trained detection module from the medical image by using a preprocessing module, Generates a probability value related to the presence of a nodule in at least one region of interest and first position information of the at least one region of interest based on the input image by using the detection module, Determines second position information related to a suspected nodule present in the medical image from the first position information based on the probability value related to the presence of the nodule by using a postprocessing module, Generates a first probability value and the first position information related to the at least one region of interest based on a plurality of two-dimensional medical images by using a first sub-detection module included in the detection module, Estimates a second probability value related to the at least one region of interest based on a three-dimensional medical image and the first position information by using a second sub-detection module included in the detection module, Device.

Images