Method for analyzing lesions in medical images

Abstract

According to one embodiment of the present disclosure, a method for analyzing lesions in medical images, executed in a computing device, is disclosed, which may include obtaining location information of suspicious nodules present in a medical image, generating a mask of the suspicious nodule based on a patch of the medical image corresponding to the location information, and classifying a state of the suspicious nodule based on the patch of the medical image and the mask of the suspicious nodule.

Description

[Technical field]

[0001] The present invention relates to a method for processing medical images, and more particularly to a method for utilizing artificial intelligence to detect and evaluate lesions associated with specific diseases present in medical images. [Background technology]

[0002] Medical imaging is the documentation that allows understanding of the physical state of various organs of the human body. Medical imaging includes digital radiography (X-ray), computed tomography (CT), and magnetic resonance imaging (MRI).

[0003] Research and technological development on methods for automating the detection of lesions of specific diseases using medical images has been continuously conducted. However, since conventional technologies focus on identifying lesions present in medical images, they are unable to generate and process information related to lesions in a form suitable for diagnosing specific diseases. In other words, in terms of providing necessary information suitable for diagnosing specific diseases, conventional technologies are unable to provide performance that meets the purpose of detecting and evaluating lesions.

[0004] U.S. Patent No. 7,305,111 (December 4, 2007) discloses an automated method for lung nodule detection for lung cancer screening. Summary of the Invention [Problem to be solved by the invention]

[0005] The present disclosure has been devised in light of the above background art, and aims to provide a method for detecting and assessing lesions present in medical images for diagnosing specific diseases. [Means for solving the problem]

[0006] According to an embodiment of the present disclosure for achieving the above object, a method for analyzing lesions in medical images executed by a computing device is disclosed, which may include the steps of obtaining location information of suspicious nodules present in a medical image, generating a mask of the suspicious nodule based on a patch of the medical image corresponding to the location information, and classifying a state of the suspicious nodule based on the patch of the medical image and the mask of the suspicious nodule.

[0007] In an alternative embodiment, obtaining location information of the suspicious nodule may include using a pre-trained first model to generate a probability value for the presence of a nodule in at least one region of interest included in the medical image and candidate location information; and using the first model to determine location information of the suspicious nodule from the candidate location information based on the probability value for the presence of a nodule in the at least one region of interest.

[0008] In an alternative embodiment, generating a mask of the suspicious nodule may include extracting patches from the medical image corresponding to the location information; and using the second model to generate a first mask of the entire area of ​​the suspicious nodule and a second mask of areas representing specific attributes of the suspicious nodule based on the patches.

[0009] In an alternative embodiment, the method may further comprise the steps of: generating, based on the first mask, first numerical information including at least one of a diameter and a volume of an entire area of ​​the suspicious nodule; and generating, based on the second mask, second numerical information including at least one of a diameter and a volume of an area representing a particular attribute of the suspicious nodule.

[0010] In an alternative embodiment, the method may further include calculating an assessment score for the suspicious nodule based on the class relating to the suspicious nodule's status and the first numerical information, with reference to an auxiliary indicator for lung disease diagnosis.

[0011] In an alternative embodiment, the method may further include a step of calculating an evaluation score for the suspicious nodule based on the class of the suspicious nodule's status, the first numerical information, and the second numerical information, when the class of the suspicious nodule's status corresponds to a predetermined type related to a particular attribute of the suspicious nodule, using auxiliary indicators for lung disease diagnosis as a reference.

[0012] In an alternative embodiment, the method may further include generating a user interface based on at least one of the location information of the suspicious nodule, the mask, the class relating to the status of the suspicious nodule, the first numerical information, the second numerical information, and the evaluation score.

[0013] In an alternative embodiment, the method may further include a step of determining whether a subject in the medical image corresponds to a subject in the analyzed image; and if the subject in the medical image corresponds to a subject in the analyzed image, modifying the evaluation score of the medical image or the evaluation score of the analyzed image based on the time of capture of the medical image and the analyzed image using a pre-trained fourth model.

[0014] In an alternative embodiment, the step of modifying the evaluation score of the medical image or the evaluation score of the analyzed image may include the steps of: performing registration between the medical image and the analyzed image using the fourth model; matching a suspicious nodule present in the medical image with a suspicious nodule present in the analyzed image using the fourth model and identifying modification information of the matched suspicious nodule; and modifying the evaluation score of the medical image or the analyzed image based on the modification information.

[0015] In an alternative embodiment, the step of modifying the evaluation score of the medical image or the evaluation score of the analyzed image may include the step of modifying the evaluation score of the analyzed image using the fourth model based on the evaluation score of the medical image if the medical image was taken at a time before the analyzed image was taken; and the step of modifying the evaluation score of the medical image using the fourth model based on the evaluation score of the analyzed image if the medical image was taken at a time after the analyzed image was taken.

[0016] In an alternative embodiment, the method may further include estimating the malignancy of the suspicious nodule based on the location information, condition class, and numerical information of the suspicious nodule using a fifth pre-trained model.

[0017] In an alternative embodiment, the method may further comprise estimating a malignancy level of the suspicious nodule based on the patches and the mask using a fifth pre-trained model.

[0018] In an alternative embodiment, the method may further include generating a user interface based on at least one of the location information of the suspicious nodule, the mask, the class of the suspicious nodule's status, the first numerical information, the second numerical information, and the malignancy level.

[0019] In an alternative embodiment, the step of classifying the state of the suspicious nodule may include using a third model including at least one pre-trained sub-model to determine at least one of the attributes of the suspicious nodule: type, presence or absence of spiculation, and presence or absence of calcification based on the patch and mask using different sub-models.

[0020] According to another embodiment of the present disclosure for achieving the above object, a method for analyzing lesions in medical images, which is executed by a computing device, is disclosed, which may include the steps of: determining whether a subject in the medical image corresponds to a subject in an analyzed image; calculating an evaluation score for the suspicious nodule based on class and numerical information related to the state of the suspicious nodule present in the medical image; and, if the subject in the medical image corresponds to the subject in the analyzed image, correcting the evaluation score of the medical image or the evaluation score of the analyzed image based on the capture time of the medical image and the analyzed image.

[0021] In an alternative embodiment, the method may further comprise storing an assessment score of the medical image if the subject of the medical image and the subject of the analyzed image do not correspond.

[0022] In an alternative embodiment, the method may further include generating a user interface based on the stored rating scores or the revised rating scores.

[0023] In an alternative embodiment, the step of modifying the evaluation score of the medical image or the evaluation score of the analyzed image may include the steps of: performing registration between the medical image and the analyzed image using a pre-trained fourth model; matching suspicious nodules present in the medical image with suspicious nodules present in the analyzed image using the fourth model and identifying modification information of the matched suspicious nodules; and modifying the evaluation score of the medical image or the analyzed image based on the modification information.

[0024] In an alternative embodiment, the step of modifying the evaluation score of the medical image or the evaluation score of the analyzed image may include a step of modifying the evaluation score of the analyzed image based on the evaluation score of the medical image using a pre-trained fourth model if the medical image was taken at a time before the analyzed image was taken; and a step of modifying the evaluation score of the medical image based on the evaluation score of the analyzed image using the fourth model if the medical image was taken at a time after the analyzed image was taken.

[0025] According to an embodiment of the present disclosure for achieving the above object, a computer program stored on a computer-readable storage medium is disclosed, which, when executed on one or more processors, performs the following operations for analyzing lesions in a medical image, which may include: obtaining location information of suspicious nodules present in the medical image; generating a mask of the suspicious nodule based on a patch of the medical image corresponding to the location information; and classifying a state of the suspicious nodule based on the patch of the medical image and the mask of the suspicious nodule.

[0026] According to an embodiment of the present disclosure for solving the above-mentioned problems, a computing device for analyzing lesions in medical images is disclosed, the device includes a processor including at least one core, a memory including program code executable on the processor, and a network unit for receiving medical images including a chest region, the processor is capable of obtaining position information of suspicious nodules present in the medical images, generating a mask for the suspicious nodules based on a patch of the medical images corresponding to the position information, and classifying a state of the suspicious nodules based on the patch of the medical images and the mask for the suspicious nodules.

[0027] In order to achieve the above object, a user terminal for providing a user interface is disclosed according to an embodiment of the present disclosure. The user terminal may include a processor including at least one core; a memory; a network unit for receiving a user interface generated based on analysis information of a lesion included in a medical image from a computing device; and an output unit for providing the user interface. In this case, the analysis information of the lesion may include at least one of location information of a suspicious nodule, a mask related to the suspicious nodule, a class related to a state of the suspicious nodule, numerical information of the suspicious nodule, evaluation information for the suspicious nodule, and a malignancy level of the suspicious nodule. Effect of the Invention

[0028] The present disclosure can provide methods for detecting and assessing lesions present in medical images to diagnose specific diseases. [Brief description of the drawings]

[0029] [Figure 1] FIG. 1 is a block diagram of a computing device for analyzing lesions in medical images in accordance with one embodiment of the present disclosure. [Diagram 2]FIG. 2 is a schematic diagram illustrating a network function in one embodiment of the present disclosure. [Diagram 3] FIG. 3 is a block diagram illustrating a process by which a computing device interprets lesions in accordance with one embodiment of the present disclosure. [Figure 4] FIG. 4 is a flow chart illustrating a process by which a computing device reads lesions in one embodiment of the present disclosure. [Diagram 5] FIG. 5 is a block diagram illustrating a process by which a computing device reads and evaluates lesions in accordance with one embodiment of the present disclosure. [Figure 6] FIG. 6 is a block diagram illustrating a process by which a computing device modifies lesion assessment results in accordance with one embodiment of the present disclosure. [Figure 7] FIG. 7 is a flow chart of a method for analyzing a lesion in a medical image in accordance with one embodiment of the present disclosure. [Figure 8] FIG. 8 is a flow chart of a method for analyzing lesions in medical images in an alternative embodiment of the present disclosure. [Figure 9] FIG. 9 is a schematic diagram illustrating a computing environment in one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030] Various embodiments are described below with reference to the drawings, in which like reference numerals are used to represent like components throughout the drawings. Various descriptions are provided herein to facilitate understanding of the present disclosure. However, the embodiments can undoubtedly be practiced without these specific descriptions.

[0031] As used herein, terms such as "component," "module," and "system" refer to computer-related entities, hardware, firmware, software, a combination of software and hardware, or software execution. For example, a component can be, but is not limited to, a procedure running on a processor, a processor, an object, a thread of execution, a program, and / or a computer. For example, both an application running on a computing device and the computing device can be a component. One or more components can reside within a processor and / or thread of execution, and a component can be localized within one computer or distributed across two or more computers. Such components can also execute from various computer-readable media having various data structures stored therein. Components can communicate, for example, through local and / or remote processing, by signals (e.g., data from one component interacting with other components in a local system, a distributed system, and / or data transmitted over a network such as the Internet to other systems) carrying one or more data packets.

[0032] The term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, "X utilizes A or B" is intended to mean one of the natural inclusive permutations. That is, if X utilizes A; X utilizes B; or X utilizes both A and B, then "X utilizes A or B" can apply to any of these. Additionally, the term "and / or" as used herein should be understood to refer to and include all possible combinations of one or more of the associated listed items.

[0033] Additionally, the predicate "comprises" and / or the modifier "comprises" should be understood to mean that the feature and / or component is present. However, the predicate "comprises" and / or the modifier "comprises" should be understood to not exclude the presence or addition of one or more other further features, components and / or groups thereof. Additionally, in the present specification and claims, the singular should generally be construed to mean "one or more," unless a specific number is specified or the context is clear to indicate the singular form.

[0034] Furthermore, the term "at least one of A or B" should be interpreted as meaning "a case in which only A is included," "a case in which only B is included," or "a case in which a combination of A and B is included."

[0035] Those skilled in the art should further appreciate that the various exemplary logical blocks, configurations, modules, circuits, means, logic, and algorithm steps described in accordance with the embodiments disclosed herein can be realized by electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the various exemplary components, blocks, configurations, means, logic, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design limitations of the overall system. Skilled artisans can implement the described functionality in various ways for each particular application. However, such implementation decisions should not be interpreted as departing from the scope of the present disclosure.

[0036] The description of the embodiments set forth herein is provided to enable one of ordinary skill in the art to use or practice the present disclosure. Various modifications to these embodiments will be apparent to those of ordinary skill in the art to which the disclosure pertains, and the general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present disclosure is not intended to be limited by the embodiments set forth herein, but is to be accorded the broadest scope consistent with the principles and novel features disclosed herein.

[0037] In this disclosure, the terms network function, artificial neural network, and neural network may be used interchangeably.

[0038] In contrast, the terms "image" or "image data" as used in the detailed description and claims of this invention refer to multidimensional data made up of discrete picture elements (e.g., pixels in a two-dimensional image), in other words, to either a visually perceptible object (e.g., as displayed on a video screen) or a digital representation of that object (e.g., a file corresponding to the pixel output of a CT or MRI machine).

[0039] For example, an "image" or "picture" may be a medical image of a subject acquired by computed tomography (CT), magnetic resonance imaging (MRI), ultrasound, or any other medical imaging system known in the art. The image need not necessarily be provided for a medical purpose and may be provided for a non-medical purpose, such as, for example, an x-ray taken for security screening.

[0040] In the detailed description and claims of this invention, the "DICOM (Digital Imaging and Communications in Medicine)" standard is a collective term for all standards used in digital image representation and communication in medical equipment, and is published by a joint committee consisting of the American College of Radiology (ACR) and the National Electrical Manufacturers Association (NEMA).

[0041] In addition, in the detailed description and claims of the present invention, "Picture Archiving and Communication System (PACS)" is a term referring to a system that stores, processes, and transmits medical images in accordance with the DICOM standard, and is capable of storing medical image data obtained using digital medical imaging devices such as X-rays, CT scans, and MRIs in accordance with the DICOM standard and transmitting it to terminals within the hospital or outside the hospital via a network, and capable of adding interpretation results and medical records to the data.

[0042] FIG. 1 is a block diagram of a computing device for analyzing lesions in medical images in accordance with one embodiment of the present disclosure.

[0043] 1 is merely a simplified example, and in one embodiment of the present disclosure, the computer device (100) may include other components for implementing the computing environment of the computer device (100), and the computer device (100) may be configured with only some of the disclosed components.

[0044] The computer device (100) may include a processor (110), a memory (130), and a network unit (150).

[0045] The processor (110) may be configured with one or more cores, and may include a processor for data analysis and deep learning, such as a central processing unit (CPU) of a computing device, a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU). The processor (120) may read a computer program stored in the memory (130) and execute data processing for machine learning in an embodiment of the present disclosure. According to an embodiment of the present disclosure, the processor (120) may perform calculations for neural network learning. The processor (120) may execute calculations for neural network learning, such as processing input data for learning in deep learning (DL), extracting features from the input data, calculating errors, and updating weights of the neural network using backpropagation. At least one of the CPU, GPGPU, and TPU of the processor (120) may process learning of a network function. For example, the CPU and GPGPU may both perform learning of a network function or classifying data using the network function. In addition, in an embodiment of the present disclosure, the processors of multiple computing devices may be used together to learn the network function or classify data using the network function. Also, the computer program executed in the computing device in an embodiment of the present disclosure may be a program executable by a CPU, a GPGPU, or a TPU.

[0046] In one embodiment of the present disclosure, the processor 110 may use at least one pre-trained machine learning model to interpret lesions associated with a particular disease from a medical image. The processor 110 may input the medical image into a detection-based neural network model to identify location information of a lesion present in the medical image. The processor 110 may generate a patch corresponding to a portion of the medical image based on the location information of the lesion. The processor 110 may input the patch corresponding to the location information of the lesion into a segmentation-based neural network model to generate a mask for the lesion. In this case, the mask may refer to a collection of data including information related to an area where a lesion exists in the medical image. The processor 110 may input the patch and the mask into a classification-based neural network model to generate class information indicating the state of the lesion. Through such operations, the processor 110 can generate information related to lesions (e.g., the location of the lesion in the body, its condition, etc.) that can serve as diagnostic criteria for specific diseases based on the medical images.

[0047] For example, the processor (110) can input a medical image including a chest region input to the network unit (150) to the pre-trained first model. In this case, the medical image including the chest region can include a three-dimensional CT image including at least one lung tissue. The processor (110) can input the medical image including the chest region to the first model and obtain position information of a suspicious nodule present in the medical image. The position information of the suspicious nodule can include a central coordinate system of a region identified as a suspicious nodule in the medical image. When the medical image is a three-dimensional CT image, the position information of the suspicious nodule can include the (X, Y, Z) coordinate values ​​of the center of the region determined to be a suspicious nodule.

[0048] The processor (110) can extract patches corresponding to the position information from a medical image including a chest region based on the position information of the suspicious nodule obtained using the first model. The processor (110) can input the patches corresponding to the position information of the suspicious nodule to a pre-trained second model. The processor (110) can input the patches generated from the medical image to the second model to generate a mask related to the suspicious nodule. That is, the processor (110) can use the second model to extract information related to the area where the suspicious nodule exists in the patch.

[0049] The processor (110) can input the previously extracted patch together with the mask of the suspicious nodule generated using the second model to a pre-trained third model. The processor (110) can input both the mask of the suspicious nodule and the patch to the third model and perform a class for the suspicious nodule status. In this case, the suspicious nodule status can include features, attributes, etc. of the suspicious nodule that are the basis for determining pulmonary disease. In other words, the processor (110) can use the third model to identify the suspicious nodule status in the patch of the medical image to obtain lesion information for diagnosing pulmonary disease.

[0050] In one embodiment of the present disclosure, the processor (110) can evaluate a lesion read from a medical image based on an auxiliary indicator for diagnosing a specific disease. The processor (110) can calculate a numerical value related to an area where a lesion exists based on a mask generated using a segmentation-based neural network model. The processor (110) can calculate an evaluation score for the lesion based on numerical information including a numerical value of the lesion and class information related to the state of the lesion based on the auxiliary indicator for diagnosing a specific disease. The processor (110) can also predict the malignancy of the lesion using a pre-trained neural network model. The processor (110) can estimate the malignancy of the lesion based on the position information of the lesion, class information related to the state of the lesion, and numerical information of the lesion using a regression-based neural network model. Through this operation, the processor (110) can generate evaluation information for the lesion that can be used as a diagnostic indicator for a specific disease based on the medical image.

[0051] For example, the processor (110) can generate numerical information related to the area where the suspicious nodule exists in the medical image based on the mask of the suspicious nodule generated using the second model. In this case, the numerical information can include numerical values ​​related to the diameter, volume, etc. of the suspicious nodule. The processor (110) can calculate an evaluation score of the suspicious nodule based on the auxiliary indicators of lung disease diagnosis stored in the memory (130) based on the numerical information of the suspicious nodule and the class related to the state of the suspicious nodule classified using the third model. In this case, the auxiliary indicators of lung disease diagnosis can include classification indicators based on Lung-RADS (Lung CT Screening Reporting and Data System), etc. In other words, the processor (110) can determine an evaluation score of the suspicious nodule using both structural information and attribute information of the suspicious nodule according to a criterion determined based on the auxiliary indicators of lung disease diagnosis. The evaluation score determined by the processor 110 can be used for diagnosing lung diseases and predicting prognosis of subjects in medical images.

[0052] The processor (110) can input the position information of the suspicious nodule generated by the first model and the class information related to the state of the suspicious nodule generated by the third model and the numerical information of the suspicious nodule generated based on the mask to the pre-trained fifth model. The processor (110) can input the position information, class information, and numerical information of the suspicious nodule to the fifth model to estimate the malignancy of the suspicious nodule. The processor (110) can also input the patch extracted from the medical image and the mask generated by the second model to the fifth model to estimate the malignancy of the suspicious nodule. In other words, the processor (110) can estimate the malignancy of the suspicious nodule using the quantitative information of the suspicious nodule, and can estimate the malignancy of the suspicious nodule using image information related to the suspicious nodule. The processor (110) can use the fifth model to predict the malignancy of a suspicious nodule affecting a pulmonary disease by considering all of the position information, structural information, and attribute information of the suspicious nodule present in the medical image including the chest region. The malignancy predicted by the processor (110) can be utilized for diagnosing a pulmonary disease and predicting a prognosis of the subject in the medical image.

[0053] In one embodiment of the present disclosure, the processor (110) can use a pre-trained neural network model to modify the lesion-related evaluation score based on the time-series medical images of a specific subject. When the processor (110) inputs medical images of the same subject as the analyzed images into the computing device (100), the processor (110) can use the pre-trained machine learning model to match the lesions present in the analyzed images with the lesions present in the medical images input later to grasp the changed information. The processor (110) can then reflect the changed information in the lesion-related evaluation score and modify the evaluation score. If there is no changed information, the processor (110) can maintain the existing evaluation score without modifying the lesion-related evaluation score.

[0054] For example, the processor (110) may perform the above-described operations to calculate an assessment score for a suspicious nodule based on a medical image received via the network unit (150) and store the calculated assessment score in the memory (130). When a new medical image is received via the network unit (150), the processor (110) may determine whether a subject in the new medical image corresponds to a subject in an analyzed medical image. That is, the processor (110) may determine whether an identifier (ID) used to identify a subject in the new medical image matches one of the identifiers of the plurality of analyzed medical images. If the identifier of the new medical image matches one of the identifiers of the plurality of analyzed medical images, the processor (110) may use the pre-trained fourth model to match a suspicious nodule present in an existing image with a suspicious nodule present in the new image. The processor (110) may use the fourth model to identify modification information of the matched suspicious nodule and modify the assessment score for the suspicious nodule based on the modification information. Through such operations, the processor 110 can effectively track changes in pathology in a particular subject and improve the accuracy of information required to determine the prognosis of lung disease.

[0055] In one embodiment of the present disclosure, the memory 130 may store any type of information generated or determined by the processor 110 and any type of information received by the network unit 150.

[0056] In one embodiment of the present disclosure, the memory (130) may include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), Random Access Memory (RAM), Static Random Access Memory (SRAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Programmable Read-Only Memory (PROM), magnetic memory, magnetic disk, and optical disk. The computing device (100) may also operate in conjunction with a web storage that performs the storage function of the memory (130) over the Internet. The above description of the memory is merely exemplary, and the present disclosure is not limited thereto.

[0057] The network unit 150 according to an embodiment of the present disclosure may use any form of known wired and wireless communication system.

[0058] The network unit (150) can receive medical images showing human organs from a medical imaging system. For example, the medical images showing human organs can be training data or inference data for a neural network model trained with two-dimensional or three-dimensional features. The medical images showing human organs can be three-dimensional CT regions including at least one lung region. The medical images showing human organs are not limited to the above examples, and can include all images related to human organs obtained by imaging, such as X-ray images, MR images, etc.

[0059] In addition, the network unit (150) can transmit and receive information processed by the processor (110), a user interface, etc., through communication with other terminals. For example, the network unit (150) can provide a user interface generated by the processor (100) to a client (e.g., a user terminal). In addition, the network unit (150) can receive an external input from a user to a client and transfer it to the processor (110). In this case, the processor (100) can process operations such as output, correction, modification, and addition of information provided through the user interface based on the external input from the user received from the network unit (150).

[0060] Meanwhile, in one embodiment of the present disclosure, the computing device (100) may include a server as a computing system that transmits and receives information through communication with a client. In this case, the client may be any type of terminal that can access the server. For example, the computing device (100) serving as a server may receive medical images from a medical imaging system, analyze lesions, and provide a user interface including the analysis results to a user terminal. In this case, the user terminal may output the user interface received from the computing device (100) serving as a server, and receive and process information input through interaction with a user.

[0061] The user terminal is capable of displaying a user interface provided to provide analysis information of a lesion (e.g., a suspicious nodule, etc.) included in a medical image sent from the server computing device (100). Although not shown in the figure, the user terminal may include a network unit that receives the user interface from the computing device (100), a processor including at least one core, a memory, an output unit that provides the user interface, and an input unit that receives external input made by a user.

[0062] In additional embodiments, the computing device 100 may include any type of terminal that receives data resources generated at any server and performs additional information processing.

[0063] FIG. 2 is a schematic diagram illustrating network functions used to provide diagnostic relevant information for medical data in one embodiment of the present disclosure.

[0064] Throughout this specification, the terms computational model, neural network, network function, and neural network may be used interchangeably. A neural network is often composed of a collection of interconnected computational units, generally called nodes. Such nodes may also be referred to as neurons. A neural network is composed of at least one or more nodes. The nodes (or neurons) that make up a neural network may be interconnected by one or more links.

[0065] In a neural network, one or more nodes connected via links can form a relationship of input node and output node relative to each other. The concept of input node and output node is relative, and any node that is an output node for one node can be an input node in relation to another node, and vice versa. As mentioned above, the relationship between input node and output node can be established around the link. One or more output nodes can be connected to one input node via links, and vice versa.

[0066] In the relationship between an input node and an output node connected via a link, the data of the output node can be determined based on the data input to the input node. Here, the node interconnecting the input node and the output node can have a weight. The weight can be variable and can be changed by a user or an algorithm so that the neural network performs a desired function. For example, when one or more input nodes are interconnected to one output node by each link, the output node can determine the value of the output node based on the value input to the input node connected to the output node and the weight set to the link corresponding to each input node.

[0067] As mentioned above, in a neural network, one or more nodes are interconnected through one or more links to form a relationship between an input node and an output node in the neural network. In a neural network, the characteristics of the neural network can be determined by the number of nodes and links, the correlation between the nodes and links, and the weight value assigned to each link. For example, if there are two neural networks that have the same number of nodes and links but different link weight values, the two neural networks can be recognized as different.

[0068] A neural network may be composed of a set of one or more nodes. A subset of the nodes in a neural network may form a layer. A portion of the nodes in a neural network may form a layer based on their distance from a first input node. For example, a set of nodes whose distance from a first input node is n may form an n-layer. The distance from the first input node may be defined based on the minimum number of links that must be traversed to reach the node from the first input node. However, this definition of a layer is given arbitrarily for the purpose of explanation, and the configuration of layers in a neural network may be defined in a manner different from that described above. For example, a layer of nodes may be defined based on their distance from a final output node.

[0069] The first input node may refer to one or more nodes in a neural network to which data is directly input without passing through a link in relation to other nodes. Alternatively, the first input node may refer to a node that does not have other input nodes connected via a link in relation between nodes based on links in a neural network. Similarly, the final output node may refer to one or more nodes in a neural network that do not have an output node in relation to other nodes. In addition, the hidden node may refer to a node that is neither a first input node nor a final output node and that constitutes a neural network.

[0070] The neural network according to an embodiment of the present disclosure may be a neural network in which the number of nodes in the input layer is the same as the number of nodes in the output layer, and the number of nodes decreases once and then increases again as the input layer progresses to the hidden layer. The neural network according to an embodiment of the present disclosure may be a neural network in which the number of nodes in the input layer is smaller than the number of nodes in the output layer, and the number of nodes decreases as the input layer progresses to the hidden layer. Furthermore, the neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes in the input layer is greater than the number of nodes in the output layer, and the number of nodes increases as the input layer progresses to the hidden layer. The neural network according to another embodiment of the present disclosure may be a neural network in which the above-mentioned neural networks are combined.

[0071] Deep neural networks (DNNs) can refer to neural networks that include multiple hidden layers in addition to input and output layers. Deep neural networks can be used to understand the latent structures of data. In other words, they can understand the latent structures of photos, text, videos, audio, and music (e.g., whether an object is in the photo, what is the content and emotion of the text, what is the content and emotion of the audio, etc.). Deep neural networks can include convolutional neural networks (CNNs), recurrent neural networks (RNNs), autoencoders, generative adversarial networks (GANs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), Q networks, U networks, Siamese networks, generative adversarial networks (GANs), etc. The above-mentioned deep neural networks are merely examples and the present disclosure is not limited thereto.

[0072] In one embodiment of the present disclosure, the network function may include an autoencoder. An autoencoder may be a type of artificial neural network for outputting output data similar to input data. An autoencoder may include at least one hidden layer, and an odd number of hidden layers may be disposed between input and output layers. The number of nodes in each layer may decrease from the number of nodes in the input layer toward an intermediate layer called a bottleneck layer (encode), and may expand from the bottleneck layer toward an output layer (symmetrical to the input layer) in a manner that contrasts with shrinkage. An autoencoder may perform nonlinear dimensionality reduction. The number of input layers and output layers may correspond to the dimensions after preprocessing of the input data. In the autoencoder structure, the number of nodes in the hidden layer included in the encoder may decrease as it gets farther from the input data. If the number of nodes in the bottleneck layer (the layer with the fewest number of nodes located between the encoder and the decoder) is too small, a sufficient amount of information may not be transmitted, so the number may be maintained at a certain number or more (e.g., more than half of the input layer).

[0073] A neural network can be trained by at least one of supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Training a neural network can be a process of providing the neural network with knowledge to perform a specific operation.

[0074] Neural networks can be trained in a direction that minimizes the error of the output. In training a neural network, training data is repeatedly input to the neural network, the error of the neural network output and the target regarding the training data is calculated, and the weight value of each node of the neural network is updated by backpropagating the error of the neural network from the output layer of the neural network to the input layer as a direction to reduce the error. In the case of supervised learning, training data in which the correct answer is labeled is used for each training data (i.e., labeled training data), and in the case of unsupervised learning, the correct answer may not be labeled for each training data. That is, for example, the training data in supervised learning for data classification can be data in which each training data is labeled with a category. The labeled training data is input to the neural network, and the error can be calculated by comparing the output (category) of the neural network with the label of the training data. As another example, in the case of unsupervised learning for data classification, the error can be calculated by comparing the input training data with the output of the neural network. The calculated error is backpropagated in the reverse direction (i.e., from the output layer to the input layer) in the neural network, and the connection weight value of each node in each layer of the neural network can be updated through backpropagation. The amount of change in the connection weight value of each node to be updated can be determined by a learning rate. The calculation of the neural network for the input data and the backpropagation of the error can constitute a learning cycle (epoch). The application method of the learning rate can be changed depending on the number of iterations of the neural network learning cycle. For example, in the early stage of neural network learning, the learning rate can be increased to increase efficiency by allowing the neural network to quickly achieve a certain level of performance, and in the later stage of learning, the learning rate can be decreased to increase accuracy.

[0075] In the training of a neural network, the training data may generally be a subset of the actual data (i.e., data to be processed using the trained neural network), and therefore there may be a training cycle in which the error associated with the training data decreases, but the error associated with the actual data increases. Overfitting is a phenomenon in which errors increase in the actual data due to excessive learning of the training data. For example, a neural network that has learned about cats by looking at a yellow cat may be unable to recognize that it is a cat when it sees a cat of a color other than yellow, which may be a type of overfitting. Overfitting may cause an increase in errors in machine learning algorithms. To prevent overfitting, various optimization methods may be applied. To prevent overfitting, methods such as increasing the training data, regularization, dropout, which deactivates some of the nodes of the network during the training process, and the use of a batch normalization layer may be applied.

[0076] FIG. 3 is a block diagram illustrating a process by which a computing device interprets lesions in accordance with one embodiment of the present disclosure.

[0077] Referring to FIG. 3, in one embodiment of the present disclosure, the processor (110) of the computing device (100) can obtain location information (13) of a suspicious nodule present in a medical image (11) using a pre-trained first model (210). For example, the processor (110) can input the medical image (11) to the first model (210) and generate a probability value and candidate location information related to the presence of a nodule in at least one region of interest included in the medical image (11). In this case, the region of interest can mean an area in the medical image that corresponds to a group of candidates for a suspicious nodule. Also, the candidate location information can mean location information of the region of interest. The processor (110) can determine the location information (13) of the suspicious nodule based on the probability value related to the presence of a nodule in the at least one region of interest using the first model (210). The processor (110) can determine whether each of the probability values ​​related to the presence of a nodule in a plurality of regions of interest is equal to or greater than a predetermined threshold. The processor (110) can determine location information of at least one region of interest that corresponds to a probability value equal to or greater than a predetermined threshold as location information (13) of a suspicious nodule.

[0078] The processor (110) can extract, from the medical image (11), a patch (15) that corresponds to input data for the pre-trained second model (220) or the third model (230) based on the position information (13) of the suspicious nodule. That is, the processor (110) can extract an area that corresponds to the position information (13) of the suspicious nodule from the medical image (11) and generate the patch (15). Thus, the patch (15) can be image data that contains meaningful information related to the suspicious nodule, with areas unnecessary for lesion analysis filtered from the medical image. The processor (110) can generate a mask (17) related to the suspicious nodule based on the patch (15) using the second model (220). For example, the processor (110) may input the patch (15) into the second model (220) to generate a mask (17) for at least one of the entire area of ​​the suspicious nodule or an area representing a particular attribute of the suspicious nodule, where the particular attribute may be, by way of example only, an attribute type representative of the suspicious nodule condition, and the attributes may be non-limiting examples of such attributes. Through operation with the second model (220), the processor (110) may filter out unnecessary information from the medical image (11) and effectively extract information related to the area in which the suspicious nodule is present.

[0079] The processor (110) can use a pre-trained third model (230) to generate class information (19) related to the state of a suspicious nodule based on the patch (15) and a mask (17) corresponding to the output of the second model (220). By receiving both the patch (15) and the mask (17) as input, the third model (230) can output class information (19) with higher accuracy than when only the patch (15) is input. For example, the processor (110) can input both the patch (15) and the mask (17) to a third model (230) including at least one sub-model, and generate class information (19) indicating at least one of a type related to the attributes of a suspicious nodule, the presence or absence of spiculation, and the presence or absence of calcification, using different sub-models. Specifically, the processor (110) can use a first sub-model of the third model (230) to determine a type of individual attribute of the suspicious nodule based on the patch (15), a mask related to the entire area of ​​the suspicious nodule, and a mask related to an area representing a specific attribute in the suspicious nodule. The processor (110) can use a second sub-model of the third model (230) to determine the presence or absence of spicules in the suspicious nodule based on the patch (15), a mask related to the entire area of ​​the suspicious nodule, and a mask related to an area representing a specific attribute in the suspicious nodule. The processor (110) can use a third sub-model of the third model (230) to determine the presence or absence of calcification in the suspicious nodule based on the patch (15), a mask related to the entire area of ​​the suspicious nodule, and a mask related to an area representing a specific attribute in the suspicious nodule. As described above, the multiple sub-models included in the third model (230) can operate in parallel for each type of class, but the third model (230) can also operate as a single model to classify multiple types of classes independently. The third model (230) can also include all of the three sub-models described above, or only some of them.

[0080] FIG. 4 is a flow chart illustrating a process by which a computing device reads lesions in one embodiment of the present disclosure.

[0081] Referring to FIG. 4, at step S110, the computing device (100) can receive a medical image of a human organ to be analyzed for lesions from a medical imaging system. For example, the medical image can be a CT image of a chest region including the lungs. The computing device (100) can obtain location information of lesions present in the medical image using a first model, which is a pre-trained neural network model. For example, the computing device (100) can obtain center coordinate values ​​of suspicious nodules from the CT image of the chest region using the first model.

[0082] In S120, the computing device (100) can extract image units of a predetermined standard from the medical image based on the position information of the lesion. The predetermined standard can be optimized according to the input of the pre-trained neural network model. For example, the computing device (100) can extract a patch centered on the central coordinate value of the suspicious nodule obtained using the first model. The computing device (100) can generate a mask for the lesion based on the image units extracted from the medical image using a second model, which is a pre-trained neural network model. For example, the computing device (100) can input the patch corresponding to the central coordinate value of the suspicious nodule into the second model to generate a mask including information related to the suspicious nodule.

[0083] In step S130, the computing device (100) may use a third model, which is a pre-trained neural network model, to perform classification of the state of the lesion based on the image units and the mask extracted in step S120. For example, the computing device (100) may input the patch corresponding to the center coordinate value of the suspicious nodule and the mask generated by the second model into the third model to perform classification of the state of the suspicious nodule. In this case, the state of the suspicious nodule may include physical attributes (e.g., solid, partially solid, non-solid, etc.) and characteristics (e.g., spicules, calcification, etc.) of the suspicious nodule.

[0084] FIG. 5 is a block diagram illustrating a process by which a computing device reads and evaluates lesions in accordance with one embodiment of the present disclosure.

[0085] Referring to FIG. 5, in one embodiment of the present disclosure, a processor (110) of a computing device (100) can input a medical image (21) including at least one lung region into a first model (210) to generate location information (22) of suspicious nodules present in the lung region. The processor (110) can input a 3D patch (23) extracted from the medical image (21) based on the location information (22) of the suspicious nodule into a second model (220) to generate a number of masks (24, 25). In this case, the first mask (24) can be a mask including information regarding the entire area of ​​the suspicious nodule. The second mask (25) can be a mask including information regarding an area of ​​the entire area of ​​the suspicious nodule that represents a specific attribute of the suspicious nodule (e.g., solid, etc.). The processor (110) can input the 3D patch (23) and the multiple masks (24, 25) into a third model (230) to generate class information (27) indicating a typology of suspicious nodule attributes, the presence or absence of spicules, the presence or absence of calcification, etc.

[0086] Meanwhile, the processor (110) can generate numerical information (26) including at least one of the diameter and volume of the suspicious nodule based on the masks (24, 25) related to the suspicious nodule. In this case, the numerical information (26) includes first numerical information generated based on the first mask (24), and can further include second numerical information generated based on the second mask (25) when the suspicious nodule is classified into a specific class. The first numerical information can include a numerical value representing at least one of the diameter and volume of the entire area of ​​the suspicious nodule present in the medical image (21). The second numerical information can include a numerical value representing at least one of the diameter and volume of an area representing a specific attribute of the suspicious nodule present in the medical image (21). The processor (110) can calculate structural numerical values ​​related to the shape, size, etc. of the area corresponding to the suspicious nodule in the 3D patch (23) based on the information included in the first mask (24). However, if the class of the suspicious nodule's condition corresponds to a predefined typology of a particular attribute of the suspicious nodule (e.g., partially solid), the processor (110) may calculate the aforementioned numerical value based on the information contained in the first mask (24) and the information contained in the second mask (25).

[0087] The processor (110) can calculate an evaluation score (28) of a suspicious nodule based on the auxiliary indicators (30) for lung disease diagnosis, the numerical information (26) and the class information (27). For example, the processor (110) can consider the numerical information (26) and the class information (27) of a suspicious nodule detected from a medical image (21) based on the auxiliary indicators (30) for lung cancer diagnosis, and determine one of a plurality of scores defined by the auxiliary indicators (30) as an evaluation for the suspicious nodule. Specifically, the processor (110) can determine which of the six categories in the lung-RADS classification system the suspicious nodule falls into based on a numerical value related to at least one of the diameter, volume, etc. of the entire area of ​​the suspicious nodule and the area representing the solid attributes included in the numerical information (26), and information related to the type of solid attributes, the presence or absence of spicules, and the presence or absence of calcification included in the class information (27). When the type of the solid attribute of the suspicious nodule in the class information (27) is solid or non-solid, the processor (110) can use the first numerical information included in the numerical information (26) to determine which of the six categories in the lung-RADS classification system the suspicious nodule belongs to. When the type of the solid attribute of the suspicious nodule in the class information (27) is partially solid, the processor (110) can use both the first numerical information and the second numerical information included in the numerical information (26) to determine which category in the lung-RADS classification system the suspicious nodule belongs to. Based on the result of such determination, the processor (110) can determine one of the six categories in the lung-RADS classification system as the assessment score (28) of the suspicious nodule.

[0088] The processor (110) can predict the degree of impact of the suspicious nodule on the lungs as a cause of lung disease based on the information on the suspicious nodule output through the first model (210), the second model (220), and the third model (230). The processor (110) can estimate the malignancy level (29) of the suspicious nodule based on the position information (22), the numerical information (26), and the class information (27) of the suspicious nodule using the pre-trained fifth model (250). For example, the processor (110) can input the center coordinate value of the nodule included in the position information (22), the size value of the nodule included in the numerical information (26), the type of the attribute of the solid included in the class information (27), the presence or absence of spicules, the presence or absence of calcification, and other information into the fifth model (250) to calculate the malignancy level (29) of the suspicious nodule.

[0089] Although not shown in Fig. 5, the processor (110) can also estimate the malignancy degree (29) of the suspicious nodule based on the 3D patch (23) extracted from the medical image (21) and the masks (24, 25) generated using the second model (220) using a pre-trained fifth model (250). In other words, the processor (110) can estimate the malignancy degree (29) by directly inputting the quantitative information (22, 26, 27) related to the suspicious nodule extracted from the medical image (21) to the fifth model (250), but can also estimate the malignancy degree (29) by inputting the image information (23, 24, 25) generated by processing the medical image (21) to the fifth model.

[0090] FIG. 6 is a block diagram illustrating a process by which a computing device modifies lesion assessment results in accordance with one embodiment of the present disclosure.

[0091] Referring to FIG. 6, the processor (110) of the computing device (100) in one embodiment of the present disclosure can modify the assessment score for the suspicious nodule based on a plurality of medical images taken in chronological order of the same subject. The processor (110) can modify the assessment score for the suspicious nodule based on the time of each of the plurality of medical images taken in chronological order of the same subject. When medical images are sequentially input to the computing device (100), the processor (110) can determine whether there is a history of an image of the same subject as the medical image (41) being input and analyzed. If it is determined that there is no history of an image of the same subject as the medical image (41), the processor (110) recognizes the medical image (41) as a medical image of a new subject, and can calculate the assessment score (45) for the suspicious nodule through step A (50). In this case, step A (50) can be considered to correspond to the calculation process of the assessment score (28) shown in FIG. 5.

[0092] If it is determined that an analyzed image of the same subject as the medical image (41) exists, the processor (110) can use the pre-trained fourth model (240) to perform registration between the analyzed image and the medical image (41). In this case, registration means an operation of adjusting the relative positional relationship between the analyzed image and the medical image (41) that have a time difference. The processor (110) can match a suspicious nodule present in the analyzed image after the registration is completed with a suspicious nodule present in the medical image (41) using the fourth model (240). Although not shown in FIG. 6, the processor (110) can perform step A (50) on the medical image (41) and identify change information between the matched suspicious nodules. In this case, the step A (50) can be considered to correspond to the calculation process of the evaluation score (28) shown in FIG. 5. The processor 110 can modify the evaluation score of the medical image 41 or the evaluation score of the analyzed image based on the modification information. If the medical image 41 is taken before the analyzed image is taken, the processor 110 can modify the evaluation score of the analyzed image based on the modification information. Conversely, if the medical image 41 is taken after the analyzed image is taken, the processor 110 can modify the evaluation score of the medical image based on the modification information. That is, the processor 110 can compare the time points of the medical image 41 and the analyzed image and modify the evaluation score of the image taken relatively recently. That is, the processor 110 can modify the evaluation score for the image taken relatively recently in order to effectively track the time changes of the suspicious nodule. Through this process, the processor 110 can finally generate a modified evaluation score 49 for the suspicious nodule for the specific subject. Such assessment score modification operations may be performed repeatedly as new medical images (41) are input to the computing device (100).

[0093] FIG. 7 is a flow chart of a method for analyzing a lesion in a medical image in accordance with one embodiment of the present disclosure.

[0094] Referring to FIG. 7, when a medical image including a chest region is input in step S210, the computing device (100) in one embodiment of the present disclosure can determine whether an image having the same identifier as the input image exists among the analyzed images. In this case, the identifier refers to identification information for the subject of the image. For example, the computing device (100) can determine whether the subject of the input CT image corresponds to the subject of the analyzed CT image. If the subject of the input CT image corresponds to the subject of the analyzed CT image, the computing device (100) can perform a series of operations to modify the previously calculated and stored evaluation score of the suspicious nodule. If the subject of the input CT image does not correspond to the subject of the analyzed CT image, the computing device (100) can consider that a medical image of a new subject has been input, and perform a series of operations to read and evaluate the suspicious nodule based on the input CT image.

[0095] The following describes a general process for modifying the assessment score of a suspicious nodule that is performed when an object in an input image is determined to correspond to an object in an existing image.

[0096] At step S221, the computing device (100) can obtain center position information of a suspicious nodule present in lung tissue based on the input image. For example, the computing device (100) can use a pre-trained detection-based neural network model to generate information related to at least one candidate region of a suspicious nodule based on the input image. The computing device (100) can use the detection-based neural network model to generate center position information of the suspicious nodule based on the information related to the candidate region.

[0097] In step S222, the computing device (100) can extract image patches based on center position information of the suspicious nodule from the input image. The computing device (100) can generate a mask of the suspicious nodule based on the image patch extracted from the input image. The computing device (100) can generate numerical information including structural values ​​of the suspicious nodule based on the mask of the suspicious nodule. For example, the computing device (100) can generate multiple masks of the suspicious nodule based on the image patch using a pre-trained segmentation-based neural network model. The computing device (100) can calculate numerical values ​​related to the diameter, volume, etc. of the suspicious nodule based on the information included in the multiple masks to generate numerical information. In this case, the numerical information can include at least one of first numerical information including structural information related to the entire area of ​​the suspicious nodule and second numerical information including structural information related to an area representing a specific attribute (e.g., a solid attribute) in the suspicious nodule.

[0098] In step S223, the computing device (100) may classify the suspicious nodule status based on the image patch and the mask generated in step S222. The suspicious nodule status classes may include a first class representing a type of suspicious nodule attribute, a second class representing the presence or absence of spicules in the suspicious nodule, and a third class representing the presence or absence of calcification in the suspicious nodule. The computing device (100) may classify the suspicious nodule status into the first class, the second class, and the third class, respectively. For example, the computing device (100) may classify the suspicious nodule status into a type of solid attribute into solid, partially solid, and non-solid based on the image patch and the masks using a first sub-model of the pre-trained classification-based neural network model. The computing device (100) can use a second sub-model of the classification-based neural network model to classify the suspicious nodules as spicules or non-spicules based on the image patches and the plurality of masks, and the computing device (100) can use a third sub-model of the classification-based neural network model to classify the suspicious nodules as calcified or non-calcified based on the image patches and the plurality of masks.

[0099] In step S224, the computing device (100) can calculate an evaluation score and a malignancy level of the suspicious nodule based on the central position information of the suspicious nodule generated in step S221, the numerical information generated in step S222, and the class information generated in step S223. For example, the computing device (100) can calculate the numerical information and the class information according to a criterion defined in a predetermined diagnostic auxiliary index, and calculate an evaluation score of the suspicious nodule. The computing device (100) can estimate the malignancy level of the suspicious nodule based on the central position information, the numerical information, and the class information of the suspicious nodule using a pre-trained regression-based neural network model. The computing device (100) can also estimate the malignancy level of the suspicious nodule based on the image patch and mask generated in step S222 using a pre-trained regression-based neural network model.

[0100] In step S225, the computing device (100) can perform registration to match the relative positions of the input image and the existing image. The computing device (100) can match the suspicious nodules in the input image read in the above step with the suspicious nodules in the read and stored existing image to grasp changes in the suspicious nodules. For example, the computing device (100) can perform registration between the input image and the existing image using a pre-trained machine learning model. The computing device (100) can match at least one suspicious nodule present in each of the two images for which registration has been completed, and determine whether a change has occurred between the matched suspicious nodules.

[0101] In the step of S226, if it is determined that a change has occurred in the suspicious nodule matched between the input image and the existing image, the computing device (100) can reflect the evaluation score of the suspicious nodule calculated from the input image and the evaluation score of the suspicious nodule calculated from the existing image in the mutual evaluation scores to calculate a corrected evaluation score. In this case, the image whose evaluation score is corrected can be determined based on the time of image capture. For example, if the input image is an image of a specific subject taken in 2009 and the existing image is an image of a specific subject taken in 2015, the computing device (100) can also correct the evaluation score of the existing image by reflecting the evaluation score of the input image in the evaluation score of the existing image. Conversely, if the input image is an image of a specific subject taken in 2015 and the existing image is an image of a specific subject taken in 2009, the computing device (100) can also correct the evaluation score of the input image by reflecting the evaluation score of the existing image in the evaluation score of the input image. If it is determined that there has been no change in the matched suspicious nodules between the input image and the existing image, the computing device (100) may leave the existing assessment scores unchanged without modification.

[0102] On the other hand, steps S231 to S234, which are related to the process of reading and evaluating suspicious nodules when it is determined that the subjects in the input image and the existing image do not correspond to each other, correspond to the above-mentioned steps S221 to S224, and therefore a detailed description thereof will be omitted.

[0103] FIG. 8 is a flow chart of a method for analyzing lesions in medical images in accordance with an alternative embodiment of the present disclosure.

[0104] Referring to Fig. 8, when a medical image including a chest region is input, a computing device (100) in an alternative embodiment of the present disclosure may perform reading and evaluation of suspicious nodules as a priority. Unlike Fig. 7, in Fig. 8, the reading and evaluation of suspicious nodules are performed before determining whether the input image is an image of the same subject as the existing image. In other words, it can be understood that the method shown in Fig. 7 and the method shown in Fig. 8 have different orders of timing for making a judgment to modify the evaluation score of a suspicious nodule. Therefore, detailed descriptions of each step (steps S310 to S370) in Fig. 8 that correspond to those in Fig. 7 will be omitted.

[0105] However, in accordance with one embodiment of the present disclosure, a computer-readable storage medium having a data structure stored thereon is disclosed.

[0106] A data structure can refer to the organization, management, and storage of data that allows efficient access and modification of the data. A data structure can refer to the organization of data to solve a specific problem (e.g., data retrieval, data storage, data modification in the shortest time). A data structure can also be defined as a physical or logical relationship between data elements designed to support a specific data processing function. A logical relationship between data elements can include a connection relationship between data elements as thought by a user. A physical relationship between data elements can include an actual relationship between data elements that is physically stored in a computer-readable storage medium (e.g., a hard disk). A data structure can specifically include a collection of data, a relationship between data, and a function or command that can be applied to the data. An effectively designed data structure allows a computing device to perform calculations while minimizing the use of computing device resources. Specifically, a computing device can increase the efficiency of operations, reading, inserting, deleting, comparing, exchanging, and searching through an effectively designed data structure.

[0107] Data structures can be classified into linear data structures and non-linear data structures according to the type of data structure. A linear data structure can be a structure in which only one piece of data is linked after another. Linear data structures can include lists, stacks, queues, and deques. A list can refer to a series of data sets that are internally ordered. A list can include linked lists. A linked list can be a data structure in which data is linked in a manner in which each piece of data is linked in a row with a pointer. In a linked list, the pointer can contain information about the connection to the next or previous piece of data. A linked list can be expressed as a singly linked list, a doubly linked list, or a circularly linked list depending on the type of data structure. A stack can be a data list structure that allows limited access to data. A stack can be a linear data structure in which data can be manipulated (e.g., inserted or deleted) only at one end of the data structure. Data stored in a stack can be a LIFO (Last in, First Out) data structure. A queue is a data structure that allows limited access to data, and unlike a stack, it can be a data structure where the slowest data stored is the slowest data available (FIFO - First in First out). A deck can be a data structure where data can be processed on both ends of the data structure.

[0108] A nonlinear data structure may be a structure in which multiple pieces of data are concatenated after one piece of data. A nonlinear data structure may include a graph data structure. A graph data structure may be defined by vertices and edges, and the edges may include a line connecting two different vertices. A graph data structure may include a tree data structure. A tree data structure may be a data structure in which a path connecting two different vertices among multiple vertices included in a tree is one data structure. In other words, a graph data structure may be a data structure that does not form a loop.

[0109] Throughout this specification, the terms computation model, neural network, network function, and neural network can be used interchangeably. (Hereinafter, they will be uniformly described as neural network.) The data structure may include a neural network. The data structure including a neural network may be stored in a computer-readable storage medium. The data structure including a neural network may also include data input to the neural network, weights of the neural network, hyperparameters of the neural network, data acquired from the neural network, activation functions associated with each node or layer of the neural network, and a loss function for training the neural network. The data structure including a neural network may include any of the components disclosed above. That is, the data structure including a neural network may include all or any combination of data input to the neural network, weights of the neural network, hyperparameters of the neural network, data acquired from the neural network, activation functions associated with each node or layer of the neural network, and a loss function for training the neural network. In addition to the above-mentioned configurations, the data structure including a neural network may include any other information that determines the characteristics of the neural network. Furthermore, the data structure may include all forms of data used or generated in the process of computing a neural network, and is not limited to the above. The computer-readable storage medium may include a computer-readable recording medium and / or a computer-readable transmission medium. A neural network may be comprised of a collection of interconnected computing units, generally referred to as nodes. Such nodes may be referred to as neurons. A neural network is comprised of at least one or more nodes.

[0110] The data structure may include data to be input to the neural network. The data structure including the data to be input to the neural network may be stored in a computer-readable storage medium. The data to be input to the neural network may include training data input during the training process of the neural network and / or input data to be input to the neural network after training has been completed. The data to be input to the neural network may include data that has undergone pre-processing and / or data to be pre-processed. Pre-processing may include a data processing process for inputting data to the neural network. Thus, the data structure may include data to be pre-processed and data generated in pre-processing. The above data structures are merely examples, and the present disclosure is not limited thereto.

[0111] The data structure may include weights of the neural network. (In this specification, weights and parameters may be used interchangeably.) The data structure including the weights of the neural network may be stored in a computer-readable storage medium. The neural network may include a plurality of weights. The weights are variable and may be varied by a user or an algorithm to allow the neural network to perform a desired function. For example, when one or more input nodes are interconnected to an output node by respective links, the output node may determine an output node value based on values ​​input to the input nodes connected to the output node and parameters set for the links corresponding to the respective input nodes. The above data structures are merely examples, and the present disclosure is not limited thereto.

[0112] By way of example and not limitation, the weights may include weights that are variable during the neural network training process and / or weights at which neural network training has been completed. The weights that are variable during the neural network training process may include weights at the start of a learning cycle and / or weights that are variable during a learning cycle. The weights at which neural network training has been completed may include weights at which a learning cycle has been completed. Thus, a data structure including weights of a neural network may include a data structure including weights that are variable during the neural network training process and / or weights at which neural network training has been completed. Thus, the above-mentioned weights and / or combinations of each weight are included in a data structure including weights of a neural network. The above-mentioned data structures are merely exemplary and the present disclosure is not limited thereto.

[0113] The data structure including the neural network weights may be stored in a computer-readable storage medium (e.g., memory, hard disk) after a serialization process. Serialization may be a process of converting a data structure into a form that can be stored in the same or another computing device and later reconstructed for use. The computing device may serialize the data structure and transmit and receive the data over a network. The serialized data structure including the neural network weights may be reconstructed in the same or another computing device through deserialization. The data structure including the neural network weights is not limited to serialization. Furthermore, the data structure including the neural network weights may include a data structure for increasing the efficiency of calculations while minimizing the use of computing device resources (e.g., nonlinear data structures such as B-Tree, Trie, m-way search tree, AVLtree, Red-Black Tree). The above is merely an example, and the present disclosure is not limited thereto.

[0114] The data structure may include hyper-parameters of the neural network. The data structure including the hyper-parameters of the neural network may be stored in a computer-readable storage medium. The hyper-parameters may be variables that are changed by a user. The hyper-parameters may include, for example, a learning rate, a cost function, a number of learning cycle iterations, weight initialization (e.g., setting a range of weights to be initialized), and the number of Hidden Units (e.g., the number of hidden layers, the number of nodes in the hidden layers). The above data structures are merely examples, and the present disclosure is not limited thereto.

[0115] FIG. 9 is a simplified general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure can be implemented.

[0116] Although the present disclosure has been described above as generally being embodied in a computing device, those skilled in the art will appreciate that the present disclosure can also be embodied in combination with computer-executable instructions and / or other program modules capable of being executed on one or more computers and / or as a combination of hardware and software.

[0117] Generally, modules herein include routines, programs, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Additionally, those skilled in the art will appreciate that the methods of the present disclosure can be practiced with other computer system configurations, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, etc., each of which can operate in conjunction with one or more associated devices.

[0118] The embodiments described in this disclosure may also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0119] A computer includes a variety of computer-readable media. Any medium accessible by a computer can be a computer-readable medium, including volatile and non-volatile media, transitory and non-transitory media, mobile and non-mobile media. By way of example and not limitation, computer-readable media can include computer-readable storage media and computer-readable transmission media. Computer-readable storage media include volatile and non-volatile media, transitory and non-transitory media, mobile and non-mobile media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, DVD (digital video disk) or other optical disk storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be accessed by a computer and used to store information.

[0120] Computer-readable transmission media typically embodies computer-readable instructions, data structures, program modules or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes all information delivery media. The term modulated data signal means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, computer-readable transmission media includes wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, RF, infrared and other wireless media. Any combination of any of the above media should also be included within the scope of computer-readable transmission media.

[0121] An exemplary environment (1100) for implementing various aspects of the present disclosure is shown including a computer (1102) including a processing unit (1104), a system memory (1106), and a system bus (1108). The system bus (1108) couples system components, including, but not limited to, the system memory (1106), to the processing unit (1104). The processing unit (1104) can be any of a variety of commercially available processors. Dual processors and other multi-processor architectures can also be utilized as the processing unit (1104).

[0122] The system bus (1108) can be any of several types of bus structures that can be further interconnected to a memory bus, a peripheral bus, and a local bus using any of a variety of commercially available bus architectures. The system memory (1106) includes read only memory (ROM) (1110) and random access memory (RAM) (1112). The basic input / output system (BIOS) is stored in non-volatile memory (1110), such as ROM, EPROM, or EEPROM, and contains the basic routines that support the exchange of information between the various components of the computer (1102), such as during start-up. The RAM (1112) can also include high-speed RAM, such as static RAM, for caching data.

[0123] The computer 1102 also includes an internal hard disk drive (HDD) 1114 (e.g., EIDE, SATA) - although the internal hard disk drive 1114 can also be configured for external use in a suitable chassis (not shown) - a magnetic floppy disk drive (FDD) 1116 (e.g., for reading from and writing to a removable diskette 1118), and an optical disk drive 1120 (e.g., for reading from CD-ROM disks 1122 and for reading from and writing to other high capacity optical media such as DVDs). The hard disk drive 1114, magnetic disk drive 1116, and optical disk drive 1120 can be connected to the system bus 1108 by a hard disk drive interface 1124, a magnetic disk drive interface 1126, and an optical drive interface 1128, respectively. The interface (1124) for implementing an external drive includes, for example, at least one or both of the Universal Serial Bus (USB) and IEEE 1394 interface technologies.

[0124] These drives and their associated computer-readable media provide non-volatile storage of data, data structures, computer-executable instructions, and the like. In the case of computer 1102, the drives and media accommodate the storage of any data in a suitable digital format. While the above description of computer-readable storage media refers to HDDs, removable magnetic disks, and removable optical media such as CDs or DVDs, those skilled in the art will appreciate that other types of computer-readable storage media, such as zip drives, magnetic cassettes, flash memory cards, cartridges, and the like, can also be used in the exemplary operating environment, and further that any such media can contain computer-executable instructions for performing the methods of the present disclosure.

[0125] A number of program modules may be stored in the drives and RAM 1112, including an operating system 1130, one or more application programs 1132, other program modules 1134, and program data 1136. All or portions of the operating system, applications, modules, and / or data may also be cached in RAM 1112. It will be appreciated that the present disclosure may be implemented with various commercially available operating systems or combinations of operating systems.

[0126] A user may enter commands and information into the computer 1102 through one or more wired or wireless input devices, such as a keyboard 1138 and a pointing device, such as a mouse 1140. Other input devices (not shown) may include a microphone, an IR remote control, a joystick, a game pad, a stylus pen, a touch screen, and the like. These and other input devices are often connected to the processing unit 1104 through an input device interface 1142 that is connected to the system bus 1108, but may also be connected through other interfaces, such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, and the like.

[0127] A monitor 1144 or other type of display device is also connected to the system bus 1108 through an interface, such as a video adapter 1146. In addition to the monitor 1144, computers typically include other peripheral output devices (not shown), such as speakers, printers, etc.

[0128] The computer 1102 can operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) 1148, via wired and / or wireless communications. The remote computer(s) 1148 can be a workstation, a server computer, a router, a personal computer, a handheld computer, a microprocessor-based entertainment device, a peer device or other conventional network node, and typically includes many or all of the components depicted for the computer 1102, although for simplicity, only a memory storage device 1150 is illustrated. The logical connections depicted include wired and wireless connections in a local area network (LAN) 1152 and / or larger networks, such as a wide area network (WAN) 1154. Such LAN and WAN networking environments are commonplace in offices and companies and facilitate enterprise-wide computer networks, such as intranets, all of which may connect to a global computer network, such as the Internet.

[0129] When used in a LAN networking environment, the computer 1102 connects to the local network 1152 through a wired and / or wireless communication network interface or adapter 1156. The adapter 1156 can facilitate wired or wireless communication to the LAN 1152, which may also include a wireless access point installed thereon for communicating with the wireless adapter 1156. When used in a WAN networking environment, the computer 1102 can include a modem 1158 or have other means for establishing communications over the WAN 1154, such as connecting to a communications server on the WAN 1154 or through the Internet. The modem 1158, which can be internal or external and can be a wired or wireless device, connects to the system bus 1108 through the serial port interface 1142. In a networked environment, program modules described for computer 1102, or portions thereof, may be stored in a remote memory / storage device 1150. It will be readily appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.

[0130] The computer 1102 is operable to communicate with any wireless device or unit that is deployed and operates in wireless communication, such as printers, scanners, desktop and / or handheld computers, portable data assistants (PDAs), communication satellites, any equipment or location associated with a wirelessly detectable tag, and telephones. This includes at least Wi-Fi and Bluetooth wireless technologies. Thus, communication can be in a predefined structure, such as a traditional network, or simply ad hoc communication between at least two devices.

[0131] Wi-Fi (Wireless Fidelity) allows devices to connect to the Internet without being wired. Wi-Fi is a wireless technology like a cell phone that allows such devices, e.g., computers, to send and receive data indoors and outdoors, anywhere within the range of a base station. Wi-Fi networks use IEEE 802.11 (a, b, g, etc.) radio technology to provide secure, reliable, and fast wireless connections. Wi-Fi can be used to connect computers to each other, to the Internet, and to wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in the unlicensed 2.4 and 5 GHz radio bands, e.g., at data rates of 11 Mbps (802.11a) or 54 Mbps (802.11b), or in products that include both bands (dual bands).

[0132] Those of ordinary skill in the art will appreciate that information and signals may be represented using any of a variety of different technologies and techniques. For example, the data, instructions, commands, information, signals, bits, symbols, and chips referred to in the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields, etc. or particles, optical fields, etc. or particles, or any combination thereof.

[0133] Those of ordinary skill in the art will appreciate that the various exemplary logic blocks, modules, processors, means, circuits, and algorithm steps described in the description of the embodiments disclosed herein can be implemented by electronic hardware, various forms of program or design code (for convenience, referred to herein as "software"), or a combination of all of these. To clearly illustrate such interoperability of hardware and software, the various exemplary components, blocks, modules, circuits, and steps have been generally described above with a focus on their functionality. Whether such functionality is implemented in hardware or software depends on the design constraints imposed on the particular application and the overall system. Those of ordinary skill in the art will appreciate that the described functionality can be implemented in various ways for each particular application, and such implementation decisions should not be interpreted as departing from the scope of the present disclosure.

[0134] The various embodiments described herein may be implemented by a method, apparatus, or article of manufacture using standard programming and / or engineering techniques. The term "article of manufacture" includes a computer program, carrier, or media accessible by any computer-readable device. For example, computer-readable storage media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical disks (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROM, cards, sticks, key drives, etc.). Additionally, the various storage media described herein include one or more devices and / or other machine-readable media for storing information.

[0135] It should be understood that the particular order or hierarchy of the stages in the processes depicted is an example of an exemplary approach. Based on design priorities, it should be understood that the particular order or hierarchy of the stages in the processes can be rearranged within the scope of the present disclosure. The accompanying method claims provide elements of the various stages in a sample order, but are not meant to be limited to the particular order or hierarchy depicted.

[0136] The description of the illustrated embodiments is provided to enable any person of ordinary skill in the art to which the disclosure pertains to the present disclosure to utilize or practice the present disclosure. Various modifications to such embodiments will be apparent to those of ordinary skill in the art to which the disclosure pertains, and the general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present disclosure is not intended to be limited by the embodiments shown herein, but is to be accorded the broadest scope consistent with the principles and novel features disclosed herein.

[0137] As described above, the relevant contents have been described in the best mode for carrying out the invention. [Industrial Applicability]

[0138] The invention may be used in computing devices, such as those that analyze lesions in medical images.

Claims

1. 1. A method for analyzing lesions in medical images executed by a computing device including at least one processor, comprising: generating a probability value and candidate location information for the presence of a nodule in at least one region of interest contained in the medical image using the pre-trained first model; determining location information of a suspected nodule from the candidate location information using the first model based on the probability value for the presence of the nodule in the at least one region of interest; generating a mask based on patches of the medical image corresponding to the location information, the mask including the overall area of ​​the suspicious nodule and areas representative of specific attributes of the suspicious nodule; and performing a class classification of the suspicious nodule status based on the patches and the mask of the medical image; Including, method.

2. In claim 1, The step of generating the mask comprises: extracting patches from the medical image corresponding to the location information; and generating, based on the patch, a first mask for the entire area of ​​the suspicious nodule and a second mask for areas representing specific attributes of the suspicious nodule using a second pre-trained model; Including, method.

3. In claim 2, generating a first numerical value based on the first mask, the first numerical value including at least one of a diameter and a volume of the entire area of ​​the suspicious nodule; and generating second numerical information including at least one of a diameter and a volume of a region representing a particular attribute of the suspicious nodule based on the second mask; Further comprising: method.

4. In claim 3, calculating an evaluation score for the suspicious nodule based on a class related to the suspicious nodule's condition and the first numerical information, with reference to an auxiliary indicator for lung disease diagnosis; Further comprising: method.

5. In claim 4, generating a user interface based on at least one of the suspicious nodule location information, the mask, the suspicious nodule status class, the first numerical information, the second numerical information, and an evaluation score; Further comprising: method.

6. In claim 4, determining whether a subject in the medical image corresponds to a subject in the analyzed image; and When the subject of the medical image corresponds to the subject of the analyzed image, correcting the evaluation score of the medical image or the evaluation score of the analyzed image based on the capture time of the medical image and the analyzed image using a pre-trained fourth model; Further comprising: method.

7. In claim 3, calculating an evaluation score for the suspicious nodule based on the class related to the suspicious nodule's state, the first numerical information, and the second numerical information, when the class related to the suspicious nodule's state corresponds to a predetermined type related to a specific attribute of the suspicious nodule, using auxiliary indicators for lung disease diagnosis as a reference; Further comprising: method.

8. In claim 7, generating a user interface based on at least one of the suspicious nodule location information, the mask, the suspicious nodule status class, the first numerical information, the second numerical information, and an assessment score; Further comprising: method.

9. In claim 7, determining whether a subject in the medical image corresponds to a subject in the analyzed image; and When the subject of the medical image corresponds to the subject of the analyzed image, correcting the evaluation score of the medical image or the evaluation score of the analyzed image based on the capture time of the medical image and the analyzed image using a pre-trained fourth model; Further comprising: method.

10. In claim 9, The step of modifying the evaluation score of the medical image or the evaluation score of the analyzed image comprises: performing registration between the medical image and the analyzed image using the fourth model; using the fourth model to match suspicious nodules present in the medical image with suspicious nodules present in the analyzed image and identify modification information of the matched suspicious nodules; and modifying an evaluation score of the medical image or the analyzed image based on the change information; Including, method.

11. In claim 9, The step of modifying the evaluation score of the medical image or the evaluation score of the analyzed image comprises: if the medical image was taken at a time before the analyzed image was taken, modifying the evaluation score of the analyzed image based on the evaluation score of the medical image using the fourth model; and if the medical image was taken at a time after the analyzed image was taken, modifying the assessment score of the medical image based on the assessment score of the analyzed image using the fourth model; Including, method.

12. In claim 3, estimating a malignancy level of the suspicious nodule based on location information, condition class and numerical information of the suspicious nodule using a pre-trained fifth model; Further comprising: method.

13. In claim 12, generating a user interface based on at least one of the suspicious nodule location information, the mask, the suspicious nodule status class, the first numerical information, the second numerical information, and a malignancy level; Further comprising: method.

14. In claim 3, estimating a malignancy level of the suspicious nodules based on the patches and the mask using a pre-trained fifth model; Further comprising: method.

15. In claim 14, generating a user interface based on at least one of the suspicious nodule location information, the mask, the suspicious nodule status class, the first numerical information, the second numerical information, and a malignancy level; Further comprising: method.

16. In claim 1, The step of classifying the suspicious nodule status comprises: using a third model including at least one pre-trained sub-model to determine at least one of attributes of the suspicious nodule, including a type, the presence or absence of spiculation, and the presence or absence of calcification, based on the patch and the mask, using different sub-models; Including, method.

17. 1. A computing device for analyzing lesions in medical images, comprising: a processor including at least one core; A memory containing program code executable on said processor; and a network portion for receiving the medical image; Including, The processor, generating a probability value and candidate location information for the presence of a nodule in at least one region of interest contained in the medical image using the pre-trained first model; determining location information of a suspected nodule from the candidate location information using the first model based on the probability value for the presence of the nodule in the at least one region of interest; generating a mask based on patches of the medical image corresponding to the location information, the mask including an overall area of ​​the suspicious nodule and an area representative of a particular attribute of the suspicious nodule; performing a class classification of the suspicious nodule status based on the patches and the mask of the medical image; Device.

18. A user terminal, a processor including at least one core; memory; a network unit for receiving from a computing device a user interface generated based on analysis information of a lesion contained in the medical image; and an output unit for providing the user interface; Including, The analysis information of the lesion includes: the location information of the suspicious nodule obtained by the computing device; a mask relating to the entire area of ​​the suspicious nodule and an area representing a specific attribute of the suspicious nodule, the mask being generated by the computing device; a class relating to the state of the suspicious nodule determined by the computing device; numerical information of the suspicious nodule generated by the computing device based on the mask; evaluation information for the suspicious nodule calculated by the computing device based on the mask and the numerical information; and a malignancy level of the suspicious nodule estimated by the computing device based on the mask and the numerical information, The location information of the suspicious nodule, generating a probability value and candidate location information for the presence of the nodule in at least one region of interest included in the medical image using a pre-trained first model; and determining location information of the suspect nodule from the candidate location information using the first model based on the probability value of the presence of the nodule in the at least one region of interest; obtained by the computing device based on Terminal.

Images