Method, computer program, and apparatus for a logistic model for determining the connectivity of 3D lesions in the z-direction (Logistic model for determining the connectivity of 3D lesions in the z-direction)

Abstract

To satisfy a need to further improve developed automated image analysis mechanisms so as to provide more efficient and correct analysis on medical image data for detecting lesions in an imaged anatomy (e.g., liver or other organs).SOLUTION: This mechanism for providing a mechanism for implementing a trained machine learning computer model that determines connectivity of lesions in a z direction identifies, with respect to a given slice in a three-dimensional medical image, a first lesion within the given slice and a second lesion within an adjacent slice in the three-dimensional medical image. This mechanism determines a first crossover value between the first and second lesions with respect to the first lesion and determines a second crossover value between the first and second lesions with respect to the second lesion. This mechanism determines whether or not the first and second lesions belong to the same three-dimensional lesion, on the basis of the first and second crossover values.SELECTED DRAWING: Figure 15

Description

[Technical Field]

[0001] This application relates generally to improved data processing apparatus and methods, and more particularly to a mechanism for providing a logistic model for determining 3D z-direction lesion connectivity. [Background technology]

[0002] Liver lesions are groups of abnormal cells within the liver, a biological entity sometimes called a mass or tumor. Noncancerous, or benign, liver lesions are common and do not spread to other areas of the body. Such benign liver lesions generally do not cause any health problems. However, some liver lesions result from cancer. Patients with certain medical conditions may be more likely than others to have cancerous liver lesions. These medical conditions include, for example, hepatitis B or C, cirrhosis, iron storage disease (hemochromatosis), obesity, or exposure to toxic chemicals (such as arsenic or aflatoxin).

[0003] Liver lesions are typically identifiable only through medical imaging, such as ultrasound, magnetic resonance imaging (MRI), computerized tomography (CT), or positron emission tomography (PET) scans. Such medical imaging must be viewed by a human subject matter expert (SME), who uses their expertise and the human ability to understand patterns in images to determine whether the medical imaging shows any lesions. If a possible cancerous lesion is identified by the human SME, the patient's physician can perform a biopsy to determine whether the lesion is cancerous.

[0004] Contrast-enhanced (CE) CT of the abdomen is the current standard in the evaluation of various abnormalities (e.g., lesions) of the liver. These lesions may be evaluated as malignant (hepatocellular carcinoma, cholangiocarcinoma, angiosarcoma, metastases, and other malignant lesions) or benign (hemangioma, focal nodular hyperplasia, adenoma, cyst or lipoma, granuloma, etc.) by human SMEs. Manual evaluation of such images by human SMEs is important in guiding subsequent treatment interventions. To correctly evaluate lesions in CE CT, multiple-phase studies have been conducted numerous times, and multiple-phase studies provide medical images of different stages of enhancement of healthy liver parenchyma and comparison with the enhancement of lesions to determine differential detection. Subsequently, human SMEs can determine the diagnosis of lesions based on these differences.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Non-Patent Documents

[0006]

Non-Patent Document 1

Non-Patent Document 2

Non-Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0007] Although some automated image analysis mechanisms have been developed, there is still a need to improve such automated image analysis mechanisms to provide more efficient and accurate analysis of medical image data for detecting lesions within imaged anatomical structures (e.g., the liver or other organs).

[0008] This "Summary of the Invention" is provided to introduce the selection of concepts in a simplified form and is further described in the "Detailed Description of the Invention" herein. This "Summary of the Invention" is not intended to identify the key factors or essential features of the claimed subject matter, nor is it intended to be used in limiting the scope of the claimed subject matter.

[0009] In one exemplary embodiment, in a data processing system comprising at least one processor and at least one memory, a method is provided, wherein the at least one memory comprises instructions for being executed by the at least one processor to implement a trained machine learning computer model for determining the connectivity of lesions in the z - direction. The method includes identifying a first lesion in a given slice within a 3 - D medical image and a second lesion in an adjacent slice within the 3 - D medical image with respect to the given slice. The method further includes determining a first intersection value between the first lesion and the second lesion with respect to the first lesion and determining a second intersection value between the first lesion and the second lesion with respect to the second lesion. The method further includes determining, based on the first intersection value and the second intersection value, whether the first lesion and the second lesion belong to the same 3 - D lesion. This has the advantage of connecting lesions through the three dimensions. The exemplary embodiment provides an algorithm for training a machine learning model to automatically connect lesions in the z - direction in a 3 - D medical image using overlapping values. The machine learning computer model uses two features (the first intersection value and the second intersection value) to accurately determine lesions in adjacent slices that are the same lesion within the 3 - D.

[0010] In one example embodiment, determining the first intersection value includes calculating the first intersection value r0 as follows.

[0011]

Number

[0012] Here, A is the first lesion, B is the second lesion, |A| represents the area of the first lesion, |B| represents the area of the second lesion, and |A∩B| represents the area of the common set of the first lesion and the second lesion. In another embodiment, determining the second intersection value includes calculating the second intersection value r1 as follows.

[0013]

Number

[0014] Here, A is the first lesion, B is the second lesion, |A| represents the area of the first lesion, |B| represents the area of the second lesion, and |A∩B| represents the area of the common set of the first and second lesions. These embodiments provide the advantage of calculating the characteristic values of the first intersection value and the second intersection value for training and applying a machine learning computer model. These embodiments calculate the intersection value of the two lesions with respect to the smaller of the two lesions and calculate the intersection value of the two lesions with respect to the larger of the two lesions, thus providing two characteristic values. Two different overlapping characteristic values are used to realize a more accurate model.

[0015] In another exemplary embodiment, determining whether the first lesion and the second lesion belong to the same three-dimensional lesion includes providing the first intersection value and the second intersection value as features to a trained machine learning computer model. The trained machine learning computer model generates a probability that the first lesion and the second lesion belong to the same three-dimensional lesion. In yet another exemplary embodiment, the trained machine learning computer model is a logistic regression model. In an alternative embodiment, determining whether the first lesion and the second lesion belong to the same three-dimensional lesion includes determining whether the probability that the first lesion and the second lesion belong to the same three-dimensional lesion is greater than a predetermined threshold value t. These embodiments have the advantage of using a logistic regression model that generates a probability that can be compared with a threshold value. Thereby, the threshold value can be adjusted to increase or decrease the sensitivity of the model.

[0016] In yet another exemplary embodiment, the method further includes, for all slices of the three-dimensional medical image, determining whether each first lesion within each slice and each second lesion within each adjacent slice belong to the same three-dimensional lesion, and connecting the lesions belonging to the same three-dimensional lesion within the three-dimensional medical image along the z-axis. This has the advantage of connecting the lesions through three dimensions based on the results of a trained machine learning computer model.

[0017] In another exemplary embodiment, the trained machine learning computer model learns to linearly combine a first intersection value r0 and a second intersection value r1 as follows.

[0018]

Equation

[0019] Here, c0, c1, and b are operation parameters learned from a training volume by a machine learning training operation. This embodiment provides a plurality of operation parameters for optimizing the loss function of the machine learning computer model, and in this way, accurate results are realized. Further, two different overlapping feature values are used to realize a more accurate model.

[0020] In other exemplary embodiments, a computer-usable medium including a computer-readable program or a computer program product comprising a computer-readable medium is provided. When the computer-readable program is executed on a computing device, the computing device is caused to perform various operations and combinations thereof among the operations outlined above with respect to the exemplary embodiments of the method.

[0021] In yet another exemplary embodiment, a system / apparatus is provided. The system / apparatus may comprise one or more processors and a memory coupled to the one or more processors. The memory may contain instructions which, when executed by the one or more processors, cause the one or more processors to perform various operations and combinations thereof among the operations outlined above with respect to the exemplary embodiments of the method.

[0022] These and other features and advantages of the present invention will be described and will become apparent to those skilled in the art in view of the following detailed description of exemplary embodiments of the invention.

Brief Description of the Drawings

[0023] The present invention, as well as its most common uses, other objects, and advantages, will be best understood by reference to the following detailed description of exemplary embodiments, read in conjunction with the accompanying drawings.

[0024]

Fig. 1

Fig. 2

Fig. 3A

Fig. 3B

Fig. 3C

Fig. 4A

Fig. 4B

Fig. 4C

Fig. 5

Fig. 6

Fig. 7

Fig. 8

Fig. 9

Fig. 10A

Fig. 10B

Fig. 10C

Fig. 10D

Fig. 11A

Fig. 11B

Fig. 12

Fig. 13A

Fig. 13B

Fig. 13C

Fig. 14A

Fig. 14B

Fig. 15

Fig. 16

Fig. 17

Fig. 18A

Fig. 18B

Fig. 18C

Fig. 19

Fig. 20

Fig. 21

[0025] The detection of lesions or groups of abnormal cells is, in current drug therapies, mainly a manual process. Since this is a manual process, it involves sources of error due to human limitations regarding the ability of individuals to detect portions of digital medical images that exhibit such lesions, especially considering the large demand for individuals to evaluate an ever - increasing number of images in a shorter time. Although some automated image - analysis mechanisms have been developed, there is still a need to improve such automated image - analysis mechanisms to provide more efficient and accurate analysis of medical - image data for detecting lesions within imaged anatomical structures (e.g., the liver or other organs).

[0026] Exemplary embodiments are particularly directed to improved computing tools that provide automated computer-based artificial intelligence medical image analysis, which analysis, via machine learning / deep learning computer processes, detects anatomical structures, detects biological structures of lesions or other objects within or associated with such anatomical structures, performs special segmentation of the detected lesions or other biological structures, performs false positive removal based on the special segmentation, performs classification of the detected lesions or other biological structures, and provides the results of the detection of the lesions / biological structures to a downstream computing system for performing additional computer operations. The following description of the exemplary embodiments assumes embodiments that are particularly related to the mechanisms of exemplary embodiments that are particularly trained with respect to liver lesions as biological structures of interest, but the exemplary embodiments are not limited to such embodiments. Rather, one of ordinary skill in the art will recognize that the machine learning / deep learning-based artificial intelligence mechanisms of the exemplary embodiments may be implemented with respect to many other types of biological structures / lesions within or associated with other anatomical structures represented in medical image data without departing from the spirit and scope of the present invention. Further, the exemplary embodiments may be described with respect to medical image data that is computer tomography (CT) medical image data, but the exemplary embodiments may be implemented using any digital medical image data from a variety of types of medical imaging techniques including, but not limited to, positron emission tomography (PET) and other nuclear medicine images, ultrasound, magnetic resonance imaging (MRI), elastography, photoacoustic imaging, echocardiography, magnetic particle imaging, functional near-infrared spectroscopy, elastography, fluoroscopy, and various other X-ray imaging including fluorescence imaging.

[0027] As a whole, the exemplary embodiments provide an improved artificial intelligence (AI) computer pipeline that includes a plurality of specially configured and trained AI computer tools (e.g., neural networks, cognitive computing systems, or other AI mechanisms trained based on a finite set of data to perform a particular task). The configured and trained AI computer tools are each specially configured / trained to perform a particular type of artificial intelligence processing of a volume of input medical images, represented as one or more populations of data and / or metadata that define the medical images captured by medical imaging techniques. Generally, these AI tools employ machine learning (ML) / deep learning (DL) computer models (or simply, ML models) to perform tasks, and the ML / DL computer models learn patterns and relationships between data that represent particular results (e.g., image classifications or labels, data values, treatment recommendations, etc.) while emulating human thought processes with respect to the generated results, using different computer processes specific to the computer tools and in particular the ML / DL computer models. The ML / DL computer models are essentially the functions of elements that include machine learning algorithms, configurations of the machine learning algorithms, features of the input data identified by the ML / DL computer models, and labels (or outputs) generated by the ML / DL computer models. By specifically adjusting the functions of these elements through the machine learning process, an instance of a particular ML / DL computer model is generated. Different ML models may be specially configured and trained to perform different AI functions with respect to the same or different input data.

[0028] Since an artificial intelligence (AI) pipeline implements multiple ML / DL computer models, it should be understood that these ML / DL computer models are trained via an ML / DL process for a specific purpose. Therefore, as an overview of the training process of an ML / DL computer model, it should be understood that machine learning is related to the design and development of technologies that receive empirical data (such as medical image data) as input and recognize complex patterns within the input data. One pattern common to machine learning techniques is the use of an underlying computer model M, and based on the input data, the parameters of M are optimized to minimize a cost function associated with M. For example, in the context of classification, the model M could be a straight line that separates the data into two classes (e.g., labels) such that M = a*x + b*y + c and the cost function is the number of points that are misclassified. In that case, the learning process operates by adjusting the parameters a, b, c such that the number of misclassified points is minimized. After this optimization phase (or learning phase), the model M can be used to classify new data points. In many cases, M is a statistical model, and based on the input data, the cost function is inversely proportional to the likelihood of M. This is merely a simple example to provide a general explanation of machine learning training, and other types of machine learning using different patterns, cost (or loss) functions, and optimizations may be used in conjunction with the mechanisms of the exemplary embodiments without departing from the spirit and scope of the present invention.

[0029] For the purpose of anatomical structure detection, lesion detection, or both (where a lesion is an "abnormality" within medical image data), the learning machine can construct an ML / DL computer model of normal structure representation and detect data points within a medical image that deviate from this normal structure representation ML / DL computer model. For example, a given ML / DL computer model (e.g., a supervised model, an unsupervised model, or a semi-supervised model) may be used to generate an abnormality score and report it to another device, provide a classification output indicating one or more classes to which the input is classified, generate probabilities or scores associated with various classes, etc. Examples of machine learning techniques that may be used to construct and analyze such ML / DL computer models include nearest neighbor (NN) techniques (e.g., k-NN models, replicator NN models, etc.), statistical techniques (e.g., Bayesian networks, etc.), clustering techniques (e.g., k-means, etc.), neural networks (e.g., reservoir networks, artificial neural networks, etc.), support vector machines (SVMs), etc., but are not limited thereto.

[0030] The processor-implemented artificial intelligence (AI) pipeline of the illustrative embodiments typically includes one or both of a machine learning (ML) computer model and a deep learning (DL) computer model. In some cases, one or the other of ML and DL may be used or implemented to achieve a particular result. Conventional machine learning can include or use algorithms such as, in particular, Bayesian decision, regression, decision trees / forests, support vector machines, or neural networks. Deep learning can be based on deep neural networks and can use multiple layers such as convolutional layers. Such DL using hierarchical networks, etc., can be efficient in its implementation and can achieve improved accuracy compared to conventional ML techniques. Conventional ML can generally be distinguished from DL in that the performance of the DL model can exceed the performance of the classical ML model, but the DL model may consume a relatively large amount of processing resources or power resources or both. In the context of the illustrative embodiments, references herein to one or the other of ML and DL can be understood to encompass AI processing in one or both forms.

[0031] Regarding the exemplary embodiments, the ML / DL computer models of the AI pipeline are configured and executed after the ML / DL training process to perform complex computer-aided medical image analysis to detect anatomical structures within the input medical images, target biological structures of interest (hereinafter assumed to be liver lesions for the purposes of the description of the exemplary embodiments), their classification, contours that specify the locations where these target biological structures of interest (e.g., liver lesions) are present within the input medical images (hereinafter assumed to be CT medical image data), and other information that specifically identifies other information that assists human subject matter experts (SMEs), such as radiologists and physicians, in understanding the patient's medical condition from the perspective of the captured input medical images. Further, these outputs can be provided to other downstream computer systems for performing additional artificial intelligence operations, such as treatment recommendations and other decision-making support actions based on classification, contours, etc.

[0032] First, the artificial intelligence (AI) pipeline of the exemplary embodiments receives an input volume of computed tomography (CT) medical image data and detects which part of the body of the biological entity is shown in the CT medical image data. A "volume" of a medical image is a three-dimensional representation of the internal anatomical structure of a biological entity composed of a stack of two-dimensional slices, which may be individual medical images captured by medical imaging techniques. The stack of slices may be referred to as a "slab" and may differ from the slices themselves in that these stacks represent portions of anatomical structures that have thickness, and the stack of slices or slab generates a three-dimensional representation of the anatomical structure.

[0033] For the purposes of this description, it is assumed that the biological entity is a human, although the invention may operate on medical images of various types of biological entities. For example, in veterinary medicine, the biological entity may be various types of small animals (e.g., pets such as dogs, cats, etc.) or large animals (e.g., horses, cows, or other livestock). In the case of an implementation where the AI pipeline is specifically trained for the detection of liver lesions, the AI pipeline determines whether the input CT medical image data represents a scan of the abdomen that is present within the CT medical image data, and if not, the operation of the AI pipeline with respect to the input CT medical image data ends because it is not directed to the correct part or portion of the human body. There may be different AI pipelines according to exemplary embodiments trained to process input medical images of different parts of the body and different target biological structures, and the input CT medical image may be an input to each of the AI pipelines, or may be routed to the AI pipeline based on the classification of the part or portion of the body shown in the input CT medical image. For example, the classification of the input CT medical image with respect to the part or portion of the body represented in the input CT medical image may be performed first, and then the corresponding trained AI pipeline for processing the input CT medical image may be selected from a plurality of trained AI pipelines of the type described herein. For the purposes of the following description, a single AI pipeline trained to detect liver lesions is described, although this extension to a series of AI pipelines or a population of AI pipelines will be apparent to those skilled in the art in view of this description.

[0034] Assuming that the volume of the input CT medical image includes a medical image of the abdomen of the human body (for the purpose of detecting liver lesions), the processing of the input CT medical image is further performed in two initial stages, which may be performed substantially in parallel with each other, or in sequence, or both, depending on the desired implementation. The two initial stages include a phase classification stage and an anatomical structure detection stage (e.g., a liver detection stage if the AI pipeline is configured to perform liver lesion detection).

[0035] The phase classification stage determines whether the volume of the input CT medical image contains a single imaging phase or multiple imaging phases. A "phase" in a medical image is an indication of the ingestion of a contrast agent. For example, in some medical imaging techniques, a phase may be defined with respect to when a contrast agent that enables the capture of a medical image including capturing the path of the contrast agent is introduced into a biological entity. For example, phases may include a pre-contrast phase, an arterial phase, a portal / venous phase, and a delayed phase, and the medical image is captured at any or all of these phases. Phases are typically related to the timing after injection and the characteristics of the enhancement of structures within the image. Timing information may be considered to "reorder" the possible phases (e.g., the delayed phase is always acquired after the portal phase) and to estimate the possible phases of a given image. Regarding the use of the characteristics of the enhancement of structures within the image, one example of using this type of information to determine a phase is described in the co-pending U.S. Patent Application No. 16 / 926,880, "Method of Determining Contrast Phase of a Computerized Tomography Image," filed on July 13, 2020, by the same applicant. Further, the timing information may be used in conjunction with other information (sampling, reconstruction kernel, etc.) to extract the best representation of each phase (a given acquisition may be reconstructed in multiple ways).

[0036] Based on the timing or characteristics or both of the enhancement, after the images within the input volume are assigned or classified to the corresponding phase, based on the phase classification, it can be determined whether the volume contains images of a single phase (e.g., there is a portal / venous phase but no arterial phase) or an examination of multiple phases (e.g., portal / venous and arterial). If the phase classification indicates that a single phase exists in the volume of the input CT medical image, the processing by the AI pipeline is further executed as described below. If multiple phases are detected, the volume is not further processed by the AI pipeline. However, in some exemplary embodiments, this filter of the volume based on single / multiple phases accepts only volumes containing images from a single phase and rejects volumes of multiple phases, while in other exemplary embodiments, the AI pipeline processing described herein may filter out images of volumes that are not classified into the target phase of interest, e.g., while filtering out images of volumes not classified as part of the portal / venous phase, the images of the portal / venous phase within the volume may be maintained, thereby modifying the input volume to be a modified volume containing only a subset of the images classified into the target phase. Further, as described above, different AI pipelines may be trained for different types of volumes, and in some exemplary embodiments, the phase classification of the images within the input volume may be used to route or distribute the images of the input volume to the corresponding AI pipelines trained and configured to process images of different phases, thereby enabling the input volume to be further divided into sub-volumes that are components and routed to the corresponding AI pipelines for processing, e.g., for processing, a first sub-volume corresponding to the images of the portal / venous phase is sent to the first AI pipeline, and a second sub-volume corresponding to the arterial phase is sent to the second AI pipeline.If the volume of the input CT medical image contains a single phase, or if the AI pipeline processes the image of a single-phase input volume or sub-volume, after filtering the sub-volume and optionally routing it to the corresponding AI pipeline, the volume (or sub-volume) is passed to the next stage of the AI pipeline for further processing.

[0037] The second initial stage is the detection stage of the target anatomical structure (in this embodiment, the liver). In this stage, the part of the volume showing the target anatomical structure is identified and passed to the next downstream stage of the AI pipeline. The detection stage of the target anatomical structure (hereinafter referred to as the liver detection stage according to the embodiment) includes a machine learning (ML) / deep learning (DL) computer model that is specifically trained and configured to perform computer medical image analysis to identify the part of the input medical image corresponding to the target anatomical structure (e.g., the liver). Such medical image analysis may include training the ML / DL model on labeled training medical image data as input to determine whether the input medical image (during training, the training image) contains the target anatomical structure (e.g., the liver). Based on the ground truth of the image labels, the operating parameters of the ML / DL model may be adjusted to reduce the loss or error in the results generated by the ML / DL model until convergence is reached (i.e., the loss is minimized). Through this process, the ML / DL model is trained to recognize the pattern of medical image data indicating the presence of the target anatomical structure (the liver in this example). After being trained, when new input medical image data is available, the ML / DL model can be run on the new input data to determine whether it contains a pattern indicating the presence of the anatomical structure. If the probability is greater than a predetermined threshold, it can be determined that the medical image data contains the target anatomical structure.

[0038] Therefore, at the liver detection stage, the AI pipeline uses a trained ML / DL computer model to determine whether the volume of the input CT medical image contains an image indicating the liver. The portion of the volume indicating the liver is passed to the decision-making stage of the AI pipeline along with the results of the phase classification stage. The decision-making stage determines whether there is a single phase in the medical image and whether at least a predetermined amount of the target anatomical structure is present in the portion of the volume indicating the target anatomical structure (e.g., the liver). The determination of whether at least a predetermined amount of the target anatomical structure is present may be determined based on known measurement mechanisms that determine the measurements of the structure from the medical image (e.g., calculating the size of the structure from the differences in the positions of the pixels in the image). If the measurements represent at least a predetermined amount or portion of the anatomical structure, the measurements may be compared to the predetermined size (e.g., the average size) of the anatomical structure of similar patients with similar demographics so that further processing can be performed by the AI pipeline. In one exemplary embodiment, this determination determines, for example, whether at least 1 / 3 of the liver is present in the portion of the volume of the input CT medical image determined to indicate the liver. Although 1 / 3 is used in this exemplary embodiment, any predetermined amount of the structure determined to be suitable for a particular implementation may be used without departing from the spirit and scope of the present invention.

[0039] In one exemplary embodiment, to determine whether a predefined amount of a target anatomical structure exists in the volume of the input CT medical image, a slice corresponding to a medical image within a volume containing a first representation of the target anatomical structure (e.g., the liver), i.e., the first slice containing the liver (FSL), is given a slice score of 0, and the last slice containing the liver (LSL) has a score of 1, such that an axial score is defined. Assuming a human biological entity, moving from the lowest slice in the volume (MISV) (closest to the lower extremities (e.g., feet)) to the highest slice in the volume (MSSV) (closest to the head), the first and last slices are defined. A pair of slice scores (s sup and s inf ) corresponding to the slice scores of the MSSV slice and the MISV slice, respectively, defines a liver axial score estimate (LAE). The ML / DL computer model is specifically configured and trained to determine the slice scores s sup and s inf of the volume of the input CT medical image, as will be described in more detail later. The mechanism of the exemplary embodiment recognizes these slice scores and, from the above definition, recognizes that the liver extends from 0 to 1 and can determine a fragment of the liver within the field of view of the volume of the input CT medical image.

[0040] In some exemplary embodiments, the slice scores s sup and s inf first divide the volume of the input CT medical image into sections, and then determine the highest slice (closest to the head) and the lowest slice (closest to the feet) s' sup and s' infTo determine, for each section, the ML / DL computer model configured and trained may be run against a slice of the section to indirectly detect by estimating the height for each slice. s' sup And s' inf Given the estimated values of and s', since it is known how the sections are positioned with respect to the entire volume of the input CT medical image, s sup And s inf are detected by extrapolation. This approach is based on a robust estimator of the height of any slice from the input volume (or sub-volume associated with the target phase). Such an estimator can be obtained, for example, by learning a regression model using a deep learning model that performs an estimation of the height from a chunk (set of consecutive slices). For example, a long short-term memory (LSTM) type artificial neural network model is suitable for these tasks and has the ability to encode the ordering of slices containing the liver and abdominal biological structures. For each volume, note that there are n estimated values of s sup And s inf , where n is the number of sections per volume. In one exemplary embodiment, the final estimated value is obtained by taking the unweighted average of these n estimated values, but in other exemplary embodiments, other functions of the n estimated values may be used to generate the final estimated value.

[0041] s of the volume of the input CT medical image sup And s infDetermine the final estimated values, and based on these values, fragments of the target anatomical structure (e.g., the liver) are calculated. This task is enabled by estimating the height of each slice. Assuming that from the estimation of the height (h1) of the first slice of the liver and the height (h2) of the last slice in the input volume, the actual height of the first slice and the last slice of the liver are H1 and H2 (regardless of whether they are included in the input volume), the visible part of the liver in the input volume can be expressed as (min(h1, H1)-max(h2, H2)) / (H1-H2). Next, this calculated fragment may be compared with a predefined threshold to determine whether a predefined minimum amount of the target anatomical structure is present in the volume of the input CT medical image (e.g., at least 1 / 3 of the liver is present in the volume of the input CT medical image).

[0042] If this determination ends with the determination that there are multiple phases, or that a predefined amount of the target anatomical structure is not present in the portion of the volume of the input CT medical image showing the anatomical structure, or both, further processing of the volume may be interrupted. If this determination ends with the determination that the volume of the input CT medical image contains a single phase and at least a predefined amount of the target anatomical structure (e.g., 1 / 3 of the liver is shown in the image), the portion of the volume of the input CT medical image showing the anatomical structure is transferred to the next stage of the AI pipeline for processing.

[0043] In the next stage of the AI pipeline, the AI pipeline performs lesion detection on a portion of the volume of the input CT medical image that represents the anatomical structure of interest (e.g., the liver). This liver and lesion detection stage of the AI pipeline uses a collection of ML / DL computer models to detect the liver and lesions within the liver represented in the volume of the input CT medical image. The collection of ML / DL computer models performs liver and lesion detection using ML / DL computer models that are differently trained, and the ML / DL computer models are trained using a loss function to balance false positives and false negatives in lesion detection. Further, the collection of ML / DL computer models of the group is configured such that a third loss function forces the outputs of the ML / DL computer models to match each other.

[0044] Assuming that liver detection and lesion detection are performed at this stage of the AI pipeline, a first ML / DL computer model is executed on the volume of input CT medical images to detect the presence of the liver. This ML / DL computer model may be the same as the ML / DL computer model employed in the stage of the AI pipeline prior to the detection of the anatomical structure of interest, and thus, previously obtained results may be utilized. A plurality (two or more) of other ML / DL computer models are configured and trained to perform lesion detection on the portion of the medical image showing the liver. The first ML / DL computer model is configured using two loss functions. The first loss function penalizes the error in false negatives (i.e., classifications that incorrectly indicate the absence of a lesion (normal anatomical structure)). The second loss function penalizes the error in false positive results (i.e., classifications that incorrectly indicate the presence of a lesion (abnormal anatomical structure)). The second ML / DL is trained to detect lesions using an adaptive loss function that penalizes the false positive error within slices of the liver containing normal tissue and penalizes the false negative error within slices of the liver containing lesions. The detections output from the two ML / DL models are averaged to produce the final lesion detection.

[0045] The results of the liver / lesion detection stage of the AI pipeline include one or more contours (contour lines) of the liver and a detection map (e.g., a per-voxel map of liver lesions detected within the volume of an input CT medical image) that identifies portions of the medical image data elements corresponding to the detected lesions. Next, the image map is input into the lesion segmentation stage of the AI pipeline. The lesion segmentation stage uses watershed techniques to split the detection map, as will be described in more detail later, and generates a segmentation of the image elements (e.g., voxels) of the input CT medical image. The liver lesion segmentation stage identifies all of the contours corresponding to lesions present in the slices of the volume of the input CT medical image based on this segmentation and performs operations to identify which contours in 3D correspond to the same lesion. Lesion segmentation aggregates the mutually associated lesion contours and generates a 3D segmentation of the lesions. Lesion segmentation focuses individually on each lesion and uses the lesion image elements (e.g., voxels) represented in the medical image and the impainting of tissues other than the liver to perform active contour analysis. In this way, individual lesions can be identified and processed without biasing the analysis due to other lesions in the medical image or due to portions of the image outside the liver.

[0046] The result of lesion segmentation is a list of lesions that includes the corresponding contour lines or contours within the volume of the input CT medical image. These outputs may include detection results that are not actual lesions. To minimize the impact of these false positives, these outputs are provided to the next stage of the AI pipeline that targets false positive removal using a trained false positive removal model. The false positive removal model of the AI pipeline functions as a classifier to identify which outputs are actual lesions and which detection results are false positives. The input consists of a volume of images (VOI) centered on the detected detection results, associated with a mask obtained from the refinement of lesion segmentation. The false positive removal model is trained using data that is the result of the detection / segmentation stage. Objects that are lesions from the ground truth detected by the detection algorithm are used to represent the lesion class during training, and detections that do not match any lesions from the ground truth are used to represent the non-lesion (false positive) class.

[0047] To further improve the overall performance, a dual operating point strategy was adopted in the lesion detection model and the false positive model. This idea focuses on the fact that the output of the AI pipeline can be interpreted at different levels. First, the output of the AI pipeline can be used to convey whether the examination volume (i.e., the input volume or volume of images (VOI)) contains a lesion. Second, the output of the AI pipeline aims to maximize the detection of lesions regardless of whether the lesions are included in the same patient / examination / volume. To clarify, the measurements taken for the examination are referred to as "patient level" in this specification, and the measurements taken for the lesions are referred to as "lesion level" in this specification. Maximizing the sensitivity at the "lesion level" results in a decrease in the specificity at the "patient level" (since only one detection is sufficient for a patient to be said to contain a lesion). This ultimately may be sub-optimal for clinical use as it is necessary to choose between having insufficient specificity at the patient level or low sensitivity at the lesion level.

[0048] Taking this into account, the exemplary embodiments use a dual operating point approach for both lesion detection and false positive removal. This principle is to first perform processing using a first operating point that gives reasonable performance at the patient level. Next, for patients having at least one lesion detected from the first execution, a second operating point is used to reinterpret / process the detected lesions. This second operating point is selected to be more sensitive. This second operating point has a lower specificity than the first operating point, but this loss of specificity is included at the patient level because all patients in which no lesion was detected using the first operating point are kept as they are regardless of whether the second operating point would have detected additional lesions. Thus, the patient level specificity is determined by the first operating point only. The patient level sensitivity is between the sensitivity of the first operating point selected alone and the sensitivity of the second operating point (certain false negative cases from the first operating point may be changed to true positives by the second operating point). At the lesion side, the actual lesion level sensitivity is improved compared to using only the first operating point. When processed using only the first operating point, no false positives occur, so the lesion specificity is better than that of the second operating point with a low specificity selected alone.

[0049] The exemplary embodiments assume a particular configuration and the use of a dual operating point approach, but it should be understood that the dual operating point approach can be used in other configurations and for other purposes when there is an interest in measuring performance at the group level (in the exemplary embodiments, this group level is the "patient level") and the element level (in the exemplary embodiments, this element level is the "lesion level"). In the exemplary embodiments, the dual operating point approach is applied to both lesion detection and false positive removal, but it can be understood that the dual operating point approach can be extended beyond these stages of the AI pipeline. For example, lesion detection may be performed at the voxel level (element) and volume level (group) instead of the patient level and lesion level. As another example, the voxel level or lesion level may be used as the element level, and a slab (a set of slices) may be used as the group level. In yet another example, instead of a single volume, all volumes of the examination may be used as the group level. It should be understood that this approach may be applied to 2D images (e.g., 2D x-ray images such as chest, mammography) for analyzing images rather than 3D volumes. Specificity such as the average number of false positives per patient / group can be used in the selection of the operating point. Further, the exemplary embodiments are described as being applicable to lesion detection and classification, but the dual operating point-based approach may be applied to other structures (clips, stents, implants, etc.) and beyond medical images.

[0050] The results of the detection and false positive removal based on the dual operating points lead to the identification of a final list of filtered lesions that are further processed by the lesion classification stage of the AI pipeline. In the lesion classification stage of the AI pipeline, a configured and trained ML / DL computer model is executed against a list of lesions and corresponding contour data, thereby classifying the lesions into one of a plurality of predefined lesion classifications. For example, each lesion and its attributes (e.g., contour data) within the final list of filtered lesions may be input into a trained ML / DL computer model, and then the trained ML / DL computer model operates on this data and classifies the lesion as a particular type of lesion. This classification can be performed by using a classifier (e.g., a trained neural network computer model) previously trained against ground truth data in combination with the results of previous processing steps of the AI pipeline. This classification task can be more or less complex and may, for example, involve providing labels such as benign, malignant, or indeterminate, or in another example, providing the actual types of lesions such as cysts, metastases, hemangiomas, etc. The classifier can be, for example, a classifier of a computer model based on a neural network (e.g., SVM, decision tree, etc.) or a deep learning computer model. The actual input to this classifier is the lesion-centered portion, and in some embodiments, it may be augmented using a lesion mask or a contour line (contour).

[0051] After classifying the lesions according to the lesion classification stage of the AI pipeline, the AI pipeline outputs a list of the lesions and their classifications, along with any contour attributes of the lesions. Additionally, the AI pipeline may output liver contour information of the liver. The information generated by this AI pipeline may be provided to a further downstream computing system for further processing and generation of the representation of the subject's anatomical structure and any detected lesions present in the anatomical structure. For example, a graphic representation of the volume of the input CT medical image may be generated in a medical image viewer or other computer application, and the anatomical structure and detected lesions may be overlaid on the graphic representation or otherwise highlighted using the contour information generated by the AI pipeline. In other exemplary embodiments, the downstream processing of the information generated by the AI pipeline may include diagnostic decision support operations, automated medical image report generation, based on the detected list of lesions, classifications, and contours. In other exemplary embodiments, different treatment recommendations may be generated for review and consideration by a physician based on the classification of the lesions.

[0052] In some exemplary embodiments, the list of lesions, their classifications, and contours may be stored in a historical data structure associated with a patient that corresponds to the volume of the input CT medical image, storing multiple executions of the AI pipeline for different volumes of the input CT medical image associated with that patient, allowing for evaluation over time. For example, differences between the list of lesions or their associated classifications or both and the contours may be determined to evaluate the progression of a patient's illness or medical condition and present such information to medical experts to assist in the patient's treatment.

[0053] Other downstream computing systems and processes of the specific anatomical structures and lesion detection information generated by the AI mechanism of the exemplary embodiments may be implemented without departing from the spirit and scope of the present invention. For example, the output of the AI pipeline may be used by another downstream computing system that processes the anatomical structures and lesion information within the output of the AI pipeline to identify discrepancies with other sources of information (e.g., radiology reports) in order to alert clinical staff to potentially missed detection results.

[0054] Accordingly, an exemplary embodiment provides a mechanism for implementing a plurality of configured and trained ML / DL computer models that perform various artificial intelligence operations at various stages of an AI pipeline to identify anatomical structures within a volume of an input medical image and lesions associated with these anatomical structures, to determine contours associated with such anatomical structures and lesions, to determine a classification of such lesions, and to generate a list of such lesions and contours of the lesions and anatomical structures for further downstream computer processing of information generated by the AI from the AI pipeline. The operation of the AI pipeline is automated so that there is no human intervention at any stage of the AI pipeline. Instead, particularly configured and trained ML / DL computer models trained via a machine learning / deep learning computer process are employed to perform specific AI analyses at various stages. The only times when there may be human intervention are before the input of the volume of the input medical image (e.g., during the imaging of a patient's medical image) and after the output of the AI pipeline (e.g., when displaying an enhanced medical image presented via a computer image display application based on the output of the list of lesions and contours generated by the AI pipeline). Accordingly, the AI pipeline is particularly related to an improved automated computer tool implemented as artificial intelligence using specific machine learning / deep learning processes that exist only within a computer environment, performs operations that cannot be performed by humans as an intellectual process, and does not orchestrate any human activities.

[0055] Before continuing with the description of the exemplary embodiments and various aspects of the improved computer operations performed by the exemplary embodiments, it should first be understood that throughout this description, the term "mechanism" is used to refer to an element of the present invention that performs various operations, functions, etc. A "mechanism" may be an implementation of a function or aspect of an exemplary embodiment in the form of an apparatus, a procedure, or a computer program product. In the case of a procedure, the procedure is implemented by one or more devices, apparatuses, computers, data processing systems, etc. In the case of a computer program product, the logic represented by computer code or instructions embodied within or on the computer program product is executed by one or more hardware devices to implement the functions associated with a particular "mechanism" or to perform the operations associated with a particular "mechanism". Thus, the mechanisms described herein may be implemented as special hardware, software executed on the hardware to configure the hardware to implement special functions of the present invention that the hardware could not otherwise perform, software instructions stored on a medium to enable the instructions to be readily executed by the hardware and thereby configure the hardware to perform the functions shown and the particular computer operations described herein, procedures or methods for performing functions, or any combination thereof.

[0056] In this specification and the claims, the terms "one," "at least one of," and "one or more of" may be used with respect to particular features and elements of the exemplary embodiments. It should be understood that these terms and phrases are intended to state that there is at least one of the particular features or elements present in a particular exemplary embodiment, but there may also be more than one. That is, these terms / phrases are not intended to limit this specification or the claims to a single feature / element that exists, nor are they intended to require that such a feature / element exist in multiple numbers. On the contrary, these terms / phrases require only at least a single feature / element, and such a feature / element may exist in multiple numbers within this specification and the claims.

[0057] Furthermore, when the term "engine" is used herein with respect to describing the embodiments and features of the present invention, it should be understood that it is not intended to limit any particular implementation for realizing, executing, or both, operations, steps, processes, etc. that are caused by or performed by or both the engine. The engine may include, but is not limited to, any use of a general processor or a special processor or both, combined with appropriate software that is loaded into or stored in a machine-readable memory and executed by the processor, and may be software, hardware, or firmware, or a combination thereof, that performs the specified function. Furthermore, all names associated with a particular engine are for reference convenience only and are not intended to limit a particular implementation, unless otherwise specified. Additionally, any function resulting from an engine may be incorporated into, combined with, or both, the function of another engine of the same or different type, or may be distributed across one or more engines of various configurations and executed in the same manner by multiple engines.

[0058] In addition, in the following description, it should be understood that various examples of the various elements of the exemplary embodiments are used to further illustrate the exemplary implementation of the exemplary embodiments and to assist in the understanding of the mechanisms of the exemplary embodiments. These examples are intended to be non-limiting and do not cover all the various possibilities of implementing the mechanisms of the exemplary embodiments. Considering this description, it will be apparent to those skilled in the art that there are many other alternative implementations that can be used in addition to or in place of the examples provided herein without departing from the spirit and scope of the present invention with respect to these various elements.

[0059] The present invention may be a system, a method, or a computer program product, or a combination thereof. The computer program product may include one or more computer-readable storage media including computer-readable program instructions for causing a processor to execute aspects of the present invention.

[0060] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. The computer-readable storage medium can be, for example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof, but is not limited thereto. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory, static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or raised structures in grooves in which instructions are recorded, and any suitable combination thereof. As used herein, a computer-readable storage medium should not be construed to be a transient signal such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., an optical pulse passing through an optical fiber cable), or an electrical signal transmitted via a wire.

[0061] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof). This network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface within each computing / processing device receives the computer-readable program instructions from the network and transfers them for storage on a computer-readable storage medium within each computing / processing device.

[0062] Computer-readable program instructions for executing the operation of the present invention may be in any combination of one or more programming languages, including assembly instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or object-oriented programming languages such as Java (registered trademark), Smalltalk (registered trademark), C++, and conventional procedural programming languages such as the "C" programming language or similar programming languages, either in source code or object code. The computer-readable program instructions may be executed entirely on the user's computer, partially executed as a stand-alone software package on the user's computer, partially executed on the user's computer and a remote computer respectively, or executed entirely on a remote computer or a server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, to execute aspects of the present invention, an electronic circuit, including, for example, a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA), may utilize the state information of the computer-readable program instructions to execute the computer-readable program instructions by customizing the electronic circuit.

[0063] Aspects of the invention will be described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0064] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions may be stored in a computer-readable storage medium that includes instructions for causing a computer, programmable data processing apparatus, or other device to function in a particular manner so that the product includes instructions which implement the aspects of the functions / acts specified in one or more blocks of the flowchart and / or block diagram.

[0065] These computer-readable program instructions may be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to implement the functions / acts specified in one or more blocks of the flowchart and / or block diagram.

[0066] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, depending on the functionality involved, or may sometimes be executed in the reverse order. It should also be noted that each block of the block diagrams or flowchart diagrams, or combinations of blocks in the block diagrams or flowchart diagrams, or both, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or by a combination of dedicated hardware and computer instructions.

[0067] [Overview of the Lesion Detection and Classification AI Pipeline]

[0068] FIG. 1 is an exemplary block diagram of a lesion detection and classification artificial intelligence (AI) pipeline (hereinafter simply referred to as the "AI pipeline") that implements a plurality of ML / DL computer models specifically configured and trained to perform identification of anatomical structures and lesion detection within input medical image data. For illustrative purposes only, the AI pipeline shown is specifically described as being directed to liver detection and liver lesion detection within medical image data. As described above, the exemplary embodiments are not limited to such examples and may be applied to any object's anatomical structure and lesions associated with such object's anatomical structure, which may be represented by image elements of medical image data captured by medical imaging technology and corresponding computing systems. For example, the mechanisms of the exemplary embodiments may be applied to the detection, contour identification, classification, etc. of other anatomical structures such as the lungs, heart, and lesions associated with the lungs, heart, or other object's anatomical structures.

[0069] Furthermore, it should be understood that the following description explains the overview of the AI pipeline from the level shown in FIG. 1, and the sections after this description explain additional details regarding the individual stages of the AI pipeline. Each of the stages of the AI pipeline is implemented as a configured and trained ML / DL computer model, such as a neural network of a deep learning neural network, as represented by the symbol 103 at various stages of the AI pipeline 100 in some exemplary embodiments. These different ML / DL computer models are specifically configured and trained to perform certain AI operations described herein, such as body part identification, liver detection, phase classification, minimum liver detection, liver / lesion detection, lesion segmentation, false positive removal, lesion classification, and the like. These additional sections of the following description show specific embodiments for implementing the various stages of the AI pipeline that provide new technologies, mechanisms, and methods for performing AI operations at various stages, but it should be understood that other equivalent technologies, mechanisms, or methods may be used without departing from the spirit and scope of the exemplary embodiments in the context of the overall AI pipeline. These other equivalent technologies, mechanisms, or methods will be apparent to those skilled in the art in view of this description and are intended to be within the spirit and scope of the present invention.

[0070] As shown in FIG. 1, an artificial intelligence (AI) pipeline 100, according to one exemplary embodiment, receives as input a volume 105 of input medical images, which in the example shown is a volume of input computed tomography (CT) medical images represented as one or more data structures, and then this volume is automatically processed by various stages of the AI pipeline 100 and ultimately generates an output 170 that includes a list of lesions along with classification and contour information and contour information regarding the anatomical structure of the object (e.g., the liver in the example shown). The volume 105 of input medical images may be captured by medical imaging technology 102 using any of a number of well-known or later-developed medical imaging techniques and devices that depict images of the internal anatomical structures of a biological entity (i.e., a patient) as one or more medical image data structures. In some exemplary embodiments, this volume 105 of input medical images includes two-dimensional slices (individual medical images) of portions of the anatomical structure of a portion of the patient's body, and then these slices are combined to generate slabs (a combination of slices along an axis to provide a population of medical images having a thickness along the axis), and these slabs are combined to generate a three-dimensional representation (i.e., a volume) of the anatomical structure of the portion of the body.

[0071] In the logic 110 of the first stage of the AI pipeline 100, the AI pipeline 100 determines (112) the part of the patient's body corresponding to the input volume 105 of CT medical image data, and the target body part determination logic 114 determines whether this part of the patient's body represents the part of the patient's body corresponding to the target anatomical structure (e.g., it is an abdominal scan rather than a cranial scan, a lower body scan, etc.). This evaluation serves as an initial filter for the use of the AI pipeline 100 with respect to only the volume 105 of the input CT medical image data (hereinafter referred to as the "input volume" 105). The AI pipeline 100 is specifically configured and trained to perform the identification and contouring of anatomical structures, as well as the identification, contouring, and classification of lesions, on the volume 105 of the input CT medical image data. This detection of the body part represented within the input volume 105 may refer to a field specifying the scanned region of the patient's body, which may be specified by the source medical image technology computing system 102 when performing the medical image scan, and may refer to the metadata associated with the input volume 105. Alternatively, the logic 110 of the first stage of the AI pipeline 100 may implement an ML / DL computer model specifically configured and trained for body part detection 112 that performs medical image classification for a specific part of the patient's body. This medical image classification performs computerized pattern analysis on the medical image data of the input volume 105 and predicts the classification of the medical image data with respect to one or more predefined parts of the patient's body. In some exemplary embodiments, this evaluation may be binary (e.g., whether it is a medical image volume of the abdomen or not), or may be a more complex multi-class evaluation that specifically identifies probabilities or scores with respect to the classification of, for example, multiple different body parts (e.g., abdomen, cranium, lower limbs, etc.).

[0072] If the body part determination logic 114 of the first-stage logic 110 of the AI pipeline 100 determines that the input volume 105 does not represent a part of the patient's body where the target anatomical structure can be detected (e.g., the abdomen of the body where the liver can be detected), the processing of the AI pipeline 100 may be interrupted (a rejection case). If the body part determination logic 114 of the first-stage logic 110 of the AI pipeline 100 determines that the input volume 105 represents a part of the patient's body where the target anatomical structure can be detected, then, as will be described later, the processing of the input volume 105 by the AI pipeline 100 is further executed. In some exemplary embodiments, it should be understood that a plurality of different instances of the AI pipeline 100 may be provided, each configured and trained to process input volumes 105 corresponding to different anatomical structures that may be present in different parts of the patient's body. Thus, the first-stage logic 110 may be provided external to the AI pipeline 100 and may act as routing logic for routing the input volume 105 to the corresponding AI pipeline 100 that is specifically configured and trained to process input volumes 105 of a particular classification. For example, one instance of an AI pipeline may be for the detection / classification of the liver and liver lesions, another instance of an AI pipeline may be for the detection / classification of the lungs and lung lesions, and a third instance of an AI pipeline may be for the detection / classification of the heart and heart lesions, etc. Thus, the first-stage logic 110 may include routing logic, which stores a mapping of which instance 100 of the AI pipeline corresponds to different body parts / target anatomical structures and, based on the detection of the body part represented in the input volume 105, can automatically route the input volume 105 to the corresponding instance 100 of the AI pipeline that is specifically configured and trained to process input volumes 105 corresponding to the detected body part.

[0073] Assuming that it is detected that the input volume 105 represents a body part of a patient in which the target anatomical structure is present (e.g., for the purpose of detecting liver lesions, an abdominal scan is present in the input volume 105), in the second stage of logic 120, the input volume 105 is further processed by the AI pipeline 100. This second stage of logic 120 includes two initial sub - stages 122 and 124 that may be executed substantially in parallel with each other, or sequentially, or both, depending on the desired implementation (in FIG. 1, parallel execution is shown as an example). The two initial sub - stages 122, 124 include a phase classification sub - stage 122 and an anatomical structure detection sub - stage 124 (e.g., a liver detection sub - stage 124 if the AI pipeline 100 is configured to perform liver lesion detection).

[0074] The phase classification sub - stage 122 determines whether the input volume 105 contains a single imaging phase (e.g., pre - contrast phase, arterial phase, portal / venous phase, delayed phase, etc.). Again, the phase classification sub - stage 122 may be implemented as logic that evaluates metadata associated with the input volume 105 that may include a field specifying the phase of a medical image study that matches the medical image, which may be generated by the medical imaging technology computing system 102 when performing the imaging of the medical image. Alternatively, the exemplary embodiment may implement an ML / DL computer model that is specifically trained and configured to detect patterns of medical images indicating different phases of a medical image study, whereby these medical images of the input volume 105 may be classified as to which phase they correspond to. The output of the phase classification sub - stage 122 may be binary indicating whether the input volume 105 contains one or more phases, or may be the classification of each phase represented within the input volume 105, and may then be used to determine whether a single phase or multiple phases are represented.

[0075] If the phase classification indicates that a single phase exists in the input volume 105, then as will be described later, the AI pipeline 100 further performs processing through the downstream stages 130-170. If multiple phases are detected, the input volume 105 may not be further processed by the AI pipeline 100, or as described above, it may be filtered into sub-volumes, or divided, or both, and each sub-volume may contain only one sub-volume corresponding to the target phase that is processed by the AI pipeline 100, or the sub-volume may be routed to a corresponding AI pipeline configured and trained to process the input volume of images corresponding to a specific phase classification, or both may be performed, and contains an image of a corresponding single phase. It should be understood that the input volume may be rejected for several reasons (e.g., the liver does not exist in the image, it is not a single-phase input volume, not enough liver exists in the image, etc.). Depending on the actual root cause of the rejection, the cause of the rejection may be communicated to the user via, for example, a user interface. For example, the output of the AI pipeline 100 in response to the rejection may indicate the reason for the rejection and may be utilized by a downstream computing system (e.g., a viewer or additional automated processing system) to communicate the reason for the rejection via the output. For example, if the liver is not detected in the input volume, the input volume may be silently ignored without communicating the rejection to the user, while in the case of an input volume that contains the liver but contains an input volume of multiple phases, in the user interface generated by the downstream computing system of the viewer, for example, to avoid being mistaken for an input volume that does not contain any detection results due to the input volume containing images of two or more phases, the rejection may be communicated to the user (e.g., a radiologist) by clearly stating that the input volume was not processed by the AI pipeline 100.

[0076] The second initial lower stage 124 is a detection lower stage for detecting a target anatomical structure (in this embodiment, the liver) in a portion of the input volume 105. That is, slices, slabs, etc. within the input volume 105 that particularly show the target anatomical structure (liver) are identified and evaluated to determine whether a predetermined minimum amount of the target anatomical structure (liver) is present in these slices, slabs, or the input volume as a whole. As described above, the detection lower stage 124 includes an ML / DL computer model 125 that is specifically trained and configured to perform computer medical image analysis to identify portions of the input medical image corresponding to the target anatomical structure (e.g., the human liver).

[0077] Accordingly, in the liver detection lower stage 124, the AI pipeline 100 uses the trained ML / DL computer model 125 to determine whether the volume of the input CT medical image contains an image showing the liver. The portion of the volume showing the liver, along with the result of the phase classification lower stage 122, is passed to a decision lower stage 126 that includes the single-phase decision logic 127 and the structural minimum amount decision logic 128 of the AI pipeline 100. The phase decision logic 127 determines whether there is a single phase of the medical image, and the structural minimum amount decision logic 128 determines whether at least a predetermined amount of the target anatomical structure is present in the portion of the volume showing the target anatomical structure (e.g., the liver). As described above, the determination of whether a predetermined amount of the target anatomical structure is present may be determined based on known measurement mechanisms that calculate the size of the structure from, for example, differences in pixel positions in the image, determine measurement values of the structure from the medical image, compare these measurement values with one or more predetermined thresholds, and determine whether a minimum amount of the target anatomical structure (e.g., the liver) is present in the input volume 105 (e.g., whether one-third of the liver is present in the portion of the input volume 105 determined to show the liver).

[0078] In one exemplary embodiment, the axial score mechanism described above may be used to determine whether a predetermined amount of a target anatomical structure (liver) is present in the input volume 105, and to evaluate the portion of the anatomical structure present in the input volume 105. As described above, the ML / DL computer model has slice scores s sup and s inf respectively corresponding to the slice scores of the MSSV slices and MISV slices of the input volume 105, and may be configured and trained to estimate them. In some exemplary embodiments, the slice scores s sup and s inf may be indirectly detected by first dividing the input volume 105 into sections, and then, for each section, running the configured and trained ML / DL computer model against the slices of the section to estimate the slice scores s' sup and s' inf of the first and last slices within the section. Given the estimated values of s' sup and s' inf , and knowing how the sections are positioned with respect to the entire volume of the input CT medical image, the estimated values of s sup and s inf are detected by extrapolation. Note that for each input volume 105, there are n estimated values of s sup and s inf , where n is the number of sections per volume. In one exemplary embodiment, the final estimated value is obtained by taking the unweighted average of these n estimated values, although in other exemplary embodiments, other functions of the n estimated values may be used to generate the final estimated value.

[0079] The s sup and s infDetermine the final estimated values, and based on these values, calculate a fragment of the target anatomical structure (e.g., the liver). Next, this calculated fragment may be compared with a predetermined threshold to determine whether a predetermined minimum amount of the target anatomical structure is present in the volume of the input CT medical image (e.g., at least 1 / 3 of the liver is present in the volume of the input CT medical image).

[0080] If the determinations of decision logics 127 and 128 indicate that there are multiple phases, or that a predetermined amount of the target anatomical structure is not present in the portion of the input volume 105 showing the liver, or both, further processing of the input volume 105 by the AI pipeline 100 regarding stages 130-170 may be interrupted (i.e., the input volume 105 is rejected). If the determinations of decision logics 127 and 128 end with the determination that the input volume 105 contains a single-phase image and shows at least a predetermined amount of the liver, the portion of the input volume 105 showing the anatomical structure is transferred to the next stage 130 of the AI pipeline 100 for processing. This exemplary embodiment further transfers the lower portion of the input volume containing the liver for further processing, but in other exemplary embodiments, the background around the liver may be provided, which can be performed by adding a predetermined amount of margin above and below the selected liver region. Depending on the amount of background required by subsequent processing operations, the margin can be increased up to the entire extent of the original input volume.

[0081] At the next stage 130 of the AI pipeline 100, the AI pipeline 100 performs lesion detection on a portion of the input volume 105 that represents a target anatomical structure (e.g., the liver). This liver / lesion detection stage 130 of the AI pipeline 100 uses a population of ML / DL computer models 132-136 to detect the liver and lesions within the liver represented within the input volume 105. The population of ML / DL computer models 132-136 performs liver and lesion detection using differently trained ML / DL computer models 132-136, and the ML / DL computer models 132-136 are trained using a loss function to balance false positives and false negatives in lesion detection. Further, the population of ML / DL computer models 132-136 is configured such that a third loss function forces the outputs of the ML / DL computer models 132-136 to agree with each other.

[0082] In one exemplary embodiment, the configured and trained ML / DL computer model 132 is executed against the input volume 105 to detect the presence of the liver. This ML / DL computer model 132 may be the same as the ML / DL computer model 125 employed in the previous stage 120 of the AI pipeline, and thus, the previously obtained results may be utilized. A plurality (two or more) of other ML / DL computer models 134 - 136 are configured and trained to perform lesion detection in the portion of the medical image of the input volume 105 that shows the liver. The first ML / DL computer model 134 is configured and trained to operate directly on the input volume 105 to generate a lesion prediction. The second ML / DL computer model 136 is configured using two different decoders that implement two different loss functions, one loss function penalizing errors in false negatives (i.e., classifications that incorrectly indicate the absence of a lesion (normal anatomical structure)), and the other loss function penalizing errors in false positive results (i.e., classifications that incorrectly indicate the presence of a lesion (abnormal anatomical structure)). The first decoder of the ML / DL computer model 136 is trained to identify patterns representing a relatively large number of different lesions at the expense of having a large number of false positives. The second decoder of the ML / DL computer model 136 is trained to be less sensitive to the detection of lesions, but the detected lesions are very likely to be accurately detected. A third loss function for the population of ML / DL computer models compares the results of the decoders of the ML / DL computer model 136 as a whole to each other and forces those results to be consistent with each other. The lesion prediction results of the first ML / DL computer model 134 and the second ML / DL computer model 136 are combined to generate the final lesion prediction for the population, while the other ML / DL computer model 132 that generates a prediction of the liver mask provides an output representing the liver and its contour. An exemplary architecture of these ML / DL computer models 132 - 136 will be described in further detail later with respect to FIG. 6.

[0083] The results of the liver / lesion detection stage 130 of the AI pipeline 100 include a detection map that identifies one or more contours (outline lines) of the liver and portions of the medical image data elements corresponding to the detected lesions 135 (e.g., a per-voxel map of liver lesions detected within the input volume 105). Next, the detection map is input into the lesion segmentation stage 140 of the AI pipeline 100. The lesion segmentation stage 140 uses watershed techniques and a corresponding ML / DL computer model 142, as will be described in more detail later, to split the detection map and generate a segmentation of the image elements (e.g., voxels) of the medical image (slices) of the input volume 105. The liver lesion segmentation stage 140 identifies all of the contours corresponding to the lesions present in the slices of the input volume 105 based on this segmentation and provides other mechanisms, such as an ML / DL computer model 144, to perform operations to identify which contours in 3D correspond to the same lesion. The lesion segmentation stage 140 further provides mechanisms, such as an ML / DL computer model 146, to aggregate the mutually associated lesion contours to generate a 3D segmentation of the lesions. Lesion segmentation focuses on each lesion individually and uses the lesion image elements (e.g., voxels) represented in the medical image and the impainting of tissues other than the liver to perform active contour analysis. In this way, individual lesions can be identified and processed without biasing the analysis due to other lesions in the medical image or due to portions of the image outside the liver.

[0084] The result of the lesion segmentation 140 is a list 148 of lesions including the corresponding contour lines or contours within the input volume 105. These outputs 148 are provided to the false positive removal stage 150 of the AI pipeline 100. The false positive removal stage 150 uses a configured and trained ML / DL computer model with a double operating point strategy to reduce false positive lesion detections in the list of lesions generated by the lesion segmentation stage 140 of the AI pipeline 100. The first operating point is selected to be highly sensitive to false positives by configuring the ML / DL computer model of the false positive removal stage 150 to remove as many lesions as possible. After the highly sensitive removal of false positives, a determination is made as to whether fewer than a predetermined number of lesions remain in the list. If any remain, the other lesions in the removed list are reexamined using a second operating point with a relatively low sensitivity to false positives. The results of both approaches identify a final filtered list of lesions that are further processed by the lesion classification stage of the AI pipeline.

[0085] After removing false positives from the list of lesions and their contours generated by the lesion segmentation stage 140, the resulting filtered list 155 of lesions is provided as an input to the lesion classification stage 160 of the AI pipeline 100, and a configured and trained ML / DL computer model is executed against the list of lesions and their corresponding contour data, thereby classifying the lesions into one of a plurality of predetermined lesion classifications. For example, each lesion and its attributes (e.g., contour data) in the final filtered list of lesions may be input into the trained ML / DL computer model of the lesion classification stage 160, after which the trained ML / DL computer model operates on this data to classify the lesion as a particular predetermined type or class of lesion.

[0086] After the classification of lesions by the lesion classification stage 160 of the AI pipeline 100, the AI pipeline 100 generates an output 170 that includes a completed list of the lesions and their classifications, along with any contour attributes of the lesions. Further, the output 170 of the AI pipeline 100 may include liver contour information of the liver obtained from the liver / lesion detection stage 130. The output generated by this AI pipeline 100 may be provided to a further downstream computing system 180 for further processing and generation of a representation of the anatomical structure of the subject and any detected lesions present in the anatomical structure. For example, a graphic representation of the input volume may be generated in a medical image viewer or other computer application of the downstream computing system 180, and the anatomical structure and detected lesions may be overlaid on the graphic representation or otherwise highlighted using the contour information generated by the AI pipeline. In other exemplary embodiments, downstream processing by the downstream computing system 180 may include diagnostic decision support operations, automated medical image report generation, based on the detected list of lesions, classifications, and contours. In other exemplary embodiments, different treatment recommendations may be generated for review and consideration by a physician based on the classification of the lesions. In some exemplary embodiments, the list of lesions, their classifications, and contours may be stored in a historical data structure of the downstream computing system 180 associated with a patient identifier, storing multiple executions of the AI pipeline 100 for different input volumes 105 associated with the same patient, allowing for evaluation over time. For example, to evaluate the progression of a patient's illness or medical condition and present such information to a medical professional to assist in the patient's treatment, differences between lists of lesions or their associated classifications or both and the contours may be determined. Other downstream computing systems 180 and processing of the specific anatomical structure and lesion detection information generated by the AI pipeline 100 of the exemplary embodiments may be implemented without departing from the spirit and scope of the present invention.

[0087] Figure 2 is an exemplary flowchart showing an overview of exemplary operations of an AI pipeline according to one exemplary embodiment. The operations outlined in Figure 2 may be implemented by various stages of the logic, including the configured and trained ML / DL computer models described above using specific exemplary embodiments described later in separate sections of this description shown in Figure 1. It should be understood that this operation is particularly directed to an automated artificial intelligence pipeline implemented in one or more data processing systems including one or more computing devices specifically configured to implement the mechanisms of automated computer tools. It should be understood that there is no human intervention in the operations outlined in Figures 1 and 2, except in the creation of volumes of medical images and the use of outputs from downstream computing systems. The present invention particularly provides an automated artificial intelligence computing mechanism improved to perform the described operations, which operations are particularly different from any previous manual processes, and is particularly directed to providing a new improved process that provides the logic and data structures that enable the improved artificial intelligence computing mechanism of the present invention to be implemented in automated computing tools, thereby avoiding interaction with humans and reducing potential errors resulting from previous manual processes.

[0088] As shown in FIG. 2, the operation starts by receiving an input volume of medical images from a medical image technology computing system (for example, a computing system that provides computed tomography (CT) medical images) (step 210). The AI pipeline operates on the received input volume to perform body part detection (step 212), enabling a determination to be made as to whether the target body part exists in the received input volume (step 214). If the target body part (for example, the abdomen in the case of detection and classification of liver lesions) does not exist in the input volume, the operation ends. If the target body part exists in the input volume, phase classification and minimum anatomical structure evaluation are performed either continuously or in parallel.

[0089] That is, as shown in FIG. 2, phase classification is performed on the input volume (step 216) to determine whether the input volume contains medical images of a single phase of a medical image (for example, pre-contrast imaging, partial contrast imaging, delayed phase, etc.) or multiple phases of medical images (slices). Next, a determination is made as to whether the phase classification indicates either a single phase or multiple phases (step 218). If the input volume contains medical images targeting multiple phases, the operation ends; otherwise, if the input volume contains medical images targeting a single phase, the operation proceeds to step 220.

[0090] In step 220, it is determined whether a minimum amount of anatomical structure exists in the input volume, and detection of the target anatomical structure (for example, the liver in the example shown) is performed to enable the subsequent stages of the AI pipeline operation to be executed accurately. A determination is made as to whether a minimum amount of anatomical structure exists (for example, at least 1 / 3 of the liver is represented within the input volume) (step 222). If the minimum amount does not exist, the operation ends; otherwise, the operation proceeds to step 224.

[0091] In step 224, liver / lesion detection is performed to generate lesion contours and a detection map. These contours and detection map are provided to the lesion segmentation logic, which performs lesion segmentation (e.g., liver lesion segmentation in the example shown in the figure) based on these contours and detection map (step 226). Lesion segmentation results in a list of lesions and their contours, as well as the generation of detection and contour information for anatomical structures (e.g., the liver) (step 228). Based on this list of lesions and their contours, a false positive removal operation is performed on the lesions in the list to remove false positives and generate a list of filtered lesions and their contours (step 230).

[0092] The list of filtered lesions and their contours is provided to the lesion classification logic, which performs lesion classification and generates a complete list of lesions, their contours, and the lesion classification (step 232). This complete list is provided to the downstream computing system along with the liver contour information (step 234), and the downstream computing system may operate on this information to generate medical image displays for a medical image viewer application, generate treatment recommendations based on the classification of the detected lesions, evaluate the past progression of lesions over time for the same patient based on a comparison of the complete lists of lesions generated by the AI pipeline at different times, etc.

[0093] Accordingly, the exemplary embodiments outlined above provide an automated artificial intelligence mechanism and ML / DL computer model that operate on an input volume of medical images and generate a list of lesions, their contours, and classifications while minimizing false positives. The exemplary embodiments provide an automated artificial intelligence computer tool that specifically identifies, in a given set of voxels of an image of the input volume, which of the plurality of voxels correspond to a portion of the anatomical structure of interest (e.g., the liver) and which of those voxels correspond to a lesion (e.g., a liver lesion) within the anatomical structure of interest. The exemplary embodiments provide a clear improvement over previous techniques, both manually and automatically, in that the exemplary embodiments can be integrated into a fully automated computer tool within a clinician's workflow. In practice, based on an initial stage of the AI pipeline design of the exemplary embodiments that accepts only an input volume of a single phase (e.g., an abdominal scan) and rejects an input volume that does not show the anatomical structure of interest (e.g., the liver) or shows less than a predetermined amount of the anatomical structure of interest (e.g., too little liver), only meaningful input volumes are processed by the automated AI pipeline, thereby preventing a radiologist from expending valuable manual resources on results or incorrect results when viewing an input volume outside of the anatomical structure of interest (e.g., other than the liver). In addition to preventing the radiologist from being sent large amounts of unhelpful information, the automated AI pipeline of the exemplary embodiments also ensures a smooth integration of information technology by avoiding congestion of the AI pipeline and downstream computing systems (such as networks, archives, and review computing systems) by data associated with cases where the data does not correspond to the anatomical structure of interest or where the data does not provide a sufficient amount of the anatomical structure of interest.Furthermore, as described above, the automated AI pipeline of the exemplary embodiments enables accurate detection, measurement, and characterization of lesions in a fully automated manner, which is technically enabled by components based on the structure of one or more of the automated AI pipelines of the exemplary embodiments and the corresponding automated ML / DL computer models.

[0094] ML / DL computer model for detecting the minimum amount of anatomical structure present in the input volume

[0095] As described above, as part of the processing of the input volume 105, it is important to ensure that the input volume 105 represents a single phase of a medical image and that at least a minimum amount of the subject's anatomical structure is represented within the input volume 105. To determine whether a minimum amount of the subject's anatomical structure is present in the input volume 105, in one exemplary embodiment, the decision logic 128 implements a specially configured and trained ML / DL computer model that estimates a slice score to determine portions of the anatomical structure (e.g., the liver) present in the input volume 105. Below, an example embodiment of this configured and trained ML / DL computer model will be described based on the defined axial scoring technique.

[0096] FIG. 3A is an exemplary diagram showing an exemplary input volume (medical image) of the abdomen of a human patient according to one exemplary embodiment. In the depiction of FIG. 3A, a two-dimensional representation of a three-dimensional volume is shown. The slices are horizontal lines within the two-dimensional representation shown in FIG. 3A, but are represented as planes that extend within the page, outside the page, or both, representing flattened two-dimensional slices of the human body, and the stacking of these planes forms a three-dimensional image.

[0097] As shown in FIG. 3A, an exemplary embodiment defines an axial score for slices over the range of 0 to 1. The axial score is defined such that the slice corresponding to the first slice (FSL) containing the liver has a slice score of 0, and the slice corresponding to the last slice (LSL) containing the liver has a score of 1. In the example shown in the figure, the first slice and the last slice are defined in relation to the lowest slice in the volume (MISV) and the highest slice in the volume (MSSV), and lower and upper are determined along a given axis of the volume (e.g., the y-axis in the example shown in FIG. 3A). Thus, in this shown example, the MSSV is the slice with the highest y-axis value, and the MISV is the slice with the lowest y-axis value. For example, the MISV may be closest to the lower limb of the biological entity (e.g., the foot of a human subject), and the MSSV may be closest to the upper part of the biological entity (e.g., the head of a human subject). The FSL is the slice showing the anatomical structure (e.g., the liver) of the subject that is relatively closest to the MISV. The LSL is the slice showing the anatomical structure of the subject that is relatively closest to the MSSV. In one exemplary embodiment, a trained ML / DL computer model (e.g., a neural network) may assign an axial score by receiving a chunk of slices as input and outputting the height (axial score) of the central slice within the chunk. This trained ML / DL computer model is trained using a cost function that minimizes the error (e.g., the least squares error) at the actual height. Next, this trained ML / DL computer model is applied to all chunks covering the input volume (in some cases, the chunks may overlap slightly).

[0098] A pair of slice scores (s sup and s inf ) corresponding to the slice scores of the MSSV slice and the MISV slice, respectively, defines a liver axial score estimate (LAE). The ML / DL computer model of the decision logic 128 of FIG. 1 has a slice score s of the input volume 105 supand s inf is particularly configured and trained to determine, and the mechanism of the exemplary embodiment can recognize these slice scores and determine a fragment of the liver within the field of view of the input volume 105.

[0099] In some exemplary embodiments, the slice scores s sup and s inf are first determined by dividing the input volume 105 into sections (e.g., sections containing X slices (e.g., 20 slices)), and then, for each section, running a configured and trained ML / DL computer model against the slices of the section to obtain slice scores s' sup and s' inf for the first and last slices within the section, which may be detected indirectly, and "first" and "last" may be determined according to the direction of progression along the axis of the three-dimensional volume 105 (e.g., the direction of progression from the slice with the minimum y-axis value to the slice with the maximum y-axis value, from the first slice to the last slice along the y-axis). Given the estimated values of s' sup and s' inf , since it is known how the sections are positioned with respect to the entire volume 105, the estimated values of s sup and s inf are detected by extrapolation. Note that for each volume, there are n estimated values of s sup and s inf , where n is the number of sections per volume. In one exemplary embodiment, the final estimated value is obtained by taking the unweighted average of these n estimated values, but in other exemplary embodiments, other functions of the n estimated values may be used to generate the final estimated value.

[0100] For example, FIG. 3B shows the corresponding axial scores s' inf and s' supAnother representation of the input volume of FIG. 3A is shown, including the section of the slice represented together with. As shown in FIG. 3B, the section is defined as 20 slices separated by 5 mm in this example. For each section of the 20 slices of the volume, slice scores s' sup and s' inf are estimated, and from the extrapolation of these s' sup and s' inf values along a given range (e.g., a range from 0 to 1, a range from -0.5 to 1.2, or any other desired default range suitable for a particular implementation), s sup and s inf are obtained. In this example, assuming a default range of -0.5 to 1.2, by applying the ML / DL computer model and extrapolation, if s sup is estimated to be about 1.2 and s inf is estimated to be -0.5, these values indicate that the entire liver is included in the volume. Similarly, if s sup is estimated to be 1.2 and s inf is estimated to be 0.5, these values indicate that approximately the upper 50% of the axial liver range is included in the volume, for example the coverage rate is (1.2 - 0.5) / (1.2 - (-0.5)) = 0.41. As an additional example, in another exemplary embodiment where the liver starts at -2.0 and ends at 0.8 (i.e., s sup is estimated to be 0.8 and s inf is estimated to be 2.0), since the upper limit of the liver is lower than 1.2, the upper part of the liver is cut off, and since the lower limit is lower than -0.5, the lower part of the liver is completely covered. This indicates that approximately the lower 80% of the axial liver range is included in the volume, i.e., the coverage rate is (0.8 - max(-2, -0.5)) / (1.2 - (-0.5)) = 0.76.

[0101] FIG. 3C is an exemplary diagram of the input volume of FIG. 3A axially divided into n completely overlapping sections of volume. In the example shown in the figure, there are 7 sections indicated by arrows. Note that in this example, the last two sections (arrows at the top of the figure) are almost identical. As described above, for each of these sections, s' sup and s' inf values are estimated and used to extrapolate the s sup and s inf values of the MSSV slices and MISV slices, and the s sup and s inf values can then be used to determine the amount of the anatomical structure of interest present in the input volume 105.

[0102] Thus, the s sup and s inf values of MSSV and MISV are indirectly detected by first dividing the input volume 105 into sections and then, for each section, estimating the slice scores s' sup and s' inf of the first and last slices within the section. Given these estimated values and knowing how the sections are positioned with respect to the entire input volume 105, the s sup and s inf values are estimated by extrapolation. There are n estimated values of s sup and s inf extrapolated from each section, where n is the number of sections per volume. The final estimated value may be obtained by any suitable combination function that evaluates the n estimated values, such as the unweighted average of the n estimated values or any other suitable combination function.

[0103] FIGS. 4A - 4C relate to one exemplary embodiment and show s' sup and s' infAn exemplary diagram of one exemplary embodiment of an ML / DL computer model configured and trained to estimate the value of is shown. The ML / DL computer models of FIGS. 4A-4C are merely examples of an architecture of an ML / DL computer model, and many modifications to this architecture may be made without departing from the spirit and scope of the present invention, such as changing the tensor size of the input slice of the input volume, changing the number of nodes within a layer of the ML / DL computer model, changing the number of layers, and so on. Those skilled in the art will recognize, in view of this description, how to modify the ML / DL computer model of the exemplary embodiment to a desired implementation.

[0104] As shown in FIG. 4A, a series of 20 slices representing section 410 or “slab” of input volume 105 are provided as input to processing blocks (PB) 420-430. In the illustrated exemplary embodiment, PB 420-430 are blocks of logic that mix convolutional layers and LSTM layers, as shown in FIGS. 4B and 4C. Features are extracted from the convolutional layers of PB 420, 430, and then these features are fed as input to the LSTM layers of PB 420, 430. This is a kind of intelligent / lite modeling of the fact that the slices have a specific order within an anatomical region or the anatomical structure of an object (e.g., abdomen / liver), driven by the biological structure (e.g., in addition to the biological structure of the liver itself, the relative positions of the liver, kidneys, heart, etc.). In the example shown in the figure, first, the tensor size of the 20 slices 410 is 128x128 in this example. The first processing block 420 reduces the tensor size by a factor of 8 in this exemplary embodiment to generate a section of 20 slices (the number of slices within the section is implementation-specific and should be understood to be changeable without departing from the spirit and scope of the present invention), and these slices have dimensions 16x16x32, where 32 is the number of filters. The second processing block 430 converts the slices of the input section into a section of 20 slices including slices having dimensions 2x2x64, where 64 is the number of filters. A subsequent neural network 440 composed of a flattening layer, a dense layer, and a linear layer is configured and trained to generate estimates s' sup and s' inf of the input section 410 of input volume 105. FIG. 4B shows the composition of the processing blocks (PB) with respect to the convolutional layer and the LSTM layer according to one exemplary embodiment, and FIG. 4C shows an exemplary configuration of each of these convolutional layers and LSTM layers of each PB.

[0105] As an example, in the architecture of the ML / DL computer model of FIGS. 4A - 4C, during the training of this ML / DL computer model, in one exemplary embodiment, medical image data (e.g., data for communicating digital images in medicine (DICOM: Digital Imaging and Communications in Medicine)) is assembled into an input volume (e.g., a 3D array of size S i x512x512 with 32 - bit floating - point numbers (the Hounsfield Unit value is a normalized physical value indicating the X - ray attenuation of the material presented at a given position (e.g., voxel))). S i is the number of slices in the i - th volume, where i ranges from 0 to N - 1 and N is the total number of volumes. Each input volume is processed by a body - part detector, and as described above, an approximate region corresponding to the abdomen (in the case of liver detection) is extracted. The abdomen is defined, for example, as a continuous region between axial scores - 30 and 23 from the body - part detector. Slices outside this continuous region are rejected, and the ground truth may be defined as the appropriately adjusted positions of FSL and LSL. For example, assuming the range of the input volume is a to b, if there is no overlap between [a:b] and [-30,23], the input volume is rejected. In other words, when b > 23 or a < - 30.

[0106] The input section 410 or "slab" is sliced again at a default slice separation (e.g., 5 mm). The input section 410 is reformed to 128x128 in the x, y dimensions with shape M iN sections 410 of x128x128 are obtained. This is called downsampling of the data within the input volume. Since the ordering of the slices within the input volume depends on rough information (e.g., the size of the organ), the AI pipeline still operates well on the downsampled data, and since the size of the downsampled data is reduced, both the processing time and the training time of the AI pipeline are improved.

[0107] Input sections 410 that contain fewer slices than a given number (e.g., 20) or have a pixel size less than a given pixel size (e.g., 55mm) are rejected, and N' M i sections of x128x128 are obtained. The values within the sections are clipped and normalized using a linear transformation from the acquisition range (e.g., -1024, 2048) to the range (0, 1). At this point, the processed N' M i sections of x128x128 constitute the training set as described above, and for this training set, the neural network 440 is trained to generate the estimated values of s' sup and s' inf of the input sections.

[0108] Regarding performing inference using the trained neural network 440, the above operations for processing the input volume 105 by body part detection, selection of slices corresponding to the body part of interest, reslicing, reforming, rejection of specific sections not meeting the predefined requirements, and generation of the clipped and normalized sections are performed again for the new sections of the input volume 105. After generation of the clipped and normalized sections, the input volume 105 is divided into R-ceil(M - 10) / 10 subvolumes or sections containing 20 slices, thereby generating a division of slices containing overlapping chunks. For example, if there is a volume of N' = 31 slices (slices numbered 0 to 30), three sections or subvolumes are defined that include the overlapping slices 0 to 19, 10 to 29, and 11 to 30. The sections or subvolumes typically have at least about 50% overlap.

[0109] Accordingly, the ML / DL computer model is provided, configured, and trained to estimate the values of s sup and s inf of the input volume based on the estimated values of s' sup and s' inf for the sections of the volume corresponding to a predefined number of slices (medical images), assuming a defined axial score range of 0 to 1. From these estimated values, a determination can be made as to whether the input volume contains medical slices that together constitute at least a predefined amount of the anatomical structure of interest (e.g., the liver). This determination may be part of the decision logic 128 of the AI pipeline 100 for determining whether a sufficient representation of the anatomical structure is present in the input volume 105 in order to enable accurate liver / lesion detection, lesion segmentation, etc. at further downstream stages of the AI pipeline 100 as described above.

[0110] FIG. 5 is a flowchart showing an overview of exemplary operations of liver detection and a predefined amount of anatomical structure determination logic of an AI pipeline according to one exemplary embodiment. As shown in FIG. 5, the liver detection operation of the AI pipeline begins by receiving an input volume (step 510), and the input volume is divided into a plurality of overlapping sections of a predefined number of slices per section (step 520). The slices per section are input into a trained ML / DL computer model that estimates the axial scores of the first and last slices within each section (step 530). The axial scores of the first and last slices are used to extrapolate to the scores of the lowest slice (MISV) and the highest slice (MSSV) within the volume of the input volume (step 540). Thereby, a plurality of estimated values of the axial scores of the MISV and MSSV are obtained, and these estimated values are then combined by a function of the individual estimated values, thereby generating estimated values of the axial scores of the MISV and MSSV of the input volume (e.g., weighted average, etc.) (step 550). Based on the estimated values of the axial scores of the MISV and MSSV, the axial scores are compared to a criterion for determining whether a predefined amount of a target anatomical structure (e.g., liver) is present in the input volume (step 560). Thereafter, the operation ends.

[0111] [Liver / lesion detection]

[0112] As described above, assuming that the input volume 105 is determined to represent a single phase and that the input volume 105 represents a predetermined amount of the target anatomical structure in a slice of the input volume 105, liver / lesion detection is performed on the portion of the input volume 105 that includes the target anatomical structure. In one exemplary embodiment, the liver / lesion detection logic at stage 130 of the AI pipeline 100 employs a configured and trained ML / DL computer model (and in some exemplary embodiments, this can be the same ML / DL computer model 125 used at stage 120 for liver detection) that operates to detect the target anatomical structure (e.g., the liver) in a slice of the input volume 105. The liver / lesion detection logic at stage 130 of the AI pipeline 100 is also configured to detect lesions in an image of the target anatomical structure (the liver) and includes a population of a plurality of other configured and trained ML / DL computer models.

[0113] FIG. 6 is an exemplary diagram of a population of ML / DL computer models used to perform lesion detection in a target anatomical structure (e.g., the liver) according to one exemplary embodiment. The population 600 of ML / DL computer models includes a first ML / DL computer model 610 for detecting a target anatomical structure, e.g., the liver, and generating a corresponding mask. The population 600 of ML / DL computer models also includes a second ML / DL computer model 620, which is configured and trained to process a liver mask input and generate a lesion prediction using two competing loss functions implemented in two decoders of the second ML / DL computer model 620. One loss function is configured to penalize false positive errors (resulting in low sensitivity but high precision), while the other is configured to penalize false negative errors (resulting in high sensitivity but low precision). An additional loss function, called the consistency loss 627 in FIG. 6, is employed in the second ML / DL computer model 620 to make the outputs generated by the two competing decoders similar (consistent) to each other. The population of ML / DL computer models also includes a third ML / DL computer model 630 that is configured and trained to directly process the input volume 105 and generate a lesion prediction.

[0114] As shown in FIG. 6 and as described above, the group 600 specifically includes a first configured and trained ML / DL computer model 610 configured and trained to identify a target anatomical structure in an input medical image. In some exemplary embodiments, this first ML / DL computer model 610 includes a U-Net neural network model 612 configured and trained to perform image analysis to detect the liver in a medical image. However, the exemplary embodiments are not limited to this particular neural network model, and it should be understood that any ML / DL computer model capable of performing segmentation can be utilized without departing from the spirit and scope of the present invention. U-Net is a convolutional neural network developed for the segmentation of medical biological images in the Department of Computer Science at Freiburg University in Germany. The U-Net neural network is based on a fully convolutional network that has been modified and extended so that its architecture works with fewer training images and also provides more accurate segmentation. Since U-Net is generally known in the art, it will not be described in more detail herein.

[0115] As shown in FIG. 6, in one exemplary embodiment, a first ML / DL computer model 610 may be trained to process a predetermined number of slices when, for example, through an empirical process, the number of slices is determined to be suitable for a desired implementation, such as when it is determined that 3 slices yield good results. For example, in other implementations, different slice dimensions may be used without departing from the spirit and scope of the exemplary embodiment, but the slices of this input volume were, in one exemplary embodiment, 512x512 pixel medical images. The U-Net generates a segmentation of the anatomical structure in the input slice that yields one or more segments corresponding to the anatomical structure of interest, such as the liver. As part of this segmentation, the first ML / DL computer model 610 generates a segment corresponding to the liver mask 614. This liver mask 614 is provided as an input to at least one of the other ML / DL computer models 620 in the population 600 to focus the processing by the ML / DL computer model 620 on only the portion of the input slice of the input volume 105 corresponding to the liver. By preprocessing the input to the ML / DL computer model with the liver mask 614, this processing by the ML / DL computer model can be focused on the portion of the input slice corresponding to the anatomical structure of interest and not on the "noise" in the input image. The others of the ML / DL computer models, such as the ML / DL computer model 630, directly receive the input volume 105 without liver mask processing, using the liver mask 614 generated by the first ML / DL computer model 610.

[0116] In an exemplary embodiment of the illustrated population 600, a third ML / DL computer model 630 is composed of an encoder section 634-636 and a decoder section 638. The ML / DL computer model 630 receives nine slice slabs of the input volume 105, which are configured to be divided into three groups 631-633 of three slices each, where each group 631-633 is an input to a corresponding encoder network 634-636. Each encoder 634-636 is a convolutional neural network (CNN), such as DenseNet-121 (D121), without a fully connected head, that is pre-trained to recognize various types of objects (e.g., lesions) present in the input slice and output a classification output indicating the detected types of objects present in the input slice, e.g., as an output classification vector. The CNNs 634-636 can operate on, for example, three channels of the input slice, and the resulting output features of the CNNs 634-636 are provided to the concatenated NHWC logic 637, where NHWC refers to the number of image sheets (N) in the batch, the height (H) of the image, the width (W) of the image, and the number of channels (C) of the image. The architecture of the original DenseNet network includes many convolutional layers and skip-true connections that downsample three slice full-resolution inputs to many more feature channels of a smaller resolution. From there, a fully connected head aggregates all the features and maps them to multiple classes in the final output of the DenseNet. Since the DenseNet network is used as an encoder in the shown architecture, this head is removed and only the downsampled features are retained. Thereby, in the concatenated NHWC logic 637, all of the feature channels are concatenated to transfer them to a decoder stage 638 that serves to upsample the image until all reach the desired (e.g., 512x512) output probability map resolution.

[0117] Encoders 634 - 636 share the same parameters optimized through the training process, such as weights, sampling for lesion types during training, weights for loss, augmentation type, etc. In training the ML / DL computer model 630, two different loss functions are used. The main loss function is an adaptive loss specifically configured to penalize false positive errors in slices where the ground truth has no lesions and to penalize false negative errors in slices where the ground truth has lesions. The loss function is a revised version of the Tversky loss as follows. For each output slice: TP = sum(prediction * target) FP = sum((1 - target) * prediction) FN = sum((1 - prediction) * target) LOSS = 1 - ((TP + 1) / (TP + 1 + alpha * FN + beta * FP)) Here, "prediction" is the output probability of the ML / DL computer model 630, and "target" is the ground truth lesion mask. The output probability values are in the range of 0 to 1. Target is either 0 or 1 for each pixel in the slice. In slices where there are no lesions, the term "alpha" is small (e.g., zero), and "beta" is large (e.g., 10). In slices where there are lesions, the term "alpha" is large (e.g., 10), and "beta" is small (e.g., 1).

[0118] The second loss function 639 is a function connected to the outputs of the encoders 634 - 636. Since the input to this loss comes from the middle of the ML / DL computer model 630, it is also called "deep supervision" 639. Deep supervision has been found to force the encoder neural networks 634 - 636 to better learn the representation of the input data during training. In one exemplary embodiment, this second loss is a simple mean squared error in predicting whether there is a lesion in a slice. Therefore, the mapping network is used to map the output features of the encoders 634 - 636 to nine values between 0 and 1 corresponding to the probability of a lesion in each of the nine slice inputs. The decoder 638 generates an output that specifies a probability map for the lesions detected in the input image.

[0119] The second ML / DL computer model 620 receives, from the input volume, pre - processed input of three slices that have been pre - processed by the liver mask 614 generated by the first ML / DL computer model 610 to identify a part of three slices corresponding to the liver mask 614. The resulting pre - processed input slices (in the illustrated exemplary embodiment shown, those of size 192x192x3) are provided to a second ML / DL computer model 620 comprising a DenseNet - 169 (D169) encoder 621 connected to two decoders (represented as 2D DEC, where the decoder consists of two - dimensional neural network layers). The D169 encoder 621 is a neural network feature extraction device that is widely used in computer vision applications. It consists of a series of convolutional layers where the features extracted from each layer are also connected in a feed - forward manner to any other layer. The features extracted in the encoder 621 are transferred to two independent decoders 622, 623, where each decoder 622, 623 is composed of two - dimensional convolutional layers and up - sampling layers (referred to as 2D DEC in FIG. 6). Each decoder 622, 623 is trained to detect lesions (e.g., liver lesions) in the input slices. Both decoders 622, 623 are trained to perform the same task, i.e., lesion detection, but the key difference in their training is that the two decoders 622, 623 utilize different loss functions to drive the detection training in two competing directions, as described previously and also below. The final detection map of the second ML / DL model 620 is combined with the final detection map of the third ML / DL model 630 using an average operation 640. This procedure is applied across all of the input slabs of the input volume 105 to generate a final detection map (e.g., of liver lesions).

[0120] As described above, a second ML / DL computer model 620, which attempts to achieve opposing detection operating point performance, is trained using two different loss functions. That is, one encoder 622 uses a loss function for training that penalizes errors in false negative lesion detection and thus produces high-sensitivity detection with relatively low accuracy, while the other encoder 623 uses a loss function for training that penalizes errors in false positive lesion detection and results in high-accuracy detection albeit with low sensitivity. An example of these loss functions can be the Focal Tversky Loss, where the parameters are adjusted according to high or low penalties for false positives and false negatives according to one exemplary embodiment (see "A Novel Focal Tversky Loss function with Improved Attention U-Net for Lesion Segmentation" by Abraham et al., arXiv:1810.07842 [cs], October 2018). A third loss function, the consistency loss 627, is used to enforce the consistency between the predicted detections of each decoder 622, 623. In the consistency loss logic 627, the outputs 624, 625 of the two encoders 622, 623 are compared with each other and forced to be similar to each other. This loss can be, for example, the mean squared error loss between two predicted detections, the structural similarity loss, or any other loss that enforces consistency / similarity between the predicted detections being compared.

[0121] At runtime, these opposing operating point encoders 622, 623 are used such that the second ML / DL computer model 620 generates two lesion outputs 624, 625 that are input into slice averaging (SLC AVG) logic 623 that produces the average value of the lesion outputs. Thereby, the average value of this lesion output is resampled to produce an output that matches the dimensions of the output of a third ML / DL computer model 630 being compared (note that this process consists of reversing the liver masking operation and thereby calculating the lesion output at the original 512x512x3 resolution).

[0122] At runtime, in the Slice Averaging (SLC AVG) logic 626, the final detection map of the ML / DL model 620 is generated for the lesion prediction outputs 624 and 625 of the encoders 622, 623. Although a consistency loss 627 was applied during training to drive each decoder 622, 623 to learn consistent detections, at runtime this consistency loss is no longer utilized. Instead, it should be understood that the ML / DL model 620 outputs two detection maps that need to be aggregated by the SLC AVG module 626. The result of the SLC AVG logic 626 is resampled again to produce an output having dimensions that match the input slab (512x512x3). All the detections of the ML / DL model 620 generated for each slab of the input volume 105 are combined with the detections of the ML / DL model 630 generated through the Volume Averaging (VOL AVG) logic 640. In this logic, the average value of the two detection masks is calculated at the voxel level. This result is the Final Lesion mask 650 corresponding to the lesions detected in the input volume 105.

[0123] Thus, after training the ML / DL computer models 620, 630, when presented with a new slice of the new input volume 105, the first ML / DL computer model 610 generates a liver mask 614 to preprocess the input to the second ML / DL computer model 620, and the two ML / DL computer models 620, 630 process the input slice to generate a lesion prediction that is averaged over its volume by volume averaging logic 640. The result is a final lesion output 650 with a liver mask output 660 based on the operation of the first ML / DL computer model 610. These outputs can be provided as the output of the liver / lesion detection logic stage 130 of the AI pipeline 100 and provided to the lesion segmentation logic stage 140 of the AI pipeline 100 as described above and in more detail below. Thus, the mechanism of the exemplary embodiment provides a population 600 approach for anatomical structure identification and lesion detection in a medical image (slice) input volume 105.

[0124] As shown in FIG. 6, a population architecture provides improved performance over the use of only one ML / DL computer model. That is, it has been observed that improved detection specificity at the same sensitivity level as that of only one ML / DL computer model is achieved through the combination of the detection outputs of multiple ML / DL computer models in the population. That is, when the detection outputs are averaged due to errors (false positives) committed by the ML / DL models 620, 630 at various locations, the signal from true positive lesions is dominant while the signal from false positives is reduced, leading to performance improvement.

[0125] FIG. 7 is a flowchart showing an overview of exemplary operations of liver / lesion detection logic in an AI pipeline according to one exemplary embodiment. As shown in FIG. 7, the operation begins by receiving an input volume (step 710) and performing anatomical structure detection, e.g., liver detection, using a first trained ML / DL computer model such as a U-Net computer model configured and trained to identify anatomical structures (e.g., the liver). The result of the anatomical structure detection is a segmentation of the input volume to identify a mask for the anatomical structure (e.g., a liver mask) (step 730). This input volume is also processed through a first trained ML / DL computer model of a population specifically configured and trained to perform lesion detection (step 740). The first trained ML / DL computer model generates a first set of lesion detection prediction outputs based on its processing of the input volume (step 750).

[0126] The second trained ML / DL computer model of the group receives the masked input generated by applying the generated anatomical structure mask to the input volume, thereby identifying in the input volume the portion of the medical image corresponding to the anatomical structure of interest (step 760). The second trained ML / DL computer model processes the masked input through two different, competing loss functions, for example, one that penalizes errors in false positive lesion detection and another that penalizes errors in false negative lesion detection, via two different decoders (step 770). The result is then two sets of lesion prediction outputs that are combined through combinatorial logic to generate the lesion prediction output of the second ML / DL computer model (step 780). If necessary, the second lesion prediction output is resampled and combined with the first lesion prediction output generated by the first ML / DL computer model of the group to generate the final lesion prediction output (step 790). Thereby, the final lesion prediction output is output with the anatomical structure mask (step 795), and the operation ends.

[0127] [Lesion Segmentation]

[0128] As described above, lesion prediction outputs are generated through the logical stages of the AI pipeline, including the operations of various ML / DL computer models, as well as body part detection, determination of the body part of the subject, phase classification, identification of the anatomical structure of the subject, and anatomical structure / lesion detection. For example, in the AI pipeline 100 shown in FIG. 1, the results of the liver / lesion detection stage 130 of the AI pipeline 100 include one or more contours (contour lines) of the liver, and further a detection map that identifies the portion of the medical image data element corresponding to the detected lesion 135, for example, a map for each voxel of the liver lesion detected in the input volume 105. Thereby, the detection map is input to the lesion segmentation stage 140 of the AI pipeline 100.

[0129] As described above, in the lesion segmentation logic, for example, in the lesion segmentation stage 140 in FIG. 1, the detection map is divided to generate an image element segmentation of the medical image (slice) of the input volume using watershed technology and the corresponding ML / DL computer model. The liver lesion segmentation stage also provides other mechanisms such as one or more other ML / DL computer models that identify all of the contours corresponding to the lesions present in the slices of the input volume based on the image element segmentation and perform operations to identify which contours in 3D correspond to the same lesion. In this lesion segmentation stage, it is further equipped with mechanisms such as one or more additional ML / DL computer models that aggregate the mutually associated lesion contours to generate a 3D lesion segmentation.

[0130] In lesion segmentation, each lesion is individually focused on, and the lesion image elements represented in the medical image and the impainting of tissues other than the liver are used to perform active contour analysis. In this way, individual lesions may be identified and processed without biasing the analysis due to other lesions in the medical image or due to portions of the image outside the liver. The result of this lesion segmentation is a list of the lesions by their corresponding contour lines or contours in the input volume.

[0131] FIG. 8 shows a block diagram of an overview of aspects of a lesion segmentation process performed by a lesion segmentation logic according to one exemplary embodiment. As shown in FIG. 8, lesion segmentation involves a mechanism for splitting two-dimensional detection, i.e., detecting lesions in two-dimensional slices for each slice (block 810), connecting two-dimensional lesions along the z-axis (block 820), and refining the contours for each slice (block 830). Each of these blocks will be described in more detail below with respect to the subsequent drawings. The segmentation process shown in FIG. 8 is implemented as a process for identifying all lesions in a given input volume under analysis and differentiating lesions that are close to each other in the images (slices) of the input volume. For example, two lesions that appear to fuse during the pixel period in one or more images may need to be identified as separate lesions for other downstream processing of the detected lesions, such as during lesion classification or when outputting a list of lesions for downstream computing system operations, such as those equipped with a medical viewing application, performing a treatment recommendation operation, or performing a decision support operation.

[0132] As part of the per-slice 2D image segmentation in block 810, in the mechanism of the exemplary embodiment, existing watershed techniques are used to segment a detection map from the previous lesion detection stage of the AI pipeline, e.g., the detection map 135 generated by the liver / lesion detection logic 130 of the AI pipeline 100 in FIG. 1. In the watershed algorithm, seed definition is required to perform mask segmentation. As a result, as shown in FIGS. 10A and 10C, the mask is divided into the same number of regions as the seeds, with exactly one seed placed approximately at the center of each region. In automatic segmentation, the seeds within the mask can be obtained as the maxima of its distance map (the distance to the mask contour). However, such an approach is prone to noise and may lead to too many seeds, thereby potentially over-segmenting the mask. Therefore, it is necessary to intervene in the segmentation by re-grouping some of the regions. Considering the empirical observation that most lesions are bubble-shaped, the principle guiding the re-grouping of regions is to result in approximately circular new regions. For example, in the case of the mask shown in FIG. 10C, the mechanism will fuse the two regions identified by seeds 1051 and 1061 respectively, thereby resulting in a new mask segmentation consisting of only two approximately round regions. Thus, for detected lesions defined in the detection map 135, such as the lesion shown on the left side of FIG. 9 described hereinafter, they can be segmented into several bubble-shaped lesions as shown on the right side of FIG. 9. They will be regarded as cross-sections of 3D lesions on the slice.

[0133] Watershed segmentation is a region-based method whose origin lies in mathematical morphology. In watershed segmentation, an image is regarded as a local landscape with ridges and valleys. The altitude values of this landscape are usually defined by the shade values of each pixel or the magnitude of their gradients, thereby regarding the two-dimensional representation as a three-dimensional representation. In the watershed transformation, the image is decomposed into "catchment basins". For each minimum value, the catchment basin includes all points whose steepest descent path ends at that minimum value. Watersheds separate the basins from each other. In the watershed transformation, the image is completely decomposed and each pixel is assigned to either a region or a watershed.

[0134] Watershed segmentation requires the selection of at least one marker, also called a "seed" point, inside each object of the image. The seed points can be selected by an operator. In one embodiment, the seed points are selected by an automatic procedure that takes into account the knowledge specific to the use of that object. Having received the objects marked, they can grow using the morphological watershed transformation, as will be described in more detail below. Lesions are usually in a "bubble" shape. This exemplary embodiment provides a technique for fusing the watershed-segmented regions based on this assumption.

[0135] Next, in block 820, the mechanism of the exemplary embodiment aggregates the voxel segmentation for each of the slices along the z-direction so as to generate a three-dimensional output. To that end, this mechanism needs to determine whether two sets of image elements in different slices, for example voxels, belong to the same lesion, that is, whether they are aligned in three dimensions. The mechanism calculates the measurements between the lesions in adjacent slices based on the common set and union of those lesions, and applies a regression model to determine whether two lesions in adjacent slices are part of the same region. Each lesion can be regarded as a set of voxels, and the mechanism determines the intersection of two lesions as the intersection of two sets of voxels and the union of two lesions as the union of two sets of voxels.

[0136] This results in a 3D segmentation of this lesion, but the contours may not fit well with the actual image. There may be over-segmented lesions. In an exemplary embodiment, it is proposed to use active contouring, which is a conventional framework for addressing segmentation problems. Such an algorithm requires iteratively refining the contour to gradually fit it well to the image data, while ensuring that it maintains some desirable properties such as shape smoothness. In block 830, the mechanism of the exemplary embodiment initiates an active contour where the segmentation is obtained from the first stage 810 and the second stage 820, focusing on one lesion at a time. Otherwise, an active contour or a random segmentation method in operation in a similar lesion may result in them being merged into one contour, which is unproductive as it would effectively cancel out the benefits brought about by the previous segmentation stages. This mechanism focuses on one lesion and performs "inpainting" on the lesion voxels near the lesion and tissues other than the liver under the focus.

[0137] This chain of three-stage processing enables a process that is not biased by other lesions in the image or by pixels outside the liver, i.e., by the lesions.

[0138] [Segmentation of 2D Detection per Slice]

[0139] FIG. 9 shows the results of lesion detection and slice-by-slice segmentation according to one exemplary embodiment. As seen on the left side of FIG. 9, a lesion region 910 is detected through the above-described AI pipeline process up to that point and can be defined in the output of a contour and detection map, such as 135 in FIG. 1, from a lesion detection logic, such as 130 in FIG. 1. According to one exemplary embodiment, the logic of block 810 in FIG. 8 attempts to divide this region into three lesions 911, 912, and 913, as shown on the right side of FIG. 9. The segmentation mechanism of this exemplary embodiment is based on existing watershed techniques that operate to divide a detection map from the previous lesion detection stage of the AI pipeline. Watershed algorithms are mainly used for image processing for segmentation purposes. The fundamental principle behind these known watershed algorithms is that a grayscale image can be viewed as a geographical surface where high intensities indicate peaks and hills while low intensities indicate valleys. The watershed technique starts by filling each isolated valley (minimum value) with water (label) of a different color. As the water level rises according to the nearby peaks (gradients), water from valleys of different colors begins to merge. To avoid this, a partition is built at the location where the water merges. The operation of filling the water and building the partitions continues until all the peaks are below the water surface at which point the created partitions give the segmentation result. Again, since the watershed technique is generally known, it will not be described in detail herein. Without departing from the spirit and scope of the present invention, any known technique can be used to divide 2D images slice by slice.

[0140] In the context of lesion segmentation, the empirical observation that most lesions are circular strongly suggests that a segmentation resulting in a set of circular regions is likely a good segmentation. However, as previously stated, the quality of watershed segmentation is determined by the quality of the seeds. In fact, any set of seeds does not necessarily lead to a set of circular regions. For example, in FIG. 10C, a watershed segmentation induced by three seeds that each contain only one approximately circular region is shown. The other two regions are not circular. However, their union is also approximately circular. Since the diagonal segmentation in the figure divides another circular region into two smaller non-circular regions, such a configuration is said to be over-segmentation. Therefore, it is desirable to have an algorithm that can correct over-segmentation. This is done by a seed relabeling mechanism that fuses some of the over-segmented regions into a more heterogeneous segmentation that contains only circular regions. For example, for the segmentation in FIG. 10C, this mechanism decides that by fusing the two regions identified by seeds 1051 and 1061, a new, more circular region is created.

[0141] This exemplary embodiment fuses regions in the segmentation to create a more circular and larger region that can correspond to a physical lesion. By this segmentation, an area is divided into smaller regions, or as described herein, by this segmentation, a mask is divided into smaller regions. In terms of the contour, the segmentation generates a set of smaller contours from a larger contour (see the left and right of FIG. 9).

[0142] The seed is obtained by extracting the maximum value from the distance map calculated from the input mask to be segmented. In the map, for each pixel, the Euclidean distance to the mask contour is measured. Depending on the topology of the input mask, the maximum value derived from this distance map may lead to over-fragmented segmentation by the watershed algorithm. In this case, the watershed is said to be over-segmented, which may be desirable depending on the application, but tends to generate non-circular regions that are not ideal for lesion segmentation. In FIG. 10C, a synthetic input mask with three maximum values in its distance map is shown. Thereby, the watershed results in a segmentation that includes three regions, only one of which (corresponding to seed 1071) is approximately circular. The other two regions are not circular. The region with seed 1051 is only semi-circular. Thereby, the seed relabeling mechanism checks all seed pairs and determines that the two regions corresponding to seeds 1051 and 1061 should fuse such that they together form a more complete bubble. Such an operation leads to a new segmentation that includes only two regions, and the shapes of both regions are approximately circular.

[0143] The maximum value is a point whose distance to the contour is the longest compared to its directly adjacent points. The maximum value is a point, and its distance to the contour is known. As a result, the mechanism of the exemplary embodiment can draw a circle centered on this point. The radius of the circle is this distance. Thereby, for two maximum values, the mechanism can calculate the overlap of their respective circles. This is shown in FIGS. 10A and 10B.

[0144] By relabeling the seeds, it is determined whether the following two regions should be merged. In the case of two regions whose associated seeds are directly adjacent seeds, merging occurs; otherwise, the mechanism bases its decision on a hypothesis verification procedure. For example, referring to FIG. 10A, in the example shown, the distance map can result in a situation that leads to two distinguishable maxima, assuming that each maximum corresponds to the center of a distinguishable circular lesion. It should be noted here that the distance map can also tell how far the maxima are from the contour (boundary) for the mechanism of the exemplary embodiment. This distance is represented by the dotted line segment connecting the maximum value on the contour and the point in FIG. 10B. Therefore, if the hypothesis is maintained, the spatial extent of these two lesions can be inferred due to the assumption that the lesions are approximately round or in a "bubble" shape. Thus, the mechanism of the exemplary embodiment can draw two perfect circles as shown in FIG. 10B. Then, the mechanism measures the overlap of the two circles (e.g., by the classical Dice index) and compares it with a predefined threshold. If the value of the overlap index is greater than this threshold, the mechanism concludes that the two bubbles overlap excessively and are indistinguishable, and merging will occur. In other words, thereby, the mechanism of the exemplary embodiment concludes that the two maxima correspond to the two "centers" of the same lesion. However, in the conventional watershed, there is no such seed (i.e., maximum) relabeling mechanism. As a result, mask over-segmentation frequently occurs.

[0145] This overlap can be measured in several ways. In one exemplary embodiment, the mechanism uses the Dice coefficient. For two perfect circles corresponding to two maxima as shown in FIG. 10B, the mechanism can calculate the Dice index of these two circles. In this way, upon receiving that the Dice index becomes greater than the threshold, the mechanism can learn from the training dataset what optimal threshold should actually be applied such that the two maxima are actually the centers of the same lesion.

[0146] In FIGS. 10C and 10D, two partially fused circles provide an example of a lesion mask shape that is different from the lesion mask shapes of FIGS. 10A and 10B, which are more similar to each other in FIG. 10A than in FIG. 10C. Due to the distance map that can be very sensitive to the mask shape, there are three seeds in the lesion mask shape example of FIG. 10C. According to the above inference, in the lesion segmentation algorithm, the lesion represented in FIG. 10C will be divided into two separate lesions, but not into three separate lesions that may occur in the watershed technique without re-labeling the seeds.

[0147] In FIGS. 10C and 10D, seed 1051 and seed 1061 represent more extreme cases than those shown in FIGS. 10A and 10B. Without using the seed re-labeling technique of the exemplary embodiment, a division that separates them will be performed (represented by the diagonal solid line). However, with the seed re-labeling mechanism of the exemplary embodiment, this undesirable result can be practically avoided. On the contrary, since seed 1071 is far enough from seeds 1051 and 1061, the same hypothesis verification procedure described above helps to accept the assumption that seed 1071 corresponds to the center of a distinguishable bubble, leading to a vertical division as shown in FIGS. 10C and 10D. Similarly, this results in a conversion from the labels assigned to seeds 1051 and 1061 to a different label for seed 1071. However, similar to the situation in FIGS. 10A and 10B, the hypothesis verification procedure of the seed re-labeling technique of the exemplary embodiment will determine that seeds 1051 and 1061 correspond to the same lesion.

[0148] Figure 11A is a block diagram showing a mechanism for lesion segmentation and relabeling according to one exemplary embodiment. As shown in Figure 11A, it can be implemented as a computer model including one or more algorithms, machine learning computer models, etc., executed by one or more processors of one or more computing devices, and a mechanism operating on an input volume of one or more medical image data structures receives a two-dimensional lesion mask 1101 and performs a distance transform (block 1102) to generate a distance map 1111. This distance transform (block 1102) is an operation performed on a binary mask that calculates the shortest distance to the mask contour (boundary) for each point within the lesion mask. The more one moves towards the inside of the lesion mask, the more the other moves away from its contour (boundary). Thus, in the distance transform, the center point of the lesion mask, i.e., the point with a longer distance than others, is identified. In one embodiment, the mechanism optionally performs Gaussian smoothing on the distance map 1111.

[0149] Next, the mechanism performs identification of maxima (block 1103) to generate seeds 1112. As described above, these maxima are the points in the distance map 1111 that are at the longest distance from the contour or boundary. The mechanism performs a watershed technique based on the seeds 1112 (block 1104) to generate a watershed segmentation lesion mask 1113. As previously explained, this segmented lesion mask 1113 can be over-segmentation, resulting in regions that do not match the assumed bubble shape of the lesion. Therefore, the mechanism performs relabeling of the seeds based on the distance map 1111, the seeds 1112, and the segmented 2D lesion mask 1113 (block 1120) to generate an updated segmented lesion mask 1121. The relabeling of the seeds will be described in more detail below with reference to Figure 11B. As a result, the resulting updated segmented lesion mask 1121 will have regions that are fused to form a region that fits exactly the assumed bubble shape of the lesion.

[0150] Figure 11B is a block diagram showing a seed relabeling mechanism according to one exemplary embodiment. As shown in Figure 11B, it can be implemented as a computer model including one or more algorithms, machine learning computer models, etc., executed by one or more processors of one or more computing devices, and a mechanism that operates on the input volume of one or more medical image data structures receives a distance map 1111 and seeds 1112. More specifically, the mechanism considers each pair of seeds (seed A and seed B) at the seeds 1112. The mechanism determines whether seed A and seed B are seeds that are directly adjacent (block 1151). If seed A and seed B are seeds that are directly adjacent, the mechanism assigns the same label to seed A and seed B (block 1155). In other words, seed A and seed B are grouped so as to correspond to only one region.

[0151] In block 1151, if seed A and seed B are not directly adjacent, the mechanism performs a spatial range estimation based on the distance map 1111 (1152) and determines the pairwise affinity for seed A and seed B as follows. According to this exemplary embodiment, in the spatial range estimation, it is assumed that a certain region is in a "bubble" shape. Thereby, the mechanism assumes that each seed corresponds to a circle whose distance from the distance map is the radius of the circle.

[0152] Thereby, the mechanism calculates an overlap index for the circles corresponding to seed A and seed B (block 1153). In one exemplary embodiment, the mechanism uses the Dice index as follows.

[0153]

Equation

[0154]

Number

[0155] The mechanism determines whether the overlap metric is greater than a predefined threshold (block 1154). In block 1154, if the overlap metric is greater than the threshold, the mechanism fuses the corresponding regions within the segmented 2D lesion mask 1113 (block 1155).

[0156] If the affinity between two seeds is greater than this threshold, the same label is assigned to them. Otherwise, at this stage, it is not known whether they should belong to the same group. This decision-making is entrusted to the same module used in the z-direction connection, the label propagation stage (block 1512 in FIG. 15), as described below.

[0157] In a situation where there are more than two seeds, the same operation in FIG. 11B is repeated for all seed pairs before label propagation that generates seed groups. For example, although the seed pairs (a, b) and (b, c) are determined to belong to the same group, there may be a situation where the seed pair (a, c) fails the inspection as shown in FIG. 11B. Thereby, in label propagation, it is necessary to put a, b, and c in the same group, that is, the regions corresponding to seeds a and c will still merge. However, if there are seeds a, b, c, and d, and it is shown that in affinity calculation (performed for a total of six pairs), only (a, b) and (c, d) pass the inspection, label propagation will result in two groups each containing (a, b) and (c, d), respectively. Therefore, when a seed pair fails the inspection, it means that it is not known whether they should be put in the same group or should belong to different groups.

[0158] For example, in FIG. 10C, there are three seed pairs (1051 - 1061, 1051 - 1071, 1061 - 1071), and the mechanism should determine that the same label should be assigned to seeds 1051 and 1061 (belong to the same group). Thereby, in the label propagation step, these three seeds are clustered into two groups where the first group contains only 1071 and the second group contains both 1051 and 1061.

[0159] FIG. 12 is a flowchart showing an overview of an exemplary operation of lesion segmentation according to one exemplary embodiment. The operations outlined in FIG. 12 can be performed by the mechanisms described so far with respect to FIGS. 11A-11B. As shown in FIG. 12, the operation begins (step 1200), and the mechanism generates a distance map for the two-dimensional lesion mask (block 1101). As described so far, this distance map can be generated by performing a distance transform operation on the two-dimensional lesion mask and, optionally, performing Gaussian smoothing to remove noise. Thereby, the mechanism uses the identification of maxima to generate a grouping of data points, e.g., a grouping of maxima for each group (step 1202). The mechanism performs lesion segmentation based on the maxima to generate regions (step 1203). Next, the mechanism uses the distance map to re-label the seeds based on pairwise affinities (step 1204). Thereby, the mechanism fuses the regions corresponding to the seeds with the same label (step 1205). Due to the re-labeling of the seeds performed by the mechanism of the exemplary embodiment, it should be understood that the output of the segmented lesion mask in step 1205 does not have the over-segmentation problem associated with the watershed technique, which was caused by incorrect labels being associated with the data points associated with each of the lesion shapes, as described above. After that, the operation ends (step 1206).

[0160] [Z-direction connection of lesions]

[0161] The above process during lesion segmentation and seed relabeling is performed for each 2D image or 2D slice of the input volume, thereby generating a lesion mask appropriately labeled for each lesion represented in the corresponding 2D image. However, the input volume represents a 3D representation of the internal anatomical structure of a biological entity, and lesions that may appear to be associated with the same lesion when considered in 3D may actually be associated with different lesions. Therefore, to be able to correctly identify separate lesions within the biological entity when represented in a 3D input volume, the exemplary embodiment provides a mechanism for connecting 2D lesions along the z-axis, i.e., in 3D.

[0162] This mechanism for connecting 2D lesions along the z-axis, called z-direction connection of lesions, includes a logistic regression model that is executed to determine 3D z-direction lesion detection for the segmented lesion output generated by the above mechanism. The mechanism connects two lesions in adjacent image slices. When the logistic regression model determines that the two lesions represent the same lesion, the two lesions are connected. For example, for any 2D lesion on adjacent image slices, i.e., slices having z-axis coordinates that are sequentially ordered without interruption along the z-axis in a 3D organized set of slices, the mechanism determines whether these 2D lesions belong to the same 3D lesion, as described below.

[0163] Figures 13A - 13C show a process for z - direction connection of lesions according to one exemplary embodiment. In Figure 13A, a lesion mask input is shown. In Figure 13B, lesions after slice - by - slice lesion segmentation are shown, where an improved lesion segmentation mechanism for relabeling of the exemplary embodiment as described above may be employed. As shown in Figures 13A and 13B, slice 1310 has lesions 1311 and 1312, slice 1320 has lesion 1321, and slice 1330 has lesions 1331 and 1332. The z - direction connection mechanism of the lesions, i.e., the logistic regression model, is executed to compare each lesion in a given slice with each lesion in the pair of adjacent slices for the segmented lesion masks of adjacent slice pairs in the input volume. For example, the z - direction connection mechanism of the lesions compares lesion 1311 (lesion A) in slice 1310 with lesion 1321 (lesion B) in slice 1320. For each comparison, the mechanism considers each lesion as a set of voxels and determines the common set between lesion A (the set of voxels in lesion A) and lesion B (the set of voxels in lesion B) with respect to the size of lesion A and with respect to the size of lesion B. The z - direction connection mechanism of the lesions then uses a logistic regression model to determine whether lesion A and lesion B are connected based on the following two overlap ratios.

Number

[0164] Here, |A| represents the area of the circle corresponding to seed A, |B| represents the area of the circle corresponding to seed B, and |A∩B| represents the area of the common set of the circles corresponding to seed A and seed B. The mechanism trains a logistic regression model to use these two ratios as input features to determine the probability that lesion A and lesion B are connected. That is, using a machine learning process such as those described above, the logistic regression model is trained to generate, for each combination of slice pairs per unit for the volume of the training images, a prediction regarding the probability that a lesion in one slice is the same lesion or a different lesion from the lesion represented in an adjacent slice for the volume of the training images. This prediction is compared with the ground-truth indication of whether the lesions are the same lesion or different lesions to generate a loss or error. Next, the operating parameters of the logistic regression model, such as coefficients or weights, are modified to reduce this loss or error until a predefined number of epochs of training is performed or a predefined stopping condition is met.

[0165] Logistic regression models are widely used to solve binary classification problems. In the context of the exemplary embodiments, this logistic regression model predicts the probability that two cross-sections of a lesion are part of the same lesion. For this purpose, logistic regression uses two overlap ratios γ0, γ1 as described above. Specifically, the logistic regression model learns to linearly combine two features as follows.

Equation

[0166] There are two extreme cases. First, when the threshold t is set to 0, the z-connection mechanism of the exemplary embodiment always determines that the lesions are the same lesion, i.e., the cross sections are connected. This results in both a true positive rate and a false positive rate of 1. Second, when the threshold t is set to 1, the z-connection mechanism does not identify any cross sections of the connected lesion. In this case, both the true positive rate and the false positive rate are 0. Therefore, only when the threshold t is in the interval (0, 1) does the logistic regression model make a decision as to whether the lesion cross sections are associated with the same lesion across adjacent slices. With an ideal logistic regression model, the true positive rate would be equal to 1 (all true connections would be identified) and the false positive rate would be 0 (zero false connections would be made).

[0167] Thus, upon receiving that the logistic regression model has been trained, new slice pairs are input as input features to the trained logistic regression model such that for each of these pairs, these ratios are calculated for the pair and predictions are generated for each of these pairs. In this way, they can be evaluated. Next, if the predicted probability is greater than or equal to a predetermined threshold probability, lesions A and B are considered to be associated with the same lesion in three dimensions. Thereby, appropriate relabeling of the lesions across the slices can be performed to correctly associate the lesions in the two-dimensional slices with the same lesion representation in other adjacent slices, thereby identifying the three-dimensional lesions within the input volume.

[0168] There is a logical basis for supporting the two ratio input features used to train the logistic regression model. For example, if lesions A and B are sufficiently different in size, they are probably not part of the same lesion. Also, in the case of lesion 1312 in slice 1310 and lesion 1321 in slice 1320, if lesions A and B do not intersect, the features γ0 and γ1 will have zero values. As described above, the logistic regression model performs regression based on the two feature values γ0 and γ1 and outputs a probability value between 0 and 1, representing the likelihood that lesion A and lesion B are part of the same lesion.

[0169] In FIG. 13C, a cross-sectional connection between slices according to one exemplary embodiment is shown. As shown in FIG. 13C, a mechanism determines that lesion 1311 in slice 1310 and lesion 1321 in slice 1320 are part of the same lesion by executing a trained logistic regression model of an exemplary embodiment that predicts lesion commonality based on the overlap rate as described above. Similarly, the mechanism also determines that lesion 1321 in slice 1320 and lesion 1331 in slice 1330 are part of the same lesion. In this way, the mechanism propagates the intersecting lesions along the z-axis and performs the z-axis connection of the lesions.

[0170] Based on the per-slice evaluation of slices in the input volume for identifying the z-direction connection of lesions across 2D slices and the determination by a trained logistic regression model of whether the lesions are connected along the z-axis, re-labeling of the lesions is performed so that the same label for a lesion can be ensured to be applied to each of the lesion masks existing in each slice of the input volume, for example, to all of the lesion masks across a set of slices in the input volume. In this case, these lesion masks are determined by the logistic regression model to be associated with the same lesion A and can be re-labeled to specify that they are part of the same lesion A. This is performed for each lesion cross-section in each slice of the input volume, thereby enabling the generation of a 3D association of lesion masks for one or more lesions present in the input volume. Thereby, using this information, all cross-sections associated with the same lesion can be correctly labeled in the subsequent 3D in the input volume so that the lesion can be represented or otherwise processed, such as during subsequent downstream computing system operations.

[0171] Figures 14A and 14B show the results of a trained logistic regression model according to one exemplary embodiment. Figure 14A shows the Receiver Operating Characteristic (ROC) curves for the maximum overlap rate (γ0) + minimum overlap rate (γ1) metric and for the maximum overlap rate metric. The ROC curve is a graphical plot that shows the diagnostic ability of a binary classifier system as its discrimination threshold changes. The ROC curve is created by plotting the True Positive Rate (TPR) against the False Positive Rate (FPR) at various threshold settings. Figure 14B shows the precision-recall curves for the maximum overlap rate + minimum overlap rate metric and for the maximum overlap rate metric. The precision-recall curve is a plot of precision (y-axis) and recall (x-axis) for various thresholds, which is very similar to the ROC curve. Here, precision is the ratio of relevant instances among the instances obtained, and recall (or sensitivity) is the ratio of the total amount of relevant instances actually obtained. As shown in these figures, the two-feature logistic model outperforms its one-feature counterpart. Therefore, both features bring valuable information to this prediction task.

[0172] Looking at the maximum overlap rate (γ0) + minimum overlap rate (γ1) metric curve in Figure 14A, it can be seen that at an appropriate threshold t, the trained logistic regression model can generate a true positive rate ~= 95% at the expense of a false positive rate of approximately 3%. Looking at Figure 14B, it shows that the plotted results can identify the trained logistic regression model in terms of precision and recall, and the selection of the correct threshold t for both measurements can bring very good results.

[0173] FIG. 15 is a flowchart showing an overview of an exemplary operation of a mechanism for connecting two-dimensional lesions along the z-axis according to one exemplary embodiment. As shown in FIG. 15, the operation begins (step 1500), the mechanism selects a first image X from the input volume (step 1501), and selects a first lesion A in the image X (step 1502). In some exemplary embodiments, the images or slices in the input volume may be processed using the above-described segmentation and relabeling mechanisms, but this is not required. Conversely, the mechanisms of the exemplary embodiments directed to z-direction connection of lesions can be performed for any input volume in which the lesion masks have been identified.

[0174] Next, the z-direction connection mechanism of the exemplary embodiment selects a first lesion B in the adjacent image Y (step 1503). Next, the mechanism determines the intersection of lesion A and lesion B with respect to lesion A and determines the intersection of lesion A and lesion B with respect to lesion B (step 1504). The mechanism applies a trained logistic regression model to the common set of lesion A and lesion B for γ0 features and γ1 features, and based on the two intersection values, determines whether lesion A and lesion B belong to the same lesion, generates a prediction or probability that lesion A and lesion B belong to the same lesion, and then compares this probability with a threshold probability (step 1505). Based on the result of this determination, the lesion cross-sections in the image can be labeled or relabeled to indicate whether they are part of the same lesion.

[0175] The mechanism determines whether the lesion B in the image Y is the last lesion in the image Y (step 1506). If lesion B is not the last lesion, the mechanism considers the next lesion B to be in the adjacent image Y (step 1507), and the operation returns to step 1504 to determine the intersection of lesion A and the new lesion B.

[0176] In step 1506, if lesion B is the last lesion in the adjacent slice or image Y, the mechanism determines whether lesion A is the last lesion in image X (step 1508). If lesion A is not the last lesion in image X, the mechanism considers the next lesion A in image X (step 1509), and the operation returns to step 1502 to consider the first lesion B in the adjacent image Y.

[0177] In step 1508, if lesion A is the last lesion in image X, the mechanism determines whether image X is the last image to be considered (step 1510). If image X is not the last image, the mechanism considers the next image X (step 1511), and the operation returns to step 1502 to consider the first lesion A in the new image X.

[0178] In step 1510, if image X is the last image to be considered, the mechanism propagates the lesions that intersect between images along the z-axis. Here, propagation means that the labels associated with the same lesions determined through the above process are set to the same value to indicate that they are part of the same lesion (step 1512). This is performed for each separate lesion identified in the input volume such that the cross-sections in each of the images associated with the same lesion are properly labeled, and thus, a three-dimensional representation of each lesion is generated through the z-direction connection of the cross-sections. Then, the operation ends (step 1513).

[0179] [Contour refinement]

[0180] In the above process, the number and relative positions of lesions, as well as the surfaces of lesions that are connected across a two-dimensional space (within an image or slice) and a three-dimensional space (across images or slices in the input volume), yield accurate results. However, the lesion contours (boundaries) are not always well-defined and need improvement. Exemplary embodiments provide a mechanism for improving lesion contour accuracy. This additional mechanism may be employed in the above mechanism as part of lesion segmentation, or in other exemplary embodiments that do not require the above specific lesion detection, lesion segmentation and relabeling, or z-direction connection mechanism or combinations thereof.

[0181] Existing contour algorithms work well only when there is a lesion at the center of an anatomical structure with no lesions around it, but there are various situations that lead to the problem of "leakage" where two or more adjacent lesions initially merge their distinguishable contours into a single encompassing contour, thereby not working well when completely eliminating the advantages brought about by early two-dimensional lesion mask segmentation. In some cases where a lesion is near an anatomical structure boundary, such as the liver boundary, the contouring algorithm differentiates the pixels of the anatomical structure from the pixels of other anatomical structures in the image, such as organs, rather than differentiating one lesion from another since most are distinguishable by the contouring algorithm.

[0182] The mechanism of the exemplary embodiment inpaints areas that are not the object in the image or slice. FIG. 16 shows an example of contours for two lesions in the same image according to an exemplary embodiment. On the left side of FIG. 16, using an effective contour algorithm, contours 1611 and 1612 are determined for the two lesions. An effective contour algorithm is a class of algorithms that iteratively evolves the contour to better match the image content.

[0183] According to this exemplary embodiment, the mechanism impaints tissues other than the liver that are within the contour 1612 and near the contour 1611 but not within the contour 1611. However, if this impainting means that the pixel values for the pixels of healthy tissue (non-lesion tissue) within and around the contour 1612 and further around the contour 1611 are set to a specific value such that they all have the same value, then the inside of the contour 1611 is not impainted. For example, this value can be the average tissue value in an area that has been identified as not associated with a lesion, i.e., in healthy tissue of an anatomical structure such as the liver.

[0184] This impainting can be performed on a selected lesion contour 1611 such that the impainting is applied to healthy tissue and other lesions, e.g., the lesion 1612 in the image. In this way, the contour and pixels associated with the selected lesion, e.g., 1611, are considered separately from other parts of the image when re-evaluating the contour 1611. Thereby, the contour 1611 can be re-evaluated, and a determination can be made as to whether the re-evaluation of the contour 1611 results in an improved definition of the contour 1611. That is, an initial determination of the contrast and variance between the pixels associated with the selected lesion contour 1611 and the pixels near the selected lesion contour 1611 can be generated. After calculating this contrast and variance before impainting, the impainting can be performed on the selected lesion 1611 such that other lesion contours, e.g., 1612, and the pixels associated with the area of the anatomical structure corresponding to healthy tissue in the image are impainted with the average pixel intensity value of the healthy tissue.

[0185] The variance of a set of values is determined as follows. Consider, so to speak, a set of voxels composed of n voxels. First, their intensity values are summed, and then the arithmetic mean is calculated by dividing the resulting sum by n. This is represented by the number A as the result. Second, these voxel values are individually squared, and then the arithmetic mean is calculated. This result is represented by B. Next, the variance is defined as B - A * A, i.e., the difference between B and the square of A.

[0186] Therefore, the variance of a set of n values {x1, ..., x n} is defined as follows.

[0187]

Mathematics

[0188] This variance is calculated between the inner and outer voxels of a given contour. The inner voxels of the contour are the voxels enclosed by the contour, and the outer voxels refer to the outer voxels that are outside the contour but remain within a predefined distance from the contour.

[0189] The mechanism recalculates the contour 1611 of the selected lesion after inpainting using an effective contour formation algorithm as described above, recalculates the contrast or variance or both of the new contour 1611, and determines whether these values have been improved (higher contrast values or lower variance values inside or outside or both inside and outside the lesion). If the contrast and variance have been improved, the newly calculated contour 1611 is maintained as the contour of the corresponding lesion. Thereby, this process can be performed on the lesion 1612 as the selected lesion by inpainting the pixels associated with the healthy tissue near the lesion 1611 and the contour 1612. Thus, each lesion generates a contour for each lesion, and thereby is evaluated separately to prevent leakage of lesions into each other.

[0190] The mechanism for calculating the contour of a lesion after impainting may be based on the Chan-Vese segmentation algorithm, which is designed to segment objects without well-defined boundaries. This algorithm is repeatedly developed based on a set of levels that minimize the energy defined by the weighted values corresponding to the sum of the difference intensities from the average value outside the segmented region, the sum of the differences from the average value inside the segmented region, and the length of the boundary of the segmented region. Initialization is performed using the segmented detection map (to solve the problem of the minimum value of energy).

[0191] Upon receiving that the mechanism has segmentation, the mechanism initializes the contour with the previous estimate and determines whether the new contour is better, for example, whether the contrast and variance of the contour have been improved. If the original contour is better, the original contour is maintained. If the new contour is better, for example, if the contrast and variance of the contour have been improved, the mechanism uses the new contour. In some exemplary embodiments, the mechanism determines which contour is better based on calculating a homogeneous area or variance. If the variance is reduced both inside and outside the contour, the mechanism uses the new contour; otherwise, the mechanism uses the previous contour. In another exemplary embodiment, the mechanism determines whether the contrast (average value inside the contour versus average value near the contour) has been improved. Without departing from the spirit and scope of the exemplary embodiments, other techniques using various metrics may be used to select either the previous contour or the new contour.

[0192] FIG. 17 is a flowchart showing an exemplary operation of a mechanism for refining the contour for each slice according to an exemplary embodiment. As shown in FIG. 17, the operation begins with a given contour in a segmented image that shows a lesion, for example, in the liver (step 1700), and the mechanism determines the first contrast and variance for the initial contour (step 1701). The mechanism inpaints lesion pixels (or 3D voxels) near the lesion (step 1702). Next, the mechanism determines the contour around the lesion (step 1703). Next, the mechanism determines the second contrast and variance for the new contour (step 1704). The mechanism determines whether the second contrast and variance correspond to an improvement compared to the first contrast and variance (step 1705). If the second contrast and variance correspond to an improvement, the mechanism uses the updated contour to represent the lesion (step 1706). Thereafter, the operation ends (step 1708).

[0193] In step 1705, if the second contrast and variance do not correspond to an improvement, the mechanism returns to the initial contour (step 1707). Thereafter, the operation ends (step 1708). This process can be repeated for each lesion identified in the input slice or input volume or both to recalculate and improve the contour associated with each lesion present in the image / input volume.

[0194] [False positive removal]

[0195] After performing lesion segmentation to generate a list of lesions and their contours, AI pipeline 100 performs a false-positive processing stage 150 to remove incorrectly indicated lesions from the list of lesions. This false-positive stage 150 can take many forms, such as reducing the number of incorrectly identified lesions in the list of lesions, e.g., the contours and maps 135 in FIG. 1 output by liver / lesion detection logic 130 and then fused by segmentation and relabeling performed in lesion segmentation logic 140. The following description demonstrates a novel false-positive removal mechanism that can be used to perform this false-positive removal, although this specific false-positive removal is not required. Also, the false-positive removal mechanism described hereinafter can be used independently of the other mechanisms described above and can be applied to any list of identified objects in an image, and exemplary embodiments utilize such false-positive removal specifically in the case of lesions in medical images. That is, the false-positive removal mechanism described in this section can be implemented independently and distinctly from the other mechanisms described herein.

[0196] For ease of explanation, we will assume that a false positive removal mechanism is implemented as part of AI pipeline 100 and as part of AI pipeline 100's false positive removal logic 150. Thus, in false positive stage 150, the false positive removal mechanism described in this section operates on the list of lesions resulting from the liver / lesion detection logic and lesion segmentation and relabeling, taking into account the three-dimensional nature of the input volume in the case of lesion z-connectivity and contour refinement described above. This list 148 in FIG. 1 is input to false positive removal logic stage 150, which processes list 148 as described below and outputs a filtered or corrected list of lesions to lesion classification stage 160, in which incorrectly identified lesions have been minimized in the corrected list of lesions. The lesion classification stage thereby classifies the various lesions indicated in the corrected list of lesions.

[0197] That is, capturing all lesions at the front end of the AI pipeline 100 can lead to an increase in the sensitivity setting that causes the AI pipeline 100 to misidentify pixels that do not actually represent lesions as part of the lesions. As a result, there may be false positives to be removed. The false positive stage 150 includes logic that operates on the list of lesions and their contours to remove false positives. It should be understood that such false positive removal also needs to balance the risk that, during the examination (at the input volume level for the lesion level), if the false positive removal is not properly performed, it may result in lesions that go undetected. This can be a problem because it may prevent the doctor and the patient from noticing the lesions that require treatment. It should be understood that one examination may theoretically include images of several volumes for the same patient. However, in some exemplary embodiments with a single-phase detection implementation AI pipeline, only images of one volume are processed, and it is assumed that this processing is performed on a single volume. For clarity, since "patient level" is the subject for this exemplary embodiment, it will be used hereafter instead of "examination level" (whether the patient has lesions). In other exemplary embodiments, it should be understood that the operations described herein can extend to the examination level where multiple volumes of images for the same patient can be evaluated.

[0198] In an exemplary embodiment, assuming that the output of the previous stage of the AI pipeline 100 (such as slices, masks, multiple lesions, lesion and anatomical structure contours, etc.) is the input 148 to the false positive removal stage 150, the false positive removal stage 150 is implemented at a very specific operating point at the patient level (input volume level) to allow only a few patient-level false positives (for a normal patient / volume where at least one lesion is detected). This point can be obtained from the analysis of the receiver operating characteristic (ROC) analysis of the patient (sensitivity at the patient level versus specificity at the patient level). In this specification, the operating point OP at the patient level patientIn a volume that uses a very specific operating point, called the lesion-level operating point OP herein, which results in at least some lesions, a more sensitive operating point lesion is used at the lesion level called. The lesion-level operating point OP lesion can be identified from the analysis of the lesion-level ROC curve (lesion sensitivity vs. lesion specificity) to maximize the number of lesions to be retained.

[0199] Two operating points, namely, OP patient and OP lesion can be implemented in one or more trained ML / DL computer models. One or more trained ML / DL computer models are trained to classify an input volume or a list of its lesions (result of segmentation logic) or both with respect to whether the identified lesions are true lesions or false lesions, i.e., true positives or false positives. One or more trained ML / DL computer models can be implemented as a binary classifier whose output indicates for each lesion whether it is a true positive or a false positive. A set of outputs including binary classification for all lesions in the input list of lesions can be used to filter the list of lesions to remove false positives. In one exemplary embodiment, one or more trained ML / DL computer models first implement an operating point at the patient level to determine whether the result of the classification indicates that any of the lesions in the list of lesions is a true positive while filtering out false positives. After filtering at the patient level (input volume level), if true positives remain in the list of lesions that was first filtered, the lesion-level operating point is used to filter out any remaining false positives, if any. As a result, a filtered list of lesions with minimized false positives is generated.

[0200] The implementation of the operating point can be for only one trained ML / DL computer model or for multiple trained ML / DL computer models. For example, using a single trained ML / DL computer model, the operating point can be the setting of the operating parameters of the ML / DL computer model that can be dynamically switched. For example, the input to the ML / DL computer model may be processed using the patient-level operating point to generate a result indicating whether the list of lesions contains true positives after each classification of the lesions, and then, if true positives are included, the operating point of the ML / DL computer model may be switched to the lesion-level operating point, and the input that is reprocessed as false positives each time it passes through the ML / DL computer model is removed from the final list of lesions output in the false positive removal stage. Alternatively, in some exemplary embodiments, at least one true positive indicating the result of the first ML / DL computer model causes the input through the second ML / DL computer model and the false positives identified by removing both models from the final list of lesions output by the false positive removal stage of the AI pipeline to be processed, such that two separate ML / DL computer models can be trained, one for the patient-level operating point and the other for the lesion-level operating point.

[0201] Training of an ML / DL computer model may involve machine learning training operations in which the ML / DL computer model processes training inputs that include an image volume and a list of corresponding lesions. In this case, the list of lesions includes lesion masks or contours so as to generate a classification for each lesion in the image as to whether it is a true positive or a false positive. The training input further indicates whether the image contains a lesion, and then the output generated by the ML / DL computer model is evaluated to determine a loss or error, which is then used to modify the operating parameters of the ML / DL computer model so as to reduce the determined loss / error, and is associated with ground truth information. Thus, the ML / DL computer model learns the features of the input that represent true positive / false positive lesion detection. This machine learning is performed for each of the operating points, i.e., OP patient and OP lesion such that the operating parameters of the ML / DL computer model are learned taking into account patient-level sensitivity / specificity or lesion-level sensitivity / specificity or both.

[0202] When classifying a lesion as to whether it is a true positive or a false positive, the input volume (corresponding to the patient at the "patient level") is considered positive if it contains at least one lesion. The input volume is considered negative if it contains no lesions. Keeping this in mind, a true positive is defined as a positive input volume, i.e., an input volume with at least one detection result classified as a lesion that is actually a lesion. A true negative is defined as a negative input volume, i.e., an input volume with no lesions and no detection results classified as lesions. A false positive is defined as a negative input volume with no lesions, but this input indicates a lesion in the detection result. That is, the AI pipeline lists a lesion when there is no lesion present. A false negative is defined as a positive input volume with a lesion, but the AI pipeline does not indicate a lesion in the detection result. The trained ML / DL computer model classifies the lesion in the input as to whether it is a true positive or a false positive. False positives are removed by the output filter generated by false positive removal. The detection of false positives is performed at two different operating points, the patient level and the lesion level, i.e., at various sensitivity / specificity levels.

[0203] The two different operating points at the patient level and the lesion level can be determined based on ROC curve analysis. The ROC curve can be calculated using ML / DL computer model validation data consisting of several input volumes (e.g., several input volumes corresponding to various patient examinations) that may contain several lesions (0 to K lesions per examination). The input to the trained ML / DL computer model, or "classifier", is the detection result detected so far in the input as to whether it is an actual lesion or a false positive, e.g., the output of the lesion detection stage and the segmentation stage of the AI pipeline. The first operating point, i.e., the operating point OP at the patient level patientIt is defined to maintain at least X% of the lesions identified as true positives. That is, while removing some false positives, almost all of the true positives are retained. The value of X can be set based on the analysis of the ROC curve and can be any value suitable for a particular implementation. In one exemplary embodiment, the value of X is set to 98% such that almost all of the true positives are maintained while some false positives are removed.

[0204] The second operating point, i.e., the lesion-level operating point OP lesion is defined such that the lesion sensitivity is higher than the lesion sensitivity obtained for the first operating point, i.e., the patient-level operating point OP patient and the specificity is higher than Y%. Here, Y is determined by the actual performance of the trained ML / DL computer model. In one exemplary embodiment, Y is set to 30%. An example of the ROC curve for determining the patient-level and lesion-level operating points is shown in FIG. 18A. As shown in FIG. 18A, the lesion-level operating point is selected along the lesion-level ROC curve such that the lesion sensitivity is higher than the lesion sensitivity for the patient-level operating point.

[0205] FIG. 18B is an exemplary flowchart of an operation for performing false positive removal based on the operating points at the patient level and the lesion level according to one exemplary embodiment. As shown in FIG. 18B, the result of the segmentation stage logic of the AI pipeline is the input 1810 to the first trained ML / DL computer model 1820 that implements the first operating point. The input 1810 includes an input volume (or volume of images (VOI)) and a list of lesions, where the list of lesions includes a lesion mask or contour data that specifies the pixels or voxels corresponding to each of the lesions identified in the image data of the volume of images, and a list of lesions including labels associated with the pixels that indicate which lesions in the three-dimensional space of the input volume, i.e., the three-dimensional space of the output of the segmentation, z-connection, and contour refinement described above, correspond. The input can be denoted as set S. The first trained ML / DL computer model 1820 implements an operating point at the patient level in its training to classify the features extracted from the input at X%, for example 98%, of the true positives maintained in the filtered list of lesions obtained as a result of the classification by the trained ML / DL computer model 1820 and a part of the false positives removed in the resulting list. The resulting list includes a subset S + that includes true positive lesions classified by the first ML / DL computer model 1820 and a subset S - that includes false positive lesions classified by the first ML / DL computer model 1820.

[0206] ]As the false positive removal logic, there is further true positive evaluation logic 1830 that determines whether the true positive subset output by the first ML / DL computer model 1820 is empty. That is, the true positive evaluation logic 1830 determines whether there is nothing in the elements from S that is classified as a true lesion by the first ML / DL computer model 1820. If the true positive subset is empty, the true positive evaluation logic 1830 sets the true positive subset S +is output as a list 1835 of filtered lesions, i.e., there will be no lesions identified in the output sent to the lesion classification stage of the AI pipeline. The true positive evaluation logic 1830 determines that the true positive subset S + is not empty, a second ML / DL computer model 1840 is executed with the input S received, and this second ML / DL computer model 1840 implements the second operating point during its training, i.e., the lesion-level operating point OP lesion As described above, two ML / DL computer models 1820 and 1840 are shown for convenience of explanation, but it should be understood that the second ML / DL computer model can be the same ML / DL computer model as 1820, but with different operating parameters corresponding to the second operating point, and these two operating points can be implemented in various sets of trained operating parameters for configuring the same ML / DL computer model for the processing of the input S.

[0207] The second ML / DL computer model 1840 processes the input with the trained operating parameters corresponding to the second operating point so as to regenerate the classification of the lesion as to whether the lesion is a true positive or a false positive. The result is a subset S' + containing predicted lesions (true positives) and a subset S' - containing predicted false positives. Thereby, a list 1845 of filtered lesions is output as the subset S' + thereby substantially eliminating the false positives designated in the subset S' -

[0208] ​Regarding the exemplary embodiments shown in FIGS. 18A and 18B, they will be described from the perspective of the operating points at the patient level and the lesion level. It should be understood that the false positive removal mechanism can be implemented by operating points at various different levels. For example, similar operations can be performed on the image volume level operating point and the voxel-level operating point in the "per-voxel" false positive removal operation. FIG. 18C is an exemplary flowchart of operations for performing per-voxel false positive removal based on the input volume level and the voxel-level operating points according to one exemplary embodiment. The operations in FIG. 18C are similar to the operations in FIG. 18B, but these operations can be performed on the voxels in the input set S. In per-voxel false positive removal, the first operating point can still be an operating point at the patient level or the input volume level, while the second operating point can be at the voxel-level operating point OP voxel Therein. In this case, true positives and false positives are evaluated at the voxel level. Any voxel is shown as being associated with a lesion, and if it is actually associated with a lesion, it is a true positive, but if a voxel is shown as being associated with a lesion and it is not actually associated with a lesion, it is considered a false positive. Appropriate setting of the operating point can be regenerated again based on the corresponding ROC curve so that a similar balance between sensitivity and specificity is achieved as described above.

[0209] The above exemplary embodiments of the false positive removal mechanism assume a single input volume from a patient examination, but it should also be understood that this exemplary embodiment can be applied to any grouping of one or more images (slices). For example, false positive removal may be applied to a single slice, a set of slices smaller than the input volume, or even multiple input volumes from the same examination.

[0210] FIG. 19 is a flowchart showing an overview of an exemplary operation of false positive removal logic of an AI pipeline according to one exemplary embodiment. As shown in FIG. 19, the operation starts with receiving an input S from a previous stage of the AI pipeline (step 1910) (step 1900). Here, the input includes, for example, an input volume of an image and a corresponding list of lesions including masks, contours, etc. The input is processed by a trained first trained ML / DL computer model implementing a first operating point, for example, a patient-level operating point with relatively high specificity and low sensitivity, to generate a first set of classifications for lesions including a true positive subset and a false positive subset (step 1920). A determination is made as to whether the true positive subset is empty (step 1930). If the true positive subset is empty, in the operation, the true positive subset is output as a filtered list of lesions (step 1940), and this operation ends. If the true positive subset is not empty, the input S is processed by a trained second ML / DL computer model implementing a second operating point with relatively high sensitivity and low specificity compared to the first operating point (step 1950). As described above, in some exemplary embodiments, the first ML / DL computer model and the second ML / DL computer model can be the same model, but can be configured with different operating parameters corresponding to different trainings implementing different operating points. The result of the processing by the second ML / DL computer model is a second set of classifications for lesions including a second true positive subset and a second false positive subset. Thereby, this second true positive subset is output as a filtered list of lesions (step 1960), and this operation ends.

[0211] [Example of computer system environment]

[0212] The exemplary embodiments can be utilized in many different kinds of data processing environments. To provide background for the description of certain elements and functionality of the exemplary embodiments, FIGS. 20 and 21 are provided hereinafter as an exemplary environment in which aspects of the exemplary embodiments can be implemented. It should be understood that FIGS. 20 and 21 are not meant to assert or imply any limitation with respect to the environment in which aspects or embodiments of the present invention can be implemented. Many modifications can be made to the shown environment without departing from the spirit and scope of the present invention.

[0213] In FIG. 20, for some embodiments, a schematic diagram of one exemplary embodiment of a cognitive system 2000 implementing a request processing pipeline 2008 is shown, which, depending on the embodiment, uses a composite artificial intelligence mechanism that approximates a human through various computer-specific processes, but through a question answering (QA) pipeline, a treatment recommendation pipeline, a medical image enhancement pipeline, or any other artificial intelligence (AI) or cognitive computing-based pipeline that processes requests through a process on the generated result. For the sake of convenience of this description, it is assumed that the request processing pipeline 2008 is implemented as a QA pipeline that operates on structured requests or unstructured requests or both in an input question form. One example of question processing operations that can be used in conjunction with the principles described herein is described in U.S. Patent Application Publication No. 2011 / 0125734, which is hereby incorporated by reference in its entirety.

[0214] The cognitive system 2000 is implemented in one or more computing devices 2004A - D (including one or more processors and one or more memories, and optionally, any other computing device elements commonly known in the art such as buses, storage devices, communication interfaces, etc.) connected to a computer network 2002. For the sake of simplicity of explanation, FIG. 20 shows the cognitive system 2000 implemented only in the computing device 2004A, but as described above, the cognitive system 2000 may be distributed across multiple computing devices such as the multiple computing devices 2004A - D. The network 2002 includes multiple computing devices, namely, the computing devices 2004A - D that can operate as server computing devices, and the computing devices 2010 - 2012 that can operate as client computing devices, which communicate with each other and communicate with other devices or components via one or more wired data communication links or wireless communication links or both. In this case, each communication link includes one or more of wires, routers, switches, transmitters, receivers, etc. In some exemplary embodiments, the cognitive system 2000 and the network 2002 enable one or more cognitive system - user - targeted question processing and answer generation (QA) functionality via their respective computing devices 2010 - 2012. In other embodiments, the cognitive system 2000 and the network 2002 can provide other types of cognitive operations including, but not limited to, request processing and cognitive response generation, such as acquisition of cognitive information, user training / instructions, cognitive evaluation of data, etc., which can take many different forms depending on the desired implementation. Other embodiments of the cognitive system 2000 may be used with components, systems, subsystems or devices or combinations thereof other than those described herein.

[0215] The cognitive system 2000 is configured to implement a request processing pipeline 2008 that receives inputs from various sources. The requests can be posed in the form of natural language questions, natural language requests for information, or natural language requests for the execution of cognitive operations. For example, the cognitive system 2000 receives inputs from the network 2002, one or more electronic document corpora 2006, the cognitive system user, or other data and other possible input sources or combinations thereof. In one embodiment, some or all of the inputs to the cognitive system 2000 are routed through the network 2002. The various computing devices 2004A - D in the network 2002 include access points for content producers and the cognitive system user. Some of the computing devices 2004A - D include devices for a database that stores one or more data corpora 2006 (shown as separate entities in FIG. 20 for ease of explanation). A portion of the one or more data corpora 2006 can also be provided by one or more other network-connected storage devices, one or more databases, or other computing devices not clearly shown in FIG. 20. The network 2002 includes local network connections and remote connections in various embodiments so that the cognitive system 2000 can operate in an environment of any scale, including local and global, such as the Internet.

[0216] In one embodiment, the content creator creates content in one or more documents of the data corpus 2006 for use as part of the cognitive system 2000. Documents include any file, text, article, or data source for use in the cognitive system 2000. The cognitive system user accesses the cognitive system 2000 via a network connection to the network 2002 or an Internet connection and inputs into the cognitive system 2000 a question / request to be answered / processed based on the content within one or more data corpora 2006. In one embodiment, the question / request is formed using natural language. The cognitive system 2000 parses, interprets, and via the pipeline 2008 provides a response to the cognitive system user, e.g., the cognitive system user 2010, that includes one or more answers to the presented question, responses to the request, results of processing the request, and the like. In some embodiments, the cognitive system 2000 provides the response to the user within a ranked list of candidate answers / responses, and in other exemplary embodiments, the cognitive system 2000 provides a single final answer / response, or a combination of the final answer / response and a ranked list of other candidate answers / responses.

[0217] The cognitive system 2000 implements a pipeline 2008 that includes multiple stages for processing the input question / request based on information obtained from one or more data corpora 2006. The pipeline 2008 generates an answer / response to the input question or request based on the input question / request and the processing of one or more data corpora 2006.

[0218] In some exemplary embodiments, the cognitive system 2000 can be an IBM Watson (trademark) cognitive system available from International Business Machines Corporation of Armonk, New York, augmented by the mechanisms of the exemplary embodiments described hereinafter. As outlined previously, the pipeline of the IBM Watson (trademark) cognitive system receives an input question or request, then parses the question or request to extract the key features of the question / request, and then formulates a query that is applied to one or more data corpora 2006 using these key features. Based on the application of the query to the one or more data corpora 2006, a set of hypotheses, or candidate answers / responses to the input question / request are generated. This generation is done by searching through portions of one or more data corpora 2006 (hereinafter simply referred to as corpus 2006) that have some likelihood of containing a valuable response to the input question / response (hereinafter assumed to be the input question). Thereby, the pipeline 2008 of the IBM Watson (trademark) cognitive system performs a deep analysis on the language of the input question and the language used in each such portion of the corpus 2006 identified upon application of the query, using various inference algorithms.

[0219] Thereby, scores obtained from various inference algorithms are weighted against a statistical model that aggregates the level of confidence that, in this example, the pipeline 2008 of the IBM Watson™ cognitive system 2000 has with respect to the evidence that possible candidate answers are inferred by the question. This process is then repeated for each of the candidate answers to generate a ranked list of candidate answers that can be presented to the user who submitted the input question, e.g., the user of the client computing device 2010, or from which a final answer is selected and presented to the user. Information about the pipeline 2008 of the IBM Watson™ cognitive system 2000 can be further obtained, for example, from the IBM Corporation website, IBM Redbooks, etc. For example, information about the pipeline of the IBM Watson™ cognitive system can be verified in “Watson And Healthcare” by Yuan et al., IBM developerWorks, 2011 and “The Era of Cognitive Systems: An Inside Look at IBM Watson and How it Works” by Rob High, IBM Redbooks, 2012.

[0220] As described above, the input from the client device to the cognitive system 2000 can be presented in the form of a natural language question, but the exemplary embodiments are not limited to such. Rather, the input question can actually be formatted or structured as any suitable type of request that can be parsed and analyzed using structured input analysis or unstructured input analysis or both, including but not limited to natural language parsing and analysis mechanisms of cognitive systems such as IBM Watson (trademark), which serves as the basis for performing cognitive analysis and can determine the basis for resulting in the results of this cognitive analysis. For example, a doctor, a patient, etc. can issue a request to the cognitive system 2000 via the client computing device 2010 regarding specific medical image-based operations, such as "identifying liver lesions present in patient ABC" or "providing treatment recommendations for the patient" or "identifying changes in the liver lesions of patient ABC". According to the exemplary embodiments, such a request may be particularly directed to cognitive computer operations that employ the lesion detection and classification mechanisms of the exemplary embodiments to provide a list of lesions, lesion contours, lesion classifications, and contours of the target anatomical structure, and the cognitive system 2000 operates on this list to provide a cognitive computing output. For example, the request processing pipeline 2008 processes a request such as "identifying liver lesions present in patient ABC", parses this request, whereby the target anatomical structure can be identified as the "liver", a specific input volume is the medical image volume for patient "ABC", and the "lesions" within the anatomical structure are to be identified. Based on this parsing, a specific medical image volume corresponding to patient "ABC" is retrieved from the corpus 2006 and can be input into the lesion detection and classification AI pipeline 2020. The lesion detection and classification AI pipeline 2020 operates on this input volume as described above to identify a list of liver lesions. The list of liver lesions is output to the cognitive computing system 2000 for further evaluation through the request processing pipeline 2008, for example, to generate a medical image viewer application output, etc.

[0221] As shown in FIG. 20, one or more of the computing devices, such as server 2004, can be specifically configured to implement a lesion detection and classification AI pipeline 2020, such as the AI pipeline 100 in FIG. 1. The configuration of the computing device can include the provision of application-specific hardware, firmware, etc. to facilitate the implementation of the operations described herein and the generation of outputs for exemplary embodiments. The configuration of the computing device can further or alternatively include a software application stored in one or more storage devices and loaded into the memory of the computing device, such as server 2004, to configure the one or more hardware processors of the computing device to perform the operations described herein and generate outputs. Also, any combination of application-specific hardware, firmware, software applications executed in hardware, etc. can be used without departing from the spirit and scope of the exemplary embodiments.

[0222] When a computing device is configured in one of these ways, it should be understood that the computing device becomes a special computing device specifically configured to implement the mechanisms of the exemplary embodiments and is not a general-purpose computing device. Also, as described herein, the implementation of the mechanisms of the exemplary embodiments improves the functionality of the computing device and provides useful and specific results that facilitate not only the automatic detection of lesions in the anatomical structure of interest but also the classification of such lesions, thereby reducing errors and increasing efficiency compared to manual processes.

[0223] As described above, the mechanisms of the exemplary embodiments utilize a specially configured computing device or data processing system to perform operations for identifying anatomical structures, detecting and classifying lesions. These computing devices, or data processing systems, may comprise various hardware elements specially configured to implement one or more of the systems / subsystems described herein by hardware configuration, software configuration, or a combination of hardware and software. FIG. 21 is a block diagram of just one example of a data processing system in which aspects of the exemplary embodiments may be implemented. The data processing system 2100 is an example of a computer such as the server 2004 in FIG. 20, in which computer-usable code or instructions implementing the processes and aspects of the exemplary embodiments of the present invention may be arranged or executed, or both, to provide the operations, outputs, and external effects of the exemplary embodiments as described herein.

[0224] In the depicted example, the data processing system 2100 employs a hub architecture that includes a North Bridge and Memory Controller Hub (NB / MCH) 2102 and a South Bridge and Input / Output (I / O) Controller Hub (SB / ICH) 2104. A processing unit 2106, a main memory 2108, and a graphics processor 2110 are connected to the NB / MCH 2102. The graphics processor 2110 may be connected to the NB / MCH 2102 through an Accelerated Graphics Port (AGP).

[0225] In the example shown, the local area network (LAN) adapter 2112 is connected to the SB / ICH 2104. The audio adapter 2116, the keyboard and mouse adapter 2120, the modem 2122, the read-only memory (ROM) 2124, the hard disk drive (HDD) 2126, the CD-ROM drive 2130, the universal serial bus (USB) ports and other communication ports 2132, and further the PCI / PCIe devices 2134 are connected to the SB / ICH 2104 through the buses 2138 and 2140. Examples of PCI / PCIe devices include, for example, an Ethernet (registered trademark) adapter, an add-in card, or a PC card for a notebook computer. In PCI, a card bus controller is used, but not in PCIe. The ROM 2124 can be, for example, a flash basic input / output system (BIOS).

[0226] The HDD 2126 and the CD-ROM drive 2130 are connected to the SB / ICH 2104 through the bus 2140. For the HDD 2126 and the CD-ROM drive 2130, for example, an integrated drive electronics (IDE) or a serial advanced technology attachment (SATA) interface can be used. The super I / O (SIO) device 2136 can be connected to the SB / ICH 2104.

[0227] The operating system operates in the processing unit 2106. This operating system coordinates the various components within the data processing system 2100 in FIG. 21 and provides its control. As a client, the operating system can be a commercially available operating system such as Microsoft® Windows 10®. An object-oriented programming system such as the Java® programming system can operate in conjunction with the operating system and provide calls to the operating system from Java® programs or applications executed in the data processing system 200.

[0228] As a server, the data processing system 2100 can be, for example, an IBM eServer® System p® computer system, a Power® processor-based computer system that operates the Advanced Interactive Executive (AIX®) operating system, or the LINUX® operating system. The data processing system 2100 can be a symmetric multi-processor (SMP) system that includes multiple processors in the processing unit 2106. Alternatively, a single-processor system may be employed.

[0229] The operating system, object-oriented programming system, and application or program instructions are located in a storage device such as the HDD 2126 and are read into the main memory 2108 for execution by the processing unit 2106. The processes of the exemplary embodiments of the present invention can be performed by the processing unit 2106 using computer-usable program code located in a memory such as the main memory 2108, ROM 2124, or, for example, one or more peripheral devices 2126 and 2130.

[0230] A bus system, such as bus 2138 or bus 2140 shown in FIG. 21, may be composed of one or more buses. Of course, the bus system can be implemented using any type of communication fabric or communication architecture that provides data transfer between various components or devices attached to the fabric or architecture. Examples of communication units such as the modem 2122 or network adapter 2112 in FIG. 21 can include one or more devices used to transmit and receive data. The memory can be, for example, main memory 2108, ROM 2124, or cache, such as that found in NB / MCH 2102 in FIG. 21.

[0231] As described above, in some exemplary embodiments, the mechanisms of the exemplary embodiments can be implemented as application software stored in a storage device such as HDD 2126 and loaded into a memory such as main memory 2108 for execution by one or more hardware processors such as processing unit 2106, or by specific-purpose hardware or firmware, etc. Thus, the computing device shown in FIG. 21 will be specifically configured to implement the mechanisms of the exemplary embodiments and will be specifically configured to perform the operations described herein and generate outputs with respect to the lesion detection and classification artificial intelligence pipeline.

[0232] Those skilled in the art will understand that the hardware in FIGS. 20 and 21 may vary depending on the implementation. Other internal hardware or peripheral devices such as flash memory, equivalent non-volatile memory, or optical disk drives may be used in addition to or instead of the hardware shown in FIGS. 20 and 21. Also, the processes of this exemplary embodiment may be applied to multiprocessor data processing systems other than the SMP systems described above without departing from the spirit and scope of the present invention.

[0233] Also, the data processing system 2100 can take any form of several different data processing systems, including client computing devices, server computing devices, tablet computers, laptop computers, telephones or other communication devices, personal digital assistants (PDAs), and the like. In some exemplary examples, the data processing system 2100 can be a portable computing device configured with flash memory that provides non-volatile memory for storing, for example, operating system files or user-generated data or both. In essence, the data processing system 2100 can be any known or yet-to-be-developed data processing system without constructional constraints.

[0234] As described above, it should be understood that the exemplary embodiments can take the form of an all-hardware embodiment, an all-software embodiment, or an embodiment that includes both hardware elements and software elements. In one example embodiment, the mechanisms of the exemplary embodiments can be implemented in software or program code, including but not limited to firmware, resident software, microcode, and the like.

[0235] A data processing system suitable for storing or executing or both program code will include at least one processor directly or indirectly coupled to a memory element through a communication bus, such as a system bus. Memory elements can include local memory actually employed during actual execution of the program code, a mass storage area, and a cache memory that provides at least some temporary storage of some program code to reduce the number of times the code must be retrieved from the mass storage area during execution. The memory can be of various types including, but not limited to, ROM, PROM, EPROM, EEPROM, DRAM, SRAM, flash memory, solid-state memory, and the like.

[0236] Input / output devices, i.e., I / O devices (including, but not limited to, keyboards, displays, pointing devices, etc.), can be coupled to this system either directly or through intervening wired or wireless I / O interfaces or controllers or both. The I / O devices can take many different forms other than conventional keyboards, displays, pointing devices, etc., such as communication devices coupled through wireless or wired connections, including but not limited to smartphones, tablet computers, touch screen devices, voice recognition devices, etc. Any I / O device known or developed in the future is intended to be within the scope of the exemplary embodiments.

[0237] A network adapter can be coupled to the system to couple the data processing system through an intervening private or public network to other data processing systems or remote printers or storage devices. Modems, cable modems, and Ethernet cards are only a small part of the currently available network adapter types for wired communication. Network adapters based on wireless communication, including but not limited to 802.11a / b / g / n wireless communication adapters, Bluetooth (registered trademark) wireless adapters, etc., can also be utilized. Any network adapter known or developed in the future is intended to be within the spirit and scope of the present invention.

[0238] The description of the present invention is presented for purposes of illustration and description, and is not intended to be exhaustive or to limit the invention to the forms disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The embodiments were chosen and described in order to best explain the principles of the invention and its practical application, and to enable others skilled in the art to understand the invention for various embodiments with various modifications suitable for the particular uses contemplated. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application that exceeds the technology seen in the market or the technical improvement, or to enable others skilled in the art to understand the embodiments disclosed herein. 。 [Item 1] A method in a data processing system comprising at least one processor and at least one memory, wherein the at least one memory contains instructions for implementing a trained machine learning computer model executed by the at least one processor to determine the connectivity of lesions in the z - direction, the method comprising: identifying a first lesion in a given slice within a three - dimensional medical image and a second lesion in an adjacent slice within the three - dimensional medical image with respect to the given slice; determining a first intersection value between the first lesion and the second lesion with respect to the first lesion; determining a second intersection value between the first lesion and the second lesion with respect to the second lesion; determining whether the first lesion and the second lesion belong to the same three - dimensional lesion based on the first intersection value and the second intersection value A method as described above. [Item 2] The step of determining the first intersection value has the step of calculating the first intersection value r 0 as follows,

Number

Number

Number

Number

Number

Number

Number

Number

Number

Explanation of Symbols

[0239] 100 AI Pipeline 128 Structure Minimum Determination 130 Liver / Lesion Detection Logic Stage 132 Group of ML / DL Computer Models 133 Group of ML / DL Computer Models 134 Group of ML / DL Computer Models 135 Group of ML / DL Computer Models, Lesion, Detection Map 136 Group of ML / DL Computer Models 140 Lesion Segmentation Logic Stage 148 List, Input 150 False Positive Processing Stage 160 Lesion Classification Stage 600 Group of ML / DL Computer Models 610 First ML / DL Computer Model 612 U-Net Neural Network Model 614 Liver Mask 620 Second ML / DL Computer Model 621 DenseNet-169 (D169) Encoder 622 Decoder, Encoder 623 Decoder, Encoder, Slice Averaging (SLC AVG) Logic 624 Output, Lesion Output, Lesion Prediction Output 625 Output, Lesion Output, Lesion Prediction Output 626 SLC AVG Module 627 Loss of consistency, consistency loss logic 630 Third ML / DL computer model 634 Encoder section, encoder, encoder neural network, CNN 635 Encoder section, encoder, encoder neural network, CNN 636 Encoder section, encoder, encoder neural network, CNN 638 Decoder stage, decoder 639 Second loss function, deep supervision 640 Average operation, volume averaging (VOL AVG) logic 650 Final Lesion mask, final lesion output 660 Liver mask output 810 First stage 820 Second stage 910 Lesion area 911 Lesion 912 Lesion 913 Lesion 1051 Seed 1061 Seed 1071 Seed 1101 2D lesion mask 1111 Distance map 1112 Seed 1113 Watershed segmentation lesion mask 1121 Updated segmented lesion mask 1310 Slice 1311 Lesion 1312 Lesion 1320 Slice 1321 Lesion 1330 Slice 1331 Lesion 1332 Lesion 1611 Contour 1612 Contour 1810 Input 1820 First trained ML / DL computer model 1830 True Positive Evaluation Logic Circuit 1835 List of Filtered Lesions 1840 Second ML / DL Computer Model 1845 List of Filtered Lesions 2000 Cognitive System 2002 Computer Network 2004A Computing Device, Server 2004B Computing Device 2004C Computing Device 2004D Computing Device 2006 Corpus 2006 Electronic Document Corpus or Electronic Document Corpora, Data Corpus or Data Corpora 2007 Cognitive System 2008 Request Processing Pipeline 2010 Cognitive System User 2010 - 2012 Client Computing Device 2020 Lesion Detection and Classification AI Pipeline 2100 Data Processing System 2102 Memory Controller Hub (NB / MCH) 2104 Input / Output (I / O) Controller Hub (SB / ICH) 2106 Processing Unit 2108 Main Memory 2110 Graphics Processor 2112 Local Area Network (LAN) Adapter 2116 Audio Adapter 2120 Keyboard and Mouse Adapter 2122 Modem 2124 Read-Only Memory (ROM) 2126 Hard Disk Drive (HDD), Peripheral Device 2130 CD-ROM Drive, Peripheral Device 2132 Universal Serial Bus (USB) ports and other communication ports 2134 PCI / PCIe devices 2136 Super I / O (SIO) devices 2138 Bus 2140 Bus

Claims

1. 1. A method in a data processing system comprising at least one processor and at least one memory, wherein the at least one memory contains instructions, when executed by the at least one processor, for implementing a trained machine learning computer model for determining z-direction lesion connectivity, the method comprising: For a given slice in a three-dimensional medical image, identifying a first lesion in the given slice and a second lesion in an adjacent slice in the three-dimensional medical image; determining, for the first lesion, a first intersection value between the first lesion and the second lesion; determining a second intersection value between the first lesion and the second lesion for the second lesion; determining whether the first lesion and the second lesion belong to the same three-dimensional lesion based on the first intersection value and the second intersection value; It is equipped with determining the first intersection value comprises calculating the first intersection value r 0 as follows: 【Number 1】 A is the first lesion, B is the second lesion, |A| denotes the area of the first lesion, |B| denotes the area of the second lesion, and |A∩B| denotes the area of the intersection of the first and second lesions; determining the second intersection value comprises calculating the second intersection value r 1 as follows: 【Number 2】 A is the first lesion, B is the second lesion, |A| denotes the area of the first lesion, |B| denotes the area of the second lesion, and |A∩B| denotes the area of the intersection of the first and second lesions; determining whether the first lesion and the second lesion belong to the same three-dimensional lesion comprises providing the first intersection value and the second intersection value as features to the trained machine learning computer model, wherein the trained machine learning computer model generates a probability that the first lesion and the second lesion belong to the same three-dimensional lesion; the trained machine learning computer model learns to linearly combine the first intersection value r 0 and the second intersection value r 1 as follows: [Equation 3] c 0 , c 1 , and b are the operational parameters learned from the training volume by the machine learning training operation; method.

2. The step of determining whether the first lesion and the second lesion belong to the same three-dimensional lesion includes determining whether the probability that the first lesion and the second lesion belong to the same three-dimensional lesion is greater than a predetermined threshold value t. The method according to claim 1.

3. For all slices of the three-dimensional medical image, determining whether each first lesion in each slice and each second lesion in each adjacent slice belong to the same three-dimensional lesion; Connecting the lesions belonging to the same three-dimensional lesion in the three-dimensional medical image along the z-axis The method according to claim 1 or 2, further comprising:

4. A procedure for implementing a trained machine learning computer model for determining the connectivity of lesions in the z-direction on a computing device, wherein the trained machine learning computer model For a given slice in the three-dimensional medical image, identifying a first lesion in the given slice in the three-dimensional medical image and a second lesion in an adjacent slice; Determining a first intersection value between the first lesion and the second lesion with respect to the first lesion; Determining a second intersection value between the first lesion and the second lesion with respect to the second lesion; Based on the first intersection value and the second intersection value, determining whether the first lesion and the second lesion belong to the same three-dimensional lesion Executing the procedure, Determining the first intersection value includes calculating the first intersection value r0 as follows: [Number 4] A is the first lesion, B is the second lesion, |A| represents the area of the first lesion, |B| represents the area of the second lesion, and |A∩B| represents the area of the common set of the first lesion and the second lesion. Determining the second intersection value includes calculating the second intersection value r1 as follows: 【Number 5】 A is the first lesion, B is the second lesion, |A| represents the area of the first lesion, |B| represents the area of the second lesion, and |A∩B| represents the area of the common set of the first lesion and the second lesion. Determining whether the first lesion and the second lesion belong to the same three-dimensional lesion includes providing the trained machine learning computer model with the first intersection value and the second intersection value, and the trained machine learning computer model generates a probability that the first lesion and the second lesion belong to the same three-dimensional lesion. The trained machine learning computer model learns to linearly combine the first intersection value r0 and the second intersection value r1 as follows: [Equation 6] c0, c1, and b are operation parameters learned from the training volume by a machine learning training operation. Computer program. **Claim 5** The computer program according to claim 4, wherein determining whether the first lesion and the second lesion belong to the same three-dimensional lesion includes determining whether the probability that the first lesion and the second lesion belong to the same three-dimensional lesion is greater than a predetermined threshold value t. **Claim 6** The trained machine learning computer model for each slice of the three-dimensional medical image, determining whether each first lesion in each slice and each second lesion in each adjacent slice belong to the same three-dimensional lesion; connecting the lesions belonging to the same three-dimensional lesion in the three-dimensional medical image along the z-axis is executed. The computer program according to claim 4 or 5. **Claim 7** An apparatus comprising a processor and a memory coupled to the processor, wherein the memory includes instructions that, when executed by the processor, cause the processor to implement a trained machine learning computer model for determining the connectivity of lesions in the z-direction, and when the trained machine learning computer model is executed, for a given slice in a three-dimensional medical image, identifying a first lesion in the given slice in the three-dimensional medical image and a second lesion in an adjacent slice; determining a first intersection value between the first lesion and the second lesion with respect to the first lesion; determining a second intersection value between the first lesion and the second lesion with respect to the second lesion; ​ determining whether the first lesion and the second lesion belong to the same three-dimensional lesion based on the first intersection value and the second intersection value; Run Determining the first intersection value includes calculating the first intersection value r 0 as follows: 【Number 7】 A is the first lesion, B is the second lesion, |A| denotes the area of the first lesion, |B| denotes the area of the second lesion, and |A∩B| denotes the area of the intersection of the first and second lesions; Determining the second intersection value includes calculating the second intersection value r 1 as follows: [Equation 8] determining whether the first lesion and the second lesion belong to the same three-dimensional lesion includes providing the first intersection value and the second intersection value as features to the trained machine learning computer model, wherein the trained machine learning computer model generates a probability that the first lesion and the second lesion belong to the same three-dimensional lesion; the trained machine learning computer model learns to linearly combine the first intersection value r 0 and the second intersection value r 1 as follows: [Equation 9] c 0 , c 1 , and b are the operational parameters learned from the training volume by the machine learning training operation; Device.

Images