Method, computer program and apparatus for relabeling seeds for seed-based segmentation of medical images (Relabeling seeds for seed-based segmentation of medical images)

Abstract

To improve an image analysis mechanism automatized so as to detect an affected part in an imaged anatomical structure.SOLUTION: An affected part mask of a volume of a three-dimensional medical image is received. Each affected part that is detected includes an affected part outline. A distance map of a specific two-dimensional slice in volume of the medical image is generated on the basis of the affected part mask affected part mask. The distance map includes a distance to the affected part outline every voxel of the specific two-dimensional slice. A maximum value is identified, and a set of maximum value is selected from the distance map so as to have larger value than a point each maximum value is directly adjacent to. A seed is relabeled on the basis of the distance map and the set of maximum value to generate a set of seeds. Each seed expresses a center of an individual structural element of the affected part outline. Segmentation of an image is performed with respect to the affected part mask affected part mask on the basis of the set of seeds to form the affected part mask divided into an affected part mask.SELECTED DRAWING: Figure 11A

Description

[Technical Field]

[0001] FIELD OF THE INVENTION This application relates generally to improved data processing apparatus and methods, and more particularly to a mechanism for relabeling seeds for seed-based segmentation of medical images. [Background technology]

[0002] Liver lesions are groups of abnormal cells within the liver, a biological entity sometimes called a mass or tumor. Noncancerous, or benign, liver lesions are common and do not spread to other areas of the body. Such benign liver lesions generally do not cause any health problems. However, some liver lesions result from cancer. Patients with certain medical conditions may be more likely than others to have cancerous liver lesions. These medical conditions include, for example, hepatitis B or C, cirrhosis, iron storage disease (hemochromatosis), obesity, or exposure to toxic chemicals (such as arsenic or aflatoxin).

[0003] Liver lesions are typically identifiable only through medical imaging, such as ultrasound, magnetic resonance imaging (MRI), computerized tomography (CT), or positron emission tomography (PET) scans. Such medical imaging must be viewed by a human subject matter expert (SME), who uses their expertise and the human ability to understand patterns in images to determine whether the medical imaging shows any lesions. If a possible cancerous lesion is identified by the human SME, the patient's physician can perform a biopsy to determine whether the lesion is cancerous.

[0004] Abdominal contrast-enhanced (CE) CT is the current standard for evaluating various liver abnormalities (e.g., lesions). These lesions may be assessed by human SMEs as malignant (hepatocellular carcinoma, cholangiocarcinoma, angiosarcoma, metastases, and other malignant lesions) or benign (hemangioma, focal nodular hyperplasia, adenoma, cyst or lipoma, granuloma, etc.). Manual evaluation of such images by human SMEs is important in guiding subsequent therapeutic interventions. To properly evaluate lesions in CE CT, multiple phase studies have been conducted. These studies provide medical images of different phases of enhancement of healthy liver parenchyma and comparison with lesion enhancement to determine differential detection. Human SMEs can then determine the diagnosis of the lesion based on these differences. Summary of the Invention [Problem to be solved by the invention]

[0005] Although several automated image analysis mechanisms have been developed, there is still a need to improve such automated image analysis mechanisms to provide more efficient and correct analysis of medical image data to detect lesions within the imaged anatomical structure (e.g., the liver or other organs). [Means for solving the problem]

[0006] This Summary is provided to introduce a selection of concepts in a simplified form that are further described herein in the Detailed Description. This Summary is not intended to identify key elements or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0007] In one example embodiment, a data processing system including at least one processor and at least one memory is provided, wherein the at least one memory includes instructions, when executed by the at least one processor, for implementing a trained machine learning computer model for seed relabeling of slice-by-slice lesion segmentation based on seeds. The trained machine learning computer model performs the method, which includes receiving a lesion mask for a volume of a three-dimensional medical image. The lesion mask corresponds to detected lesions in the volume of the medical image, and each detected lesion includes a lesion contour. The method further includes generating a distance map for a particular two-dimensional slice in the volume of the medical image based on the lesion mask. The distance map includes, for each voxel of the particular two-dimensional slice, a distance to the lesion contour. The method further includes performing local maximum identification and selecting a set of local maxima from the distance map such that each local maximum has a value greater than its immediately neighboring points. The method further includes performing seed relabeling based on the distance map and the set of local maxima to generate a set of seeds, each seed representing the center of a distinct component of the lesion contour. The method further includes performing image segmentation on the lesion mask based on the set of seeds to form a lesion mask segmented lesion mask, which has the advantage of providing a lesion mask segmented lesion mask that is not overly segmented. The method of example embodiments combines regions in the lesion mask segmented lesion mask that are likely to be part of the same lesion.

[0008] In one example embodiment, generating the distance map includes performing Gaussian smoothing on the distance map, which has the advantage of smoothing distance values ​​that result in irrelevant maxima that could lead to further over-segmentation within the segmented lesion mask. Performing Gaussian smoothing also reduces the number of candidate points that may be considered local maxima, potentially reducing the number of distinct regions that must be combined during seed relabeling.

[0009] In another example embodiment, performing seed relabeling includes grouping the first and second maxima in response to determining that the first and second maxima are immediately adjacent, which has the advantage of eliminating immediately adjacent maxima as candidates for segmenting the lesion, since maxima that are very close together are unlikely to be centers of individual lesions.

[0010] In yet another example embodiment, performing seed relabeling includes determining a circle centered on each local maximum with a radius equal to the corresponding distance in the distance map, calculating an overlap index for a first circle centered on the first local maximum and a second circle centered on the second local maximum, and grouping the first and second local maximums if the overlap index is greater than a predetermined threshold. This embodiment assumes that the shape of the lesion is substantially circular, like a bubble, and determines whether the local maxima are centers of overlapping bubbles. This has the advantage of grouping local maxima together if they are likely to represent the same lesion.

[0011] In yet another example embodiment, the overlap measure is calculated as follows:

[0012]

number

[0013] where |S1| denotes the area of ​​the first circle, |S2| denotes the area of ​​the second circle, and |S1∩S2| denotes the area of ​​the intersection of the first and second circles. In an alternative example embodiment, the overlap index is calculated as follows:

[0014]

number

[0015] where |S1| denotes the area of ​​the first circle, |S2| denotes the area of ​​the second circle, |S1∩S2| denotes the area of ​​the intersection of the first and second circles, and |S1∪S2| denotes the area of ​​the union of the first and second circles. These embodiments provide the advantage of assigning a value to the index to determine whether local maxima define separate lesions or are likely to correspond to the same lesion. These embodiments provide an alternative formula for calculating the value of the overlap index that can be compared to a threshold.

[0016] In another example embodiment, performing image segmentation includes performing a watershed algorithm on the lesion mask based on the set of local maxima to form an initial lesion mask segmented lesion mask defining the first set of lesions. In yet another example embodiment, performing image segmentation further includes merging lesions in the first set of lesions based on results of the seed relabeling to form a modified lesion mask segmented lesion mask. These embodiments provide the advantage of using the known watershed algorithm to segment the image into lesion mask segmented lesion masks while also merging regions that are likely to be part of the same lesion, thus preventing over-segmentation.

[0017] In another example embodiment, a computer program product is provided that includes a computer usable or computer readable medium containing a computer readable program that, when executed on a computing device, causes the computing device to perform various and combinations of the operations outlined above with respect to the example method embodiments.

[0018] In yet another example embodiment, a system / apparatus is provided that may include one or more processors and a memory coupled to the one or more processors. The memory may include instructions that, when executed by the one or more processors, cause the one or more processors to perform various and combinations of the operations outlined above with respect to the example method embodiment.

[0019] These and other features and advantages of the present invention will be described, or will become apparent to those skilled in the art, in view of the following detailed description of exemplary embodiments of the invention. [Brief explanation of the drawings]

[0020] The present invention, together with its most preferred uses and other objects and advantages, will be best understood by reference to the following detailed description of exemplary embodiments, read in conjunction with the accompanying drawings.

[0021] [Figure 1] FIG. 1 is an exemplary block diagram of an AI pipeline implementing multiple ML / DL computational models specifically configured and trained to perform anatomical structure identification and lesion detection within input medical image data, according to one example embodiment.

[0022] [Figure 2] 1 is an exemplary flowchart outlining an exemplary operation of an AI pipeline, according to an example embodiment.

[0023] [Figure 3A] 1 is an exemplary diagram illustrating an exemplary input volume of a slice (medical image) of a patient's abdomen, according to an example embodiment;

[0024] [Figure 3B] FIG. 3B shows another representation of the input volume of FIG. 3A, including sections of slices represented with corresponding axial scores s′inf and s′sup.

[0025] [Figure 3C] FIG. 3B is an exemplary illustration of the input volume of FIG. 3A, where the volume is divided axially into n completely overlapping sections.

[0026] [Figure 4A] FIG. 1 is an exemplary diagram of one embodiment of a trained ML / DL computer model configured to estimate values ​​of s'sup and s'inf for a section of an input volume of a medical image, according to one example embodiment. [Figure 4B] FIG. 1 is an exemplary diagram of one embodiment of a trained ML / DL computer model configured to estimate values ​​of s'sup and s'inf for a section of an input volume of a medical image, according to one example embodiment. [Figure 4C] FIG. 1 is an exemplary diagram of one embodiment of a trained ML / DL computer model configured to estimate values ​​of s'sup and s'inf for a section of an input volume of a medical image, according to one example embodiment.

[0027] [Figure 5] 1 is a flowchart outlining an exemplary operation of the liver detection and predetermined amount of anatomy determination logic of the AI ​​pipeline, according to an example embodiment.

[0028] [Figure 6] FIG. 1 is an exemplary diagram of a population of ML / DL computer models used to perform lesion detection in a target anatomy (e.g., liver), according to an example embodiment.

[0029] [Figure 7] 1 is a flowchart outlining an exemplary operation of liver / lesion detection logic in an AI pipeline, according to an example embodiment.

[0030] [Figure 8]FIG. 1 is a block diagram illustrating aspects of lesion segmentation, according to an example embodiment.

[0031] [Figure 9] FIG. 10 illustrates lesion detection and slice-by-slice segmentation results according to an example embodiment.

[0032] [Figure 10A] FIG. 10 illustrates seed positioning according to an example embodiment. [Figure 10B] FIG. 10 illustrates seed positioning according to an example embodiment. [Figure 10C] FIG. 10 illustrates seed positioning according to an example embodiment. [Figure 10D] FIG. 10 illustrates seed positioning according to an example embodiment.

[0033] [Figure 11A] FIG. 1 is a block diagram illustrating a lesion segmentation mechanism according to an example embodiment.

[0034] [Figure 11B] FIG. 2 is a block diagram illustrating a seed relabeling mechanism according to an example embodiment.

[0035] [Figure 12] 1 is a flowchart outlining an exemplary operation of lesion segmentation according to an example embodiment.

[0036] [Figure 13A] FIG. 10 illustrates z-direction connectivity of lesions according to an example embodiment. [Figure 13B] FIG. 10 illustrates z-direction connectivity of lesions according to an example embodiment. [Figure 13C] FIG. 10 illustrates z-direction connectivity of lesions according to an example embodiment.

[0037] [Figure 14A]FIG. 10 illustrates trained model results for lesion connectivity in z-direction, according to an example embodiment. [Figure 14B] FIG. 10 illustrates trained model results for lesion connectivity in z-direction, according to an example embodiment.

[0038] [Figure 15] 10 is a flowchart outlining an exemplary operation of a mechanism for connecting two-dimensional lesions along the z-axis, according to an example embodiment.

[0039] [Figure 16] FIG. 10 is a diagram illustrating an example including contours of two lesions in the same image, according to an example embodiment.

[0040] [Figure 17] 10 is a flowchart outlining an exemplary operation of a mechanism for slice-by-slice contour refinement according to an example embodiment.

[0041] [Figure 18A] FIG. 10 illustrates example ROC curves for patient-level and lesion-level operating point determination according to an example embodiment.

[0042] [Figure 18B] FIG. 10 is an exemplary flow diagram of operations for performing false positive rejection based on patient-level and lesion-level operating points according to an example embodiment.

[0043] [Figure 18C] FIG. 10 is an exemplary flow diagram of operations for performing voxel-wise false positive removal based on an input volume-level and voxel-level operating point, according to an example embodiment.

[0044] [Figure 19] 1 is a flowchart outlining an exemplary operation of the false positive removal logic of an AI pipeline, according to an example embodiment.

[0045] [Figure 20] 1 is an exemplary diagram of a distributed data processing system in which aspects of example embodiments may be implemented;

[0046] [Figure 21] 1 is an exemplary block diagram of a computing device in which aspects of example embodiments may be implemented. DETAILED DESCRIPTION OF THE INVENTION

[0047] The detection of lesions or groups of abnormal cells is a primarily manual process in modern medical therapy. As a manual process, it is subject to sources of error due to human limitations in an individual's ability to detect portions of digital medical images that exhibit such lesions, especially given the increasing demands on individuals to evaluate increasingly large numbers of images in shorter periods of time. While several automated image analysis mechanisms have been developed, there remains a need to improve such automated image analysis mechanisms to provide more efficient and accurate analysis of medical image data to detect lesions within imaged anatomical structures (e.g., the liver or other organs).

[0048] Example embodiments are particularly directed to improved computing tools that provide automated, computer-based, artificial intelligence medical image analysis, specifically trained via machine learning / deep learning computational processes to: detect anatomical structures; detect lesions or other biological structures of interest within or associated with such anatomical structures; perform specialized segmentation of the detected lesions or other biological structures; perform false positive rejection based on the specialized segmentation; perform classification of the detected lesions or other biological structures; and provide the results of lesion / biological structure detection to a downstream computing system for performing additional computational operations. The following description of example embodiments assumes an embodiment specifically related to the example embodiment mechanism being specifically trained with respect to liver lesions as the biological structure of interest, but the example embodiments are not limited to such an embodiment. Rather, those skilled in the art will recognize that the example embodiment machine learning / deep learning-based artificial intelligence mechanism may be implemented with respect to many other types of biological structures / lesions within or associated with other anatomical structures represented within medical image data without departing from the spirit and scope of the present invention. Additionally, although example embodiments may be described with respect to medical image data that is computed tomography (CT) medical image data, example embodiments may be implemented using any digital medical image data from various types of medical imaging techniques, including, but not limited to, positron emission tomography (PET) and other nuclear medicine imaging, ultrasound, magnetic resonance imaging (MRI), elastography, photoacoustic imaging, echocardiography, magnetic particle imaging, functional near-infrared spectroscopy, elastography, various x-ray imaging including fluoroscopy, and the like.

[0049] Overall, example embodiments provide an improved artificial intelligence (AI) computer pipeline that includes a plurality of specifically configured and trained AI computer tools (e.g., neural networks, cognitive computing systems, or other AI mechanisms trained based on a finite set of data to perform a specific task). The configured and trained AI computer tools are each specifically configured / trained to perform a specific type of artificial intelligence processing of a volume of input medical images, represented as one or more collections of data and / or metadata defining the medical images captured by medical imaging technology. Generally, these AI tools employ machine learning (ML) / deep learning (DL) computer models (or simply, ML models) to perform their tasks. ML / DL computer models use different computer processes specific to the computer tools and, in particular, the ML / DL computer models, to learn patterns and relationships among data that represent specific results (e.g., image classifications or labels, data values, treatment recommendations, etc.) while emulating human thought processes with respect to the generated results. An ML / DL computer model is essentially a function of elements, including a machine learning algorithm, configuration settings for the machine learning algorithm, input data features identified by the ML / DL computer model, and labels (or outputs) generated by the ML / DL computer model. By specifically coordinating the functions of these elements through a machine learning process, a particular instance of an ML / DL computer model is generated. Different ML models may be specifically configured and trained to perform different AI functions on the same or different input data.

[0050] It should be understood that because an artificial intelligence (AI) pipeline implements multiple ML / DL computer models, these ML / DL computer models are trained through a purpose-specific ML / DL process. Therefore, as an overview of the ML / DL computer model training process, it should be understood that machine learning involves the design and development of techniques that take empirical data (e.g., medical imaging data) as input and recognize complex patterns within the input data. One common pattern in machine learning techniques is the use of an underlying computer model, M, whose parameters are optimized to minimize a cost function associated with M given the input data. For example, in the context of classification, model M may be a line separating data into two classes (e.g., labels) such that M = a*x+b*y+c, where the cost function is the number of misclassified points. The learning process then operates by adjusting parameters a, b, and c to minimize the number of misclassified points. After this optimization phase (or learning phase), model M can be used to classify new data points. Often, M is a statistical model, and the cost function is inversely proportional to the likelihood of M given the input data. This is merely a simple example to provide a general description of machine learning training; other types of machine learning using different patterns, cost (or loss) functions, and optimizations may be used with the mechanisms of the example embodiments without departing from the spirit and scope of the present invention.

[0051] For the purposes of anatomical structure detection and / or lesion detection (lesions are "abnormalities" in medical image data), a learning machine can construct an ML / DL computer model of normal structural representation and detect data points in the medical image that deviate from this ML / DL computer model of normal structural representation. For example, a particular ML / DL computer model (e.g., a supervised, unsupervised, or semi-supervised model) may be used to generate and report an abnormality score to another device, generate a classification output indicating one or more classes into which the input is classified, generate probabilities or scores associated with various classes, etc. Examples of machine learning techniques that may be used to build and analyze such ML / DL computer models may include, but are not limited to, nearest neighbor (NN) techniques (e.g., k-NN models, replicator NN models, etc.), statistical techniques (e.g., Bayesian networks, etc.), clustering techniques (e.g., k-means, etc.), neural networks (e.g., reservoir networks, artificial neural networks, etc.), support vector machines (SVMs), etc.

[0052] The processor-implemented artificial intelligence (AI) pipelines of example embodiments typically include one or both of machine learning (ML) and deep learning (DL) computational models. In some cases, one or the other of ML and DL may be used or implemented to achieve a particular result. Traditional machine learning may include or use algorithms such as Bayesian decision making, regression, decision trees / forests, support vector machines, or neural networks, among others. Deep learning may be based on deep neural networks and may use multiple layers, such as convolutional layers. Such DL, using hierarchical networks, may be efficient in its implementation and achieve improved accuracy compared to traditional ML techniques. Traditional ML may be distinguished from DL in that the performance of DL models can generally exceed that of classical ML models, but DL models may consume relatively greater amounts of processing resources, power resources, or both. In the context of example embodiments, references herein to one or the other of ML and DL may be understood to encompass one or both forms of AI processing.

[0053] With respect to example embodiments, the ML / DL computer model of the AI ​​pipeline, after configuration and training via an ML / DL training process, executes to perform complex computational medical image analysis to detect anatomical structures within input medical images and generate outputs that, among other things, identify target biological structures of interest (hereinafter, for purposes of describing example embodiments, assumed to be liver lesions), their classification, contours specifying where these target biological structures of interest (e.g., liver lesions) reside within input medical images (hereinafter, assumed to be CT medical image data), and other information that assists human subject matter experts (SMEs), such as radiologists, physicians, etc., in understanding a patient's medical condition from the perspective of the captured input medical images. Furthermore, these outputs may be provided to other downstream computer systems to perform additional artificial intelligence operations, such as treatment recommendations and other decision support operations based on the classifications, contours, etc.

[0054] Initially, an artificial intelligence (AI) pipeline of an example embodiment receives an input volume of computed tomography (CT) medical image data and detects which parts of a biological entity's body are represented in the CT medical image data. A medical image "volume" is a three-dimensional representation of the biological entity's internal anatomical structure made up of a stack of two-dimensional slices, which may be individual medical images captured by medical imaging technology. The stacks of slices may be called "slabs," which may differ from the slices themselves in that they represent portions of the anatomical structure that have thickness; the stacks of slices or slabs generate a three-dimensional representation of the anatomical structure.

[0055] For purposes of this description, it is assumed that the biological entity is a human, although the present invention may operate on medical images of various types of biological entities. For example, in veterinary medicine, the biological entity may be various types of small animals (e.g., pets such as dogs, cats, etc.) or large animals (e.g., horses, cows, or other livestock). In an implementation where the AI ​​pipeline is specifically trained for liver lesion detection, the AI ​​pipeline determines whether the input CT medical image data represents an abdominal scan present in the CT medical image data, and if not, terminates operation of the AI ​​pipeline on the input CT medical image data because it is not directed to the correct part or portion of the human body. It should be understood that there may be different AI pipelines according to example embodiments trained to process input medical images of different body parts and different target biological structures, and that an input CT medical image may be input to each of the AI ​​pipelines or may be routed to the AI ​​pipelines based on a classification of the body part or body part depicted in the input CT medical image; for example, classification of the input CT medical image with respect to the body part or body part depicted in the input CT medical image may be performed first, and then a corresponding trained AI pipeline for processing the input CT medical image may be selected from multiple trained AI pipelines of the type described herein. For purposes of the following description, a single AI pipeline trained to detect liver lesions will be described, but this extension to a series or population of AI pipelines will be apparent to those skilled in the art in view of this description.

[0056] Assuming the volume of input CT medical images includes medical images of the abdomen of a human body (for purposes of liver lesion detection), processing of the input CT medical images is further performed in two early stages, which may be performed substantially in parallel with one another, sequentially, or both, depending on the desired implementation. The two early stages include a phase classification stage and an anatomical structure detection stage (e.g., a liver detection stage, if the AI ​​pipeline is configured to perform liver lesion detection).

[0057] The phase classification stage determines whether the input CT medical image volume contains a single imaging phase or multiple imaging phases. A "phase" in medical imaging is an indication of contrast agent uptake. For example, in some medical imaging techniques, phases may be defined with respect to when a contrast agent is introduced into a biological entity, enabling the capture of medical images, including capturing the contrast agent's path. For example, phases may include a pre-contrast phase, an arteriographic phase, a portal venous / venous phase, and a delayed phase, and medical images are captured in any or all of these phases. Phases are typically related to the timing after injection and the characteristics of enhancement of structures within the image. Timing information may be considered to "order" potential phases (e.g., the delayed phase is always acquired after the portal venous phase) and to estimate the potential phase of a particular image. With regard to using the enhancement properties of structures within an image, one example of using this type of information to determine phase is described in commonly assigned, co-pending U.S. patent application Ser. No. 16 / 926,880, entitled "Method of Determining Contrast Phase of a Computerized Tomography Image," filed July 13, 2020. Additionally, timing information can be used in conjunction with other information (sampling, reconstruction kernels, etc.) to extract the best representation of each phase (a particular acquisition may be reconstructed in multiple ways).

[0058] After the images in the input volume are assigned or classified to corresponding phases based on the timing and / or characteristics of the enhancement, it may be determined based on the phase classification whether the volume contains images from a single phase (e.g., a portal vein / venous phase is present but an arterial phase is absent) or multiple phases (e.g., a portal vein / venous and arterial) of the examination. If the phase classification indicates that a single phase is present in the volume of the input CT medical images, processing by the AI ​​pipeline is further performed as described below. If multiple phases are detected, the volume is not further processed by the AI ​​pipeline. However, in some example embodiments, this filter for single / multiple phase-based volumes only accepts volumes containing images from a single phase and rejects volumes with multiple phases, while in other example embodiments, the AI ​​pipeline processing described herein may filter out images in the volume that were not classified into a target phase of interest. For example, images of the portal vein / venous phase in the volume may be retained while filtering out images in the volume that were not classified as being part of the portal vein / venous phase, thereby modifying the input volume to result in a modified volume that contains only a subset of images classified into the target phase. Additionally, as previously mentioned, different AI pipelines may be trained on different types of volumes, and in some example embodiments, the phase classification of images in the input volume may be used to route or distribute the images of the input volume to corresponding AI pipelines trained and configured to process images of different phases, thereby allowing the input volume to be further divided into constituent sub-volumes and routed to corresponding AI pipelines for processing; for example, a first sub-volume corresponding to images in the portal vein / venous phase may be sent to a first AI pipeline for processing, and a second sub-volume corresponding to images in the arterial phase may be sent to a second AI pipeline for processing.If the volume of input CT medical images contains a single phase, or the AI ​​pipeline processes images of the input volume or sub-volume of a single phase, after filtering and optionally routing the sub-volume to a corresponding AI pipeline, the volume (or sub-volume) is passed to the next stage of the AI ​​pipeline for further processing.

[0059] The second initial stage is a target anatomical structure (in this example embodiment, the liver) detection stage, in which portions of a volume representing the target anatomical structure are identified and passed to the next downstream stage in the AI ​​pipeline. The target anatomical structure detection stage (hereinafter, according to the example embodiment, referred to as the liver detection stage) includes a machine learning (ML) / deep learning (DL) computer model specifically trained and configured to perform computational medical image analysis to identify portions of input medical images that correspond to the target anatomical structure (e.g., the liver). Such medical image analysis may include training the ML / DL model on labeled training medical image data as input to determine whether the input medical image (during training, referred to as the training image) contains the target anatomical structure (e.g., the liver). Based on ground truth of the image labels, operating parameters of the ML / DL model may be adjusted to reduce loss or error in the results generated by the ML / DL model until convergence is reached (i.e., loss is minimized). This process trains the ML / DL model to recognize patterns in medical image data that indicate the presence of an anatomical structure of interest (in this example, the liver). Once trained, the ML / DL model can then be run against new input medical image data to determine if the new input medical image data contains the pattern indicative of the presence of the anatomical structure; if the probability exceeds a predetermined threshold, it can be determined that the medical image data contains the anatomical structure of interest.

[0060] Thus, in the liver detection phase, the AI ​​pipeline uses the trained ML / DL computer model to determine whether a volume of input CT medical images contains an image showing a liver. The portion of the volume showing the liver, along with the results of the phase classification phase, is passed to a determination phase of the AI ​​pipeline, which determines whether a single phase of the medical image is present and whether at least a predetermined amount of the target anatomical structure is present in the portion of the volume showing the target anatomical structure (e.g., liver). The determination of whether a predetermined amount of the target anatomical structure is present may be determined based on a known measurement mechanism that determines a measurement of the structure from the medical image (e.g., calculating the size of the structure from the difference in pixel locations within the image). If the measurement represents at least a predetermined amount or portion of the anatomical structure, the measurement may be compared to a predetermined size (e.g., average size) of the anatomical structure in similar patients with similar demographics so that further processing can be performed by the AI ​​pipeline. In one example embodiment, this determination determines, for example, whether at least one-third of the liver is present in the portion of the volume of the input CT medical image determined to show the liver. Although 1 / 3 is used in this example embodiment, any predetermined amount of structure determined to be suitable for a particular implementation may be used without departing from the spirit and scope of the present invention.

[0061] In one example embodiment, to determine whether a predetermined amount of the target anatomical structure is present in an input CT medical image volume, axial scores are defined such that a slice in the volume corresponding to a medical image containing a first representation of the target anatomical structure (e.g., the liver) (i.e., the first slice containing the liver (FSL)) is given a slice score of 0, and the last slice containing the liver (LSL) has a score of 1. Assuming a human biological entity, the first and last slices are defined proceeding from the least significant slice in the volume (MISV) (closest to the lower extremities (e.g., feet)) to the most significant slice in the volume (MSSV) (closest to the head). A pair of slice scores (s) corresponding to the slice scores of the MSSV slice and the MISV slice, respectively, are provided. sup and s inf ) defines the liver axial score estimate (LAE). The ML / DL computer model calculates the slice scores s of the input CT medical image volume, as will be explained in more detail later. sup and s inf The mechanism of the example embodiment recognizes these slice scores, and, recognizing from the above definition that the liver spans from 0 to 1, is able to determine the liver fragments within the field of view of the input CT medical image volume.

[0062] In some example embodiments, slice scores s sup and s inf First, the input CT medical image volume is divided into sections, and then the top slice (closest to the head) and bottom slice (closest to the feet) of the liver in each section are identified as s'. sup and s' infIt may be indirectly detected by running a constructed and trained ML / DL computer model on the slices of the section to estimate the height per slice, for each section, to determine s'. sup and s' inf Given an estimate of , we know how the sections are located relative to the entire volume of the input CT medical image, so we can extrapolate to find s sup and s inf An estimate of s is found. This method is based on a robust estimator of the height of any slice from the input volume (or subvolume associated with the target phase). Such an estimator can be obtained, for example, by training a regression model using a deep learning model that performs height estimation from chunks (sets of consecutive slices). For example, long short-term memory (LSTM)-based artificial neural network models are suitable for these tasks and have the ability to encode the ordering of slices containing the liver and abdominal anatomy. For each volume, s sup and s inf It should be noted that there are n estimates of , where n is the number of sections per volume. In one example embodiment, the final estimate is obtained by taking an unweighted average of these n estimates, although in other example embodiments, other functions of the n estimates may be used to generate the final estimate.

[0063] Input CT medical image volume s sup and s infA final estimate of h is determined, and based on these values, a fraction of the target anatomical structure (e.g., the liver) is calculated. This task is made possible by estimating the height of each slice. From the estimate of the height of the first slice (h1) and the height of the last slice (h2) of the liver in the input volume, assuming that the heights of the actual first and last slices of the liver are H1 and H2 (whether or not they are included in the input volume), the visible portion of the liver in the input volume can be expressed as (min(h1, H1)-max(h2, H2)) / (H1-H2). This calculated fraction can then be compared to a predetermined threshold to determine whether a predetermined minimum amount of the target anatomical structure is present in the input CT medical image volume (e.g., at least 1 / 3 of the liver is present in the input CT medical image volume).

[0064] If this determination results in a determination that multiple phases are present and / or a predetermined amount of the anatomical structure of interest is not present in the portion of the volume of the input CT medical image depicting the anatomical structure, processing of further volumes may be halted. If this determination results in a determination that the volume of the input CT medical image contains a single phase and at least a predetermined amount of the anatomical structure of interest (e.g., one-third of the liver is visible in the image), the portion of the volume of the input CT medical image depicting the anatomical structure is forwarded to the next stage of the AI ​​pipeline for processing.

[0065] In the next stage of the AI ​​pipeline, the AI ​​pipeline performs lesion detection on a portion of the volume of the input CT medical image representing an anatomical structure of interest (e.g., the liver). This liver and lesion detection stage of the AI ​​pipeline uses an ensemble of ML / DL computer models to detect the liver and lesions within the liver represented in the volume of the input CT medical image. The ensemble of ML / DL computer models performs liver and lesion detection using differentially trained ML / DL computer models, which are trained using a loss function to balance false positives and false negatives in lesion detection. Additionally, the ensemble of ML / DL computer models are configured such that a third loss function forces the outputs of the ML / DL computer models to match each other.

[0066] Assuming that liver detection and lesion detection are performed at this stage of the AI ​​pipeline, a first ML / DL computer model is run on a volume of input CT medical images to detect the presence of a liver. This ML / DL computer model may be the same as the ML / DL computer model employed in the previous stage of the AI ​​pipeline for detecting an anatomical structure of interest, and thus may utilize previously obtained results. Multiple (two or more) other ML / DL computer models are configured and trained to perform lesion detection in portions of the medical images showing the liver. The first ML / DL computer model is configured using two loss functions. The first loss function penalizes errors in false negatives (i.e., classifications that incorrectly indicate the absence of a lesion (normal anatomical structure)). The second loss function penalizes errors in false positive results (i.e., classifications that incorrectly indicate the presence of a lesion (abnormal anatomical structure)). The second ML / DL model is trained to detect lesions using an adaptive loss function that penalizes false positive errors in liver slices containing normal tissue and false negative errors in liver slices containing lesions. The output detections from the two ML / DL models are averaged to produce the final lesion detection.

[0067] The results of the liver / lesion detection stage of the AI ​​pipeline include one or more contours (outline lines) of the liver and a detection map (e.g., a voxel-by-voxel map of liver lesions detected within the volume of the input CT medical image) that identifies portions of the medical image data elements that correspond to the detected lesions. The image map is then input to the lesion segmentation stage of the AI ​​pipeline. The lesion segmentation stage, as described in more detail below, uses a watershed technique to segment the detection map and generate a segmentation of the image elements (e.g., voxels) of the input CT medical image. Based on this segmentation, the liver lesion segmentation stage identifies all of the contours that correspond to lesions present in the slice of the volume of the input CT medical image and performs operations to identify which contours correspond to the same lesion in three dimensions. The lesion segmentation aggregates the interrelated lesion contours and generates a three-dimensional segmentation of the lesion. The lesion segmentation uses inpainting of the lesion image elements (e.g., voxels) and non-liver tissues represented within the medical image to focus on each lesion individually and perform active contour analysis. In this way, individual lesions may be identified and processed without biasing the analysis due to other lesions in the medical image or due to portions of the image outside the liver.

[0068] The result of lesion segmentation is a list of lesions with their corresponding contours or outlines in the input CT medical image volume. These outputs may contain detections that are not actual lesions. To minimize the impact of these false positives, these outputs are provided to the next stage of the AI ​​pipeline, which targets false positive removal using a trained false positive removal model. The false positive removal model in the AI ​​pipeline acts as a classifier to identify which outputs are actual lesions and which detected detections are false positives. The input consists of a volume of images (VOI) centered on the detected detections, associated with the mask obtained from the lesion segmentation refinement. The false positive removal model is trained using the data resulting from the detection / segmentation stage. Objects from the ground truth that are lesions detected by the detection algorithm are used to represent the lesion class during training, while detections that do not match any lesions from the ground truth are used to represent the non-lesion (false positive) class.

[0069] To further improve overall performance, a dual operating point strategy was adopted for the lesion detection model and the false positive model. This idea highlights that the output of the AI ​​pipeline can be interpreted at different levels. First, the output of the AI ​​pipeline can be used to communicate whether an examination volume (i.e., input volume or volume of image (VOI)) contains a lesion. Second, the output of the AI ​​pipeline aims to maximize lesion detection, regardless of whether the lesions are contained in the same patient / examination / volume. For clarity, measurements taken for examinations are referred to herein as the "patient level," and measurements taken for lesions are referred to herein as the "lesion level." Maximizing sensitivity at the "lesion level" results in reduced specificity at the "patient level" (one detection is sufficient for a patient to be said to contain a lesion). This can ultimately be suboptimal for clinical use, as one must choose between insufficient specificity at the patient level or low sensitivity at the lesion level.

[0070] Taking this into consideration, example embodiments use a dual operating point approach for both lesion detection and false positive removal. The principle is to first run processing using a first operating point that provides reasonable performance at the patient level. Then, for patients with at least one lesion detected from the first run, a second operating point is used to reinterpret / process the detected lesions. This second operating point is selected to be more sensitive. Although this second operating point has lower specificity than the first operating point, this loss of specificity is included at the patient level because all patients whose lesions were not detected using the first operating point remain as they are, regardless of whether the second operating point would have detected additional lesions. Therefore, the specificity at the patient level is determined solely by the first operating point. The patient-level sensitivity is between the sensitivity of the first operating point selected alone and the sensitivity of the second operating point (some false negative cases from the first operating point may be turned into true positives by the second operating point). On the lesion side, the actual lesion-level sensitivity is improved compared to the first operating point alone. When processed using only the first operating point, no false positives occur, resulting in better specificity for the lesion than the second operating point selected alone, which has lower specificity.

[0071] While the example embodiments assume a particular configuration and use of the dual operating point method, it should be understood that the dual operating point method can be used in other configurations and for other purposes, such as when it is of interest to measure performance at the group level (in the example embodiments, this group level is the “patient level”) and the element level (in the example embodiments, this element level is the “lesion level”). In the example embodiments, the dual operating point method is applied to both lesion detection and false positive removal, but it should be understood that the dual operating point method can be extended beyond these stages of an AI pipeline. For example, lesion detection may be performed at the voxel level (element) and volume level (group) rather than the patient and lesion levels. As another example, the voxel level or lesion level may be used for the element level, and a slab (set of slices) may be used as the group level. In yet another example, instead of a single volume, all volumes of the exam may be used as the group level. It should be understood that this method may also be applied to two-dimensional images (e.g., 2D x-rays, such as chest, mammograms, etc.) to analyze images rather than three-dimensional volumes. Specificity measures such as the average number of false positives per patient / group may be used in selecting the operating point. Furthermore, although example embodiments are described as applying to lesion detection and classification, dual operating point based methods may be applied to other structures (clips, stents, implants, etc.) and beyond medical images.

[0072] The results of the dual operating point-based detection and false positive removal lead to the identification of a final, filtered list of lesions, which is further processed by the lesion classification stage of the AI ​​pipeline. In the lesion classification stage of the AI ​​pipeline, a configured and trained ML / DL computer model is run on the list of lesions and corresponding contour data to classify the lesions into one of several predefined lesion classifications. For example, each lesion in the final, filtered list and its attributes (e.g., contour data) may be input into a trained ML / DL computer model, which then operates on this data to classify the lesion as a specific lesion type. This classification may be performed using a classifier previously trained on ground truth data (e.g., a trained neural network computer model) in combination with the results of previous processing steps in the AI ​​pipeline. This classification task can be more or less complex, for example, providing a label such as benign, malignant, or indeterminate, or, in another example, providing the actual lesion type, such as cyst, metastasis, or hemangioma. The classifier can be, for example, a neural network-based computer model classifier (e.g., SVM, decision tree, etc.) or a deep learning computer model. The actual input for this classifier is the lesion-centered portion, which in some embodiments may be augmented with a lesion mask or contour line (outline).

[0073] After lesion classification by the lesion classification stage of the AI ​​pipeline, the AI ​​pipeline outputs a list of lesions and their classifications, along with any contour attributes of the lesions. Additionally, the AI ​​pipeline may output liver contour information for the liver. The information generated by this AI pipeline may be provided to a downstream computing system for further processing and generation of a representation of the target anatomical structure and any detected lesions present in the anatomical structure. For example, a graphical representation of the input CT medical image volume may be generated in a medical image viewer or other computer application, with the anatomical structure and detected lesions superimposed or otherwise enhanced on the graphical representation using the contour information generated by the AI ​​pipeline. In other example embodiments, downstream processing of the information generated by the AI ​​pipeline may include diagnostic decision support operations, automated medical image report generation, based on the detected list of lesions, classifications, and contours. In other example embodiments, different treatment recommendations may be generated for review and consideration by a physician based on the lesion classification.

[0074] In some example embodiments, the list of lesions, their classifications, and contours may be stored in a historical data structure associated with a patient that matches a volume of input CT medical images, such that multiple runs of the AI ​​pipeline on different volumes of input CT medical images associated with that patient can be stored and evaluated over time. For example, differences between the list of lesions and / or their associated classifications and contours may be determined to assess the progression of the patient's disease or medical condition and present such information to a medical professional to aid in the patient's treatment.

[0075] Other downstream computing systems and processing of the specific anatomical and lesion detection information generated by the AI ​​mechanisms of the example embodiments may be implemented without departing from the spirit and scope of the present invention. For example, the output of the AI ​​pipeline may be used by another downstream computing system that processes the anatomical and lesion information in the output of the AI ​​pipeline to identify discrepancies with other sources of information (e.g., radiology reports) in order to make clinical staff aware of detections that may otherwise be overlooked.

[0076] Thus, example embodiments provide a mechanism for realizing an automated AI pipeline that includes multiple configured and trained ML / DL computer models implementing various artificial intelligence operations at various stages of the AI ​​pipeline to identify anatomical structures and lesions associated with those anatomical structures within a volume of input medical images, determine contours associated with such anatomical structures and lesions, determine classifications for such lesions, and generate a list of such lesions and lesion and anatomical structure contours for further downstream computer processing of the AI-generated information from the AI ​​pipeline. The operation of the AI ​​pipeline is automated such that there is no human intervention at any stage of the AI ​​pipeline; instead, specifically configured and trained ML / DL computer models trained via machine learning / deep learning computer processes are employed to perform the specific AI analyses at the various stages. Human intervention may only be present prior to the input of the volume of input medical images (e.g., during the acquisition of the patient's medical images) and after the output of the AI ​​pipeline (e.g., when displaying an augmented medical image presented via a computer image viewing application based on the output of the list of lesions and contours generated by the AI ​​pipeline). Thus, AI pipelines specifically relate to improved automated computer tools implemented as artificial intelligence using specific machine learning / deep learning processes that exist solely within a computer environment, thus performing actions that cannot be performed as intelligent processes by humans and do not orchestrate any human activity.

[0077] Before continuing with a description of the example embodiments and various aspects of the improved computer operations performed by the example embodiments, it should first be understood that throughout this description, the term "mechanism" is used to refer to elements of the present invention that perform various operations, functions, etc. A "mechanism," as the term is used herein, may be an implementation of a function or aspect of an example embodiment in the form of an apparatus, a procedure, or a computer program product. In the case of a procedure, the procedure is implemented by one or more devices, apparatuses, computers, data processing systems, etc. In the case of a computer program product, the logic represented by the computer code or instructions embodied in or on the computer program product is executed by one or more hardware devices to implement the functionality associated with, or perform the operations associated with, a particular "mechanism." Thus, the mechanisms described herein may be implemented as specialized hardware, software that executes on the hardware and thereby configures the hardware to implement specific functionality of the invention that the hardware cannot otherwise perform, software instructions stored on a medium such that the instructions are readily executable by the hardware and thereby specifically configure the hardware to perform the functions set forth and the specific computer operations described herein, procedures or methods for performing functions, or any combination thereof.

[0078] The description and claims may use the terms "a," "at least one of," and "one or more of" in reference to particular features and elements of example embodiments. It should be understood that these terms and phrases are intended to state that there is at least one of a particular feature or element present in a particular example embodiment, but that more than one may also be present. That is, these terms / phrases are not intended to limit the specification or claims to a single feature / element present, nor are they intended to require the presence of a plurality of such features / elements. To the contrary, these terms / phrases require only at least a single feature / element, and such features / elements may be present in a plurality within the specification and claims.

[0079] Furthermore, it should be understood that the use of the term “engine,” when used herein in connection with describing embodiments and features of the present invention, is not intended to limit any particular implementation for realizing and / or executing operations, steps, processes, etc., attributed to and / or performed by the engine. An engine may be, but is not limited to, software, hardware, or firmware, or a combination thereof, that performs specified functions, including, but not limited to, any use of general and / or specialized processors in combination with appropriate software loaded into or stored in machine-readable memory and executed by the processor. Furthermore, all names associated with particular engines, unless otherwise specified, are for convenience of reference and are not intended to limit a particular implementation. Furthermore, any functionality attributed to one engine may be incorporated into or combined with the functionality of another engine of the same or different type, or may be distributed across one or more engines in various configurations and performed in the same way by multiple engines.

[0080] Additionally, it should be understood that the following description uses multiple different examples of various elements of the example embodiments to further explain the exemplary implementations and aid in understanding the mechanisms of the example embodiments. These examples are intended to be non-limiting and do not exhaust the various possibilities for implementing the mechanisms of the example embodiments. In view of this description, it will be apparent to those skilled in the art that there are many other alternative implementations of these various elements that could be utilized in addition to or in place of the examples provided herein without departing from the spirit and scope of the present invention.

[0081] The present invention may be a system, a method, and / or a computer program product, which may include a computer-readable storage medium containing computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0082] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or ridge-in-groove structures on which instructions are recorded, and any suitable combination thereof. As used herein, a computer-readable storage medium should not itself be construed as a transitory signal such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted over a wire.

[0083] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or storage device over a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). This network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface within each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage on a computer-readable storage medium within each computing / processing device.

[0084] Computer-readable program instructions for carrying out the operations of the present invention may be either assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as JAVA®, Smalltalk®, C++, and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to customize the electronic circuitry to carry out aspects of the invention.

[0085] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0086] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to create a machine, where the instructions, executed by the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may be stored on a computer-readable storage medium and capable of directing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer-readable storage medium on which the instructions are stored comprises an article of manufacture containing instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0087] The computer-readable program instructions may be loaded into a computer, other programmable data processing apparatus, or other device to cause a series of operable steps to be performed on the computer, other programmable apparatus, or other device to generate a computer-implemented process, such that the instructions, which execute on a computer, other programmable apparatus, or other device, implement the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0088] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, comprising one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions shown in the blocks may occur out of the order shown in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowchart diagrams, and combinations of blocks included in the block diagrams and / or flowchart diagrams, may be implemented by a special-purpose hardware-based system that performs the specified function(s) or operation(s), or executes a combination of special-purpose hardware and computer instructions.

[0089] Overview of the Lesion Detection and Classification AI Pipeline

[0090] FIG. 1 illustrates an exemplary block diagram of a lesion detection and classification artificial intelligence (AI) pipeline (referred to herein simply as an “AI pipeline”) that implements multiple ML / DL computer models specifically configured and trained to perform anatomical structure identification and lesion detection within input medical image data, according to one example embodiment. For illustrative purposes only, the illustrated AI pipeline is specifically described as being directed to liver detection and liver lesion detection within medical image data. As previously mentioned, example embodiments are not limited to such example embodiments and may be applied to any target anatomical structures and lesions associated with such target anatomical structures that may be represented in image elements of medical image data captured by medical imaging techniques and corresponding computing systems. For example, the mechanisms of example embodiments may be applied to the detection, contour identification, classification, etc. of other anatomical structures, such as the lungs, the heart, and lesions associated with the lungs, heart, or other target anatomical structures.

[0091] It should further be understood that the following description provides an overview of the AI ​​pipeline from the level shown in FIG. 1 , and that subsequent sections of this description provide additional details regarding individual stages of the AI ​​pipeline. Each of the stages of the AI ​​pipeline, in some example embodiments, is implemented as a constructed and trained ML / DL computer model, such as a deep learning neural network, as represented by symbol 103 in the various stages of AI pipeline 100. These different ML / DL computer models are specifically constructed and trained to perform specific AI operations described herein, such as, for example, body part identification, liver detection, phase classification, minimal liver volume detection, liver / lesion detection, lesion segmentation, false positive removal, and lesion classification. While these additional sections of the following description illustrate specific embodiments for implementing the various stages of the AI ​​pipeline that provide novel techniques, mechanisms, and methods for performing the AI ​​operations of the various stages, it should be understood that other equivalent techniques, mechanisms, or methods may be used in connection with the AI ​​pipeline as a whole without departing from the spirit and scope of the example embodiments. These other equivalent techniques, mechanisms, or methods will be apparent to those skilled in the art in view of this specification and are intended to be within the spirit and scope of the present invention.

[0092] 1 , an artificial intelligence (AI) pipeline 100, according to one example embodiment, receives as input a volume of input medical images 105, which in the illustrated example is a volume of input computed tomography (CT) medical images represented as one or more data structures, which is then automatically processed by various stages of the AI ​​pipeline 100 to ultimately produce an output 170 including a list of lesions along with classification and contour information, as well as contour information about the anatomical structure of interest (e.g., the liver in the illustrated example). The volume of input medical images 105 may be captured by medical imaging technology 102 using any of a number of commonly known or later developed medical imaging techniques and equipment that render images of the internal anatomical structures of a biological entity (i.e., a patient) as one or more medical image data structures. In some example embodiments, this volume of input medical images 105 includes two-dimensional slices (individual medical images) of portions of the patient's anatomy of the patient's body part, which are then combined to generate slabs (combinations of slices along an axis to provide a collection of medical images having a thickness along the axis), and these slabs are combined to generate a three-dimensional representation (i.e., volume) of the anatomy of the body part.

[0093] In the first stage logic 110 of the AI ​​pipeline 100, the AI ​​pipeline 100 determines (112) the part of the patient's body that corresponds to the input volume 105 of CT medical image data and, via body part of interest determination logic 114, determines whether this part of the patient's body represents the part of the patient's body that corresponds to the anatomical structure of interest (e.g., an abdominal scan as opposed to a cranial scan, lower body scan, etc.). This evaluation serves as an initial filter for the AI ​​pipeline 100's use with only the input CT medical image data volume 105 (hereinafter referred to as the "input volume" 105), on which the AI ​​pipeline 100 is specifically configured and trained to perform anatomical structure identification and contouring, and lesion identification, contouring, and classification. This detection of the body part represented within the input volume 105 may reference metadata associated with the input volume 105, which may include a field specifying the region of the patient's body that was scanned, which may be specified by the source medical imaging technology computing system 102 when performing the medical image scan. Alternatively, the first stage logic 110 of the AI ​​pipeline 100 may implement an ML / DL computer model specifically configured and trained for body part detection 112 that performs medical image classification with respect to a particular part of the patient's body, performing computerized pattern analysis on the medical image data of the input volume 105 and predicting a classification of the medical image data with respect to one or more predetermined parts of the patient's body. In some example embodiments, this assessment may be binary (e.g., is it an abdominal medical image volume or not) or may be a more complex multi-class assessment that specifically identifies probabilities or scores for classification of, for example, multiple different body parts (e.g., abdomen, skull, lower extremities, etc.).

[0094] If the subject body part determination logic 114 of the first stage logic 110 of the AI ​​pipeline 100 determines that the input volume 105 does not represent a part of the patient's body where an anatomical structure of interest may be detected (e.g., the abdominal part of the body where a liver may be detected), processing of the AI ​​pipeline 100 may be interrupted (a rejection case). If the subject body part determination logic 114 of the first stage logic 110 of the AI ​​pipeline 100 determines that the input volume 105 does represent a part of the patient's body where an anatomical structure of interest may be detected, further processing of the input volume 105 by the AI ​​pipeline 100 is performed, as described below. It should be understood that in some example embodiments, multiple different instances of the AI ​​pipeline 100 may be provided, each configured and trained to process input volumes 105 corresponding to different anatomical structures that may be present in different parts of the patient's body. Thus, first stage logic 110 may be provided external to AI pipeline 100 and may operate as routing logic for routing input volumes 105 to corresponding AI pipelines 100 specifically configured and trained to process a particular classification of input volumes 105, e.g., one AI pipeline instance for liver and liver lesion detection / classification, another AI pipeline instance for lung and lung lesion detection / classification, a third AI pipeline instance for heart and cardiac lesion detection / classification, etc. Thus, first stage logic 110 may include routing logic that stores a mapping of which AI pipeline instances 100 correspond to different body parts / anatomical structures of interest, and can automatically route the input volume 105 to a corresponding AI pipeline instance 100 specifically configured and trained to process the input volume 105 that corresponds to the detected body part, based on the detection of a body part represented in the input volume 105.

[0095] Assuming that input volume 105 is detected as representing a region of a patient's body where an anatomical structure of interest is present (e.g., for liver lesion detection, an abdominal scan is present in input volume 105), further processing of input volume 105 is performed by AI pipeline 100 in second-stage logic 120. This second-stage logic 120 includes two initial substages 122 and 124 that may execute substantially in parallel with one another, sequentially, or both, depending on the desired implementation (parallel execution is depicted in FIG. 1 by way of example). The two initial substages 122, 124 include a phase classification substage 122 and an anatomy detection substage 124 (e.g., a liver detection substage 124 if AI pipeline 100 is configured to perform liver lesion detection).

[0096] The phase classification substage 122 determines whether the input volume 105 contains a single imaging phase (e.g., a pre-contrast phase, an arteriography phase, a portal / venous imaging phase, a delayed phase, etc.). Again, the phase classification substage 122 may be implemented as logic that evaluates metadata associated with the input volume 105, which may include a field specifying the phase of the medical imaging study that matches the medical image, which may be generated by the medical imaging computing system 102 when performing imaging of the medical image. Alternatively, example embodiments may implement a configured and trained ML / DL computer model specifically trained to detect patterns in medical images that indicate different phases of a medical imaging study, thereby classifying the medical images of the input volume 105 as to which phase they correspond. The output of the phase classification substage 122 may be a binary value indicating whether the input volume 105 contains one phase or multiple phases, or may be a classification of each of the phases represented in the input volume 105, which may then be used to determine whether a single phase or multiple phases are represented.

[0097] If the phase classification indicates that a single phase is present in the input volume 105, the AI ​​pipeline 100 further processes it through downstream stages 130-170, as described below. If multiple phases are detected, the input volume 105 may not be further processed by the AI ​​pipeline 100, or may be filtered and / or divided into subvolumes, as described above, each containing a corresponding single-phase image, such that only one subvolume corresponding to the target phase is processed by the AI ​​pipeline 100, and / or the subvolume is routed to a corresponding AI pipeline configured and trained to process input volumes of images corresponding to the particular phase classification. It should be understood that an input volume may be rejected for several reasons (e.g., no liver present in the image, not a single-phase input volume, not enough liver present in the image, etc.). Depending on the actual underlying cause of the rejection, the reason for the rejection may be communicated to the user, such as via a user interface. For example, an output of AI pipeline 100 responsive to a rejection may indicate the reason for the rejection, which may be utilized by a downstream computing system (e.g., a viewer or additional automated processing system) to communicate the reason for the rejection via an output. For example, if no liver is detected in the input volume, the input volume may be silently ignored, e.g., without communicating the rejection to a user. Meanwhile, in the case of an input volume that includes a liver but includes images of multiple phases, the rejection may be communicated to a user (e.g., a radiologist) in a user interface generated by a downstream viewer computing system, e.g., by explicitly stating that the input volume was not processed by AI pipeline 100, to avoid mistaking the input volume for an input volume that does not include any detections due to the input volume containing images of two or more phases.

[0098] The second initial substage 124 is a detection substage for detecting the anatomical structure of interest (in this example embodiment, the liver) in portions of the input volume 105. That is, slices, slabs, etc. within the input volume 105 that specifically depict the anatomical structure of interest (the liver) are identified and evaluated to determine whether a predetermined minimum amount of the anatomical structure of interest (the liver) is present in these slices, slabs, or the input volume as a whole. As previously mentioned, the detection substage 124 includes an ML / DL computer model 125 that is specifically trained and configured to perform computational medical image analysis to identify portions of the input medical image that correspond to the anatomical structure of interest (e.g., a human liver).

[0099] Thus, in liver detection substage 124, AI pipeline 100 uses trained ML / DL computer model 125 to determine whether the input CT medical image volume contains an image showing the liver. The portion of the volume showing the liver, along with the results of phase classification substage 122, is passed to decision substage 126 of AI pipeline 100, which includes single phase decision logic 127 and structure minimum decision logic 128, where phase decision logic 127 determines whether a single phase of the medical image is present and structure minimum decision logic 128 determines whether at least a predetermined amount of the target anatomical structure is present in the portion of the volume showing the target anatomical structure (e.g., the liver). As previously mentioned, determining whether a predetermined amount of the target anatomical structure is present may be determined based on known measurement mechanisms, such as determining measurements of the structure from the medical image, for example, by calculating the size of the structure from differences in pixel locations within the image, comparing these measurements to one or more predetermined thresholds, and determining whether a minimum amount of the target anatomical structure (e.g., the liver) is present in the input volume 105 (e.g., whether 1 / 3 of the liver is present in the portion of the input volume 105 determined to represent, for example, the liver).

[0100] In one example embodiment, the axial scoring mechanism described above may be used to evaluate the portions of the anatomy present in the input volume 105 to determine whether a predetermined amount of the anatomy of interest (liver) is present in the input volume 105. As described above, the ML / DL computer model generates slice scores s corresponding to the slice scores of the MSSV and MISV slices of the input volume 105, respectively. sup and s inf In some example embodiments, the slice score s sup and s inf first divides the input volume 105 into sections, and then, for each section, runs the constructed and trained ML / DL computer model on the slices of the section to obtain slice scores s' of the first and last slices in the section. sup and s' inf It may be indirectly detected by estimating s'. sup and s' inf Given an estimate of , we know how the sections are located relative to the entire volume of the input CT medical image, so we can extrapolate to find s sup and s inf For each input volume 105, an estimate of s sup and s inf It should be noted that there are n estimates of , where n is the number of sections per volume. In one example embodiment, the final estimate is obtained by taking an unweighted average of these n estimates, although in other example embodiments, other functions of the n estimates may be used to generate the final estimate.

[0101] Input CT medical image volume s sup and s infA final estimate of , is determined, and based on these values, a fraction of the target anatomical structure (e.g., the liver) is calculated. This calculated fraction may then be compared to a predetermined threshold to determine whether a predetermined minimum amount of the target anatomical structure is present in the volume of the input CT medical image (e.g., at least 1 / 3 of the liver is present in the volume of the input CT medical image).

[0102] If the determination of decision logic 127 and 128 indicates that multiple phases are present and / or a predetermined amount of the anatomical structure of interest is not present in the portion of input volume 105 depicting the liver, further processing of input volume 105 by AI pipeline 100 for stages 130-170 may be aborted (i.e., input volume 105 is rejected). If the determination of decision logic 127 and 128 results in a determination that input volume 105 contains images of a single phase and depicts at least a predetermined amount of liver, the portion of input volume 105 depicting the anatomical structure is forwarded to the next stage 130 of AI pipeline 100 for processing. While this example embodiment forwards a subportion of the input volume containing the liver for further processing, in other example embodiments, background around the liver may be provided, which may be done by adding a predetermined amount of margin above and below the selected liver region. Depending on the amount of background required by subsequent processing operations, the margin may be increased to the full extent of the original input volume.

[0103] In the next stage 130 of the AI ​​pipeline 100, the AI ​​pipeline 100 performs lesion detection on a portion of the input volume 105 representing an anatomical structure of interest (e.g., the liver). This liver / lesion detection stage 130 of the AI ​​pipeline 100 uses an ensemble of ML / DL computer models 132-136 to detect the liver and lesions within the liver represented in the input volume 105. The ensemble of ML / DL computer models 132-136 performs liver and lesion detection using differentially trained ML / DL computer models 132-136, which are trained using a loss function to balance false positives and false negatives in lesion detection. Additionally, the ensemble of ML / DL computer models 132-136 are configured such that a third loss function forces the outputs of the ML / DL computer models 132-136 to match each other.

[0104] In one example embodiment, a constructed and trained ML / DL computer model 132 is run on the input volume 105 to detect the presence of a liver. This ML / DL computer model 132 may be the same as the ML / DL computer model 125 employed in the previous AI pipeline stage 120, and thus previously obtained results may be utilized. Multiple (two or more) other ML / DL computer models 134-136 are constructed and trained to perform lesion detection in portions of the medical image of the input volume 105 that show the liver. A first ML / DL computer model 134 is constructed and trained to operate directly on the input volume 105 and generate a lesion prediction. The second ML / DL computer model 136 is configured using two different decoders that implement two different loss functions: one loss function penalizes errors in false negatives (i.e., classifications that incorrectly indicate the absence of a lesion (normal anatomy)) and the other loss function penalizes errors in false positive results (i.e., classifications that incorrectly indicate the presence of a lesion (abnormal anatomy)). The first decoder of the ML / DL computer model 136 is trained to identify patterns that represent a relatively large number of different lesions, at the expense of having a large number of false positives. The second decoder of the ML / DL computer model 136 is trained to be less sensitive to detecting lesions, but any lesions that are detected are highly likely to be accurately detected. The third loss function of the ensemble of ML / DL computer models compares the results of the decoders of the ML / DL computer model 136 as a whole to each other and forces their results to agree with each other. The lesion prediction results of the first and second ML / DL computer models 134, 136 are combined to generate a final lesion prediction for the population, while the other ML / DL computer model 132, which generates a liver mask prediction, provides an output representing the liver and its contours. An exemplary architecture of these ML / DL computer models 132-136 is described in further detail below with respect to FIG. 6.

[0105] The results of the liver / lesion detection stage 130 of the AI ​​pipeline 100 include one or more contours (outline lines) of the liver and a detection map (e.g., a voxel-by-voxel map of detected liver lesions within the input volume 105) that identifies portions of medical image data elements corresponding to detected lesions 135. The detection map is then input to the lesion segmentation stage 140 of the AI ​​pipeline 100. The lesion segmentation stage 140 segments the detection map using a watershed technique and a corresponding ML / DL computer model 142, as described in more detail below, to generate a segmentation of image elements (e.g., voxels) of the medical image (slice) of the input volume 105. Based on this segmentation, the liver lesion segmentation stage 140 identifies all of the contours corresponding to lesions present in the slice of the input volume 105 and provides other mechanisms, such as an ML / DL computer model 144, that perform operations to identify which contours correspond to the same lesion in three dimensions. The lesion segmentation stage 140 further provides mechanisms, such as an ML / DL computer model 146, that aggregates the interrelated lesion contours to generate a three-dimensional segmentation of the lesion. Lesion segmentation uses inpainting of lesion image elements (e.g., voxels) and non-liver tissue represented within the medical image to focus on each lesion individually and perform active contour analysis. In this way, individual lesions may be identified and processed without biasing the analysis due to other lesions within the medical image or due to portions of the image outside the liver.

[0106] The result of lesion segmentation 140 is a list 148 of lesions with corresponding contours or outlines in the input volume 105. These outputs 148 are provided to a false positive removal stage 150 of the AI ​​pipeline 100. The false positive removal stage 150 uses a configured and trained ML / DL computer model that uses a dual operating point strategy to reduce false positive lesion detections in the list of lesions generated by the lesion segmentation stage 140 of the AI ​​pipeline 100. A first operating point is selected to be highly sensitive to false positives by configuring the ML / DL computer model of the false positive removal stage 150 to remove as many lesions as possible. After the sensitive removal of false positives, a determination is made as to whether or not a predetermined number of lesions remain in the list. If so, the other lesions in the removed list are reconsidered using a second operating point that is relatively less sensitive to false positives. The results of both methods identify a final filtered list of lesions that is further processed by the lesion classification stage of the AI ​​pipeline.

[0107] After removing false positives from the list of lesions and their contours generated by the lesion segmentation stage 140, the resulting filtered list of lesions 155 is provided as input to the lesion classification stage 160 of the AI ​​pipeline 100, where a configured and trained ML / DL computer model is run on the list of lesions and their corresponding contour data to thereby classify the lesions into one of multiple pre-defined lesion classifications. For example, each lesion and its attributes (e.g., contour data) in the final filtered list of lesions may be input to the trained ML / DL computer model of the lesion classification stage 160, which then operates on this data to classify the lesion as a particular pre-defined type or class of lesion.

[0108] After lesion classification by the lesion classification stage 160 of the AI ​​pipeline 100, the AI ​​pipeline 100 generates an output 170 containing a completed list of lesions and their classifications, along with any contour attributes of the lesions. Additionally, the output 170 of the AI ​​pipeline 100 may include liver contour information of the liver obtained from the liver / lesion detection stage 130. This output generated by the AI ​​pipeline 100 may be provided to a downstream computing system 180 for further processing and generation of a representation of the target anatomical structure and any detected lesions present in the anatomical structure. For example, a graphical representation of the input volume may be generated in a medical image viewer or other computer application of the downstream computing system 180, with the anatomical structure and detected lesions superimposed or otherwise enhanced on the graphical representation using the contour information generated by the AI ​​pipeline. In other example embodiments, downstream processing by the downstream computing system 180 may include automated medical image report generation, diagnostic decision support operations based on the detected list of lesions, classifications, and contours. In other example embodiments, different treatment recommendations may be generated for review and consideration by a physician based on the lesion classification. In some example embodiments, the list of lesions, their classifications, and contours may be stored in a historical data structure in the downstream computing system 180 associated with a patient identifier, allowing multiple runs of the AI ​​pipeline 100 on different input volumes 105 associated with the same patient to be stored and evaluated over time. For example, differences between the list of lesions and / or their associated classifications and contours may be determined to assess the progression of the patient's disease or medical condition and present such information to a medical professional to aid in the patient's treatment. Other downstream computing systems 180 and processing of specific anatomical structure and lesion detection information generated by the AI ​​pipeline 100 of example embodiments may be implemented without departing from the spirit and scope of the present invention.

[0109] FIG. 2 is an exemplary flowchart outlining an exemplary operation of an AI pipeline according to one example embodiment. The operations outlined in FIG. 2 may be implemented by various stages of logic including the constructed and trained ML / DL computer model described above using the specific example embodiments shown in FIG. 1 and described later in separate sections of this description. It should be understood that the operations are particularly directed to an automated artificial intelligence pipeline implemented in one or more data processing systems including one or more computing devices specifically configured to implement the mechanisms of an automated computer tool. The operations outlined in FIGS. 1 and 2 do not involve human intervention other than during the creation of the medical image volume and the use of output from downstream computing systems. The present invention specifically provides an automated artificial intelligence computing mechanism improved to perform the described operations, which operations avoid human interaction and reduce potential errors resulting from previous manual processes by providing a new and improved process that is specifically directed to providing logic and data structures that enable the improved artificial intelligence computing mechanism of the present invention to be implemented in an automated computing tool, which differs particularly from any previous manual process.

[0110] As shown in FIG. 2, the operations begin by receiving an input volume of medical images from a medical imaging technology computing system (e.g., a computing system providing computed tomography (CT) medical images) (step 210). The AI ​​pipeline operates on the received input volume to perform body part detection (step 212) so that a determination can be made as to whether a body part of interest is present in the received input volume (step 214). If the body part of interest (e.g., the abdomen in the case of liver lesion detection and classification) is not present in the input volume, the operations end. If the body part of interest is present in the input volume, phase classification and minimal anatomy assessment are performed, either serially or in parallel.

[0111] That is, as shown in Figure 2, a phase classification is performed on the input volume (step 216) to determine whether the input volume contains medical images of a single phase (e.g., pre-contrast imaging, partial contrast imaging, delayed phase, etc.) or multiple phases (slices). A determination is then made as to whether the phase classification indicates a single phase or multiple phases (step 218). If the input volume contains medical images covering multiple phases, the operation ends; otherwise, if the input volume contains medical images covering a single phase, the operation proceeds to step 220.

[0112] At step 220, detection of the anatomical structure of interest (e.g., the liver in the illustrated example) is performed to determine whether a minimum amount of anatomical structure is present in the input volume so that subsequent stages of the operation of the AI ​​pipeline can be accurately performed. A determination is made as to whether a minimum amount of anatomical structure is present (e.g., at least 1 / 3 of the liver is represented within the input volume) (step 222). If a minimum amount is not present, the operation ends; otherwise, the operation proceeds to step 224.

[0113] At step 224, liver / lesion detection is performed to generate lesion contours and detection maps. These contours and detection maps are provided to lesion segmentation logic, which performs lesion segmentation (e.g., liver lesion segmentation in the illustrated example) based on these contours and detection maps (step 226). Lesion segmentation results in the generation of a list of lesions and their contours, as well as anatomical structure (e.g., liver) detection and contour information (step 228). Based on this list of lesions and their contours, a false positive removal operation is performed on the lesions in the list to remove false positives and generate a filtered list of lesions and their contours (step 230).

[0114] The filtered list of lesions and their contours is provided to lesion classification logic, which performs lesion classification and generates a completed list of lesions, their contours, and lesion classifications (step 232). This completed list, along with the liver contour information, is provided to a downstream computing system (step 234), which can operate on this information to generate medical image displays in a medical image viewer application, generate treatment recommendations based on the classification of the detected lesions, evaluate the historical progression of lesions over time in the same patient based on a comparison of the completed lists of lesions generated by the AI ​​pipeline at different time points, etc.

[0115] Thus, the example embodiments outlined above provide automated artificial intelligence mechanisms and ML / DL computer models that operate on an input volume of medical images and generate a list of lesions, their contours, and classifications while minimizing false positives. The exemplary embodiments provide an automated artificial intelligence computer tool that, in a particular set of voxels in the input volume of images, specifically identifies which voxels of a plurality of voxels correspond to portions of an anatomical structure of interest (e.g., the liver) and which voxels of those voxels correspond to lesions within the anatomical structure of interest (e.g., liver lesions). The example embodiments provide a distinct improvement over previous methods, both manual and automated, in that the example embodiments can be integrated into a fully automated computer tool within a clinician's workflow. In fact, based on the initial stages of the AI ​​pipeline design of example embodiments, which accepts input volumes from only a single phase (e.g., abdominal scans) and rejects input volumes that do not represent the target anatomical structure (e.g., liver) or do not represent a predetermined amount of the target anatomical structure (e.g., too little liver), only meaningful input volumes are processed by the automated AI pipeline, thereby preventing radiologists from wasting valuable manual resources on useless or erroneous results when identifying input volumes other than the target anatomical structure (e.g., other than the liver). In addition to preventing radiologists from receiving a large amount of useless information, the automated AI pipeline of example embodiments also ensures smooth integration of information technology by avoiding congestion of the AI ​​pipeline and downstream computing systems (e.g., network, archiving, and verification computing systems) with data associated with cases that do not correspond to the target anatomical structure or that do not provide a sufficient amount of the target anatomical structure.Furthermore, as previously described, the automated AI pipelines of the example embodiments enable accurate detection, measurement, and characterization of lesions in a fully automated manner, which is technically enabled by the structure of one or more of the example embodiments of the automated AI pipelines and their corresponding automated ML / DL computer model-based components.

[0116] ML / DL computational model for detecting minimal anatomical structures present in an input volume

[0117] As previously mentioned, as part of processing the input volume 105, it is important to ensure that the input volume 105 represents a single phase of the medical image and that at least a minimum amount of the target anatomy is represented within the input volume 105. To determine whether a minimum amount of the target anatomy is present in the input volume 105, in one example embodiment, the decision logic 128 implements a specially configured and trained ML / DL computer model that estimates slice scores to determine the portions of the anatomy (e.g., the liver) that are present in the input volume 105. An example embodiment of this configured and trained ML / DL computer model is described below, based on a defined axial scoring technique.

[0118] 3A is an exemplary diagram illustrating an exemplary input volume (medical image) of a human patient's abdomen, according to an example embodiment. In the depiction of FIG. 3A, a two-dimensional representation of a three-dimensional volume is shown. Slices, which are horizontal lines in the two-dimensional representation shown in FIG. 3A, are represented as planes that extend into the page, off the page, or both, representing flattened two-dimensional slices of the body; the stacking of these planes results in the three-dimensional image.

[0119] As shown in FIG. 3A, an example embodiment defines axial scores for slices ranging from 0 to 1. The axial scores are defined such that the slice corresponding to the first slice containing the liver (FSL) has a slice score of 0, and the slice corresponding to the last slice containing the liver (LSL) has a score of 1. In the example shown, the first and last slices are defined relative to the lowest slice in the volume (MISV) and the highest slice in the volume (MSSV), where inferior and superior are determined along a particular axis of the volume (e.g., the y-axis in the example shown in FIG. 3A). Thus, in this example, the MSSV is the slice with the highest y-axis value, and the MISV is the slice with the lowest y-axis value. For example, the MISV may be closest to the lower extremities of the biological entity (e.g., the feet of a subject), and the MSSV may be closest to the top of the biological entity (e.g., the head of a subject). The FSL is the slice representing the anatomical structure of the subject (e.g., the liver) that is closest relative to the MISV. The LSL is the slice representing the anatomical structure of the subject that is closest relative to the MSSV. In one example embodiment, a trained ML / DL computer model (e.g., a neural network) may assign axial scores by taking a chunk of slices as input and outputting the height (axial score) of the center slice within the chunk. This trained ML / DL computer model is trained using a cost function that minimizes the error (e.g., least squares error) in the actual height. This trained ML / DL computer model is then applied to all chunks covering the input volume (possibly with some overlap between chunks).

[0120] The slice score pairs (s sup and s inf ) defines the Liver Axial Score Estimate (LAE). The ML / DL computer model of the decision logic 128 in Figure 1 calculates the slice score s of the input volume 105. sup and sinf Specifically configured and trained to determine the slice scores, the mechanism of the example embodiment can recognize these slice scores and determine the liver fragments within the field of view of the input volume 105.

[0121] In some example embodiments, slice scores s sup and s inf first divides the input volume 105 into sections (e.g., sections containing X number of slices (e.g., 20 slices)), and then, for each section, runs the constructed and trained ML / DL computer model on the slices in the section to obtain slice scores s' of the first and last slices in the section. sup and s' inf s' may be detected indirectly by estimating s', and the "first" and "last" may be determined according to the direction of progression along the axis of the three-dimensional volume 105 (e.g., the direction of progression from the first slice to the last slice along the y-axis, going from the slice with the smallest y-axis value to the slice with the largest y-axis value). sup and s' inf Given the estimated value of , we know where the section is located relative to the entire volume 105, so we can extrapolate to find s sup and s inf For each volume, an estimate of s sup and s inf It should be noted that there are n estimates of , where n is the number of sections per volume. In one example embodiment, the final estimate is obtained by taking an unweighted average of these n estimates, although in other example embodiments, other functions of the n estimates may be used to generate the final estimate.

[0122] For example, Figure 3B shows the corresponding axial score s' inf and s' supFigure 3B shows another representation of the input volume of Figure 3A, including slice sections represented with σ. As shown in Figure 3B, the sections are defined as 20 slices spaced 5 mm apart in this example. For each 20 slice section of the volume, the ML / DL computer model generates a slice score s'. sup and s' inf are estimated and these s' along a particular range (e.g., 0 to 1, -0.5 to 1.2, or any other desired predefined range appropriate for a particular implementation). sup and s' inf From the extrapolation of the value of s sup and s inf In this example, assuming a default range of -0.5 to 1.2, s is obtained by application of the ML / DL computer model and extrapolation. sup is estimated to be about 1.2, and s inf If s is estimated to be -0.5, these values ​​indicate that the entire liver is contained in the volume. sup is estimated to be 1.2, and s inf If s is estimated to be 0.5, these values ​​indicate that approximately the upper 50% of the axial liver extent is included in the volume, e.g., coverage is (1.2 - 0.5) / (1.2 - (-0.5)) = 0.41. As an additional example, if the liver starts at -2.0 and ends at 0.8 (i.e., s sup is estimated to be 0.8, and s inf In another example embodiment (where the upper limit of the liver is less than 1.2, the liver is clipped at the top, and the lower limit is less than -0.5, the lower part of the liver is completely covered. This indicates that approximately 80% of the lower axial liver extent is included in the volume, i.e., coverage is (0.8-max(-2,-0.5)) / (1.2-(-0.5))=0.76.

[0123] Figure 3C is an exemplary illustration of the input volume of Figure 3A, where the volume has been divided axially into n completely overlapping sections. In the example shown, there are seven sections indicated by arrows. Note that in this example, the last two sections (arrows at the top of the figure) are almost completely identical. As mentioned above, the ML / DL computer model generates s' for each of these sections. sup and s' inf The values ​​of are estimated and s for the MSSV and MISV slices. sup and s inf is used to extrapolate the value of s sup and s inf The value of may then be used to determine the amount of the anatomical structure of interest that is present in the input volume 105.

[0124] Therefore, MSSV and MISV s sup and s inf The value of s' is calculated by first dividing the input volume 105 into sections, and then for each section, calculating the slice scores s' of the first and last slices in the section. sup and s' inf Given these estimates, we know how the section is positioned relative to the entire input volume 105, so we can find s sup and s inf The value of is estimated by extrapolation. sup and s inf There are n estimates of , where n is the number of sections per volume. The final estimate may be obtained by any suitable combination function that evaluates the n estimates, such as, for example, an unweighted average of the n estimates or any other suitable combination function.

[0125] 4A-4C illustrate a diagram of a section s' of an input volume of a medical image according to one example embodiment. sup and s' inf4A-4C show an exemplary diagram of one embodiment of an ML / DL computer model configured and trained to estimate values ​​of σ. The ML / DL computer model of FIGS. 4A-4C is merely one example of an ML / DL computer model's architecture, and many modifications to this architecture may be made without departing from the spirit and scope of the present invention, such as changing the tensor size of an input slice of an input volume, changing the number of nodes in a layer of the ML / DL computer model, or changing the number of layers. Those skilled in the art will recognize, in light of this description, how to modify the ML / DL computer model of the example embodiment to a desired implementation.

[0126] As shown in FIG. 4A, a series of 20 slices representing a section 410 or "slab" of the input volume 105 are provided as input to processing blocks (PBs) 420-430. In the illustrated example embodiment, PBs 420-430 are blocks of logic that blend convolutional and LSTM layers, as shown in FIGS. 4B and 4C. Features are extracted from the convolutional layers of PBs 420, 430, and these features are then provided as input to the LSTM layers of PBs 420, 430. This is a form of clever / lightweight modeling of the fact that slices have a specific order within an anatomical region or anatomical structure of interest (e.g., abdomen / liver), driven by anatomy (e.g., the relative positions of the liver, kidneys, heart, etc., in addition to the liver anatomy itself). In the illustrated example, initially, the tensor size of the 20 slices 410 is 128x128 in this example. The first processing block 420 reduces the size of the tensor by a factor of eight in this example embodiment, producing sections of 20 slices (it should be understood that the number of slices in a section is implementation specific and may vary without departing from the spirit and scope of the present invention), with these slices having dimensions 16x16x32, where 32 is the number of filters. The second processing block 430 converts the slices of the input section into sections of 20 slices, including slices having dimensions 2x2x64, where 64 is the number of filters. A subsequent neural network 440, constructed using flattening, dense, and linear layers, generates an estimate s' of the input section 410 of the input volume 105. sup and s' inf 4B illustrates the composition of processing blocks (PBs) in terms of convolutional and LSTM layers, according to one example embodiment, and FIG. 4C illustrates an example configuration of each of these convolutional and LSTM layers in each PB.

[0127] As an example, in the architecture of the ML / DL computer model of FIGS. 4A-4C, during training of the ML / DL computer model, in one example embodiment, medical image data (e.g., Digital Imaging and Communications in Medicine (DICOM) data) is converted into an input volume (e.g., S , including values ​​in Hounsfield Units (HU)). i The data is assembled into a 3D array of 32-bit floating-point numbers (x512x512 in size). (The Hounsfield unit values ​​are normalized physical values ​​that indicate the X-ray attenuation of the material present at a particular location (e.g., voxel).) i is the number of slices in the i-th volume, where i ranges from 0 to N-1, and N is the total number of volumes. Each input volume is processed by a body part detector, and an approximate region corresponding to the abdomen (in the case of liver detection) is extracted, as described above. The abdomen is defined, for example, as the contiguous region between axial scores from the body part detector -30 and 23. Slices outside this contiguous region are rejected, and ground truth may be defined as the locations of appropriately adjusted FSL and LSL. For example, assuming the input volume ranges from a to b, the input volume is rejected if there is no overlap between [a:b] and [-30,23]. In other words, if b > 23 or a < -30.

[0128] The input section 410 or "slab" is re-sliced ​​to a predetermined slice separation (e.g., 5 mm). The input section 410 is reshaped to 128x128 in the x and y dimensions, resulting in a slab of shape M i The result is N x128 x 128 sections 410. This is called downsampling of the data in the input volume. Because the ordering of slices in the input volume relies on coarse information (e.g., organ size), the AI ​​pipeline still operates well on the downsampled data, and the reduced size of the downsampled data improves both the processing time and training time of the AI ​​pipeline.

[0129] Input sections 410 containing less than a predetermined number of slices (e.g., 20) or having a pixel size less than a predetermined pixel size (e.g., 55 mm) are rejected, and N' of M i A x128x128 section is obtained. The values ​​within the section are clipped and normalized using a linear transformation from the acquisition range (e.g., -1024,2048) to the range (0,1). At this point, the processed N' M i The x128x128 sections form the training set as described above, and the neural network 440 uses the s' input section on this training set. sup and s' inf is trained to generate estimates of

[0130] With respect to performing inference using the trained neural network 440, the operations described above for processing the input volume 105 by body part detection, selection of slices corresponding to the body part of interest, reslicing, reshaping, rejecting certain sections that do not meet predetermined requirements, and generating clipped and normalized sections are performed again for the new sections of the input volume 105. After generating the clipped and normalized sections, the input volume 105 is divided into R-ceil(M-10) / 10 subvolumes or sections containing 20 slices, thereby generating a slice division containing overlapping chunks. For example, if there is a volume of N'=31 slices (slices numbered 0 to 30), three sections or subvolumes containing overlapping slices 0 to 19, 10 to 29, and 11 to 30 are defined. The sections or subvolumes typically have at least about 50% overlap.

[0131] Therefore, the ML / DL computer model calculates the s' of sections of a volume corresponding to a predefined number of slices (medical images), assuming a defined axial score range of 0 to 1.sup and s' inf Based on the estimate of the value of s of the input volume sup and s inf The AI ​​pipeline 100 may be provided, configured, and trained to estimate values ​​of . From these estimates, a determination may be made as to whether the input volume contains medical slices that together comprise at least a predetermined amount of the anatomical structure of interest (e.g., the liver). This determination may be part of the decision logic 128 of the AI ​​pipeline 100 for determining whether a sufficient representation of the anatomical structure is present in the input volume 105 to enable accurate liver / lesion detection, lesion segmentation, etc. in further downstream stages of the AI ​​pipeline 100, as described above.

[0132] FIG. 5 is a flowchart outlining an exemplary operation of the AI ​​pipeline's liver detection and predetermined amount of anatomical structure determination logic according to one example embodiment. As shown in FIG. 5, the AI ​​pipeline's liver detection operation begins by receiving an input volume (step 510) and dividing the input volume into multiple overlapping sections with a predetermined number of slices per section (step 520). The slices per section are input to a trained ML / DL computer model that estimates axial scores for the first and last slices within each section (step 530). The axial scores for the first and last slices are used to extrapolate scores for the input volume's least significant slice (MISV) and most significant slice (MSSV) (step 540). This results in multiple estimates of the MISV and MSSV axial scores, which are then combined by a function of the individual estimates (e.g., weighted average) to generate an estimate of the input volume's MISV and MSSV axial scores (step 550). Based on the estimates of the axial scores of the MISV and MSSV, the axial scores are compared to a criterion to determine whether a predetermined amount of the anatomical structure of interest (e.g., the liver) is present in the input volume (step 560). The operation then terminates.

[0133] [Liver / lesion detection]

[0134] As previously described, assuming input volume 105 is determined to represent a single phase and input volume 105 is determined to represent a predetermined amount of the target anatomy in a slice of input volume 105, liver / lesion detection is performed on the portion of input volume 105 that includes the target anatomy. In one example embodiment, the liver / lesion detection logic of stage 130 of AI pipeline 100 employs a configured and trained ML / DL computer model (and in some example embodiments, this may be the same ML / DL computer model 125 used in stage 120 for liver detection) that operates to detect the target anatomy (e.g., the liver) in the slice of input volume 105. The liver / lesion detection logic of stage 130 of AI pipeline 100 also includes a population of multiple other ML / DL computer models configured and trained to detect lesions in images of the target anatomy (liver).

[0135] 6 is an exemplary diagram of a population of ML / DL computer models used to perform lesion detection in a target anatomical structure (e.g., the liver), according to one example embodiment. The population of ML / DL computer models 600 includes a first ML / DL computer model 610 for detecting a target anatomical structure, e.g., the liver, and generating a corresponding mask. The population of ML / DL computer models 600 also includes a second ML / DL computer model 620, which is configured and trained to process a liver mask input and generate a lesion prediction using two competing loss functions implemented in two decoders of the second ML / DL computer model 620. One loss function is configured to penalize false positive errors (leading to low sensitivity but high accuracy), and the other is configured to penalize false negative errors (leading to high sensitivity but low accuracy). An additional loss function, referred to as consistency loss 627 in Figure 6, is employed in the second ML / DL computer model 620 to make the outputs produced by the two competing decoders similar (consistent) to each other. The ensemble of ML / DL computer models also includes a third ML / DL computer model 630 that is configured and trained to directly process the input volume 105 and generate lesion predictions.

[0136] As shown in FIG. 6 and described above, the population 600 specifically includes a first configured and trained ML / DL computer model 610 configured and trained to identify anatomical structures of interest in input medical images. In some example embodiments, this first ML / DL computer model 610 includes a U-Net neural network model 612 configured and trained to perform image analysis and detect livers in medical images. However, it should be understood that example embodiments are not limited to this particular neural network model, and any ML / DL computer model capable of performing segmentation may be utilized without departing from the spirit and scope of the present invention. U-Net is a convolutional neural network deployed for biomedical image segmentation at the Department of Computer Science at the University of Freiburg, Germany. The U-Net neural network is based on a fully convolutional network whose architecture has been modified and extended to work with fewer training images and result in more accurate segmentation. U-Net is generally known in the art and will not be described in further detail herein.

[0137] As shown in FIG. 6 , in one example embodiment, a first ML / DL computer model 610 may be trained to process a predetermined number of slices once that number has been determined to be appropriate for a desired implementation, e.g., through an empirical process, that three slices provides good results. For example, the slices of the input volume in one example embodiment were 512x512 pixel medical images, although other implementations may use different slice dimensions without departing from the spirit and scope of the example embodiment. The U-Net generates a segmentation of the anatomical structure in the input slices, resulting in one or more segments corresponding to the anatomical structure of interest, e.g., the liver. As part of this segmentation, the first ML / DL computer model 610 generates a segment corresponding to a liver mask 614. This liver mask 614 is provided as an input to at least one of the other ML / DL computer models 620 of the population 600 to focus processing by the ML / DL computer model 620 on only the portion of the input slice of the input volume 105 that corresponds to the liver. By preprocessing the input to the ML / DL computer model with liver mask 614, this processing by the ML / DL computer model can be focused on the portions of the input slices that correspond to the anatomical structures of interest, and not on "noise" in the input image. Others of the ML / DL computer models, such as ML / DL computer model 630, use liver mask 614 generated by the first ML / DL computer model 610 and directly receive input volume 105 without liver mask processing.

[0138] In the illustrated example embodiment of the ensemble 600, a third ML / DL computer model 630 is configured with encoder sections 634-636 and a decoder section 638. The ML / DL computer model 630 is configured to receive nine slice slabs of the input volume 105, which are in turn divided into groups 631-633 of three slices each, where each group 631-633 is an input to a corresponding encoder network 634-636. Each encoder 634-636 is a fully connected, headless convolutional neural network (CNN), such as a DenseNet-121 (D121), that is pre-trained to recognize different types of objects (e.g., lesions) present in the input slices and to output a classification output, such as an output classification vector, indicating the detected type of object present in the input slice. CNNs 634-636 can operate on, for example, three channels of input slices, and the resulting output features of CNNs 634-636 are provided to concatenated NHWC logic 637, where NHWC refers to the number of images in the batch (N), image height (H), image width (W), and number of image channels (C). The original DenseNet network architecture includes many convolutional layers and skip-true connections that downsample the three-slice full-resolution input to many smaller-resolution feature channels. From there, a fully connected head aggregates all features and maps them to multiple classes in the final DenseNet output. Because the DenseNet network is used as an encoder in the illustrated architecture, this head is removed, and only the downsampled features are kept. Thus, in concatenated NHWC logic 637, all of the feature channels are concatenated to pass them to a decoder stage 638, which is responsible for upsampling the image until it reaches the desired (e.g., 512x512) output probability map resolution.

[0139] The encoders 634-636 share the same parameters that are optimized through the training process, such as weights, training lesion type sampling, loss weights, and augmentation type. Training the ML / DL computer model 630 uses two different loss functions. The primary loss function is an adaptive loss that is specifically configured to penalize false positive errors in slices without a ground truth lesion and false negative errors in slices with a ground truth lesion. The loss function is a modified Tversky loss as follows: For each output slice: TP = sum(prediction*target) FP = sum((1-target)*prediction) FN = sum((1-prediction)*target) LOSS = 1-((TP+1) / (TP+1+alpha*FN+beta*FP)), where "prediction" is the output probability of the ML / DL computer model 630 and "target" is the ground truth lesion mask. The output probability values ​​range from 0 to 1. Target is either a value of 0 or 1 for each pixel in the slice. For slices that do not have a lesion in them, the term "alpha" is small (e.g., zero) and "beta" is large (e.g., 10). For slices that do have a lesion in them, the term "alpha" is large (e.g., 10) and "beta" is small (e.g., 1).

[0140] The second loss function 639 is a function connected to the outputs of the encoders 634-636. Because the input for this loss comes from the middle of the ML / DL computer model 630, it is also referred to as "deep supervision" 639. Deep supervision is known to force the encoder neural networks 634-636 to better learn representations of the input data during training. In one example embodiment, this second loss is a simple mean squared error in predicting whether a slice has a lesion or not. Therefore, a mapping network is used to map the output features of the encoders 634-636 to nine values ​​between 0 and 1, corresponding to the probability of a lesion being present in each of the nine slice inputs. The decoder 638 generates an output specifying a probability map for lesions detected in the input image.

[0141] The second ML / DL computer model 620 receives three preprocessed slices from the input volume, which have been preprocessed using the liver mask 614 generated by the first ML / DL computer model 610 to identify portions of the three slices that correspond to the liver mask 614. The resulting preprocessed input slices (which in the illustrated example embodiment are of size 192x192x3) are provided to the second ML / DL computer model 620, which includes a DenseNet-169 (D169) encoder 621 connected to two decoders (representing 2D DEC, where the decoder consists of two-dimensional neural network layers). The D169 encoder 621 is a neural network feature extractor widely used in computer vision applications. It consists of a series of convolutional layers, with features extracted from each layer connected to every other layer in a feed-forward fashion. The features extracted in the encoder 621 are transferred to two independent decoders 622, 623, each consisting of a two-dimensional convolutional layer and an upsampling layer (labeled 2D DEC in FIG. 6). Each decoder 622, 623 is trained to detect lesions (e.g., liver lesions) in input slices. While both decoders 622, 623 are trained to perform the same task, i.e., lesion detection, the key difference in their training is that the two decoders 622, 623 each utilize a different loss function to drive the detection training in two competing directions, as described above and below. The final detection map of the second ML / DL model 620 is combined with the final detection map of the third ML / DL model 630 using an averaging operation 640. This procedure is applied across all input slabs of the input volume 105 to produce the final detection map (e.g., liver lesions).

[0142] As previously mentioned, a second ML / DL computer model 620, which attempts to achieve opposing detection operating point performance, is trained using two different loss functions. One encoder 622 uses a training loss function that penalizes errors in false-negative lesion detection, thus producing highly sensitive detections with relatively low accuracy, while the other encoder 623 uses a training loss function that penalizes errors in false-positive lesion detection, resulting in less sensitive but more accurate detections. One example of one of these loss functions is the Focal Tversky Loss, whose parameters are tuned for high or low penalties for false positives and false negatives, depending on the example embodiment (see Abraham et al., "A Novel Focal Tversky Loss Function with Improved Attention U-Net for Lesion Segmentation," arXiv:1810.07842[cs], October 2018). A third loss function, the consistency loss 627, is used to enforce consistency between the predicted detections of each decoder 622, 623. The consistency loss logic 627 compares the outputs 624, 625 of the two encoders 622, 623 with each other and enforces that the outputs are similar to each other. This loss can be, for example, a mean squared error loss between the two predicted detections, a structural similarity loss, or any other loss that enforces consistency / similarity between the compared predicted detections.

[0143] At runtime, using these opposing operating point encoders 622, 623, the second ML / DL computer model 620 generates two lesion outputs 624, 625 that are input to slice averaging (SLC AVG) logic 623, which generates an average value of the lesion output. This average value of the lesion output is then resampled to generate an output that is dimensionally matched to the output of the third ML / DL computer model 630 to which it is compared (note that this process consists of reversing the liver masking operation, thereby computing the lesion output at the original 512x512x3 resolution).

[0144] At runtime, the slice averaging (SLC AVG) logic 626 operates to generate the final detection map for the ML / DL model 620 given the lesion prediction outputs 624 and 625 of the encoders 622 and 623. It should be understood that while a consistency loss 627 was applied during training to drive each decoder 622 and 623 to learn consistent detections, at runtime this consistency loss is no longer utilized, and instead the ML / DL model 620 outputs two detection maps that must be aggregated by the SLC AVG module 626. The results of the SLC AVG logic 626 are resampled to generate an output with dimensions commensurate with the input slab (512x512x3). All of the ML / DL model 620 detections generated for each slab of the input volume 105 are combined with the ML / DL model 630 detections generated via the volume averaging (VOL ACG) logic 640, which calculates the average of the two detection masks at the voxel level. The result is a Final Leision mask 650 that corresponds to the lesions detected in the input volume 105 .

[0145] Thus, after training the ML / DL computer models 620, 630, when presented with a new slice of a new input volume 105, the first ML / DL computer model 610 generates a liver mask 614 to preprocess the input to the second ML / DL computer model 620, and the two ML / DL computer models 620, 630 process the input slice to generate a lesion prediction that is averaged over that volume by the volume averaging logic 640. The result is a final lesion output 650 along with a liver mask output 660 based on the operation of the first ML / DL computer model 610. These outputs may be provided as liver / lesion detection logic stage 130 outputs of the AI ​​pipeline 100, which are provided to the lesion segmentation logic stage 140 of the AI ​​pipeline 100, as previously described above and described in more detail below. In this manner, the mechanisms of the example embodiment provide an ensemble 600 approach to anatomical structure identification and lesion detection in an input volume 105 of medical images (slices).

[0146] The ensemble architecture shown in Figure 6 achieves improved performance over the use of a single ML / DL computational model. That is, it is observed that through the use of an ensemble architecture, improved detection specificity at the same sensitivity level as a single ML / DL computational model is achieved through the combination of the detection outputs of multiple ML / DL computational models in the ensemble. That is, due to errors (false positives) made by the ML / DL models 620, 630 at various locations, when their detection outputs are averaged, the signals from true positive lesions dominate while the signals from false positives are reduced, leading to improved performance.

[0147] FIG. 7 is a flowchart outlining an exemplary operation of liver / lesion detection logic in an AI pipeline according to one example embodiment. As shown in FIG. 7, the operation begins by receiving an input volume (step 710) and performing anatomical structure detection, e.g., liver detection, using a first trained ML / DL computer model, such as a U-Net computer model configured and trained to identify anatomical structures (e.g., liver) (step 720). The result of the anatomical structure detection is segmentation of the input volume to identify a mask for the anatomical structure (e.g., liver mask) (step 730). This input volume is also processed through a first trained ML / DL computer model of a population specifically configured and trained to perform lesion detection (step 740). The first trained ML / DL computer model generates a first set of lesion detection prediction outputs based on its processing of the input volume (step 750).

[0148] A second trained ML / DL computer model of the population receives a masked input generated by applying the generated anatomical structure mask to an input volume, thereby identifying portions of the medical image in the input volume that correspond to the anatomical structure of interest (step 760). The second trained ML / DL computer model processes the masked input through two different decoders with two different, competing loss functions, e.g., one that penalizes errors in false-positive lesion detection and another that penalizes errors in false-negative lesion detection (step 770). The result is two sets of lesion prediction outputs that are then combined through combining logic to generate the lesion prediction output of the second ML / DL computer model (step 780). If necessary, the second lesion prediction output is resampled and combined with the first lesion prediction output generated by the first ML / DL computer model of the population to generate a final lesion prediction output (step 790). The final lesion prediction output is then output along with the anatomical structure mask (step 795), terminating the operation.

[0149] Lesion Segmentation

[0150] As previously described, lesion prediction outputs are generated through the operation of various ML / DL computational models and logic stages of the AI ​​pipeline, including body part detection, target body part determination, phase classification, target anatomical structure identification, and anatomical structure / lesion detection. For example, in the AI ​​pipeline 100 shown in FIG. 1, the results of the liver / lesion detection stage 130 of the AI ​​pipeline 100 include one or more contours of the liver, as well as a detection map, e.g., a voxel-by-voxel map of liver lesions detected in the input volume 105, that identifies portions of the medical image data elements that correspond to detected lesions 135. The detection map is then input to the lesion segmentation stage 140 of the AI ​​pipeline 100.

[0151] As previously described, the lesion segmentation logic, e.g., lesion segmentation step 140 in FIG. 1 , uses a watershed technique and corresponding ML / DL computer models to segment the detection map to generate an image element segmentation of the medical image (slice) of the input volume. The liver lesion segmentation step also provides other mechanisms, such as one or more other ML / DL computer models, that perform operations to identify all of the contours corresponding to lesions present in the slice of the input volume based on the image element segmentation and identify which contours correspond to the same lesion in three dimensions. The lesion segmentation step further includes mechanisms, such as one or more further ML / DL computer models, that aggregate the interrelated lesion contours to generate a three-dimensional lesion segmentation.

[0152] Lesion segmentation uses inpainting of lesion image elements and non-liver tissue represented in the medical image to focus on each lesion individually and perform active contour analysis. In this way, individual lesions may be identified and processed without biasing the analysis due to other lesions in the medical image or due to portions of the image outside the liver. The result of this lesion segmentation is a list of lesions with their corresponding contour lines or contours in the input volume.

[0153] FIG. 8 illustrates a block diagram overview of aspects of the lesion segmentation process performed by the lesion segmentation logic according to an example embodiment. As shown in FIG. 8, lesion segmentation encompasses mechanisms for two-dimensional detection, i.e., segmenting the detection of lesions in two-dimensional slices (block 810), connecting the two-dimensional lesions along the z-axis (block 820), and refining the contours slice by slice (block 830). Each of these blocks is described in more detail below with respect to subsequent figures. The segmentation process illustrated in FIG. 8 is implemented as a process for identifying all lesions in a given input volume under analysis and distinguishing lesions that are near another lesion in an image (slice) of the input volume. For example, two lesions that appear to merge over a pixel period in one or more images may need to be identified as two distinct regions or distinct lesions for purposes of lesion classification and other downstream processing of the detected lesions, such as separate lesion identification in the output of a list of lesions for downstream computing system operations, such as providing a medical viewing application, performing treatment recommendations, or performing decision support operations.

[0154] As part of the slice-by-slice 2D image segmentation in block 810, the example embodiment mechanism uses existing watershed techniques to segment the detection map from a previous lesion detection stage of the AI ​​pipeline, e.g., detection map 135 generated by liver / lesion detection logic 130 of AI pipeline 100 in FIG. 1 . The watershed algorithm requires seed definition to perform mask segmentation. This results in the mask being divided into as many regions as there are seeds, with exactly one seed located approximately in the center of every region, as shown in FIGS. 10A and 10C . In automatic segmentation, seeds within a mask may be obtained as local maxima in its distance map (distance to the mask contour). However, such an approach is prone to noise and can lead to too many seeds, which can over-segment the mask. Therefore, the segmentation must be refined by regrouping some of the regions. Given the empirical observation that most lesions are bubble-shaped, the guiding principle for region regrouping is to result in new regions that are approximately circular. For example, for the mask shown in Figure 10C, the mechanism would fuse the two regions identified by seeds 1051 and 1061, respectively, resulting in a new mask segmentation consisting of only two approximately round regions. Thus, for a detected lesion defined in the detection map 135, such as the lesion shown on the left side of Figure 9 described hereafter, it can be segmented into several bubble-shaped lesions, as shown on the right side of Figure 9, which would be considered as cross-sections of the 3D lesion on the slice.

[0155] Watershed segmentation is a region-based method that has its origins in mathematical morphology. In watershed segmentation, the image is viewed as a local landscape with ridges and valleys. The elevation values ​​of this landscape are typically defined by the gray value of each pixel or the magnitude of their gradient, thereby viewing the 2D representation as a 3D representation. The watershed transformation decomposes the image into "catchment basins". For each local minimum, the catchment basin contains all points whose steepest descent path ends at that minimum. Watersheds separate basins from each other. The watershed transformation completely decomposes the image and assigns each pixel to either a region or a watershed.

[0156] Watershed segmentation requires the selection of at least one marker, also referred to as a "seed" point, inside each object in the image. The seed point may be selected by an operator. In one embodiment, the seed point is selected by an automatic procedure that takes into account application-specific knowledge of the object. Once the objects are marked, they can be grown using morphological watershed transformation, as described in more detail below. Lesions are typically "bubble" shaped. This example embodiment provides a technique for fusing watershed segmented regions based on this assumption.

[0157] Next, in block 820, a mechanism in an example embodiment aggregates the voxel divisions for each of the slices along the z direction to generate a three-dimensional output. To do so, the mechanism must determine whether two sets of image elements, e.g., voxels, in different slices belong to the same lesion, i.e., whether they are aligned in three dimensions. The mechanism calculates measurements between lesions in adjacent slices based on the intersection and union of the lesions and applies a regression model to determine whether two lesions in adjacent slices are part of the same region. Each lesion can be viewed as a set of voxels, and the mechanism determines the intersection of two lesions as the intersection of the two sets of voxels, and the union of two lesions as the union of the two sets of voxels.

[0158] This results in a 3D segmentation of the lesion, but the contour may not fit well to the actual image. There may be an over-segmented lesion. In an example embodiment, we propose using a conventional framework, active contouring, to address the segmentation problem. Such algorithms iteratively refine the contour to gradually fit the image data, while ensuring that it maintains certain desirable properties, such as geometric smoothness. In block 830, the example embodiment mechanism begins the active contour segmentation resulting from the first and second stages 810 and 820, focusing on one lesion at a time; otherwise, running active contour or random segmentation methods on similar lesions may result in them being merged into a single contour, which would be counterproductive because it would effectively undo the benefits of the previous segmentation stages. This mechanism focuses on a single lesion and performs "inpainting" on the lesion voxels and non-liver tissue that are in focus and near the lesion.

[0159] These three-stage processing chains allow the process to be unbiased by other lesions present in the image or by pixels outside the liver, i.e., lesions.

[0160] [Slice-by-slice 2D detection segmentation]

[0161] FIG. 9 shows the results of lesion detection and slice-by-slice segmentation according to one example embodiment. As seen on the left side of FIG. 9, a lesion area 910 may be detected through the preceding AI pipeline process and defined in the contour and detection map, e.g., 135 in FIG. 1, output from the lesion detection logic, e.g., 130 in FIG. 1. According to one example embodiment, the logic of block 810 in FIG. 8 attempts to segment this region into three lesions 911, 912, and 913, as shown on the right side of FIG. 9. The segmentation mechanism in this example embodiment is based on existing watershed techniques that operate to segment the detection map from the preceding lesion detection stage of the AI ​​pipeline. Watershed algorithms are primarily used in image processing for segmentation purposes. The underlying principle behind these known watershed algorithms is that grayscale images can be viewed as a geographic surface, where high intensities represent peaks and hills, while low intensities represent valleys. The watershed technique begins by filling each isolated valley (local minimum) with different colored water (labels). As the water level rises along nearby peaks (slope), the water from the differently colored valleys begins to merge. To prevent this, partitions are built where the water merges. The process of filling with water and building partitions continues until all peaks are underwater, at which point the created partitions provide the segmentation result. Again, watershed techniques are commonly known and will not be described in detail herein. Any known technique may be used to segment a 2D image slice by slice without departing from the spirit and scope of the present invention.

[0162] In the context of lesion segmentation, the empirical observation that most lesions are circular strongly suggests that a segmentation that results in a set of round regions is likely a good segmentation. However, as noted above, the quality of a watershed segmentation depends on the quality of the seeds. In fact, any set of seeds need not result in a set of round regions. For example, Figure 10C shows a watershed segmentation induced by three seeds, containing only one approximately circular region. The other two regions are not circular, although their union is also approximately circular. This configuration is said to be over-segmented because the diagonal segmentation in the figure splits the other circular region into two smaller non-circular regions. Therefore, it is desirable to have an algorithm that can correct over-segmentation. A seed relabeling mechanism does this by merging some over-segmented regions into a coarser segmentation that contains only round-shaped regions. For example, for the segmentation in Figure 10C, this mechanism determines that a new, more circular region should be created by merging the two regions identified by seeds 1051 and 1061.

[0163] This example embodiment merges regions into larger, rounder regions in the segmentation, which may correspond to physical lesions. The segmentation divides an area into smaller regions, or, as described herein, divides a mask into smaller regions. In terms of contours, the segmentation generates a set of smaller contours from a larger contour (see Figure 9, left and right).

[0164] Seeds are obtained by extracting local maxima from a distance map calculated from the input mask to be segmented. The map measures, for each pixel, the Euclidean distance to the mask contour. Depending on the topology of the input mask, local maxima derived from this distance map may lead to an over-segmentation by the watershed algorithm. In this case, the watershed is said to be over-segmented, and while this may be desirable for some applications, it tends to produce non-circular regions that are not ideal for lesion segmentation. Figure 10C shows a composite input mask whose distance map has three local maxima. The watershed then results in a segmentation containing three regions, only one of which (corresponding to seed 1071) is approximately circular. The other two regions are not circular. The region containing seed 1051 is only semicircular. The seed relabeling mechanism then examines all seed pairs and determines that the two regions corresponding to seeds 1051 and 1061 should merge so that they together form a more complete bubble. This operation leads to a new segmentation containing only two regions, both of which are approximately circular in shape.

[0165] A local maximum is a point that has the greatest distance to the contour compared to its immediate neighbors. A local maximum is a point, and its distance to the contour is known. As a result, the mechanism of the example embodiment can draw a circle centered at this point. The radius of the circle is this distance. Then, for two local maxima, the mechanism can calculate the overlap of their respective circles. This is shown in Figures 10A and 10B.

[0166] Relabeling seeds determines whether two regions should be merged as follows: If two regions have directly adjacent seeds, merging occurs; otherwise, the mechanism bases its decision-making on a hypothesis testing procedure. For example, with reference to FIG. 10A , the illustrated example describes a situation in which a distance map may yield two distinct local maxima, leading to the assumption that each local maxima corresponds to the center of a distinct circular lesion. Note that the distance map also allows the mechanism of the example embodiment to tell how far a local maximum is from the contour (boundary). This distance is represented in FIG. 10B by a dotted line segment connecting the maximum and a point on the contour. Therefore, if the assumption holds, the spatial extent of these two lesions can be inferred due to the assumption that the lesions are approximately round or "bubble" shaped. This allows the mechanism of the example embodiment to draw two complete circles, as shown in FIG. 10B . From this, the mechanism measures the overlap of the two circles (e.g., by the classic Dice index) and compares it to a predefined threshold. If the value of the overlap index is greater than this threshold, the mechanism will conclude that the two bubbles overlap too much and are indistinguishable, and fusion will occur. In other words, the mechanism of the example embodiment will conclude that the two maxima correspond to two "centers" of the same lesion. However, in conventional watersheds, no such seed (i.e., maximum) relabeling mechanism exists, which frequently results in mask over-segmentation.

[0167] This overlap can be measured in several ways. In one example embodiment, the mechanism uses the Dice coefficient. For two perfect circles corresponding to two maxima, as shown in FIG. 10B, the mechanism can calculate the Dice index of these two circles. In this way, the mechanism can learn from the training data set what the optimal threshold should actually be, such that, upon receiving a Dice index greater than the threshold, the two maxima are in fact the center of the same lesion.

[0168] Figures 10C and 10D provide another example of a lesion mask shape that differs from the lesion mask shape of Figures 10A and 10B, where two partially fused circles resemble each other more closely in Figure 10A than in Figure 10C. Due to the distance map, which can be very sensitive to mask shape, the example lesion mask shape of Figure 10C has three seeds. Following the above reasoning, a lesion segmentation algorithm would segment the lesion depicted in Figure 10C into two separate lesions, rather than three separate lesions as would occur with a watershed technique that does not relabel seeds.

[0169] In Figures 10C and 10D, seeds 1051 and 1061 represent a more extreme case than that shown in Figures 10A and 10B. Without the seed relabeling technique of the example embodiment, a split would occur that separates them (represented by the solid diagonal line). However, the seed relabeling mechanism of the example embodiment may effectively avoid this undesirable result. Conversely, because seed 1071 is far enough away from seeds 1051 and 1061, the same hypothesis verification procedure described above would tend to accept the assumption that seed 1071 corresponds to the center of a distinct bubble, leading to the vertical split shown in Figures 10C and 10D. Similarly, this would convert the labels assigned to seeds 1051 and 1061 into a different label for seed 1071. However, similar to the situation in Figures 10A and 10B, the hypothesis verification procedure of the seed relabeling technique of the example embodiment would determine that seeds 1051 and 1061 correspond to the same lesion.

[0170] FIG. 11A is a block diagram illustrating a lesion segmentation and relabeling mechanism according to an example embodiment. As shown in FIG. 11A, the mechanism, which may be implemented as a computer model including one or more algorithms, machine learning computer models, etc., executed by one or more processors of one or more computing devices and operates on an input volume of one or more medical image data structures, receives a two-dimensional lesion mask 1101 and performs a distance transform (block 1102) to generate a distance map 1111. This distance transform (block 1102) is an operation performed on a binary mask that calculates, for each point in the lesion mask, the shortest distance to the mask contour (boundary). The further one moves toward the interior of the lesion mask, the further away another one is from its contour (boundary). Thus, the distance transform identifies the center point of the lesion mask, i.e., points at a greater distance than others. In one embodiment, the mechanism optionally performs Gaussian smoothing on the distance map 1111.

[0171] Next, the mechanism performs local maxima identification (block 1103) to generate seeds 1112. As described above, these local maxima are the points in the distance map 1111 that are the longest distance from the contour or boundary. The mechanism performs a watershed technique based on the seeds 1112 (block 1104) to generate a watershed segmented lesion mask 1113. As previously described, this segmented lesion mask 1113 may be over-segmented, resulting in regions that do not fit the expected bubble shape of the lesion. Therefore, the mechanism performs seed relabeling based on the distance map 1111, the seeds 1112, and the segmented 2D lesion mask 1113 (block 1120) to generate an updated segmented lesion mask 1121. Seed relabeling is described in more detail below with reference to FIG. 11B. The resulting updated segmented lesion mask 1121 will have merged regions that more closely match the expected bubble shape for the lesion.

[0172] FIG. 11B is a block diagram illustrating a seed relabeling mechanism according to an example embodiment. As shown in FIG. 11B, a mechanism, which may be implemented as a computer model including one or more algorithms, machine learning computer models, etc., executed by one or more processors of one or more computing devices and operating on an input volume of one or more medical image data structures, receives distance map 1111 and seeds 1112. More specifically, the mechanism considers each seed pair (seed A and seed B) in seeds 1112. The mechanism determines whether seed A and seed B are immediately adjacent seeds (block 1151). If seed A and seed B are immediately adjacent seeds, the mechanism assigns seed A and seed B the same label (block 1155). In other words, seed A and seed B are grouped together to represent only one region.

[0173] In block 1151, if Seed A and Seed B are not directly adjacent, the mechanism performs 1152 spatial range estimation based on the distance map 1111 to ascertain pairwise affinities for Seed A and Seed B, as follows: According to this example embodiment, spatial range estimation assumes a region is "bubble" shaped, whereby the mechanism assumes that each seed corresponds to a circle with its distance from the distance map as the radius of the circle.

[0174] The mechanism then calculates overlap indices for the circles corresponding to Seed A and Seed B (block 1153). In an example embodiment, the mechanism uses dice indices as follows:

[0175]

number

[0176]

number

[0177] The mechanism determines whether the overlap index is greater than a predetermined threshold (block 1154). If the overlap index is greater than the threshold in block 1154, the mechanism merges corresponding regions in the segmented 2D lesion mask 1113 (block 1155).

[0178] If the affinity between two seeds is greater than this threshold, they are assigned the same label. Otherwise, this stage does not know whether they should belong to the same group or not. This decision is left to the same module used for z-direction connections, the label propagation stage (block 1512 in Figure 15), as described below.

[0179] In situations where there are more than two seeds, the same operation in FIG. 11B is repeated for all seed pairs before label propagation to generate seed groups. For example, there is a situation where seed pairs (a, b) and (b, c) are determined to belong to the same group, but seed pair (a, c) fails the test shown in FIG. 11B. As a result, label propagation should place a, b, and c in the same group, i.e., the regions corresponding to seeds a and c will still be merged. However, if there are seeds a, b, c, and d, and the affinity calculation (performed for a total of six pairs) shows that only (a, b) and (c, d) pass the test, label propagation will result in two groups containing (a, b) and (c, d), respectively. Therefore, if a seed pair fails the test, it means that it is not clear whether they should be placed in the same group and not belong to different groups.

[0180] For example, in Figure 10C, there are three seed pairs (1051-1061, 1051-1071, 1061-1071), and the mechanism would determine that seeds 1051 and 1061 should be assigned the same label (belong to the same group). The label propagation step would then cluster these three seeds into two groups, with the first group containing only 1071 and the second group having both 1051 and 1061.

[0181] FIG. 12 is a flowchart outlining an exemplary operation of lesion segmentation according to an example embodiment. The operations outlined in FIG. 12 may be performed by the mechanism previously described with respect to FIGS. 11A-11B. As shown in FIG. 12, operations begin (step 1200) when the mechanism generates a distance map for a two-dimensional lesion mask (block 1101). As previously described, this distance map may be generated by performing a distance transform operation on the two-dimensional lesion mask and, optionally, Gaussian smoothing to remove noise. The mechanism then uses local maximum identification to generate groupings of data points, e.g., local maxima per group (step 1202). The mechanism then performs lesion segmentation based on the local maxima to generate regions (step 1203). Next, the mechanism uses the distance map to relabel seeds based on pairwise affinity (step 1204). This allows the mechanism to merge regions corresponding to seeds with the same label (step 1205). It should be appreciated that due to the relabeling of seeds performed by the mechanism of the example embodiment, the segmented lesion mask output in step 1205 does not suffer from the over-segmentation problem associated with the watershed technique, as previously discussed above, due to incorrect labels being associated with the data points associated with each of the lesion shapes. After this, the operation terminates (step 1206).

[0182] [Z-direction connection of lesions]

[0183] The above process of lesion segmentation and seed relabeling is performed for each two-dimensional image or two-dimensional slice of the input volume, thereby generating appropriately labeled lesion masks for each of the lesions represented in the corresponding two-dimensional image. However, the input volume represents a three-dimensional representation of the internal anatomy of a biological entity, and lesions that may appear to be associated with the same lesion when viewed in three dimensions may actually be associated with different lesions. Therefore, to be able to correctly identify separate lesions within the biological entity when represented by the three-dimensional input volume, example embodiments provide a mechanism for connecting two-dimensional lesions along the z-axis, i.e., in three dimensions.

[0184] This mechanism for connecting two-dimensional lesions along the z-axis, called lesion z-connection, includes a logistic regression model executed on the segmented lesion output generated by the above mechanism to determine three-dimensional z-direction lesion detection. The mechanism connects two lesions in adjacent image slices. If the logistic regression model determines that the two lesions represent the same lesion, the two lesions are connected. For example, for any two-dimensional lesions on adjacent image slices, i.e., slices with z-axis coordinates that are consecutively ordered along the z-axis in a three-dimensionally organized set of slices, the mechanism determines whether these two-dimensional lesions belong to the same three-dimensional lesion, as described below.

[0185] 13A-13C illustrate a process for lesion z-connection according to an example embodiment. FIG. 13A shows the lesion mask input. FIG. 13B shows the lesions after lesion segmentation per slice, which may employ the relabeling-improved lesion segmentation mechanism of the example embodiment described above. As shown in FIGS. 13A and 13B, slice 1310 contains lesions 1311 and 1312, slice 1320 contains lesion 1321, and slice 1330 contains lesions 1331 and 1332. The lesion z-connection mechanism, i.e., a logistic regression model, is run against the segmented lesion mask for each pair of adjacent slices in the input volume to compare each lesion in a particular slice with each lesion in the pair's adjacent slice. For example, the lesion z-connection mechanism compares lesion 1311 (Lesion A) in slice 1310 with lesion 1321 (Lesion B) in slice 1320. For each comparison, the mechanism considers each lesion as a set of voxels and determines the intersection of Lesion A (the set of voxels in Lesion A) and Lesion B (the set of voxels in Lesion B) with respect to the size of Lesion A and with respect to the size of Lesion B. The lesion z-connectivity mechanism uses a logistic regression model to determine whether Lesion A and Lesion B are connected based on the two overlap rates as follows:

number

[0186] Here, |A| denotes the area of ​​the circle corresponding to seed A, |B| denotes the area of ​​the circle corresponding to seed B, and |A∩B| denotes the area of ​​the intersection of the circles corresponding to seed A and seed B. The mechanism uses these two ratios as input features to train a logistic regression model to determine the probability that lesion A and lesion B are connected. That is, using a machine learning process such as that described above, the logistic regression model is trained on a volume of training images to generate a prediction for each pairwise combination of slices in each training volume regarding the probability that a lesion in one slice is the same or different from the lesion represented in the adjacent slice. This prediction is compared to a ground truth indication of whether the lesions are the same or different to generate a loss or error. The operating parameters of the logistic regression model, e.g., coefficients or weights, are then modified to reduce this loss or error until a predetermined number of training epochs have been performed or a predetermined stopping condition has been met.

[0187] Logistic regression models are widely used to solve binary classification problems. In the context of an example embodiment, this logistic regression model predicts the probability that two cross sections of a lesion are part of the same lesion. To do this, logistic regression uses two overlap rates γ0 and γ1, as described above. Specifically, the logistic regression model learns to linearly combine two features as follows:

number

[0188] There are two extreme cases. First, when the threshold t is set to 0, the z-connection mechanism of the example embodiment always determines that the lesions are the same lesion, i.e., the cross sections are connected. This results in both a true positive rate and a false positive rate of 1. Second, when the threshold t is set to 1, the z-connection mechanism does not identify any cross sections of the lesion that are connected. In this case, both the true positive rate and the false positive rate are 0. Therefore, only when the threshold t is in the interval (0, 1) does the logistic regression model make a decision as to whether the lesion cross sections correspond to the same lesion or do not span neighboring slices. With an ideal logistic regression model, the true positive rate would be equal to 1 (all true connections would be identified) and the false positive rate would be 0 (zero false connections would be made).

[0189] Thus, once the logistic regression model has been trained, new slice pairs can be evaluated by calculating these ratios for the pair and inputting them as input features into the "trained logistic regression model" to generate predictions for each of these pairs; then, if the predicted probability is greater than or equal to a predetermined threshold probability, lesions A and B are considered to be associated with the same lesion in three dimensions. Corresponding relabeling of lesions across slices can then be performed to correctly associate lesions in two-dimensional slices with the same lesion representation in other adjacent slices, thereby identifying three-dimensional lesions within the input volume.

[0190] There is rationale supporting the two ratio input features used to train the logistic regression model. For example, if lesions A and B are sufficiently different in size, they are likely not part of the same lesion. Also, if lesions A and B do not intersect, as is the case for lesion 1312 in slice 1310 and lesion 1321 in slice 1320, features γ0 and γ1 will have zero values. As previously mentioned, the logistic regression model performs a regression given the two feature values ​​γ0 and γ1 and outputs a probability value between 0 and 1 representing the likelihood lesion A and likelihood lesion B are part of the same lesion.

[0191] 13C illustrates cross-sectional connectivity between slices, according to an example embodiment. As shown in FIG. 13C, by executing the trained logistic regression model of an example embodiment that predicts lesion commonality based on overlap rates as described above, the mechanism determines that lesion 1311 in slice 1310 and lesion 1321 in slice 1320 are part of the same lesion. Similarly, the mechanism also determines that lesion 1321 in slice 1320 and lesion 1331 in slice 1330 are part of the same lesion. In this manner, the mechanism propagates the intersecting lesions along the z-axis and performs z-axis connectivity of the lesions.

[0192] Lesion relabeling can be performed based on pairwise evaluation of slices in the input volume to identify z-directional connectivity of lesions across two-dimensional slices and a trained logistic regression model's determination of whether lesions are connected along the z-axis, ensuring that the same label for a lesion is applied to each of the lesion masks present in each of the slices of the input volume, e.g., all of the lesion masks across a set of slices in the input volume, where these lesion masks are determined by the logistic regression model to be associated with the same lesion A and can be relabeled to designate them as part of the same lesion A. This can be performed for each lesion cross-section in each of the slices of the input volume, thereby generating a three-dimensional association of lesion masks for one or more lesions present in the input volume. This information can then be used to represent or otherwise process the lesion in three dimensions, such as in subsequent downstream computing system operations, because all of the cross-sections associated with the same lesion are correctly labeled in the input volume.

[0193] Figures 14A and 14B show the results of a trained logistic regression model according to one example embodiment. Figure 14A shows receiver operating characteristic (ROC) curves for the maximum overlap (γ) + minimum overlap (γ) index and for the maximum overlap index. ROC curves are graphical plots that show the diagnostic ability of a binary classifier system as its discrimination threshold is varied. ROC curves are created by plotting the true positive rate (TPR) against the false positive rate (FPR) at various threshold settings. Figure 14B shows precision-recall curves for the maximum overlap + minimum overlap index and for the maximum overlap index. Precision-recall curves are plots of precision (y-axis) and recall (x-axis) for various thresholds, similar to ROC curves. Here, precision is the proportion of relevant instances among retrieved instances, and recall (or sensitivity) is the proportion of the total number of relevant instances actually retrieved. As shown in these figures, the two-feature logistic model outperforms its one-feature counterpart, thus both features bring valuable information to this prediction task.

[0194] Looking at the maximum overlap rate (γ0) + minimum overlap rate (γ1) index curve in Figure 14A, we can see that with an appropriate threshold t, the trained logistic regression model can generate a true positive rate of ∼=95% at the expense of a false positive rate of approximately 3%. Looking at Figure 14B, the plot shown evaluates the trained logistic regression model in terms of precision and recall, and shows that both measures can yield very good results with the correct choice of threshold t.

[0195] Figure 15 is a flowchart outlining an example operation of a mechanism for connecting two-dimensional lesions along the z-axis, according to one example embodiment. As shown in Figure 15, operation begins (step 1500), with the mechanism selecting a first image X from an input volume (step 1501) and selecting a first lesion A in image X (step 1502). In some example embodiments, images or slices in the input volume may be processed using the segmentation and relabeling mechanism described above, although this is not required. Conversely, the example embodiment mechanism directed to z-direction connection of lesions may be performed for virtually any input volume for which a lesion mask has been identified.

[0196] Next, the z-direction connection mechanism of the example embodiment selects the first lesion B in adjacent image Y (step 1503). Next, the mechanism determines the intersection of lesion A and lesion B for lesion A, and determines the intersection of lesion A and lesion B for lesion B (step 1504). The mechanism determines whether lesion A and lesion B belong to the same lesion based on the two intersection values ​​by applying the trained logistic regression model to the γ0 and γ1 features for the intersection of lesion A and lesion B, generates a prediction or probability that lesion A and lesion B belong to the same lesion, and then compares this probability to a threshold probability (step 1505). Based on the result of this determination, lesion cross-sections in the image can be labeled or re-labeled to indicate whether they are part of the same lesion.

[0197] The mechanism determines whether lesion B in image Y is the last lesion in image Y (step 1506). If lesion B is not the last lesion, the mechanism considers the next lesion B in adjacent image Y (step 1507), and operation returns to step 1504 to determine the intersection of lesion A with the new lesion B.

[0198] If, in step 1506, lesion B is the last lesion in adjacent slice or image Y, the mechanism determines whether lesion A is the last lesion in image X (step 1508). If lesion A is not the last lesion in image X, the mechanism considers the next lesion A in image X (step 1509), and operation returns to step 1502 to consider the first lesion B in adjacent image Y.

[0199] In step 1508, if lesion A is the last lesion in image X, the mechanism determines whether image X is the last image to consider (step 1510). If image X is not the last image, the mechanism considers the next image X (step 1511), and operation returns to step 1502 to consider the first lesion A in the new image X.

[0200] In step 1510, if image X is the last image to be considered, the mechanism propagates intersecting lesions between images along the z-axis. Here, propagation means that labels associated with the same lesions, as determined through the above process, are set to the same value to indicate that they are part of the same lesion (step 1512). This is performed for each distinct lesion identified in the input volume, such that cross-sections in each of the images corresponding to the same lesion are labeled appropriately, and thus a three-dimensional representation of each lesion is generated through z-directional connection of the cross-sections. The operation then terminates (step 1513).

[0201] [Contour refinement]

[0202] The above process produces accurate results in terms of lesion number and relative location, and lesion connectivity across two-dimensional space (within an image or slice) and three-dimensional space (across images or slices in the input volume). However, the lesion contours (boundaries) are not always well defined and require improvement. Example embodiments provide a mechanism for improving lesion contour accuracy. This additional mechanism may be employed as part of the lesion segmentation mechanism described above, or in other example embodiments that do not require the specific lesion detection, lesion segmentation and relabeling, and / or z-connectivity mechanisms described above.

[0203] Existing contouring algorithms work well only when the lesion is located in the center of an anatomical structure without surrounding lesions, but there are various situations where two or more closely spaced lesions lead to a "leakage" problem where initially distinct contours merge into a single, comprehensive contour, thereby completely erasing the benefits provided by early 2D lesion mask segmentation. In some cases where the field lesion is near an anatomical structure boundary, e.g., the liver boundary, contouring algorithms distinguish the anatomical structure pixels against other anatomical structures in the image, e.g., organ pixels, rather than distinguishing one lesion from another, as the contouring algorithm can largely distinguish them.

[0204] The mechanism of the example embodiment inpaints areas of interest in an image or slice. Figure 16 shows an example of the case of contours for two lesions in the same image, according to an example embodiment. On the left side of Figure 16, an efficient contour algorithm is used to determine contours 1611 and 1612 for the two lesions. An efficient contour algorithm is a class of algorithms that iteratively evolve contours to better match image content.

[0205] According to this example embodiment, the mechanism inpaints non-liver tissue within contour 1612 and near contour 1611 but not within contour 1611, where this inpainting means that the pixel values ​​for pixels of healthy tissue (non-diseased tissue) within contour 1612 and near contour 1611 are set to a particular value such that they all have the same value. For example, this value may be the average tissue value in areas identified as not associated with a lesion, i.e., healthy tissue of the anatomical structure, e.g., the liver.

[0206] This inpainting may be performed on the selected lesion contour 1611 such that the inpainting is applied to healthy tissue and other lesions, e.g., lesion 1612, in the image. In this manner, the contour and pixels associated with the selected lesion, e.g., 1611, are considered separately from the rest of the image when re-evaluating the contour 1611. The contour 1611 may thereby be re-evaluated, and a determination may be made as to whether re-evaluation of the contour 1611 results in an improved definition of the contour 1611. That is, an initial determination of the contrast and variance between pixels associated with the selected lesion contour 1611 and pixels near the selected lesion contour 1611 may be generated. After calculating this contrast and variance prior to inpainting, inpainting may be performed on the selected lesion 1611 such that pixels associated with other lesion contours, e.g., 1612, and areas of the anatomical structure corresponding to healthy tissue in the image are inpainted with the average pixel intensity value of the healthy tissue.

[0207] The variance of a set of values ​​is determined as follows: Consider a voxel set consisting of, say, n voxels. First, their intensity values ​​are summed, and then the arithmetic mean is calculated by dividing the resulting sum by n. This is denoted as A. Second, these voxel values ​​are individually squared, and then the arithmetic mean is calculated. This result is denoted as B. The variance is then defined as BA*A, i.e., the difference between B and A squared.

[0208] Thus, a set of n values ​​{x1,...,x n The variance of} is defined as follows:

[0209]

number

[0210] This variance is calculated between voxels inside and outside a particular contour: inside a contour are voxels that are surrounded by the contour, and outside voxels are those that are outside the contour but remain within a predefined distance from it.

[0211] After inpainting using an effective contouring algorithm as described above, the mechanism recalculates the contour 1611 of the selected lesion, recalculates the contrast and / or variance of the new contour 1611, and determines whether these values ​​have improved (higher contrast values ​​or lower variance values ​​inside and / or outside the lesion). If the contrast and variance have improved, the newly calculated contour 1611 is retained as the contour of the corresponding lesion. This process can then be performed on lesion 1612 as the selected lesion by inpainting pixels associated with healthy tissue near lesion 1611 and contour 1612. In this way, each lesion is evaluated individually to generate a contour for each lesion, thereby preventing lesions from leaking into each other.

[0212] The mechanism for calculating the lesion contour after inpainting can be based on the Chan-Vese segmentation algorithm, which is designed to segment objects without clearly defined boundaries. This algorithm is based on a set of levels that are iteratively expanded to minimize the energy defined by the sum of the difference in intensity from the mean outside the segmented area, the sum of the difference from the mean inside the segmented area, and a weight corresponding to a term that depends on the length of the segmented area's boundary. Initialization is performed using the segmented detection map (to solve the problem of local minimum energy).

[0213] After the mechanism provides segmentation, it initializes the contour with the previous estimate and determines whether the new contour is better, e.g., whether the contour's contrast and variance are improved. If the original contour is better, the original contour is kept. If the new contour is better, e.g., if the contour's contrast and variance are improved, the mechanism uses the new contour. In some example embodiments, the mechanism determines which contour is better based on calculating the homogeneous area and variance. If the variance is reduced both inside and outside the contour, the mechanism uses the new contour; otherwise, the mechanism uses the previous contour. In another example embodiment, the mechanism determines whether the contrast (average value inside the contour vs. the average value near the contour) is improved. Other techniques using different metrics may be used to select between the previous and new contours without departing from the spirit and scope of example embodiments.

[0214] FIG. 17 is a flowchart outlining an exemplary operation of a mechanism for slice-by-slice contour refinement according to an example embodiment. As shown in FIG. 17, operation begins (step 1700) with a particular contour in an image segmented to show a lesion, such as in the liver, and the mechanism determines a first contrast and variance for the initial contour (step 1701). The mechanism inpaints lesion pixels (or three-dimensional voxels) near the lesion (step 1702). Next, the mechanism determines a contour around the lesion (step 1703). Next, the mechanism determines a second contrast and variance for the new contour (step 1704). The mechanism determines whether the second contrast and variance represent an improvement over the first contrast and variance (step 1705). If the second contrast and variance represent an improvement, the mechanism uses the updated contour to represent the lesion (step 1706). The operation then terminates (step 1708).

[0215] If, in step 1705, the second contrast and variance do not represent an improvement, the mechanism returns to the initial contour (step 1707). The operation then terminates (step 1708). This process may be repeated for each lesion identified in the input slice and / or input volume to recalculate and improve the contour corresponding to each lesion in the image / input volume.

[0216] [False positive removal]

[0217] After performing lesion segmentation to generate a list of lesions and their contours, AI pipeline 100 performs a false-positive processing stage 150 to remove incorrectly indicated lesions from the list of lesions. This false-positive stage 150 can take many forms, such as reducing the number of incorrectly identified lesions in the list of lesions, e.g., the contours and maps 135 in FIG. 1 output by liver / lesion detection logic 130 and then fused by segmentation and relabeling performed in lesion segmentation logic 140. The following description demonstrates a novel false-positive removal mechanism that can be used to perform this false-positive removal, although this specific false-positive removal is not required. Also, the false-positive removal mechanism described hereinafter can be used independently of the other mechanisms described above and can be applied to any list of identified objects in an image, and example embodiments utilize such false-positive removal specifically in the case of lesions in medical images. That is, the false-positive removal mechanism described in this section can be implemented independently and distinctly from the other mechanisms described herein.

[0218] For ease of explanation, we will assume that a false positive removal mechanism is implemented as part of AI pipeline 100 and as part of AI pipeline 100's false positive removal logic 150. Thus, in false positive stage 150, the false positive removal mechanism described in this section operates on the list of lesions resulting from the liver / lesion detection logic and lesion segmentation and relabeling, taking into account the three-dimensional nature of the input volume in the case of the lesion z-connectivity and contour refinement described above. This list 148 in FIG. 1 is input to false positive removal logic stage 150, which processes list 148 as described below and outputs a filtered or corrected list of lesions to lesion classification stage 160, in which incorrectly identified lesions have been minimized in the corrected list of lesions. The lesion classification stage thereby classifies the various lesions indicated in the corrected list of lesions.

[0219] That is, capturing all lesions in earlier stages of the AI ​​pipeline 100 may lead to increased sensitivity settings that cause the AI ​​pipeline 100 to mistakenly identify pixels that do not actually represent a lesion as part of a lesion. As a result, there may be false positives that must be removed. The false positive stage 150 includes logic that operates on a list of lesions and their contours to remove false positives. It should be understood that such false positive removal must also be balanced against the risk that, if false positive removal is not performed in a timely manner at the time of the examination (the set of input volume levels versus the lesion level), some lesions may go undetected. This can be challenging because it may lead to physicians and patients being unaware of lesions that require treatment. It should be understood that a single examination could theoretically include several volumes of images for the same patient. However, in some example embodiments with a single-phase detection implementation of the AI ​​pipeline, only one volume of images is processed, and it is assumed that this processing is performed on a single volume. For clarity, the term "patient level" will be used hereafter instead of "examination level" since this is what is of interest for this example embodiment (the patient either has a lesion or does not have a lesion). It should be appreciated that in other example embodiments, the operations described herein may extend to an examination level where multiple volumes of images may be evaluated for the same patient.

[0220] In an example embodiment, given the output of the previous stages of the AI ​​pipeline 100 (slices, masks, multiple lesions, lesion and anatomical contours, etc.) as input 148 to the false positive removal stage 150, the false positive removal stage 150 operates at a very specific operating point at the patient level (input volume level) to allow only a small number of patient-level false positives (normal patients / volumes in which at least one lesion is detected). This point can be derived from an analysis of the patient's receiver operating characteristic (ROC) (patient-level sensitivity vs. patient-level specificity) analysis. The patient-level operating point OP is referred to herein as the "patient-level operating point OP." patientFor volumes that result in at least some lesions, a more sensitive operating point is referred to herein as the lesion-level operating point OP. lesion The lesion level operating point OP lesion can be identified from an analysis of the lesion-level ROC curve (lesion sensitivity vs. lesion specificity) to maximize the number of lesions retained.

[0221] Two operating points: OP patient and OP lesion The segmentation logic may be implemented in one or more trained ML / DL computer models. The one or more trained ML / DL computer models are trained to classify an input volume and / or a list of lesions (the results of the segmentation logic) as to whether the identified lesions are true or false lesions, i.e., true positives or false positives. The one or more trained ML / DL computer models may be implemented as a binary classifier whose output indicates, for each lesion, whether it is a true positive or a false positive. The set of outputs, including the binary classifications for all of the lesions in the input list of lesions, may be used to filter the list of lesions to remove false positives. In one example embodiment, the one or more trained ML / DL computer models first implement a patient-level operating point to determine whether the classification results indicate that any of the lesions in the list of lesions are true positives, while filtering out the false positives. If true positives remain in the initially filtered list of lesions after patient-level (input volume-level) filtering, a lesion-level operating point is used to filter out any remaining false positives. The result is a filtered list of lesions that minimizes false positives.

[0222] The operating point implementation can be for only one trained ML / DL computer model or for multiple trained ML / DL computer models. For example, when using a single trained ML / DL computer model, the operating point can be a setting for the operating parameters of the ML / DL computer model that can be dynamically switched. For example, inputs to the ML / DL computer model can be processed using a patient-level operating point to generate results indicating whether a list of lesions contains true positives after each classification of the lesion. If true positives are included, the operating point of the ML / DL computer model can then be switched to a lesion-level operating point, and inputs that are again processed with false positives after each pass through the ML / DL computer model are removed from the final list of lesions output in the false positive removal stage. Alternatively, in some example embodiments, two separate ML / DL computer models can be trained, one for a patient-level operating point and one for a lesion-level operating point, such that results from a first ML / DL computer model showing at least one true positive trigger processing of the identified false positives through a second ML / DL computer model, and both models being removed from the final list of lesions output by the false positive removal stage of the AI ​​pipeline.

[0223] Training an ML / DL computer model may involve a machine learning training operation in which the ML / DL computer model processes training inputs including image volumes and corresponding lists of lesions, where the list of lesions includes lesion masks or contours, to generate a classification for each lesion in the image as to whether it is a true positive or false positive. The training input may further be associated with ground truth information that indicates whether the image contains a lesion, which may then be used to evaluate the output generated by the ML / DL computer model, determine a loss or error, and then modify the operating parameters of the ML / DL computer model to reduce the determined loss / error. In this way, the ML / DL computer model learns input features that represent true positive / false positive lesion detections. This machine learning may result in an operating point, i.e., an OP, such that the operating parameters of the ML / DL computer model are trained taking into account patient-level sensitivity / specificity, lesion-level sensitivity / specificity, or both. patient and OP lesion This can be done for each of the above.

[0224] When classifying lesions as true positives or false positives, an input volume (corresponding to a patient at the "patient level") is considered positive if it contains at least one lesion. An input volume is considered negative if it does not contain any lesions. With this in mind, a true positive is defined as a positive input volume, i.e., an input volume with at least one detection classified as a lesion that is in fact a lesion. A true negative is defined as a negative input volume, i.e., an input volume without a lesion and without any detections classified as a lesion. A false positive is defined as a negative input volume without any lesions, but this input shows a lesion in the detection results. That is, the AI ​​pipeline lists a lesion when one does not exist. A false negative is defined as a positive input volume with a lesion, but the AI ​​pipeline does not show a lesion in the detection results. The trained ML / DL computer model classifies the lesions in the input as true positives or false positives. False positives are filtered out of the output generated by false positive removal. False positive detection occurs at the patient level and at the lesion level, i.e., two different operating points and at various levels of sensitivity / specificity.

[0225] Two different operating points, one at the patient level and one at the lesion level, can be determined based on ROC curve analysis. The ROC curve can be calculated using ML / DL computer model validation data consisting of several input volumes (e.g., several input volumes corresponding to various patient exams) that may contain several lesions (0 to K lesions per exam). The inputs to the trained ML / DL computer model, or "classifier," are the detections found so far in the input that are either real lesions or false positives, e.g., the output of the lesion detection and segmentation stages of the AI ​​pipeline. The first operating point, i.e., the patient-level operating point OP patientis defined to maintain at least X% of lesions identified as true positives, i.e., retaining almost all true positives while eliminating some false positives. The value of X may be set based on analysis of the ROC curve and may be any value suitable for a particular implementation. In one example embodiment, the value of X is set to 98% so that almost all true positives are maintained while eliminating some false positives.

[0226] The second operating point, i.e., the lesion-level operating point OP lesion is the lesion sensitivity at the first operating point, i.e., the patient-level operating point OP patient The lesion-level operating point is defined to exceed the lesion sensitivity obtained for the lesion-level operating point and to have a specificity greater than Y%, where Y is determined by the actual performance of the trained ML / DL computer model. In one example embodiment, Y is set to 30%. An example of an ROC curve for determining the patient-level and lesion-level operating points is shown in FIG. 18A. As shown in FIG. 18A, the lesion-level operating point is selected along the lesion-level ROC curve such that the lesion sensitivity exceeds the lesion sensitivity for the patient-level operating point.

[0227] Figure 18B is an exemplary flow diagram of operations for performing false positive removal based on patient-level and lesion-level operating points, according to an example embodiment. As shown in Figure 18B, the result of the segmentation stage logic of the AI ​​pipeline is input 1810 to a first trained ML / DL computer model 1820 that implements a first operating point. The input 1810 includes an input volume (or volume of image (VOI)) and a list of lesions, which includes lesion mask or contour data specifying pixels or voxels corresponding to each of the lesions identified in the image data of the image volume, and labels associated with these pixels that specify the lesions corresponding to the three-dimensional space of the input volume, i.e., the output of the segmentation, z-connection, and contour refinement described above. The input may be denoted as a set S. The first trained ML / DL computer model 1820 implements a patient-level operating point in its training to classify features extracted from the input at X%, e.g., 98%, of the true positives that are retained in the resulting filtered list of lesions generated by the classification of the trained ML / DL computer model 1820, and a portion of the false positives that are removed in the resulting list. The resulting list includes a subset S that contains the true positive lesions classified by the first ML / DL computer model 1820. + and the subset S containing false-positive lesions classified by the first ML / DL computer model 1820. - This includes:

[0228] The false positive removal logic further includes true positive evaluation logic 1830 that determines whether the true positive subset output by the first ML / DL computer model 1820 is empty. That is, the true positive evaluation logic 1830 determines whether any elements from S are classified as true lesions by the first ML / DL computer model 1820. If the true positive subset is empty, the true positive evaluation logic 1830 removes the true positive subset S. +is output as a filtered list of lesions 1835, i.e., no lesions are identified in the output sent to the lesion classification stage of the AI ​​pipeline. The true positive evaluation logic 1830 selects the true positive subset S + If it determines that OP is not empty, a second ML / DL computer model 1840 is run with input S, and the second ML / DL computer model 1840 determines a second operating point during its training, i.e., a lesion-level operating point OP. lesion As mentioned above, while two ML / DL computer models 1820 and 1840 are shown for convenience of explanation, it should be understood that these two operating points may be implemented at different sets of trained operating parameters to construct the same ML / DL computer model, such that a second ML / DL computer model may be the same ML / DL computer model as 1820, but processing of input S with different operating parameters corresponding to the second operating point.

[0229] The second ML / DL computer model 1840 processes the input with the trained operating parameters corresponding to the second operating point to again generate a classification of the lesion as to whether the lesion is a true positive or a false positive. This results in a subset S containing the predicted lesions (true positives). '+ and the subset S containing predicted false positives '- The filtered list of lesions 1845 is then divided into a subset S '+ and the subset S '- Virtually eliminates false positives specified in

[0230] The example embodiments shown in Figures 18A and 18B are described in terms of patient-level and lesion-level operating points. It should be understood that the false positive removal mechanism can be implemented with a variety of different levels of operating points. For example, similar operations can be performed for image volume-level operating points and voxel-level operating points in a "voxel-by-voxel" false positive removal operation. Figure 18C is an example flow diagram of operations for performing voxel-by-voxel false positive removal based on input volume-level and voxel-level operating points, according to one example embodiment. The operations in Figure 18C are similar to those in Figure 18B, but these operations can be performed with respect to voxels in the input set S. For voxel-by-voxel false positive removal, the first operating point can again be a patient-level or input volume-level operating point, while the second operating point can be a voxel-level operating point OP. voxel In this case, true and false positives are evaluated at the voxel level, so that if any voxel is shown to be associated with a lesion and it is in fact associated with a lesion, it is a true positive, whereas if a voxel is shown to be associated with a lesion and it is not in fact associated with a lesion, it is considered a false positive. The appropriate setting of the operating point can again be generated based on the corresponding ROC curve, so that a similar balance between sensitivity and specificity is achieved as above.

[0231] While the above example embodiment of the false positive removal mechanism assumes a single input volume from a patient exam, it should also be understood that this example embodiment may be applied to any grouping of one or more images (slices). For example, false positive removal may be applied to a single slice, a set of slices smaller than the input volume, or even multiple input volumes from the same exam.

[0232] FIG. 19 is a flowchart outlining an exemplary operation of the false-positive removal logic of an AI pipeline according to an example embodiment. As shown in FIG. 19, the operation begins by receiving an input S from a previous stage of the AI ​​pipeline (step 1900). Here, the input includes, for example, an input volume of images and a corresponding list of lesions, including masks, contours, etc. The input is processed by a first trained ML / DL computer model implementing a first operating point, e.g., a patient-level operating point with relatively high specificity and low sensitivity, to generate a first set of classifications for lesions, including a true-positive subset and a false-positive subset (step 1920). A determination is made as to whether the true-positive subset is empty (step 1930). If the true-positive subset is empty, the operation outputs the true-positive subset as a list of filtered lesions (step 1940), and the operation terminates. If the true positive subset is not empty, the input S is processed by a second trained ML / DL computer model (step 1950) that implements a second operating point that is relatively more sensitive and less specific than the first operating point. As previously mentioned, in some example embodiments, the first and second ML / DL computer models may be the same model but configured with different operating parameters corresponding to different training that implement different operating points. The result of processing the second ML / DL computer model is a second set of classifications for lesions, including a second true positive subset and a second false positive subset. This second true positive subset is then output as a filtered list of lesions (step 1960), terminating the operation.

[0233] [Example of computer system environment]

[0234] The example embodiments may be utilized in many different types of data processing environments. To provide context for the description of the specific elements and functionality of the example embodiments, Figures 20 and 21 are provided hereinafter as example environments in which aspects of the example embodiments may be implemented. It should be understood that Figures 20 and 21 are not intended to assert or imply any limitations with respect to the environments in which aspects or embodiments of the present invention may be implemented. Many modifications to the depicted environments may be made without departing from the spirit and scope of the present invention.

[0235] 20 illustrates a schematic diagram of an example embodiment of a cognitive system 2000 implementing a request processing pipeline 2008, which, in some embodiments, may be a question answering (QA) pipeline, a treatment recommendation pipeline, a medical image augmentation pipeline, or any other artificial intelligence (AI) or cognitive computing-based pipeline that processes requests using a composite artificial intelligence mechanism that approximates a human through a process on generated results, but through various computer-specific processes. For purposes of this description, it is assumed that the request processing pipeline 2008 is implemented as a QA pipeline that operates on structured and / or unstructured requests in the form of input questions. One example of a question processing operation that may be used in conjunction with the principles described herein is described in U.S. Patent Application Publication No. 2011 / 0125734, which is incorporated herein by reference in its entirety.

[0236] The cognitive system 2000 is implemented in one or more computing devices 2004A-D (including one or more processors and one or more memories, and possibly any other computing device elements commonly known in the art, including buses, storage devices, communication interfaces, etc.) connected to a computer network 2002. For ease of explanation only, FIG. 20 shows the cognitive system 2000 implemented in only computing device 2004A; however, as previously mentioned, the cognitive system 2000 may be distributed across multiple computing devices, such as multiple computing devices 2004A-D. The network 2002 includes multiple computing devices, namely, computing devices 2004A-D, which may operate as server computing devices, and computing devices 2010-2012, which may operate as client computing devices, which communicate with each other and with other devices or components via one or more wired data communication links and / or wireless communication links, where each communication link may include one or more of a wire, router, switch, transmitter, receiver, etc. In some example embodiments, cognitive system 2000 and network 2002 enable question processing and answer generation (QA) functionality for one or more cognitive system users via their respective computing devices 2010-2012. In other embodiments, cognitive system 2000 and network 2002 may provide other types of cognitive operations, including, but not limited to, request processing and cognitive response generation, which may take many different forms depending on the desired implementation, such as obtaining cognitive information, training / instructing users, cognitive evaluation of data, etc. Other embodiments of cognitive system 2000 may be used with components, systems, subsystems, or devices, or combinations thereof, other than those depicted herein.

[0237] The cognitive system 2000 is configured to implement a request processing pipeline 2008 that receives input from various sources. Requests may be submitted in the form of natural language questions, natural language requests for information, or natural language requests to perform cognitive operations. For example, the cognitive system 2000 receives input from a network 2002, one or more electronic document corpora 2006, cognitive system users, or other data and other possible input sources, or a combination thereof. In one embodiment, some or all of the inputs to the cognitive system 2000 are routed through the network 2002. Various computing devices 2004A-D on the network 2002 include access points for content creators and cognitive system users. Some of the computing devices 2004A-D include database devices that store one or more data corpora 2006 (shown as separate entities in FIG. 20 for illustrative purposes only). Portions of the data corpus(s) 2006 may also be provided on one or more other network-attached storage devices, one or more databases, or other computing devices not explicitly shown in Figure 20. Network 2002 includes local network connections and remote connections in various embodiments, allowing cognitive system 2000 to operate in any scale of environment, including local and global, such as the Internet.

[0238] In one embodiment, a content creator creates content in one or more data corpus 2006 documents for use as part of a data corpus using cognitive system 2000. A document may include any file, text, article, or data source for use by cognitive system 2000. A cognitive system user accesses cognitive system 2000 via a network connection to network 2002 or an Internet connection and inputs a question / request to cognitive system 2000 to be answered / processed based on content in one or more data corpuses 2006. In one embodiment, the question / request is formulated using natural language. Cognitive system 2000 parses and interprets the question / request via pipeline 2008 and provides a response to a cognitive system user, e.g., cognitive system user 2010, including one or more answers to the posed question, a response to the request, a result of processing the request, etc. In some embodiments, the cognitive system 2000 provides a response to the user in a ranked list of candidate answers / responses, while in other example embodiments, the cognitive system 2000 provides a single final answer / response or a combination of the final answer / response and a ranked list of other candidate answers / responses.

[0239] The cognitive system 2000 implements a pipeline 2008 that includes multiple stages for processing an input question / request based on information obtained from one or more data corpora 2006. The pipeline 2008 generates an answer / response to the input question or request based on the input question / request and processing of the one or more data corpora 2006.

[0240] In some example embodiments, the cognitive system 2000 may be an IBM Watson™ cognitive system available from International Business Machines Corporation of Armonk, New York, augmented by the mechanisms of example embodiments described hereinafter. As outlined above, the IBM Watson™ cognitive system pipeline receives an input question or request, then parses the question or request to extract key features of the question / request, which in turn use the key features to formulate a query that is applied to one or more data corpora 2006. Based on the application of the query to one or more data corpora 2006, a set of hypotheses, or candidate answers / responses to the input question / request, is generated. This generation occurs by searching across one or more data corpora 2006 for portions of the one or more data corpora 2006 (hereinafter simply referred to as corpora 2006) that have some likelihood of containing a valuable response to the input question / response (hereinafter assumed to be the input question). The IBM Watson™ cognitive system's pipeline 2008 then uses various inference algorithms to perform deep analysis on the language of the input question and the language used in each of the portions of the corpus 2006 identified when the query is applied.

[0241] The scores obtained from the various inference algorithms are then weighted against a statistical model that summarizes the confidence level that the pipeline 2008 of the IBM Watson™ cognitive system 2000, in this example, has in the evidence that the likely candidate answer is inferred by the question. This process is then repeated for each of the candidate answers to generate a ranked list of candidate answers that may be presented to the user who submitted the input question, e.g., the user of the client computing device 2010, or from which a final answer is selected and presented to the user. Further information about the pipeline 2008 of the IBM Watson™ cognitive system 2000 may be obtained, for example, from the IBM Corporation website, IBM Redbook, etc. For example, information about the pipeline of the IBM Watson™ cognitive system can be found in "Watson and Healthcare" by Yuan et al., IBM developerWorks, 2011, and "The Era of Cognitive Systems: An Inside Look at IBM Watson and How it Works" by Rob High, IBM RedBooks, 2012.

[0242] As previously mentioned, input from a client device to the cognitive system 2000 may be posed in the form of a natural language question, although example embodiments are not limited to such. Rather, the input question may be formatted or structured as any suitable type of request that can be parsed and analyzed using structured and / or unstructured input analysis, including but not limited to the natural language analysis and analysis mechanisms of a cognitive system such as IBM Watson™, to determine the basis for performing a cognitive analysis and resulting in the cognitive analysis. For example, a physician, patient, or the like may issue a request via the client computing device 2010 to the cognitive system 2000 for a particular medical image-based operation, such as "identify liver lesions present in patient ABC," or "provide treatment recommendations for the patient," or "identify changes in liver lesions in patient ABC." According to example embodiments, such a request may be specifically directed to a cognitive computing operation employing the lesion detection and classification mechanisms of example embodiments to provide a list of lesions, lesion contours, lesion classifications, and contours of the target anatomical structure, upon which the cognitive system 2000 operates to provide a cognitive computing output. For example, the request processing pipeline 2008 may process a request such as "identify liver lesions present in patient ABC" to parse the request, thereby identifying the anatomical structure of interest as the "liver," and a specific input volume may be a medical image volume for patient "ABC" within which "lesions" within the anatomical structure are to be identified. Based on this parsing, a specific medical image volume corresponding to patient "ABC" may be obtained from the corpus 2006 and input into the lesion detection and classification AI pipeline 2020. The lesion detection and classification AI pipeline 2020 operates on this input volume as described above to identify a list of liver lesions. The list of liver lesions may be output to the cognitive computing system 2000 for further evaluation through the request processing pipeline 2008, such as to generate a medical image viewer application output.

[0243] As shown in FIG. 20 , one or more of the computing devices, e.g., server 2004, may be specifically configured to implement a lesion detection and classification AI pipeline 2020, e.g., AI pipeline 100 in FIG. 1 . Configuring a computing device may include providing special-purpose hardware, firmware, etc. to facilitate performing the operations and generating output described herein with respect to the example embodiments. Configuring a computing device may also or alternatively include providing software applications stored on one or more storage devices and loaded into memory of a computing device, such as server 2004, to cause one or more hardware processors of the computing device to execute software applications that configure the processors to perform the operations and generate output described herein with respect to the example embodiments. Also, any combination of special-purpose hardware, firmware, software applications executing on hardware, etc. may be used without departing from the spirit and scope of the example embodiments.

[0244] It should be understood that when a computing device is configured in one of these ways, the computing device becomes a specialized computing device that is specifically configured to implement the mechanisms of the example embodiments, and not a general-purpose computing device. Furthermore, implementation of the mechanisms of the example embodiments, as described herein, improves the functionality of the computing device, providing useful and tangible results that facilitate not only automated lesion detection in target anatomical structures, but also classification of such lesions, thereby reducing error and increasing efficiency compared to manual processes.

[0245] As previously described, the mechanisms of example embodiments utilize specially configured computing devices or data processing systems to perform operations for anatomical structure identification, lesion detection, and classification. These computing devices or data processing systems may comprise various hardware elements specifically configured to implement one or more of the systems / subsystems described herein through a hardware configuration, a software configuration, or a combination of hardware and software. FIG. 21 is a block diagram of just one example of a data processing system in which aspects of example embodiments may be implemented. Data processing system 2100 is an example of a computer, such as server 2004 in FIG. 20, in which computer-usable code or instructions implementing the processes and aspects of example embodiments of the present invention may be arranged and / or executed to produce the operations, output, and external effects of example embodiments as described herein.

[0246] In the depicted example, data processing system 2100 employs a hub architecture including a north bridge and memory controller hub (NB / MCH) 2102 and a south bridge and input / output (I / O) controller hub (SB / ICH) 2104. A processing unit 2106, main memory 2108, and a graphics processor 2110 are connected to NB / MCH 2102. Graphics processor 2110 may be connected to NB / MCH 2102 through an accelerated graphics port (AGP).

[0247] In the illustrated example, a local area network (LAN) adapter 2112 connects to the SB / ICH 2104. An audio adapter 2116, a keyboard and mouse adapter 2120, a modem 2122, a read-only memory (ROM) 2124, a hard disk drive (HDD) 2126, a CD-ROM drive 2130, a universal serial bus (USB) port and other communication ports 2132, and PCI / PCIe devices 2134 connect to the SB / ICH 2104 via bus 2138 and bus 2140. PCI / PCIe devices can include, for example, Ethernet adapters, add-in cards, and PC cards for notebook computers. PCI uses a card bus controller, but PCIe does not. The ROM 2124 can be, for example, a flash basic input / output system (BIOS).

[0248] The HDD 2126 and CD-ROM drive 2130 connect to the SB / ICH 2104 through a bus 2140. The HDD 2126 and CD-ROM drive 2130 may use, for example, an Integrated Drive Electronics (IDE) or a Serial Advanced Technology Attachment (SATA) interface. A Super I / O (SIO) device 2136 may be connected to the SB / ICH 2104.

[0249] An operating system runs on processing unit 2106. This operating system coordinates and provides control of various components within data processing system 2100 in FIG. 21. As a client, the operating system may be a commercially available operating system such as Microsoft® Windows 10®. An object-oriented programming system, such as the Java® programming system, may operate in conjunction with the operating system, providing calls to the operating system from Java® programs or applications executing in data processing system 2100.

[0250] As a server, data processing system 2100 may be, for example, an IBM® eServerSystem p™ computer system, a Power® processor-based computer system, or the like, running the Advanced Interactive Executive (AIX®) operating system or the LINUX® operating system. Data processing system 2100 may be a symmetric multiprocessor (SMP) system that includes multiple processors in processing unit 2106. Alternatively, a single processor system may be employed.

[0251] Instructions for the operating system, object-oriented programming system, and applications or program objects are located on storage devices, such as HDD 2126, and are loaded into main memory 2108 for execution by processing unit 2106. Processes of example embodiments of the present invention may be performed by processing unit 2106 using computer-usable program code, which may be located in a memory, such as main memory 2108, ROM 2124, or in one or more peripherals 2126 and 2130, for example.

[0252] A bus system, such as bus 2138 or bus 2140 as shown in FIG. 21, may be comprised of one or more buses. Of course, a bus system may be implemented using any type of communications fabric or architecture that provides for a transfer of data between various components or devices attached to the fabric or architecture. A communications unit, such as modem 2122 or network adapter 2112 in FIG. 21, may include one or more devices used to transmit and receive data. A memory may be, for example, main memory 2108, ROM 2124, or a cache, such as found in NB / MCH 2102 in FIG. 21.

[0253] As noted above, in some example embodiments, the mechanisms of the example embodiments may be implemented as, for example, special purpose hardware or firmware, or as, for example, application software stored on a storage device, such as HDD 2126, and loaded into a memory, such as main memory 2108, for execution by one or more hardware processors, such as processing unit 2106. In this manner, the computing device shown in FIG. 21 is particularly configured to implement the mechanisms of the example embodiments and to perform the operations and generate output described herein with respect to the lesion detection and classification artificial intelligence pipeline.

[0254] Those skilled in the art will appreciate that the hardware in Figures 20 and 21 may vary depending on the implementation. Other internal hardware or peripheral devices, such as flash memory, equivalent non-volatile memory, or optical disk drives, may be used in addition to or in place of the hardware depicted in Figures 20 and 21. Additionally, the processes of this example embodiment may be applied to multiprocessor data processing systems other than the SMP systems described above without departing from the spirit and scope of the present invention.

[0255] Additionally, data processing system 2100 may take the form of any of several different data processing systems, including a client computing device, a server computing device, a tablet computer, a laptop computer, a telephone or other communication device, a personal digital assistant (PDA), etc. In some illustrative examples, data processing system 2100 may be a portable computing device configured with, for example, flash memory to provide non-volatile memory for storing operating system files and / or user-generated data. In essence, data processing system 2100 may be any known or later-developed data processing system without architectural constraints.

[0256] As discussed above, it should be understood that the example embodiments can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment containing both hardware and software elements. In one example embodiment, the mechanisms of the example embodiment may be implemented in software or program code, including but not limited to firmware, resident software, microcode, etc.

[0257] Any data processing system suitable for storing and / or executing program code will include at least one processor coupled directly or indirectly to memory elements through a communications bus, such as a system bus. The memory elements may include local memory employed during the actual execution of the program code, mass storage, and cache memory, which provide temporary storage of at least some program code to reduce the number of times the code must be retrieved from mass storage during execution. The memory may be of various types, including, but not limited to, ROM, RPROM, EPROM, EEPROM, DRAM, SRAM, flash memory, solid-state memory, etc.

[0258] Input / output devices, or I / O devices (including but not limited to keyboards, displays, pointing devices, etc.), may be coupled to the system directly or through intervening wired or wireless I / O interfaces and / or controllers. I / O devices can take many different forms other than traditional keyboards, displays, pointing devices, etc., such as communication devices coupled through wireless or wired connections, including but not limited to smartphones, tablet computers, touch screen devices, voice recognition devices, etc. Any I / O device, whether known or later developed, is intended to be within the scope of example embodiments.

[0259] Network adapters may be coupled to the system to enable the data processing system to be coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modems, and Ethernet cards are just a few of the currently available types of network adapters for wired communications. Wireless communication-based network adapters may also be utilized, including, but not limited to, 802.11a / b / g / n wireless communication adapters, Bluetooth wireless adapters, and the like. Any network adapter known or hereafter developed is intended to be within the spirit and scope of the present invention.

[0260] The description of the present invention has been presented for purposes of illustration and description and is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will become apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The embodiments were chosen and described to best explain the principles and practical applications of the invention and to enable those skilled in the art to understand the invention in various embodiments with various modifications suited to the particular uses contemplated. The terminology used herein was chosen to best explain the principles of the embodiments, practical applications or technical improvements over commercially available technology, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. 1. A method in a data processing system including at least one processor and at least one memory, wherein the at least one memory includes instructions that, when executed by the at least one processor, implement a trained machine learning computer model for seed re-labeling of seed-based slice-by-slice lesion segmentation, the trained machine learning computer model performing the method, the method comprising: receiving a lesion mask of a volume of three-dimensional medical images, the lesion mask corresponding to detected lesions in the volume of three-dimensional medical images, each detected lesion having a lesion contour; generating a distance map of a particular two-dimensional slice in the volume of the three-dimensional medical image based on the lesion mask, the distance map including a distance from each voxel in the volume of the three-dimensional medical image formed by stacking the particular two-dimensional slices to a lesion contour; performing a local maximum identification to select a set of local maxima from the distance map such that each local maximum has a value greater than the values ​​of the points adjacent to said local maximum; performing seed relabeling based on the distance map and the set of local maxima to generate a set of seeds, each seed in the set of seeds representing the center of a distinct component of a lesion contour; performing image segmentation on the lesion mask based on the set of seeds to form a segmented lesion mask; Including, method.

2. The method of claim 1 , wherein generating the distance map comprises performing Gaussian smoothing on the distance map.

3. 2. The method of claim 1 , wherein performing seed relabeling comprises grouping a first maximum and a second maximum in response to determining that the first maximum and the second maximum are immediately adjacent.

4. The step of performing seed relabeling comprises: determining a circle centered on each local maximum with a radius equal to the corresponding distance in the distance map; calculating an overlap index of a first circle centered on the first local maximum and a second circle centered on the second local maximum; grouping the first maximum and the second maximum if the overlap measure is greater than a predetermined threshold; 2. The method of claim 1, comprising:

5. The overlap indicator is [Equation 8] where |S1| denotes the area of ​​the first circle, |S2| denotes the area of ​​the second circle, and |S1∩S2| denotes the area of ​​an intersection of the first circle and the second circle.

6. The overlap indicator is [Equation 9] where |S1| denotes the area of ​​the first circle, |S2| denotes the area of ​​the second circle, |S1∩S2| denotes the area of ​​the intersection of the first circle and the second circle, and |S1∪S2| denotes the area of ​​the union of the first circle and the second circle.

7. 2. The method of claim 1, wherein performing image segmentation comprises performing a watershed algorithm on the lesion mask based on the set of local maxima to form an initial lesion mask segmented lesion mask that defines a first set of lesions.

8. 8. The method of claim 7, wherein performing image segmentation further comprises merging lesions in the first set of lesions based on the results of the seed relabeling to form a modified lesion mask segmented lesion mask.

9. A computer program that, when executed by a computing device, causes the computing device to perform steps for implementing a trained machine learning computer model for seed relabeling of seed-based slice-by-slice lesion segmentation, the trained machine learning computer model comprising: receiving a lesion mask of a volume of three-dimensional medical images, the lesion mask corresponding to detected lesions in the volume of three-dimensional medical images, each detected lesion having a lesion contour; generating a distance map of a particular two-dimensional slice in the volume of the three-dimensional medical image based on the lesion mask, the distance map including a distance from each voxel in the volume of the three-dimensional medical image formed by stacking the particular two-dimensional slices to a lesion contour; performing local maximum identification to select a set of local maxima from the distance map such that each local maximum has a value greater than the values ​​of points adjacent to said local maximum; performing seed relabeling based on the distance map and the set of local maxima to generate a set of seeds, each seed in the set of seeds representing a center of a distinct component of a lesion contour; performing image segmentation on the lesion mask based on the set of seeds to form a segmented lesion mask; To execute Computer program.

10. 10. The computer program product of claim 9, wherein performing seed relabeling comprises grouping a first maximum and a second maximum in response to determining that the first maximum and a second maximum are immediately adjacent.

11. Performing seed relabeling is determining a circle centered on each local maximum with a radius equal to the corresponding distance in the distance map; calculating an overlap index of a first circle centered on the first local maximum and a second circle centered on the second local maximum; grouping the first maximum and the second maximum if the overlap measure is greater than a predetermined threshold; 10. The computer program of claim 9, comprising:

12. The overlap indicator is [Equation 10] where |S1| denotes the area of ​​the first circle, |S2| denotes the area of ​​the second circle, and |S1∩S2| denotes the area of ​​an intersection of the first circle and the second circle.

13. The overlap indicator is [0011] where |S1| denotes the area of ​​the first circle, |S2| denotes the area of ​​the second circle, |S1∩S2| denotes the area of ​​the intersection of the first circle and the second circle, and |S1∪S2| denotes the area of ​​the union of the first circle and the second circle.

14. 10. The computer program product of claim 9, wherein performing image segmentation comprises performing a watershed algorithm on the lesion mask based on the set of local maxima to form an initial lesion mask segmented lesion mask that defines a first set of lesions.

15. 15. The computer program product of claim 14, wherein performing image segmentation further comprises merging lesions in the first set of lesions based on results of the seed relabeling to form a modified lesion mask segmented lesion mask.

16. An apparatus, comprising: a processor; a memory coupled to the processor; Including, The memory includes instructions that, when executed by the processor, cause the processor to implement a trained machine learning computer model for seed re-labeling of seed-based slice-by-slice lesion segmentation, the trained machine learning computer model comprising: receiving a lesion mask of a volume of three-dimensional medical images, the lesion mask corresponding to detected lesions in the volume of three-dimensional medical images, each detected lesion having a lesion contour; generating a distance map of a particular two-dimensional slice in the volume of the three-dimensional medical image based on the lesion mask, the distance map including a distance from each voxel in the volume of the three-dimensional medical image formed by stacking the particular two-dimensional slices to a lesion contour; performing local maximum identification to select a set of local maxima from the distance map such that each local maximum has a value greater than the values ​​of points adjacent to said local maximum; performing seed relabeling based on the distance map and the set of local maxima to generate a set of seeds, each seed in the set of seeds representing a center of a distinct component of a lesion contour; performing image segmentation on the lesion mask based on the set of seeds to form a segmented lesion mask; To execute Device.

17. 17. The apparatus of claim 16, wherein performing seed relabeling comprises grouping a first maximum and a second maximum in response to determining that the first maximum and a second maximum are immediately adjacent.

18. Performing seed relabeling is determining a circle centered on each local maximum with a radius equal to the corresponding distance in the distance map; calculating an overlap index of a first circle centered on the first local maximum and a second circle centered on the second local maximum; grouping the first maximum and the second maximum if the overlap measure is greater than a predetermined threshold; 17. The apparatus of claim 16, comprising:

19. 17. The apparatus of claim 16, wherein performing image segmentation includes performing a watershed algorithm on the lesion mask based on the set of local maxima to form an initial lesion mask segmented lesion mask that defines a first set of lesions.

20. 20. The apparatus of claim 19, wherein performing image segmentation further comprises merging lesions in the first set of lesions based on results of the seed relabeling to form a modified lesion mask segmented lesion mask.

Images