Seed relabeling for seed-based segmentation of medical images

Abstract

The present application relates to seed relabeling for seed-based segmentation of medical images. A mechanism for seed relabeling for seed-based slice-wise lesion segmentation is provided. The mechanism receives a lesion mask of a three-dimensional medical image volume. The lesion mask corresponds to detected lesions in the medical image volume, and wherein each detected lesion has a lesion contour. The mechanism generates a distance map for a given two-dimensional slice in the medical image volume based on the lesion mask. The distance map includes a distance to the lesion contour for each voxel of the given two-dimensional slice. The mechanism performs local maxima identification to select a set of local maxima from the distance map such that each local maxima has a value greater than its immediate neighbor points. The mechanism performs seed relabeling based on the distance map and the set of local maxima to generate a set of seeds. Each seed represents a center of a different component of the lesion contour. The mechanism performs image segmentation on the lesion mask based on the set of seeds to form a split lesion mask.

Description

Technical Field

[0001] This application generally relates to an improved data processing apparatus and method, and more specifically to a mechanism for seed relabeling of seed-based segmentation for medical images. Background Technology

[0002] Liver lesions are groups of abnormal cells in the liver, a biological entity, and can also be called clumps or tumors. Non-cancerous or benign liver lesions are common and do not spread to other areas of the body. These benign liver lesions usually do not cause any health problems. However, some liver lesions develop due to cancer. Patients with certain medical conditions may be more likely to have cancerous liver lesions than other patients. These medical conditions include, for example, hepatitis B or hepatitis C, cirrhosis, iron storage disease (hemochromatosis), obesity, or exposure to toxic chemicals (such as arsenic or aflatoxin).

[0003] Liver lesions are typically identified only through medical imaging tests, such as ultrasound, magnetic resonance imaging (MRI), computed tomography (CT), or positron emission tomography (PET) scans. These medical imaging tests must be reviewed by a human subject matter expert (SME), who must use their own knowledge and expertise, along with human abilities, to interpret the patterns in the images to determine if the medical imaging tests reveal any lesions. If the human SME identifies a potentially cancerous lesion, the patient's physician can perform a biopsy to determine if the lesion is cancerous.

[0004] Contrast-enhanced abdominal CT (CE) is the current standard of care for evaluating various abnormalities (e.g., lesions) in the liver. These lesions can be assessed by human SME as malignant (hepatocellular carcinoma, cholangiocarcinoma, angiosarcoma, metastatic lesions, and other malignant lesions) or benign (hemangioma, focal nodular hyperplasia, adenoma, cyst or lipoma, granuloma, etc.). Manual evaluation of such images by human SME is important for guiding subsequent interventions. Often, to properly evaluate lesions on CE CT, multi-stage studies are conducted, where different levels of enhancement of healthy liver parenchyma are provided and compared with the enhancement of lesions to determine differential detection. Human SME can then determine the diagnosis of the lesion based on these differences. Summary of the Invention

[0005] This summary is provided to introduce, in a simplified form, the selection of concepts that will be further described herein in the detailed description. This summary is not intended to identify key elements or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0006] In one illustrative embodiment, a method is provided in a data processing system including at least one processor and at least one memory, the at least one memory including instructions executed by the at least one processor to implement a trained machine learning computer model for seed relabeling for seed-based slice-based lesion segmentation. The trained machine learning computer model performs the following method: receiving a lesion mask of a three-dimensional medical image volume. The lesion mask corresponds to lesions detected in the medical image volume, and wherein each detected lesion has a lesion contour. The method further includes generating a distance map of a given two-dimensional slice in the medical image volume based on the lesion mask. The distance map includes distances to the lesion contour of each voxel of the given two-dimensional slice. The method further includes performing local maximum identification to select a set of local maxima from the distance map such that each local maximum has a value greater than its immediate neighbor. The method further includes performing seed relabeling based on the distance map and the set of local maxima to generate a set of seeds, wherein each seed in the set of seeds represents the center of a different component of the lesion contour. The method further includes performing image segmentation on the lesion mask based on the set of seeds to form a split lesion mask. This has the benefit of providing a split lesion mask that is not overly segmented. The illustrative embodiment combines methods to split lesion masks, where the areas may be sites of the same lesion.

[0007] In one example embodiment, generating the distance map includes performing Gaussian smoothing on the distance map. This has the benefit of smoothing distance values ​​that would result in extraneous maxima, which would lead to further excessive splitting within the splitting lesion mask. Performing Gaussian smoothing reduces the number of candidate points that can be considered local maxima, potentially reducing the number of distinct regions that must be combined during seed relabeling.

[0008] In another example embodiment, performing seed relabeling includes grouping the first local maximum and the second local maximum in response to determining that they are direct neighbors. This has the benefit of eliminating local maximums that are direct neighbors of candidates for segmenting lesions, since very close local maximums are unlikely to be the centers of different lesions.

[0009] In yet another example embodiment, performing seed relabeling includes: determining a circle centered on each local maximum, the radius of which is equal to the corresponding distance of the circle in the distance map; calculating an overlap metric between a first circle centered on a first local maximum and a second circle centered on a second local maximum; and grouping the first and second local maximums if the overlap metric is greater than a predetermined threshold. This embodiment assumes that the lesions are substantially circular, like bubbles, and determines whether the local maximums are the centers of overlapping bubbles. This has the benefit of grouping local maximums together if they might represent the same lesion.

[0010] In a further example embodiment, the overlap metric is calculated as follows:

[0011]

[0012] Where |S1| represents the area of ​​the first circle, |S2| represents the area of ​​the second circle, and |S1∩S2| represents the area of ​​the intersection of the first circle and the second circle. In an alternative example embodiment, the overlap metric is calculated as follows:

[0013]

[0014] Wherein, |S1| represents the area of ​​the first circle, |S2| represents the area of ​​the second circle, |S1∩S2| represents the area of ​​the intersection of the first and second circles, and |S1∪S2| represents the combined area of ​​the first and second circles. These embodiments provide the benefit of assigning values ​​to metrics to determine whether local maxima define distinct lesions or may correspond to the same lesion. These embodiments provide alternative formulas for calculating overlap metric values ​​that can be compared with a threshold.

[0015] In another example embodiment, performing image segmentation includes applying a watershed algorithm to the lesion mask based on a set of local maxima to form an initial split lesion mask defining a first set of lesions. In yet another example embodiment, performing image segmentation further includes merging lesions in the first set of lesions based on the result of the seed relabeling to form a revised split lesion mask. These embodiments provide the benefit of segmenting an image into split lesion masks using known watershed algorithms while also merging regions that may be part of the same lesion, thereby avoiding over-segmentation.

[0016] In other illustrative embodiments, a computer program product is provided, comprising a computer-usable or readable medium having a computer-readable program. When executed on a computing device, the computer-readable program causes the computing device to perform various operations and combinations of operations outlined above in the illustrative embodiments of the method.

[0017] In yet another illustrative embodiment, a system / apparatus is provided. This system / apparatus may include one or more processors and memory coupled to the one or more processors. The memory may include instructions that, when executed by the one or more processors, cause the one or more processors to perform various operations and combinations of operations outlined above with respect to the illustrative embodiments of the method.

[0018] These and other features and advantages of the present invention will be described in the following detailed description of exemplary embodiments of the present invention, or will become apparent to those skilled in the art from the following detailed description of exemplary embodiments of the present invention. Attached Figure Description

[0019] The invention, its preferred mode of use, and other objects and advantages will be best understood by referring to the following detailed description of illustrative embodiments, in conjunction with the accompanying drawings:

[0020] Figure 1 This is an example block diagram illustrating an AI pipeline that implements multiple specially configured and trained ML / DL computer models to perform anatomical structure recognition and lesion detection on input medical image data, according to an illustrative embodiment.

[0021] Figure 2 This is an example flowchart overview of example operations of an AI pipeline according to an illustrative embodiment;

[0022] Figure 3A This is an example diagram illustrating an example input volume of a slice (medical image) of the abdomen of a human patient according to an illustrative embodiment;

[0023] Figure 3B It shows Figure 3A Another depiction of the input volume, in which a slice along with its corresponding axial score s' inf and s' sup together;

[0024] Figure 3C yes Figure 3A Example graph of the input volume, where the volume is axially divided into n completely overlapping segments;

[0025] Figures 4A-4CThis is an example diagram of an illustrative embodiment of an ML / DL computer model configured and trained to estimate s' of segments of an input volume of a medical image. sup and s' inf value;

[0026] Figure 5 This is a flowchart outlining example operations of liver detection and predetermined anatomical structure determination logic according to an illustrative embodiment of an AI pipeline;

[0027] Figure 6 This is an example diagram of an ensemble of ML / DL computer models for performing lesion detection in an anatomical structure of interest (e.g., liver) according to an illustrative embodiment;

[0028] Figure 7 This is a flowchart outlining example operations of liver / lesion detection logic in an AI pipeline according to an illustrative embodiment;

[0029] Figure 8 A block diagram depicts aspects of lesion segmentation according to an illustrative embodiment;

[0030] Figure 9 The results of lesion detection and slice-wise partitioning according to an illustrative embodiment are described;

[0031] Figures 10A-10D Seed positioning is illustrated according to an illustrative embodiment;

[0032] Figure 11A This is a block diagram illustrating the mechanism for lesion splitting according to an illustrative embodiment;

[0033] Figure 11B This is a block diagram illustrating a mechanism for seed relabeling according to an illustrative embodiment;

[0034] Figure 12 This is a flowchart outlining an example operation of lesion splitting according to an illustrative embodiment;

[0035] Figures 13A-13C The z-axis connection of the lesion is illustrated according to an illustrative embodiment;

[0036] Figure 14A and Figure 14B The results of a training model for z-axis lesion connectivity are illustrated according to an illustrative embodiment;

[0037] Figure 15 This is a flowchart outlining an example operation of a mechanism for connecting two-dimensional lesions along the z-axis according to an illustrative embodiment;

[0038] Figure 16 An example illustrating the outlines of two lesions in the same image according to an illustrative embodiment;

[0039] Figure 17 This is a flowchart outlining an example operation of a slice-type contour refinement mechanism according to an illustrative embodiment;

[0040] Figure 18A This is an example of a ROC curve determined based on patient-level and lesion-level operating points according to an illustrative embodiment;

[0041] Figure 18B This is an example flowchart of an operation for performing false positive removal based on patient-level and lesion-level operation points, according to an illustrative embodiment.

[0042] Figure 18C This is an example flowchart of an operation for performing voxel-based false positive removal based on an illustrative embodiment of an input volume level and voxel level operation point;

[0043] Figure 19 This is a flowchart outlining an example operation of the false positive removal logic of an AI pipeline according to an illustrative embodiment;

[0044] Figure 20 These are example diagrams of a distributed data processing system that can implement various aspects of the illustrative embodiments; and

[0045] Figure 21 These are example block diagrams of computing devices that can implement various aspects of the illustrative embodiments. Detailed Implementation

[0046] The detection of lesions or abnormal cell groups is primarily a manual process in modern medicine. Because it is a manual process, it is fraught with sources of error, especially given the human limitations in the ability to detect portions of digital medical images that reveal such lesions in an individual, and particularly considering the increasing demand for evaluating a growing number of images in a shorter timeframe for such individuals. While some automated image analysis mechanisms have been developed, there is still a need to improve these mechanisms to provide more efficient and accurate analysis of medical image data for detecting lesions in imaged anatomical structures (e.g., the liver or other organs).

[0047] The illustrative embodiments specifically address improved computational tools for providing specially trained, automated, computer-driven artificial intelligence medical image analysis. Through machine learning / deep learning computer processes, these tools detect anatomical structures, lesions, or other biological structures of interest within or associated with such anatomical structures. They perform specialized segmentation of the detected lesions or other biological structures, perform false positive removal based on this specialized segmentation, and classify the detected lesions or other biological structures. The results of lesion / biological structure detection are then provided to a downstream computational system for additional computational operations. The following description of the illustrative embodiments assumes specific relevance to embodiments of the mechanisms specifically trained with respect to liver lesions as biological structures of interest; however, the illustrative embodiments are not limited thereto. Rather, those skilled in the art will recognize that the machine learning / deep learning-based artificial intelligence mechanisms of the illustrative embodiments can be implemented with respect to a wide range of other types of biological structures / lesions within or associated with other anatomical structures represented in medical imaging data, without departing from the spirit and scope of the invention. Furthermore, illustrative embodiments may be described with respect to computed tomography (CT) medical imaging data; however, illustrative embodiments may be implemented with any digital medical imaging data from various types of medical imaging techniques, including but not limited to positron emission tomography (PET) and other nuclear medicine imaging, ultrasound, magnetic resonance imaging (MRI), elastography, photoacoustic imaging, echocardiography, magnetic particle imaging, functional near-infrared spectroscopy, elastography, various radiographic imaging including fluorescence fluoroscopy, etc.

[0048] In general, the illustrative embodiments provide an improved artificial intelligence (AI) computer pipeline comprising multiple specially configured and trained AI computer tools (e.g., neural networks, cognitive computing systems, or other AI mechanisms trained on finite datasets to perform a specified task). Each configured and trained AI computer tool is specifically configured / trained to perform a specific type of AI processing on an input medical image volume, represented as one or more sets of data and / or metadata defining a medical image captured by a medical imaging technique. Typically, these AI tools employ machine learning (ML) / deep learning (DL) computer models (or simply ML models) to perform tasks that simulate human thought processes regarding the generated outcomes while using different computer processes specific to the computer tools and, in particular, the ML / DL computer models, which learn patterns and relationships between data representing specific outcomes (e.g., image classification or labels, data values, medical treatment recommendations, etc.). An ML / DL computer model is essentially a function of elements including machine learning algorithms, configuration settings of the machine learning algorithms, features of the input data identified by the ML / DL computer model, and labels (or outputs) generated by the ML / DL computer model. By specifically tuning the functions of these elements through machine learning processes, specific ML / DL computer model instances are generated. Different ML models can be specifically configured and trained to perform different AI functions on the same or different input data.

[0049] Since artificial intelligence (AI) pipelines implement multiple ML / DL computer models, it should be understood that these ML / DL computer models are trained through ML / DL processes for a specific purpose. Therefore, as an overview of the ML / DL computer model training process, it should be understood that machine learning involves the design and development of techniques that take empirical data (such as medical image data) as input and identify complex patterns in the input data. A common pattern in machine learning techniques is to use a low-level computer model M, whose parameters, given input data, are optimized to minimize the cost function associated with M. For example, in the context of classification, model M could be a straight line that divides the data into two classes (e.g., labels) such that M = a*x + b*y + c, and the cost function would be the number of misclassified points. The learning process then operates by adjusting the parameters a, b, and c to minimize the number of misclassified points. After this optimization phase (or learning phase), model M can be used to classify new data points. Typically, given input data, M is a statistical model, and the cost function is inversely proportional to the probability of M. This is merely a general explanation of machine learning training and a simple example of other types of machine learning using different patterns, cost (or loss) functions, and optimizations can be used in conjunction with the mechanisms of the illustrative embodiments without departing from the spirit and scope of the invention.

[0050] For the purpose of anatomical structure detection and / or lesion detection (where lesions are “abnormal” in medical imaging data), learning machines can construct ML / DL computer models representing normal structures to detect data points in medical images that deviate from this normal structural representation ML / DL computer model. For example, a given ML / DL computer model (e.g., supervised, unsupervised, or semi-supervised) can be used to generate anomaly scores and report them to another device, generating classification outputs indicating one or more categories to which the input is classified, probabilities or scores associated with different categories, etc. Example machine learning techniques that can be used to construct and analyze such ML / DL computer models include, but are not limited to, nearest neighbor (NN) techniques (e.g., k-NN models, replicator NN models, etc.), statistical techniques (e.g., Bayesian networks, etc.), clustering techniques (e.g., k-means, etc.), neural networks (e.g., reservoir networks, artificial neural networks, etc.), support vector machines (SVM), etc.

[0051] The processor-implemented artificial intelligence (AI) pipeline of the illustrative embodiments typically includes one or both of machine learning (ML) and deep learning (DL) computer models. In some cases, one or the other of ML and DL may be used or implemented to achieve a specific result. Traditional machine learning may include or use algorithms such as Bayesian decision, regression, decision trees / forests, support vector machines, or neural networks. Deep learning may be based on deep neural networks and may use multiple layers, such as convolutional layers. Such DL (such as using hierarchical networks) can be efficient in their implementation and can provide enhanced accuracy relative to traditional ML techniques. Traditional ML can generally be distinguished from DL because DL models can outperform classic ML models; however, DL models can consume relatively large amounts of processing and / or power resources. In the context of the illustrative embodiments, references to one or the other of ML and DL herein are to be understood as covering one or both forms of AI processing.

[0052] Regarding the illustrative embodiments, after being configured and trained via an ML / DL training process, an ML / DL computer model is executed in the AI ​​pipeline. This ML / DL computer model performs complex computational medical imaging analysis to detect anatomical structures in input medical images and generate outputs specifically identifying target biological structures of interest (hereafter assumed to be liver lesions for the purposes of the illustrative embodiments), classifying the target biological structures of interest, specifying the contours of where these target biological structures of interest (e.g., liver lesions) are present in the input medical images (hereafter assumed to be CT medical image data), and other information to help human subject matter experts (SMEs) (such as radiologists, physicians, etc.) understand the patient's medical condition from the viewpoint of the captured input medical images. Furthermore, the output can be provided to other downstream computer systems to perform additional artificial intelligence operations (such as treatment recommendations and other decision support operations based on classification, contours, etc.).

[0053] Initially, the illustrative embodiment's artificial intelligence (AI) pipeline receives an input volume of computed tomography (CT) medical imaging data and detects which part of the body of the biological entity is depicted in the CT medical imaging data. The "volume" of a medical image is a three-dimensional representation of the internal anatomy of a biological entity, consisting of a stack of two-dimensional slices, where these slices can be individual medical images captured by medical imaging techniques. The stack of slices can also be referred to as a "slab," and unlike the slices themselves, the stack represents a portion of an anatomical structure with thickness, where the stack of slices or slabs generates a three-dimensional representation of the anatomical structure.

[0054] For the purposes of this specification, it will be assumed that the biological entity is a human being; however, the invention can be operated on for medical images of various types of biological entities. For example, in veterinary medicine, the biological entity can be different types of small animals (e.g., pets, such as dogs, cats, etc.) or large animals (e.g., horses, cattle, or other farm animals). In an embodiment where the AI ​​pipeline is specifically trained to detect liver lesions, the AI ​​pipeline determines whether the input CT medical imaging data represents an abdominal scan present in the CT medical imaging data, and if not, the operation of the AI ​​pipeline terminates with respect to the input CT medical imaging data because it does not point to the correct part or area of ​​the human body. It should be understood that, according to illustrative embodiments, there may be different AI pipelines trained to process input medical images targeting different parts of the body and different target biological structures, and input CT medical images may be input into each of the AI ​​pipelines, or routed to an AI pipeline based on the body part depicted in the input CT medical image or the classification of the body part. For example, the classification of the input CT medical image with respect to the body part or body structure represented in the input CT medical image may be performed first, and then a suitable trained AI pipeline may be selected from a plurality of trained AI pipelines of the type described herein to process the input CT medical image. For the purposes of the following description, a single AI pipeline trained to detect liver lesions will be described; however, given this specification, it will be apparent to those skilled in the art that it can be extended to a suite or collection of AI pipelines.

[0055] Assuming the input CT medical image volume includes a medical image of the human abdomen (for the purpose of liver lesion detection), further processing of the input CT medical image is performed in two primary stages. These two primary stages can be executed substantially in parallel and / or sequentially, depending on the desired implementation. The two primary stages include a phase classification stage and an anatomical structure detection stage (e.g., a liver detection stage in the case where the AI ​​pipeline is configured to perform liver lesion detection).

[0056] Phase classification determines whether the volume of the input CT medical image comprises a single imaging phase or multiple imaging phases. In medical imaging, a "phase" is an indication of contrast agent uptake. For example, in some medical imaging techniques, a phase can be defined based on when a contrast agent is introduced into a biological entity, allowing the capture of a medical image, including the path of contrast agent capture. For example, a phase can include a pre-contrast phase, an arterial phase, a portal venography phase, and a delayed phase, in which a medical image is captured in any or all of these phases. Phases are typically associated with timing following injection and with characteristics of structural enhancement within the image. Timing information can be considered to "classify" potential phases (e.g., a delayed phase would always be acquired after the portal venography phase) and to estimate the potential phase of a given image. Regarding the use of structural enhancement characteristics within an image, an example of using this type of information to determine phase is described in U.S. Patent Application Serial No. 16 / 926,880, co-assigned and co-pending, filed July 13, 2020, entitled "Method of Determining Contrast Phase of a Computerized Tomography Image". In addition, timing information can be combined with other information (sampling, kernel reconstruction, etc.) to pick the best representation of each phase (a given acquisition can be reconstructed in several ways).

[0057] Once the images in the input volume are assigned or classified to their corresponding phases based on enhanced timing and / or features, it can be determined based on phase classification whether the volume includes images of a single phase (e.g., a portal vein phase but no arterial phase) or a multiphase examination (e.g., portal vein and arterial). If the phase classification indicates the presence of a single phase in the volume of the input CT medical image, further processing by the AI ​​pipeline is performed as described below. If multiple phases are detected, the volume is not further processed by the AI ​​pipeline. However, in some illustrative embodiments, while this single / multiple phase-based volume filtering only accepts volumes with images from a single phase and rejects volumes with multiple phases, in other illustrative embodiments, the AI ​​pipeline processing described herein can filter out images of volumes not classified into the target phase of interest. For example, images of the portal vein phase in the volume can be retained while images of volumes not classified as part of the portal vein phase are filtered out, thereby modifying the input volume into a modified volume containing only a subset of images classified as the target phase. Furthermore, as previously discussed, different AI pipelines can be trained for different types of volumes. In some illustrative embodiments, phase classification of images within an input volume can be used to route or distribute images of the input volume to corresponding AI pipelines trained and configured to process images of different phases, such that the input volume can be subdivided into constituent sub-volumes and routed to their corresponding AI pipelines for processing. For example, a first sub-volume corresponding to a portal vein phase image is sent to a first AI pipeline, while a second sub-volume corresponding to an arterial phase is sent to a second AI pipeline for processing. If the volume of the input CT medical image comprises a single phase, or after filtering and optionally routing the sub-volumes to the corresponding AI pipelines such that the AI ​​pipelines process the input volume or images of sub-volumes of a single phase, the volume (or sub-volume) is then passed to the next stage of the AI ​​pipeline for further processing.

[0058] The second main level is the anatomical structure of interest (liver in an exemplary embodiment) detection level, where the volumetric portion depicting the anatomical structure of interest is identified and passed to the next downstream level of the AI ​​pipeline. The anatomical structure of interest detection level (hereinafter referred to as the liver detection level according to the example embodiment) includes a machine learning (ML) / deep learning (DL) computer model specifically trained and configured to perform computerized medical image analysis to identify portions of an input medical image corresponding to the anatomical structure of interest (e.g., the liver). Such medical image analysis may include training the ML / DL model on labeled training medical image data as input to determine whether the input medical image (the training image during training) includes the anatomical structure of interest (e.g., the liver). Based on the ground truth of the image labels, the operating parameters of the ML / DL model can be adjusted to reduce loss or error in the results generated by the ML / DL model until convergence (i.e., loss is minimized). Through this process, the ML / DL model is trained to recognize patterns in the medical image data indicating the presence of the anatomical structure of interest (liver in this example). Once trained, the ML / DL model can then be executed on new input data to determine whether the new input medical image data has a pattern indicating the presence of anatomical structures, and if the probability is greater than a predetermined threshold, the medical image data can be determined to include the anatomical structure of interest.

[0059] Therefore, at the liver detection level, the AI ​​pipeline uses a trained ML / DL computer model to determine whether the volume of the input CT medical image includes an image depicting the liver. The volume portion depicting the liver, along with the results from the phase classification level, is passed to the determination level of the AI ​​pipeline, which determines whether a single phase of medical imaging exists and whether at least a predetermined amount of the anatomical structure of interest (e.g., the liver) exists within the volume portion depicting the anatomical structure of interest. The presence of the predetermined amount of the anatomical structure of interest can be determined based on known measurement mechanisms (e.g., calculating the size of the structure from differences in pixel locations within the image) used to determine the structure from the medical image. The measurements can be compared to predetermined sizes (e.g., average sizes) of anatomical structures in similar patients with similar demographics, such that if the measurements represent at least a predetermined amount or portion of the anatomical structure, further processing can be performed by the AI ​​pipeline. In one illustrative embodiment, for example, the determination determines whether at least one-third of the liver exists within the volume portion of the input CT medical image that is determined to depict the liver. While one-third is used in the example embodiment, any predetermined amount of structure determined to be suitable for a particular implementation may be used without departing from the spirit and scope of the invention.

[0060] In one illustrative embodiment, to determine whether a predetermined number of anatomical structures of interest exist within the volume of an input CT medical image, an axial score is defined such that a slice of the medical image corresponding to a first representation of the anatomical structure of interest (e.g., the liver) within the volume, i.e., the first slice (FSL) containing the liver, is assigned a slice score of 0, and the last slice (LSL) containing the liver has a score of 1. Assuming a human biological entity, the first and last slices are defined from the lowest slice (MISV) (closest to the lower limb, e.g., the foot) to the highest slice (MSSV) (closest to the head) within the volume. The liver axial score estimate (LAE) is calculated from a pair of slice scores. sup and s inf To define this, this is the slice score s sup and s inf Slice scores, corresponding to MSSV and MISV slices respectively. As will be described in more detail below, the ML / DL computer model is specifically configured and trained to determine the slice scores of the volume of the input CT medical image. sup and s inf Knowing these slice scores and understanding from the above definition that the liver ranges from 0 to 1, the mechanism of the illustrative embodiment is able to determine the fraction of the liver within the field of view of the volume of the input CT medical image.

[0061] In some illustrative embodiments, the slice score s can be indirectly found by first dividing the volume of the input CT medical image into multiple segments, and then, for each segment, executing a configured and trained ML / DL computer model on the slices of that segment to estimate the height of each slice. sup and s inf In order to determine in segment s' sup and s' inf The top (closest to the head) and bottom (closest to the feet) liver slices. Given s' sup and s' inf The estimate is obtained by extrapolation to find s. sup and s inf The estimation is based on the known localization of segments relative to the entire volume of the input CT medical image. This method is based on a robust estimator of the height of arbitrary slices from the input volume (or a sub-volume associated with the target phase). Such an estimator can be obtained by learning a regression model, for example, by using a deep learning model that performs the estimation of heights from chunks (a set of consecutive slices). For example, Long Short-Term Memory (LSTM) type artificial neural network models are suitable for these tasks because they have the ability to encode the ordering of slices containing liver and abdominal anatomical structures. It should be noted that for each volume, there will be n ssup and s inf The estimate is the sum of the n estimates, where n is the number of segments per volume. In one illustrative embodiment, the final estimate is obtained by taking the unweighted average of these n estimates; however, in other illustrative embodiments, other functions of the n estimates may be used to generate the final estimate.

[0062] The volume s of the input CT medical image has been determined. sup and s inf The final estimate is used to calculate a score for the anatomical structure of interest (e.g., the liver) based on these values. This task is made possible by estimating the height of each slice. Based on the estimated heights (h1) and (h2) of the first and last slices of the liver in the input volume, assuming the actual heights of the first and last slices of the liver (whether or not they are included in the input volume) are H1 and H2, the portion of the liver visible in the input volume can be represented as (min(h1, H1) - max(h2, H2)) / (H1 - H2). This calculated score can then be compared with a predetermined threshold to determine whether a predetermined minimum amount of the anatomical structure of interest exists in the volume of the input CT medical image, for example, at least 1 / 3 of the liver is present in the volume of the input CT medical image.

[0063] If the determination results in the presence of multiple phases and / or the absence of a predetermined number of anatomical structures of interest depicting the anatomy within a portion of the volume of the input CT medical image, further processing of the volume may be aborted. If the determination results in the volume of the input CT medical image containing a single phase and at least a predetermined number of anatomical structures of interest (e.g., showing 1 / 3 of the liver in the image), the portion of the input CT medical image volume depicting the anatomical structure is forwarded to the next stage of the AI ​​pipeline for processing.

[0064] In the next stage of the AI ​​pipeline, lesion detection is performed on portions of the input CT medical image representing anatomical structures of interest (e.g., the liver). This liver and lesion detection stage of the AI ​​pipeline uses an ensemble of ML / DL computer models to detect liver and lesions as represented in the volume of the input CT medical image. The ensemble of ML / DL computer models uses separately trained ML / DL computer models to perform liver and lesion detection, where the ML / DL computer models are trained and a loss function is used to balance false positives and false negatives in lesion detection. Furthermore, the ensembled ML / DL computer models are configured such that a third loss function makes the outputs of the ML / DL computer models consistent with each other.

[0065] Assuming liver detection and lesion detection are performed at this stage of the AI ​​pipeline, a first ML / DL computer model is executed on the volume of the input CT medical image to detect the presence of the liver. This ML / DL computer model can be the same ML / DL computer model used in a previous AI pipeline stage for the detection of the anatomical structure of interest, and therefore, previously obtained results can be utilized. Multiple (two or more) additional ML / DL computer models are configured and trained to perform lesion detection in portions of the medical image depicting the liver. The first ML / DL computer model is configured with two loss functions. The first loss function penalizes errors in false negatives, i.e., misclassifying the absence of a lesion (normal anatomical structure). The second loss function penalizes errors in false positives, i.e., misclassifying the presence of a lesion (abnormal anatomical structure). The second ML / DL is trained to detect lesions using an adaptive loss function that penalizes false positive errors in slices of liver containing normal tissue and false negative errors in slices of liver containing lesions. The detection outputs from the two ML / DL models are averaged to produce the final lesion detection.

[0066] The results of the liver / lesion detection level in the AI ​​pipeline include one or more contours (shapes) of the liver and a detection map that identifies portions of medical imaging data elements corresponding to the detected lesions, such as a voxel-based map of liver lesions detected within the volume of an input CT medical image. This image map is then fed into the lesion segmentation level of the AI ​​pipeline. As will be described in more detail below, the lesion segmentation level uses a watersheld technique to partition the detection map to generate image element (e.g., voxel) partitions of the input CT medical image. Based on this partitioning, the liver lesion segmentation level identifies all contours corresponding to lesions present in slices of the volume of the input CT medical image and performs an operation to identify which contours correspond to the same lesion in three dimensions. The lesion segmentation aggregates the relevant lesion contours to generate three-dimensional partitions of the lesions. The lesion segmentation uses lesion image elements (e.g., voxels) represented in the medical image and inlays of non-liver tissue to focus individually on each lesion and perform active contour analysis. In this way, individual lesions can be identified and processed without bias due to other lesions in the medical image or due to portions of the image outside the liver.

[0067] The result of lesion segmentation is a list of lesions with their corresponding shapes or contours within the volume of the input CT medical image. These outputs may include findings that are not actual lesions. To minimize the impact of those false positives, a trained false positive removal model is used to feed the output to the next stage of the AI ​​pipeline involved in false positive removal. This false positive removal model in the AI ​​pipeline acts as a classifier to identify which outputs are actual lesions and which are false positives from the detected findings. The input consists of the image volume (VOI) surrounding the detected findings, associated with a mask generated from the lesion segmentation refinement. The false positive removal model is trained using data as a result of the detection / segmentation level: objects detected by the detection algorithm as lesions from the underlying facts are used to represent the lesion category during training, while detections of any lesions that do not match the underlying facts are used to represent the non-lesion (false positive) category.

[0068] To further improve overall performance, a dual-operation-point strategy is employed for lesion detection and false-positive models. The idea is that the AI ​​pipeline output can be interpreted at different levels. First, the AI ​​pipeline output can be used to distinguish the examination volume, i.e., the input volume or image volume (VOI), whether it contains a lesion or not. Second, the AI ​​pipeline output is designed to maximize lesion detection, regardless of whether they are contained within the same patient / examination / volume. For clarity, the measurement performed for the examination will be referred to here as the "patient level," and the measurement performed for the lesion will be referred to here as the "lesion level." Maximizing sensitivity at the "lesion level" will reduce specificity at the "patient level" (for a patient, one detection is sufficient to indicate the presence of a lesion). This may ultimately be suboptimal for clinical use, as one must choose between poor specificity at the patient level or low sensitivity at the lesion level.

[0069] Therefore, the illustrative embodiment uses a two-operation-point method for both lesion detection and false-positive removal. The principle is to first run the treatment using a first operation point that provides reasonable performance at the patient level. Then, for patients from the first round who have at least one detected lesion, the detected lesion is reinterpreted / treated using a second operation point. This second operation point is chosen to be more sensitive. Although the specificity of this second operation point is lower than that of the first operation point, this loss of specificity is included at the patient level because all patients without lesions detected by the first operation point remain unchanged, regardless of whether the second operation point detects additional lesions. Therefore, patient-level specificity is determined solely by the first operation point. Patient-level sensitivity is between that of the first and second operation points taken individually (a false negative case from the first operation point can be turned into a true positive by the second operation point). On the lesion side, actual lesion-level sensitivity is improved compared to the first operation point alone. Lesion specificity is better than that taken alone from the less specific second operation point because there are no false positives from cases treated only with the first operation point.

[0070] While the illustrative embodiments assume a specific configuration and use of the dual-operation-point method, it should be understood that the dual-operation-point method can be used with other configurations and for other purposes where there is interest in measuring performance at both the group level (in the illustrative embodiment, this group level is the "patient level") and the element level (in the illustrative embodiment, this element level is the "lesion level"). Although in the illustrative embodiments, the dual-operation-point method is applied to both lesion detection and false positive removal, it is understood that the dual-operation-point method can extend beyond these levels of the AI ​​pipeline. For example, lesion detection can be performed at the voxel level (element) versus the volume level (group), rather than at the patient and lesion levels. As another example, the voxel or lesion level can be used for the element level, and thick slices (sets of slices) can be used as the group level. In yet another example, the volumes of all examinations can be used as the group level instead of a single volume. It should be understood that the method can also be applied to two-dimensional images of the image to be analyzed (e.g., 2D X-rays such as chest or mammography) rather than three-dimensional volumes. Specificity (such as the average number of false positives per patient / group) can be used to select the operation point. Furthermore, although the illustrative embodiments are described as being applied to lesion detection and classification, the dual-operation-point-based method can be applied to other structures (clamps, stents, implants, etc.) and beyond medical imaging.

[0071] The results of dual-operation-point detection and false-positive removal lead to the identification of a final filtered list of lesions awaiting further processing by the lesion classification level of the AI ​​pipeline. At the lesion classification level of the AI ​​pipeline, a configured and trained ML / DL computer model is executed on the lesion list and its corresponding contour data to classify the lesions into one of several predetermined lesion categories. For example, each lesion in the final filtered lesion list and its attributes (e.g., contour data) can be fed into a trained ML / DL computer model, which then operates on this data to classify the lesion into a specific type of lesion. Classification can be performed using a classifier previously trained on ground truth data (e.g., a trained neural network computer model) combined with the results of previous processing steps in the AI ​​pipeline. The classification task can be more or less complex; for example, it can provide a label between benign, malignant, or uncertain, or, in another example, the actual lesion type (e.g., cyst, metastasis, hemangioma, etc.). The classifier can be, for example, a neural network-based computer model classifier (e.g., SVM, decision tree, etc.) or a deep learning computer model. The actual input to this classifier is a patch around the lesion, which in some embodiments can be enhanced with a lesion mask or shape (outline).

[0072] After classifying lesions through the lesion classification level of the AI ​​pipeline, the AI ​​pipeline outputs a list of lesions and their classifications, along with any contour attributes of the lesions. Additionally, the AI ​​pipeline may output liver contour information for the liver. The information generated by this AI pipeline can be provided to further downstream computing systems for further processing and generation of representations of the anatomy of interest and any detected lesions present within those anatomy. For example, a graphical representation of the volume of an input CT medical image can be generated in a medical image viewer or other computer application, where the contour information generated by the AI ​​pipeline is used to overlay or otherwise highlight the anatomy and detected lesions in the graphical representation. In other illustrative embodiments, downstream processing of the information generated by the AI ​​pipeline may include diagnostic decision support operations, automated medical imaging report generation based on the list of detected lesions, classification, and contours. In other illustrative embodiments, based on the lesion classification, different treatment recommendations can be generated for review and consideration by practicing physicians.

[0073] In some illustrative embodiments, the list of lesions, their classifications, and contours may be stored in a patient-associated historical data structure corresponding to the volume of an input CT medical image. This allows multiple executions of the AI ​​pipeline on different volumes of the input CT medical image associated with that patient to be stored and evaluated over time. For example, differences between the list of lesions and / or their associated classifications and contours can be determined to assess the progression of the patient's disease or medical condition, and such information can be presented to medical experts to assist in the patient's treatment.

[0074] Other downstream computing systems and processes can be implemented for the specific anatomical and lesion detection information generated by the AI ​​mechanism of the illustrative embodiments without departing from the spirit and scope of the invention. For example, the output of the AI ​​pipeline can be used by another downstream computing system to process the anatomical and lesion information in the AI ​​pipeline output to identify differences from other information sources (e.g., radiological reports) so that clinical staff are aware of potentially overlooked findings.

[0075] Thus, the illustrative embodiments provide a mechanism for providing an automated AI pipeline comprising multiple configured and trained ML / DL computer models that implement various artificial intelligence operations at different stages of the AI ​​pipeline to identify anatomical structures and lesions associated with these structures in a volume of input medical images, determine contours associated with such anatomical structures and lesions, determine the classification of such lesions, and generate a list of such lesions and their contours and anatomical structures for further downstream computer processing of the AI-generated information from the AI ​​pipeline. The operation of the AI ​​pipeline is automated, ensuring that no human intervention occurs at any stage of the pipeline; instead, the specified AI analysis at each stage is performed using specially configured and trained ML / DL computer models trained through machine learning / deep learning computer processes. The only points where human intervention might occur are before the input of the volume of input medical images (e.g., during medical imaging of a patient) and after the output of the AI ​​pipeline (e.g., viewing an enhanced medical image presented via a computer image viewing application based on the output of the lesion list and contours generated by the AI ​​pipeline). Thus, the AI ​​pipeline performs operations that humans cannot perform as mental processes and does not organize any human activities, because the AI ​​pipeline specifically involves the use of specified machine learning / deep learning processes that exist only within a computer environment to implement improved automated computer tools for artificial intelligence.

[0076] Before proceeding with the discussion of various aspects of the illustrative embodiments and the improved computer operations performed by the illustrative embodiments, it should first be recognized that throughout this specification, the term "mechanism" will be used to refer to the element of the invention that performs various operations, functions, etc. As used herein, the term "mechanism" can be an implementation of the functionality or aspects of the illustrative embodiments in the form of a device, program, or computer program product. In the case of a process, the process is implemented by one or more devices, apparatuses, computers, data processing systems, etc. In the case of a computer program product, the logic represented by computer code or instructions implemented in or on the computer program product is executed by one or more hardware devices to perform a function or perform an operation associated with the designated "mechanism". Thus, the mechanism described herein can be implemented as dedicated hardware, software executing on hardware to configure the hardware to perform the specific functions of the invention that the hardware would not otherwise be able to perform, software instructions stored on a medium such that the instructions can be readily executed by the hardware, thereby specifically configuring the hardware to perform the functionality and specific computer operations described herein, a process or method for performing the function, or any combination of the foregoing.

[0077] This specification and claims may use the terms "a," "at least one," and "one or more" to refer to specific features and elements in illustrative embodiments. It should be understood that these terms and phrases are intended to state the presence of at least one of a specific feature or element present in a particular illustrative embodiment, but more than one may also be present. That is, these terms / phrases are not intended to limit the specification or claims to the presence of a single feature / element or to require the presence of multiple such features / elements. Rather, these terms / phrases require only at least one single feature / element, while multiple such features / elements may be within the scope of the specification and claims.

[0078] Furthermore, it should be understood that the use of the term "engine" in this document with respect to the description of embodiments and features of the invention is not intended to limit any particular implementation for implementing and / or performing actions, steps, processes, etc., attributable to and / or performed by the engine. An engine can be, but is not limited to, software, hardware, and / or firmware, or any combination thereof, that performs a specified function, including but not limited to any use of a combination of general-purpose and / or dedicated processors with appropriate software loaded or stored in machine-readable memory and executed by the processor. Further, unless otherwise specified, any names associated with a particular engine are for ease of reference and are not intended to limit to a particular implementation. Additionally, any function attributed to an engine may be performed equally by multiple engines, incorporated into and / or combined with another engine of the same or different types, or distributed across one or more engines in various configurations.

[0079] Furthermore, it should be understood that the following description uses several different examples of different elements of the illustrative embodiments to further illustrate exemplary implementations of the illustrative embodiments and to aid in understanding the mechanisms of the illustrative embodiments. These examples are intended to be non-limiting and are not an exhaustive list of all possibilities for implementing the mechanisms of the illustrative embodiments. In view of this specification, it will be apparent to those skilled in the art that many other alternative implementations exist for these different elements, in addition to those provided herein or that may be utilized in place of the examples provided herein, without departing from the spirit and scope of the invention.

[0080] The present invention can be a system, method, and / or computer program product. The computer program product may comprise a computer-readable storage medium (or medium) having computer-readable program instructions thereon for causing a processor to perform various aspects of the invention.

[0081] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital universal disc (DVD), memory sticks, floppy disks, mechanical encoding devices (such as punched cards or raised structures in grooves with instructions recorded thereon), and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0082] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them for storage in a computer-readable storage medium within the suitable computing / processing device.

[0083] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as traditional procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet provided by an Internet service provider). In some embodiments, electronic circuitry (including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs)) may execute the computer-readable program instructions by utilizing state information from the computer-readable program instructions to personalize the electronic circuitry in order to perform various aspects of this invention.

[0084] Various aspects of the invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0085] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more boxes of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer-readable storage medium having the instructions stored therein includes an article of writing comprising instructions for implementing aspects of the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0086] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus, or other device, perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0087] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a portion of a module, segment, or instruction, which includes one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions marked in the blocks may occur in a different order than indicated in the figures. For example, depending on the functions involved, two consecutively shown blocks may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0088] Overview of AI-powered lesion detection and classification pipeline

[0089] Figure 1 This is an example block diagram illustrating an artificial intelligence (AI) pipeline (hereinafter referred to as the "AI pipeline") for lesion detection and classification, which implements multiple specially configured and trained ML / DL computer models to perform anatomical structure recognition and lesion detection in input medical image data, according to an illustrative embodiment. For illustrative purposes only, the depicted AI pipeline is specifically described for liver detection and liver lesion detection in medical image data. As mentioned above, the illustrative embodiment is not limited thereto and can be applied to any anatomical structure of interest and lesions associated with such an anatomical structure of interest, which can be represented as image elements of medical image data captured by medical imaging techniques and corresponding computing systems. For example, the mechanisms of the illustrative embodiment can be applied to other anatomical structures (such as the lungs, heart, etc.) and the detection, contour recognition, classification, etc., of lesions associated with the lungs, heart, or other anatomical structures of interest.

[0090] Furthermore, it should be understood that the following description is provided from Figure 1This description provides an overview of the AI ​​pipeline at various levels, and subsequent sections will delve into additional details regarding the various stages of the AI ​​pipeline. In some illustrative embodiments, each stage of the AI ​​pipeline is implemented as a configured and trained ML / DL computer model, such as a neural network of a deep learning neural network, as indicated by symbol 103 in the various stages of AI pipeline 100. These different ML / DL computer models are specifically configured and trained to perform the specific AI operations described herein (e.g., body part recognition, liver detection, phase classification, liver minimal detection, liver / lesion detection, lesion segmentation, false positive removal, lesion classification, etc.). Although these additional sections described below will set forth specific embodiments for implementing the various stages of the AI ​​pipeline that provides the novel technology, and the mechanisms and methods for performing the AI ​​operations at different stages, it should be understood that other equivalent techniques, mechanisms, or methods may be used in the context of the AI ​​pipeline as a whole without departing from the spirit and scope of the illustrative embodiments. In light of this specification, these other equivalent techniques, mechanisms, or methods will be readily apparent to those skilled in the art and are intended to be within the spirit and scope of the invention.

[0091] like Figure 1 As shown, according to an illustrative embodiment, an artificial intelligence (AI) pipeline 100 receives a volume 105 of an input medical image. In the depicted example, the volume 105 of the input medical image is the volume of an input computed tomography (CT) medical image represented as one or more data structures, serving as input for subsequent automated processing by various stages of the AI ​​pipeline 100 to ultimately generate an output 170 including a list of lesions and their classification and contour information, as well as contour information about anatomical structures of interest (e.g., the liver in the depicted example). The volume 105 of the input medical image can be captured by medical imaging technology 102 using any of a large number of generally known or later-developed medical imaging technologies and devices that present images of the internal anatomy of a biological entity (i.e., the patient), as one or more medical image data structures. In some illustrative embodiments, the volume 105 of the input medical image includes a two-dimensional slice (individual medical image) of a portion of the anatomical structure of a part of the patient's body, and the two-dimensional slice is then combined to generate a thick slice (a combination of slices along an axis, thereby providing a collection of medical images with thickness along the axis), and the two-dimensional slice is combined to generate a three-dimensional representation (i.e., the volume of the anatomical structure of the body part).

[0092] In the first-stage logic 110 of the AI ​​pipeline 100, the AI ​​pipeline 100 identifies 112 portions of the patient's body corresponding to the input volume 105 of CT medical imaging data, and determines via body part of interest determination logic 114 whether that portion of the patient's body represents a part of the patient's body corresponding to an anatomical structure of interest (e.g., an abdominal scan instead of a cranial scan, a lower body scan, etc.). This evaluation is performed using initial filters on the AI ​​pipeline 100 only with respect to the input volume 105 of the CT medical imaging data (hereinafter referred to as "input volume" 105), for which the AI ​​pipeline 100 is specifically configured and trained to perform anatomical structure recognition, as well as contour and lesion recognition, contouring, and classification. This detection of body parts represented in the input volume 105 can be performed by examining metadata associated with the input volume 105, which may have fields specifying the regions of the patient's body being scanned, as may be specified by the source medical imaging technology computing system 102 when performing the medical imaging scan. Alternatively, the first-stage logic 110 of the AI ​​pipeline 100 can implement a specially configured and trained ML / DL computer model for body part detection 112, which performs medical image classification on specific parts of a patient's body. The medical image classification performs computerized pattern analysis on medical image data of input volume 105 and predicts the classification of medical imaging data for one or more predetermined parts of the patient's body. In some illustrative embodiments, this evaluation can be binary (e.g., yes or no abdominal medical imaging volume) or it can be a more complex multi-class evaluation (e.g., specifically identifying probabilities or scores for multiple different body part classifications (e.g., abdomen, skull, lower limbs, etc.)).

[0093] If the body part of interest determination logic 114 of the first-level logic 110 of the AI ​​pipeline 100 determines that the input volume 105 does not represent a part of the patient's body where an anatomical structure of interest can be found (e.g., the abdominal part of the body where the liver can be found), then the processing of the AI ​​pipeline 100 can be interrupted (rejection case). If the body part of interest determination logic 114 of the first-level logic 110 of the AI ​​pipeline 100 determines that the input volume 105 does indeed represent a part of the patient's body where an anatomical structure of interest can be found, then further processing of the input volume 105 by the AI ​​pipeline 100 is performed as described below. It should be understood that in some illustrative embodiments, multiple different instances of the AI ​​pipeline 100 may be provided, each instance being configured and trained to process the input volume 105 corresponding to different anatomical structures that may be present in different parts of the patient's body. Therefore, the first-level logic 110 can be provided externally to the AI ​​pipeline 100 and can function as routing logic to route the input volume 105 to the corresponding AI pipeline 100, which is specifically configured and trained to process input volumes 105 for specific categories. For example, one AI pipeline instance might be used for liver and liver lesion detection / classification, another for lung and lung lesion detection / classification, and a third for heart and heart lesion detection / classification, etc. Thus, the first-level logic 110 can include routing logic that stores mappings of which AI pipeline instance 100 corresponds to different body parts / anatomical structures of interest. Based on the detection of body parts represented in the input volume 105, the first-level logic 110 can automatically route the input volume 105 to the corresponding AI pipeline instance 100, which is specifically configured and trained to process input volumes 105 corresponding to detected body parts.

[0094] Assuming that input volume 105 is detected as representing a part of a patient's body containing an anatomical structure of interest (e.g., an abdominal scan is present in input volume 105 for the purpose of detecting liver lesions), further processing of input volume 105 is performed by AI pipeline 100 in second-level logic 120. This second-level logic 120 includes two main sub-levels 122 and 124, which can be executed substantially in parallel and / or sequentially, depending on the desired implementation (in...). Figure 1 (This is exemplified as parallel execution). The two main sub-levels 122 and 124 include phase classification sub-level 122 and anatomical structure detection sub-level 124 (e.g., liver detection sub-level 124 in the case where AI pipeline 100 is configured to perform liver lesion detection).

[0095] Phase classification sublevel 122 determines whether the input volume 105 includes a single imaging phase (e.g., pre-angiography phase, arterial angiography phase, portal venography phase, delayed phase, etc.). Again, phase classification sublevel 122 can be implemented as logic evaluating metadata associated with the input volume 105, which may include fields specifying phases of a medical imaging study corresponding to those phases, such as those generated by the medical imaging technology computing system 102 during medical imaging. Alternatively, illustrative embodiments may implement a configured and trained ML / DL computer model specifically trained to detect patterns in medical images indicating different phases of a medical imaging study, thereby classifying the medical images of the input volume 105 regarding which phases they correspond to. The output of phase classification sublevel 122 may be binary, indicating whether the input volume 105 includes one or multiple phases, or it may be a classification of each phase represented in the input volume 105, which can then be used to determine whether it represents a single or multiple phases.

[0096] If the phase classification indicates the presence of a single phase in input volume 105, then AI pipeline 100 performs further processing via downstream stages 130-170, as described below. If multiple phases are detected, input volume 105 is not further processed by AI pipeline 100, or, as previously described, it can be filtered and / or divided into sub-volumes, each sub-volume having an image with a corresponding single phase, such that AI pipeline 100 processes only the sub-volumes corresponding to the target phase and / or routes the sub-volumes to corresponding AI pipelines configured and trained to process input volumes of images corresponding to their specific phase classifications. It should be understood that input volumes can be rejected for several reasons (e.g., no liver in the image, no single-phase input volume in the image, insufficient liver in the image, etc.). Depending on the actual underlying reason for rejection, the reason for rejection can be communicated to the user via a user interface, etc. For example, in response to such rejection, the output of AI pipeline 100 can indicate the reason for rejection and can be used by downstream computing systems (e.g., a viewer or additional automated processing system) to communicate the reason for rejection through the output. For example, if no liver is detected in the input volume, the input volume can be silently ignored, for example, when the input volume contains a liver, the rejection is not communicated to the user. However, if a multiphase input volume is included, the rejection can be communicated to the user (e.g., a radiologist) by clearly stating in the user interface generated by the viewer's downstream computing system that the input volume is not processed by the AI ​​pipeline 100 because it contains an image with more than one phase, so as not to be misunderstood by an input volume that does not contain any detected liver.

[0097] The second primary sublevel 124 is a detection sublevel for detecting anatomical structures of interest (in an exemplary embodiment, the liver) within a portion of the input volume 105. Specifically, it identifies and evaluates slices, sections, etc., within the input volume 105 that specifically depict the anatomical structure of interest (liver) to determine whether a predetermined minimum amount of the anatomical structure of interest (liver) exists as a whole within these slices, sections, or the input volume. As previously described, the detection sublevel 124 includes an ML / DL computer model 125, which is specifically trained and configured to perform computerized medical image analysis to identify portions of the input medical image corresponding to the anatomical structure of interest (e.g., a human liver).

[0098] Thus, in the liver detection sublevel 124, the AI ​​pipeline 100 uses a trained ML / DL computer model 125 to determine whether the volume of the input CT medical image includes an image depicting the liver. The portion of the volume depicting the liver, along with the result of the phase classification sublevel 122, is passed to the determination sublevel 126 of the AI ​​pipeline 100. The determination sublevel 126 includes single-phase determination logic 127 and minimum structure quantity determination logic 128. The determination sublevel 126 determines whether a single-phase medical imaging 127 exists, and whether at least a predetermined amount of the anatomical structure of interest (e.g., liver) exists in the portion of the volume depicting the anatomical structure of interest. As previously described, the presence of the predetermined amount of the anatomical structure of interest can be determined based on known measurement mechanisms for determining structure from medical images, such as calculating the size of the structure from differences in pixel locations within the image, and comparing these measurements with one or more predetermined thresholds to determine whether a minimum amount of the anatomical structure of interest (e.g., liver) exists in the input volume 105, for example, the portion of the input volume 105 determined to depict the liver contains 1 / 3 of the liver.

[0099] In one illustrative embodiment, to determine whether a predetermined amount of anatomical structures of interest (liver) exists in input volume 105, the previously described axial scoring mechanism can be used to evaluate portions of the anatomical structures present in input volume 105. As previously stated, for input volume 105, the ML / DL computer model can be configured and trained to estimate slice scores s corresponding to slice scores of MSSV and MISV slices, respectively. sup and s inf In some illustrative embodiments, the input volume 105 can be first divided into multiple segments, and then, for each segment, a configured and trained ML / DL computer model can be executed on a slice of that segment to estimate s' of that segment. sup and s' inf The slice score s is indirectly found by using the slice scores of the first and last slices in the algorithm.sup and s inf Given s' sup and s' inf The estimate is found by extrapolation. sup and s inf The estimation is based on the known localization of the segment relative to the entire volume of the input CT medical image. It should be noted that for each input volume of 10⁵, there will be n s. sup and s inf The estimate is the sum of the n estimates, where n is the number of segments per volume. In one illustrative embodiment, the final estimate is obtained by taking the unweighted average of these n estimates; however, in other illustrative embodiments, other functions of the n estimates may be used to generate the final estimate.

[0100] The volume s of the input CT medical image has been determined. sup and S inf The final estimate is then used to calculate a score for the anatomical structure of interest (e.g., the liver) based on these values. This calculated score can then be compared with a predetermined threshold to determine whether a predetermined minimum number of the anatomical structures of interest exist in the volume of the input CT medical image, for example, at least one-third of the liver is present in the volume of the input CT medical image.

[0101] If the determinations of logics 127 and 128 indicate the presence of multiple phases and / or the absence of a predetermined amount of anatomical structures of interest in the portion of the liver depicted in input volume 105, further processing of input volume 105 by AI pipeline 100 for stages 130-170 can be interrupted (i.e., input volume 105 is rejected). If the determinations of logics 127 and 128 result in the determination that input volume 105 has an image with a single phase and depicts at least a predetermined amount of liver, the portion of input volume 105 depicting the anatomical structures is forwarded to the next stage 130 of AI pipeline 100 for processing. While the exemplary illustrative embodiment forwards a sub-portion of the input volume containing the liver for further processing, in other illustrative embodiments, context surrounding the liver may also be provided, which can be accomplished by adding a predetermined amount of margin above and below the selected liver region. This margin may be increased to completely cover the original input volume, depending on how much context is required by subsequent processing operations.

[0102] In the next stage 130 of the AI ​​pipeline 100, the AI ​​pipeline 100 performs lesion detection on portions of the input volume 105 representing anatomical structures of interest (e.g., the liver). This liver / lesion detection stage 130 of the AI ​​pipeline 100 uses an ensemble of ML / DL computer models 132-136 to detect the liver and lesions within the liver as represented in the input volume 105. The ensemble of ML / DL computer models 132-136 uses separately trained ML / DL computer models 132-136 to perform liver and lesion detection, wherein the ML / DL computer models 132-136 are trained and a loss function is used to balance false positives and false negatives in lesion detection. Furthermore, the integrated ML / DL computer models 132-136 are configured such that a third loss function makes the outputs of the ML / DL computer models 132-136 consistent with each other.

[0103] In one illustrative embodiment, a configured and trained ML / DL computer model 132 is executed on input volume 105 to detect the presence of liver. This ML / DL computer model 132 may be identical to the ML / DL computer model 125 used in a previous AI pipeline stage 120, thus leveraging previously obtained results. Multiple (two or more) other ML / DL computer models 134-136 are configured and trained to perform lesion detection in portions of the liver depicted in the medical image of input volume 105. A first ML / DL computer model 134 is configured and trained to operate directly on input volume 105 and generate lesion predictions. A second ML / DL computer model 136 is configured with two different decoders implementing two different loss functions: one penalizing errors in false negatives (i.e., misclassifying the absence of lesions (normal anatomical structures)), and the second penalizing errors in false positives (i.e., misclassifying the presence of lesions (abnormal anatomical structures)). The first decoder of ML / DL computer model 136 is trained to recognize patterns representing a relatively large number of different lesions at the cost of a large number of false positives. The second decoder of ML / DL computer model 136 is trained to be less sensitive to lesion detection, but lesions that are detected are more likely to be detected accurately. A third loss function, as part of the overall ML / DL computer model integration, compares the results of the decoders of ML / DL computer model 136 with each other and makes them consistent. The lesion prediction results of the first and second ML / DL computer models 134, 136 are combined to generate a final lesion prediction for integration, while another ML / DL computer model 132, which generates a prediction of the liver mask, provides an output representing the liver and its contours. See below. Figure 6 The example architectures of these ML / DL computer models 132-136 are described in more detail.

[0104] The results of the liver / lesion detection level 130 of the AI ​​pipeline 100 include one or more contours (shapes) of the liver and a detection map (e.g., a voxel-based map of liver lesions detected in input volume 105) identifying portions of the medical imaging data elements corresponding to the detected lesions 135. The detection map is then input to the lesion segmentation level 140 of the AI ​​pipeline 100. As will be described in more detail below, the lesion segmentation level 140 partitions the detection map using watershed techniques and a corresponding ML / DL computer model 142 to generate image element (e.g., voxel) partitions of the medical image (slice) of input volume 105. The liver lesion segmentation level 140 provides additional mechanisms (such as ML / DL computer model 144) that, based on this partitioning, identify all contours corresponding to lesions present in the slices of input volume 105 and perform the operation of identifying which contours correspond to the same lesion in three dimensions. The lesion segmentation level 140 further provides mechanisms (such as ML / DL computer model 146) that aggregate related lesion contours to generate three-dimensional partitions of the lesions. Lesion segmentation utilizes image elements representing lesions in medical images (e.g., voxels) and in-painting of non-liver tissue to focus on individual lesions and perform active contour analysis. In this way, individual lesions can be identified and processed without bias due to other lesions in the medical image or due to portions of the image outside the liver.

[0105] The result of lesion segmentation 140 is a list of lesions 148 with their corresponding shapes or contours in the input volume 105. These outputs 148 are provided to the false positive removal stage 150 of the AI ​​pipeline 100. The false positive removal stage 150 uses a configured and trained ML / DL computer model that employs a two-operation-point strategy to reduce false positive lesion detections in the lesion list generated by the lesion segmentation stage 140 of the AI ​​pipeline 100. By configuring the ML / DL computer model of the false positive removal stage 150 to remove as many lesions as possible, a first operation point is selected to be sensitive to false positives. After sensitive false positive removal, it is determined whether a predetermined number or fewer lesions should remain in the list. If so, a second operation point, which is relatively less sensitive to false positives, is used to reconsider the remaining lesions removed from the list. The results of these two methods identify a final filtered list of lesions that requires further processing by the lesion classification stage of the AI ​​pipeline.

[0106] After false positives have been removed from the lesion list and its contours generated by lesion segmentation level 140, the resulting filtered lesion list 155 is provided as input to the lesion classification level 160 of the AI ​​pipeline 100. A configured and trained ML / DL computer model is executed on the lesion list and its corresponding contour data, thereby classifying the lesions into one of a plurality of predetermined lesion classifications. For example, each lesion in the final filtered lesion list and its attributes (e.g., contour data) can be input into the trained ML / DL computer model of lesion classification level 160, which then operates on this data to classify the lesion as a specific predetermined type or category of lesion.

[0107] After lesions are classified by the lesion classification level 160 of the AI ​​pipeline 100, the AI ​​pipeline 100 generates output 170, which includes a finalized list of lesions and their classifications, as well as any contour attributes of the lesions. Furthermore, the AI ​​pipeline 100 output 170 may also include liver contour information obtained from the liver / lesion detection level 130. The output generated by the AI ​​pipeline 100 can be provided to a further downstream computing system 180 for further processing and generation of representations of the anatomy of interest and any detected lesions present within the anatomy. For example, a graphical representation of the input volume can be generated in a medical image viewer or other computer application of the downstream computing system 180, wherein the contour information generated by the AI ​​pipeline is used to overlay or otherwise highlight the anatomy and detected lesions in the graphical representation. In other illustrative embodiments, downstream processing performed by the downstream computing system 180 may include diagnostic decision support operations, automated medical imaging report generation based on the list of detected lesions, classification, and contours. In other illustrative embodiments, different treatment recommendations may be generated based on the lesion classification for review and consideration by a practicing physician. In some illustrative embodiments, the list of lesions, their classifications, and contours may be stored in a historical data structure of the downstream computing system 180 in association with a patient identifier, allowing for the storage and evaluation over time of multiple executions of the AI ​​pipeline 100 on different input volumes 105 associated with the same patient. For example, differences between the list of lesions and / or their associated classifications and contours can be determined to assess the progression of the patient's disease or medical condition, and such information can be presented to medical experts to assist in the patient's treatment. Other downstream computing systems 180 and the processing of specified anatomical structures and lesion detection information generated by the AI ​​pipeline 100 can be implemented in the illustrative embodiments without departing from the spirit and scope of the invention.

[0108] Figure 2 This is an example flowchart outlining example operations of an AI pipeline according to an illustrative embodiment. Figure 2 The operations outlined in the document can be implemented at various logical levels, including the configuration and training of ML / DL computer models, such as in... Figure 1 The specific example embodiments shown and described above with reference to those described in the following separate sections of this specification are illustrated. It should be understood that this operation is specifically for automated artificial intelligence pipelines implemented in one or more data processing systems having one or more computing devices specifically configured to implement these automated computer tool mechanisms. This is in addition to the time required for medical image volume creation and the use of output from downstream computing systems. Figure 1 and 2 The operations described herein involve no human intervention. Specifically, this invention provides improved automated artificial intelligence computing mechanisms to perform the described operations, which avoid human interaction and reduce potential errors caused by previous manual processes by providing new and improved procedures that are specifically different from any previous manual processes and specifically relate to providing logical and data structures that allow the improved artificial intelligence computing mechanisms of this invention to be implemented in automated computing tools.

[0109] like Figure 2 As shown, the operation begins by receiving an input volume of medical images from a medical imaging technology computing system (e.g., a computing system providing computed tomography (CT) medical images) (step 210). The AI ​​pipeline operates on the received input volume to perform body part detection (step 212), enabling the determination of whether a body part of interest exists in the received input volume (step 214). If no body part of interest exists in the input volume (e.g., the abdomen in the case of liver lesion detection and classification), the operation terminates. If a body part of interest exists in the input volume, phase classification and minimal anatomical structure evaluation are performed either sequentially or in parallel.

[0110] That is, such as Figure 2 As shown, phase classification is performed on the input volume (step 216) to determine whether the input volume includes a single phase (e.g., precontrast imaging, partial contrast imaging, delayed phase, etc.) or multiple phases of medical images (slices) used for medical imaging. Then, it is determined whether the phase classification indicates a single phase or multiple phases (step 218). If the input volume includes medical images pointing to multiple phases, the operation terminates; otherwise, if the input volume includes medical images pointing to a single phase, the operation continues to step 220.

[0111] In step 220, a detection of the anatomical structure of interest (e.g., the liver in the depicted example) is performed to determine if a minimum amount of anatomical structure exists in the input volume to enable accurate execution of subsequent stages of the AI ​​pipeline operation. Determining the existence of a minimum amount of anatomical structure (e.g., representing at least 1 / 3 of the liver in the input volume) (step 222). If no minimum amount exists, the operation terminates; otherwise, the operation continues to step 224.

[0112] In step 224, liver / lesion detection is performed to generate lesion contours and detection maps. These contours and detection maps are provided to lesion segmentation logic, which performs lesion segmentation based on these contours and detection maps (e.g., liver lesion segmentation in the depicted example) (step 226). Lesion segmentation results in the generation of a list of lesions and their contours, as well as detection and contour information of anatomical structures (e.g., liver) (step 228). Based on this list of lesions and their contours, a false positive removal operation is performed on the lesions in the list to remove false positives and generate a filtered list of lesions and their contours (step 230).

[0113] A list of filtered lesions and their outlines is provided to lesion classification logic, which performs lesion classification to generate a final list of lesions, their outlines, and lesion classifications (step 232). This final list, along with liver outline information, is provided to a downstream computing system (step 234), which can manipulate this information to generate medical imaging views in a medical imaging viewer application, generate treatment suggestions based on the classification of detected lesions, assess the historical progression of lesions in the same patient over time based on a comparison of the final lesion lists generated by the AI ​​pipeline at different time points, and so on.

[0114] Thus, the illustrative embodiments outlined above provide automated artificial intelligence mechanisms and ML / DL computer models that operate on input volumes of medical images and generate lists of lesions, their outlines, and classifications, while minimizing false positives. The illustrative embodiments provide automated artificial intelligence computer tools that specifically identify, within a given set of image voxels in the input volume, which voxels correspond to a portion of an anatomical structure of interest (e.g., the liver), and which of these voxels correspond to lesions within that anatomical structure of interest (e.g., liver lesions). The illustrative embodiments offer significant improvements over both manual and automated methods of the past because they can be integrated into fully automated computer tools within a clinician's workflow. In practice, the early stages of the AI ​​pipeline design based on the illustrative embodiments accept input volumes of only a single phase (e.g., abdominal scans) and reject input volumes that do not depict the anatomical structures of interest (e.g., the liver) or do not depict a predetermined number of anatomical structures of interest (e.g., too small and too large for the liver). Only meaningful input volumes are processed by the automated AI pipeline, thereby preventing radiologists from expending valuable manual resources on useless or defective results when reviewing input volumes of non-anatomical structures of interest (e.g., non-liver cases). In addition to preventing radiologists from being overwhelmed with useless information, the automated AI pipeline of the illustrative embodiments also ensures smooth IT integration by avoiding congestion of the AI ​​pipeline and downstream computing systems (such as networks), using data associated with cases that do not correspond to the anatomical structures of interest or that do not provide a sufficient number of anatomical structures of interest for archiving and reviewing the computing systems. Furthermore, as mentioned above, the automated AI pipeline of the illustrative embodiments allows for the accurate detection, measurement, and characterization of lesions in a fully automated manner, which is technically possible through one or more automated AI pipeline structures of the illustrative embodiments and their corresponding components of automated ML / DL-based computer models.

[0115] ML / DL computer model for detecting the minimum amount of anatomical structure in the input volume.

[0116] As previously described, as part of the processing of input volume 105, it is important to ensure that input volume 105 represents a single phase of medical imaging and represents at least a minimum amount of anatomical structures of interest within input volume 105. To determine the minimum amount of anatomical structures of interest present in input volume 105, in one illustrative embodiment, determination logic 128 implements a specially configured and trained ML / DL computer model that estimates slice scores for determining a portion of an anatomical structure (e.g., the liver present in input volume 105). The following description provides an example embodiment of such a configured and trained ML / DL computer model based on a defined axial scoring technique.

[0117] Figure 3A This is an example diagram illustrating an example input volume (medical image) of the abdomen of a human patient according to an illustrative embodiment. Figure 3A The description shows a two-dimensional representation of a three-dimensional volume. Slices are... Figure 3A The horizontal lines shown are within a two-dimensional representation, but will be represented as planes extending into and / or out of the page to represent flat two-dimensional slices of the human body, where the stacking of these planes results in a three-dimensional image.

[0118] like Figure 3A As shown, the illustrative embodiment defines axial scores for slices ranging from 0 to 1. Axial scores are defined such that a slice corresponding to the first slice (FSL) containing the liver has a score of 0, while the last slice (LSL) containing the liver has a score of 1. In the depicted example, the first and last slices are defined as being associated with the lowest slice (MISV) and the highest slice (MSSV) in the volume, where along a given axis of the volume (e.g., ...). Figure 3A The y-axis in the depicted example determines the lowest and highest points. Thus, in this depicted example, the MSSV is located at the highest y-axis value slice and the MISV is located at the lowest y-axis value slice. For example, the MISV may be closest to the lower limb of the biological entity (e.g., the foot of a human subject), and the MSSV may be closest to the upper part of the biological entity (e.g., the head of a human subject). The FLS is a slice depicting the anatomical structure of interest (e.g., the liver) that is relatively closest to the MISV. The LSL is a slice depicting the anatomical structure of interest that is relatively closest to the MSSV. In an illustrative embodiment, a trained ML / DL computer model (e.g., a neural network) can assign an axial score by taking slice blocks as input and outputting the height (axial score) of the center slice in the block. The trained ML / DL computer model is trained with a cost function that minimizes the error of the actual height (e.g., least squares error). The trained ML / DL computer model is then applied to all blocks covering the input volume (there may be some overlap between blocks).

[0119] Liver Axial Score (LAE) estimation is derived from a pair of slice scores. sup and s inf To define this, this is the slice score s sup and s inf The slice scores correspond to MSSV and MISV slices, respectively. Figure 1 The deterministic logic 128 ML / DL computer model was specifically configured and trained to determine the slice scores of the input volume 105. sup and s inf Furthermore, knowing these slice scores, the mechanism of the illustrative embodiment is able to determine the score of the liver in a field of view with an input volume of 105.

[0120] In some illustrative embodiments, the slice score s can be found indirectly by first dividing the input volume 105 into multiple segments (e.g., segments comprising X slices (e.g., 20 slices)). sup and s inf Then, for each segment, a configured and trained ML / DL computer model is executed on the slices of that segment to estimate the slice scores s' of the first and last slices in that segment. sup and s' inf Wherein, "first" and "last" can be determined based on the direction of advance along the axis of the three-dimensional volume 105 (e.g., advancing from the slice with the smallest y-axis value to the slice with the highest y-axis value along the y-axis from the first slice to the last slice). Given s' sup and s' inf The estimate is found by extrapolation. sup and s inf The estimation is based on the known positioning of the segments relative to the entire volume 105. It should be noted that for each volume, there will be n s. sup and s inf The estimate is the sum of the n estimates, where n is the number of segments per volume. In one illustrative embodiment, the final estimate is obtained by taking the unweighted average of these n estimates; however, in other illustrative embodiments, other functions of the n estimates may be used to produce the final estimate.

[0121] For example, Figure 3B It shows Figure 3A Another depiction of the input volume, where the slice segments along with their corresponding axial scores s' inf and s' sup Together. For example... Figure 3B As shown, in this example, the segment is defined as 20 slices spaced 5 mm apart. For each 20 slice segments of volume, the slice score s' is estimated by an ML / DL computer model. supand s' inf And from these s' along a given range (e.g., from 0 to 1, from -0.5 to 1.2, or any other desired predetermined range suitable for a particular implementation). sup and s' inf extrapolation of the value yields s sup and s inf In this example, assuming a predetermined range of -0.5 to 1.2, if s is estimated by applying an ML / DL computer model... sup And the extrapolation is approximately 1.2 and s inf It is estimated to be -0.5, indicating that the entire liver is contained within that volume. Similarly, if the estimated value is s... sup It is 1.2 and s inf If the value is 0.5, then these values ​​indicate that approximately 50% of the superior axial extension of the liver is contained within the volume (e.g., coverage of (1.2 - 0.5) / (1.2 - (-0.5)) = 0.41). As another example, in another illustrative embodiment, the liver begins at -2.0 and ends at 0.8 (i.e., s sup It was estimated to be 0.8 and s inf The upper limit of the liver is estimated to be 2.0, and the lower limit is less than 1.2, so the liver is cut in the upper part, and the lower limit is less than -0.5, so that the bottom of the liver is completely covered. This indicates that about 80% of the inferior axial extension of the liver is included in the volume (i.e., the coverage is (.8 - max(-2, -.5)) / (1.2 - (-.5)) = 0.76).

[0122] Figure 3C yes Figure 3A An example diagram of the input volume, where the volume is axially divided into n completely overlapping segments. In the depicted example, there are 7 segments indicated by the arrows. It should be noted that in this example, the last two segments (arrows at the top of the diagram) are almost identical. As mentioned before, s' sup and s' inf The values ​​are estimated by the ML / DL computer model for each of these segments and are used to extrapolate the MSSV and MISV slices. sup and s inf value, then, s sup and s inf The value can be used to determine the amount of anatomical structures of interest present in the input volume 105.

[0123] Therefore, by first dividing the input volume 105 into segments and then estimating the slice scores s' of the first and last slices in each segment. sup and s' inf To indirectly find the s of MSSV and MISVsup and s inf Values. Given these estimates, estimate s by extrapolation. sup and s inf The value of is known because it is known how the segment is positioned relative to the entire input volume 105. There are n s extrapolated from each segment. sup and s inf The estimate is given by n, where n is the number of segments per volume. The final estimate can be obtained, for example, by estimating any suitable combination function of the n estimates (e.g., the unweighted average of the n estimates or any other suitable combination function).

[0124] Figures 4A-4C An example diagram illustrating an embodiment of an illustrative ML / DL computer model is shown, which is configured and trained to estimate s' of segments of the input volume of a medical image. sup and s' inf value. Figures 4A-4C The ML / DL computer model described herein is merely an example of the architecture of an ML / DL computer model, and many modifications can be made to this architecture (such as changing the tensor size of the input slice of the input volume, changing the number of nodes in the layers of the ML / DL computer model, changing the number of layers, etc.) without departing from the spirit and scope of the invention. In view of this specification, those skilled in the art will recognize how the ML / DL computer model of the illustrative embodiments can be modified to achieve desired implementations.

[0125] like Figure 4A As shown, a sequence of 20 slices representing a segment 410 or "thick slice" of input volume 105 is set as the input to processing blocks (PB) 420-430. In the illustrated embodiment, PB 420-430 is a logical block of hybrid convolutional and LSTM layers (e.g., ...). Figure 4B and 4C(As shown). Features are extracted from the convolutional layers of PB 420 and 430, and then these features are fed as input to the LSTM layers of PB 420 and 430. This is a smart / lightweight modeling of the fact that slices have a specific order in anatomical regions or structures of interest driven by anatomical structures (e.g., abdomen / liver) (e.g., the relative positions of the liver, kidneys, heart, etc., in addition to the liver anatomy itself). In the depicted example, the tensor size of the initial 20 slices 410 is 128x128. In this example embodiment, the first processing block 420 reduces the tensor size by 8 to generate 20 slice segments with a size of 16x16x32 (it should be understood that the number of slices in the segments is specific to the implementation and can be modified without departing from the spirit and scope of the invention), where 32 is the number of filters. The second processing block 430 transforms the input segment slices into 20 slice segments with a dimension of 2x2x64, where 64 is the number of filters. A subsequent neural network 440, configured with flat, compact, and linear layers, is configured and trained to generate s' for the input segment 410 with an input volume of 105. sup and s' inf estimate. Figure 4B The diagram illustrates the composition of a processing block (PB) according to an illustrative embodiment regarding convolutional and LSTM layers, and... Figure 4C An example configuration of each of these convolutional layers and LSTM layers for each PB according to an illustrative embodiment is shown.

[0126] by Figures 4A-4C Taking an ML / DL computer model architecture as an example, during the training of this ML / DL computer model, in an illustrative embodiment, medical imaging data (e.g., medical digital imaging and communication (DICOM) data) is assembled into, for example, a system with S... i The input volume of a 3D array of size x512x512 contains floating-point numbers of 32 and has a Hosfield unit (HU) value, which is a normalized physical value that describes the attenuation of X-rays from the material at a given location (e.g., a voxel). iis the number of slices in the i-th volume, where i ranges from 0 to N-1, and N is the total number of volumes. Each input volume is processed by a body part detector, and an approximate region corresponding to the abdomen (in the case of liver detection) is extracted as described above. The abdomen is defined, for example, as a continuous region between axial scores of -30 and 23 from the body part detector. Slices outside this continuous region are rejected, and the underlying truth can be defined as the location of the appropriately adjusted FSL and LSL. For example, assuming the input volume ranges from a to b, the input volume is rejected if there is no overlap between [a:b] and [-30, 23]. In other words, if b > 23 or if a < -30.

[0127] Input segment 410, or "thick slice," is overlapped at a predetermined slice interval (e.g., 5 mm). Input segment 410 is then reshaped into 128x128 slices in the x and y dimensions, resulting in N shapes of shape M. i The x128x128 segment is 410. This is referred to as downsampling of the data in the input volume. Since the sorting of slices within the input volume depends on coarse information (e.g., organ size), the AI ​​pipeline still operates well on downsampled data, and both the processing and training time of the AI ​​pipeline are improved due to the reduction in the size of the downsampled data.

[0128] Input segments 410 with fewer than a predetermined number of slices (e.g., 20) or smaller than a predetermined pixel size (e.g., 55 mm) are rejected, thereby generating N' M... i x128x128 segments. Values ​​within the segments are clipped and normalized using a linear transformation from their range (e.g., -1024, 2048) to the range (0, 1). At this point, as described above, the processed N' M... i The x128x128 segment constitutes the training set, on which the neural network 440 is trained to generate s' for the input segment. sup and s' inf The estimate.

[0129] Regarding the inference performed using the trained neural network 440, the above operations are used to process the input volume 105 through body part detection, corresponding to slice selection of the body part of interest, re-slicing, reshaping, rejection of certain slices that do not meet predetermined requirements, and the generation of slicing and normalizing segments for new segments of the input volume 105. After generating slicing and normalizing segments, the input volume 105 is divided into R-ceil(M-10) / 10 sub-volumes or segments containing 20 slices, thereby generating partitions of slices with overlapping blocks. For example, if there are N' = 31 slice volumes (slice numbers 0-30), then there are three segments or sub-volumes containing the following overlapping slices: 0-19, 10-29, and 11-30. These segments or sub-volumes will typically have at least 50% overlap.

[0130] Thus, an ML / DL computer model is provided, configured, and trained to achieve a given axial score range from 0 to 1 based on the volume of a segment corresponding to a predetermined number of slices (medical images). sup and s' inf Estimation of the value, estimating the input volume s sup and s inf Values. Based on these estimates, it can be determined whether the input volume includes medical slices that collectively constitute at least a predetermined amount of the anatomical structure of interest (e.g., liver). As discussed earlier, this determination can be part of determination logic 128 of the AI ​​pipeline 100, which is used to determine whether there are anatomical structures sufficiently represented in the input volume 105 to allow for accurate liver / lesion detection, lesion segmentation, etc., in further downstream stages of the AI ​​pipeline 100.

[0131] Figure 5 This is a flowchart outlining example operations of the liver detection and predetermined anatomical structure determination logic of an AI pipeline according to an illustrative embodiment. Figure 5As shown, the liver detection operation of the AI ​​pipeline begins by receiving an input volume (step 510) and dividing the input volume into multiple overlapping segments for a predetermined number of slices for each segment (step 520). The slices of each segment are fed into a trained ML / DL computer model, which estimates the axial scores of the first and last slices in each segment (step 530). The axial scores of the first and last slices are used to extrapolate the scores of the lowest slice in the volume (MISV) and the highest slice in the volume (MSSV) of the input volume (step 540). This results in multiple estimates of the axial scores of the MISV and MSSV, which are then combined by a function of the respective estimates to generate estimates of the axial scores of the MISV and MSSV of the input volume (e.g., a weighted average, etc.) (step 550). Based on the estimates of the axial scores of the MISV and MSSV, the axial scores are compared with criteria used to determine the presence of a predetermined number of anatomical structures of interest (e.g., the liver) in the input volume (step 560). The operation then terminates.

[0132] Liver / lesion detection

[0133] As previously described, assuming that input volume 105 is determined to have a single phase and that input volume 105 has a predetermined number of anatomical structures of interest represented in slices of input volume 105, liver / lesion detection is performed on portions of input volume 105 that include the anatomical structures of interest. In one illustrative embodiment, the liver / lesion detection logic stage 130 of the AI ​​pipeline 100 employs a configured and trained ML / DL computer model that operates to detect anatomical structures of interest (e.g., the liver) in slices of input volume 105 (again, in some illustrative embodiments, this may be the same ML / DL computer model 125 used in stage 120 for liver detection). The liver / lesion detection logic stage 130 of the AI ​​pipeline 100 also includes the integration of multiple other configured and trained ML / DL computer models to detect lesions in images of the anatomical structures of interest (liver).

[0134] Figure 6This is an example diagram of an integration of ML / DL computer models for performing lesion detection in an anatomical structure of interest (e.g., liver) according to an illustrative embodiment. The integration of ML / DL computer models 600 includes a first ML / DL computer model 610 for detecting the anatomical structure of interest (e.g., liver) and generating a corresponding mask. The integration of ML / DL computer models 600 also includes a second ML / DL computer model 620 configured and trained to process liver masking input implemented in two decoders of the second ML / DL computer model 620 and generate lesion predictions using two competing loss functions. One loss function is configured to penalize false positive errors (producing low sensitivity but high accuracy), and another loss function is configured to penalize false negative errors (producing high sensitivity but lower accuracy). An additional loss function is employed (in... Figure 6 A consistency loss (627) is used in the second ML / DL computer model 620 to make the outputs generated by the two competing decoders similar to each other (consistent). The integration of the ML / DL computer models further includes a third ML / DL computer model 630, which is configured and trained to directly process the input volume 105 and generate lesion predictions.

[0135] like Figure 6 As shown and as described above, integration 600 includes a first configured and trained ML / DL computer model 610, specifically configured and trained to identify anatomical structures of interest in input medical images. In some illustrative embodiments, the first ML / DL computer model 610 includes a U-Net neural network model 612 configured and trained to perform image analysis to detect the liver within a medical image. However, it should be understood that illustrative embodiments are not limited to this particular neural network model, and any ML / DL computer model capable of performing segmentation can be utilized without departing from the spirit and scope of the invention. U-Net is a convolutional neural network developed for biomedical image segmentation at the Computer Science Department of the University of Freiburg, Germany. The U-Net neural network is based on a fully convolutional network with an architecture modified and extended to work with fewer training images and produce more accurate segmentation. U-Net is generally known in the art, and therefore will not be explained in more detail here.

[0136] like Figure 6As shown, in one illustrative embodiment, a first ML / DL computer model 610 may be trained to process a predetermined number of slices at a time, wherein this number is determined to be suitable for the desired implementation (e.g., 3 slices are determined through an empirical process to produce good results). In one illustrative embodiment, the slices of the input volume are, for example, 512x512 pixel medical images, but other implementations may use different slice sizes without departing from the spirit and scope of the illustrative embodiment. U-Net generates segments of anatomical structures in the input slices, thereby producing one or more segments corresponding to the anatomical structure of interest (e.g., the liver). As part of these segments, the first ML / DL computer model 610 generates segments representing a liver mask 614. This liver mask 614 is provided as input to at least one of the other ML / DL computer models 620 of the integration 600 so that the processing performed by the ML / DL computer model 620 is focused only on the portion of the input slices of the input volume 105 corresponding to the liver. By preprocessing the input of the ML / DL computer model with liver mask 614, the processing performed by the ML / DL computer model can be focused on the portion of the input slice corresponding to the anatomical structure of interest, rather than on the "noise" in the input image. Other ML / DL computer models (e.g., ML / DL computer model 630) directly receive the input volume 105 without needing to perform liver masking using the liver mask 614 generated by the first ML / DL computer model 610.

[0137] In the illustrated embodiment of the integrated 600, the third ML / DL computer model 630 comprises encoder sections 634-636 and decoder section 638. The ML / DL computer model 630 is configured to receive nine thick slices of input volume 105, which are then separated into groups 631-633, each having three slices. Each group 631-633 is input to a corresponding encoder network 634-636. Each encoder 634-636 is a convolutional neural network (CNN) (such as DenseNet-121 (D121)), which, without a fully connected head, has been pre-trained to identify different types of objects (e.g., lesions) present in the input slices and output a classification output (e.g., as an output classification vector, etc.) indicating the types of detected objects present in the input slices. CNNs 634-636 can, for example, operate on 3 channels of the input slice, and the output features of CNNs 634-636 are fed into a cascaded NHWC logic 637, where NHWC refers to the number of images (N), the height (H), the width (W), and the number of channels (C) in the batch. The architecture of the original DenseNet network consists of many convolutional layers and skip true connections that downsample the full-resolution input of the 3-slice to many feature channels with a smaller resolution. From there, a fully connected head aggregates all features and maps them to multiple categories in the final output of DenseNet. Because the DenseNet network is used as the encoder in the depicted architecture, the head is removed and only the downsampled features are preserved. Then, in the cascaded NHWC logic 637, all feature channels are cascaded to pass them to the decoder stage 638, which has the function of upsampling the image until the desired (e.g., 512x512) output probability map resolution is reached.

[0138] Encoders 634-636 share the same parameters optimized through the training process (e.g., weights, sampling of lesion types during training, weights of the loss, type of enhancement, etc.). The ML / DL computer model 630 is trained using two different loss functions. The primary loss function is an adaptive loss, specifically configured to penalize false positive errors in slices without lesions in the ground truth, and also penalize false negative errors in slices with lesions in the ground truth. The loss function is a modified version of the Tversky loss as follows:

[0139] For each output slice:

[0140] TP = sum(prediction * target)

[0141] FP = sum((1 - target) * prediction)

[0142] FN = sum((1 - prediction) * target)

[0143] LOSS=1-((TP+1) / (TP+1+α*FN+β*FP))

[0144] Wherein, "prediction" is the output probability of the ML / DL computer model 630 and "target" is the base ground truth lesion mask. The output probability values ​​range between 0 and 1. For each pixel in a slice, the target has either 0 or 1. For slices in which there are no lesions, the "α" term is small (e.g., zero) and "β" is large (e.g., 10). For slices in which there are lesions, "α" is large (e.g., 10) and "β" is small (e.g., 1).

[0145] The second loss function 639 is a function connected to the output of encoders 634-636. Because the input to this loss comes from the middle of the ML / DL computer model 630, it is called “deep supervision” 639. Deep supervision has been shown to enable encoder neural networks 634-636 to learn a better representation of the input data during training. In one illustrative embodiment, this second loss is a simple mean squared error predicting whether a lesion is present in a slice. Therefore, a mapping network is used to map the output features of encoders 634-636 to nine values ​​between 0 and 1, representing the probability of having a lesion in each of the nine slice inputs. Decoder 638 generates an output map of the probability of lesions detected in the specified input image.

[0146] The second ML / DL computer model 620 receives a preprocessed input of three slices from the input volume, which has been preprocessed with a liver mask 614 generated by the first ML / DL computer model 610 to identify portions corresponding to the three slices of the liver mask 614. The resulting preprocessed input slices (with a size of 192x192x3 in the illustrated example embodiment) are provided to the second ML / DL computer model 620, which includes a DenseNet-169 (D169) encoder 621 connected to two decoders (2D DEC—indicating that the decoder consists of 2D neural network layers). The D169 encoder 621 is a neural network feature extractor widely used in computer vision applications. It consists of a series of convolutional layers, where features extracted from each layer are fed forward to any other layer. The features extracted in the encoder 621 are fed to two independent decoders 622, 623, each of which consists of two-dimensional convolutional layers and upsampling layers (in... Figure 6This is referred to as a 2D DEC. Each decoder 622, 623 is trained to detect lesions (e.g., liver lesions) in the input slices. As discussed above and below, although the two decoders 622, 623 are trained to perform the same task (i.e., lesion detection), the key difference in their training lies in the fact that the two decoders 622, 623 each utilize different loss functions to drive the detection training in two competing directions. The final detection map of the second ML / DL model 620 is combined with the final detection map of the third ML / DL model 630 through an averaging operation 640. This process is applied to all input slices of input volume 105 to generate the final detection map (e.g., liver lesions).

[0147] As described above, two different loss functions are used to train the second ML / DL computer model 620, which attempt to achieve opposite detection operation point performance. Specifically, in the case where one of the encoders 622 uses a loss function for training that penalizes errors in false-negative lesion detection, resulting in high-sensitivity detection with relatively low accuracy, the other encoders in encoder 623 use a loss function for training that penalizes errors in false-positive lesion detection, leading to low-sensitivity detection but high accuracy. An example of these loss functions could be the Focal Tversky loss (see Abraham et al., “A Novel Focal Tversky Lossfunction with Improved Attention U-Net for Lesion Segmentation”, arXiv: 1810.07842[cs], October 2018), where parameters are adjusted for high or low penalties for false positives and false negatives according to the illustrative embodiment. A third loss function (consistency loss 627) is used to enforce consistency between the predicted detections of each decoder 622, 623. The consistency loss logic 627 compares the outputs 624 and 625 of the two encoders 622 and 623 and makes these outputs similar to each other. This loss can be, for example, the mean squared error loss between the two predicted detections, the structural similarity loss, or any other loss that enforces consistency / similarity between the compared predicted detections.

[0148] During runtime, using these relative operation point encoders 622 and 623, the second ML / DL computer model 620 generates two lesion outputs 624 and 625, which are input into the slice averaging (SLC AVG) logic 623 that generates the average of the lesion outputs. This average of the lesion outputs is then resampled to generate an output that is dimensionally equivalent to the output of the third ML / DL computer model 630 for comparison (note that this process includes restoring the liver masking operation and therefore calculating the lesion output at the original 512x512x3 resolution).

[0149] At runtime, Slice-Like Average (SLC AVG) logic 626 operates on the lesion prediction outputs 624 and 625 of encoders 622 and 623 to generate the final detection map of ML / DL model 620. It should be understood that although a consistency loss 627 is applied during training to drive each decoder 622 and 623 to learn consistent detections, this consistency loss is no longer utilized at runtime; instead, the outputs of ML / DL model 620 are aggregated by the two detection maps by SLC AVG module 626. The result of SLC AVG logic 626 is resampled to generate an output with dimensions commensurate with the input slice (512x512x3). Volume-Like Average (VOLAVG) logic 640 combines all generated detections from ML / DL model 620 for each slice of input volume 105 with those generated from ML / DL model 630. This logic calculates the average of the two detection masks at the voxel level. The result is the final lesion mask 650 corresponding to the lesions detected in input volume 105.

[0150] Thus, after training ML / DL computer models 620 and 630, when a new slice of a new input volume 105 is presented, the first ML / DL computer model 610 generates a liver mask 614 for preprocessing the input of the second ML / DL computer model 620, and the two ML / DL computer models 620 and 630 process the input slice to generate a lesion prediction averaged over volume by volume averaging logic 640. The result is a final lesion output 650 and a liver mask output 660 based on the operation of the first ML / DL computer model 610. These outputs can be provided as the output of the liver / lesion detection logic level 130 of the AI ​​pipeline 100, which is provided to the lesion segmentation logic level 140 of the AI ​​pipeline 100, as discussed above and described in more detail below. Thus, the mechanism of the illustrative embodiment provides an integrated method 600 for anatomical structure recognition and lesion detection in an input volume 105 of a medical image (slice).

[0151] Through such Figure 6The integrated architecture shown achieves improved performance compared to using a single ML / DL computer model. Specifically, it has been observed that by using an integrated architecture and combining the detection outputs of multiple integrated ML / DL computer models, improved detection specificity with the same level of sensitivity as a single ML / DL computer model can be achieved. In other words, when ML / DL models 620 and 630 generate errors (false positives) at different locations, averaging the detection outputs across these different locations reduces the signal from false positives and allows the signal from true positive lesions to dominate, resulting in improved performance.

[0152] Figure 7 This is a flowchart outlining example operations of liver / lesion detection logic in an AI pipeline according to an illustrative embodiment. Figure 7 As shown, the operation begins by receiving an input volume (step 710) and performing anatomical structure detection (e.g., liver detection) using a first trained ML / DL computer model (such as a U-Net computer model configured and trained to recognize anatomical structures (e.g., liver)) (step 720). The result of the anatomical structure detection is a segmentation of the input volume to identify a mask of the anatomical structure (e.g., a liver mask) (step 730). The input volume is also processed via an integrated first trained ML / DL computer model, which is specifically configured and trained to perform lesion detection (step 740). The first trained ML / DL computer model generates a first set of lesion detection prediction outputs based on its processing of the input volume (step 750).

[0153] The integrated second-trained ML / DL computer model receives a masked input generated by applying the generated anatomical structure mask to the input volume, and thereby identifies portions of the medical image in the input volume corresponding to the anatomical structure of interest (step 760). The second-trained ML / DL computer model processes the masked input via two different decoders with two different and competing loss functions (e.g., one loss function penalizes errors in false-positive lesion detection, while the other penalty for errors in false-negative lesion detection) (step 770). The result is two sets of lesion prediction outputs, which are then combined by combinational logic to generate the lesion prediction output of the second ML / DL computer model (step 780). If necessary, the second lesion prediction output is resampled and combined with the first lesion prediction output generated by the integrated first ML / DL computer model to generate the final lesion prediction output (step 790). The final lesion prediction output is then output along with the anatomical structure mask (step 795), and the operation terminates.

[0154] lesion segmentation

[0155] As previously mentioned, lesion prediction outputs are generated through the operation of various ML / DL computer models and levels within the AI ​​pipeline logic, which includes body part detection, identification of body parts of interest, phase classification, identification of anatomical structures of interest, and anatomical structure / lesion detection. For example, in Figure 1 In the AI ​​pipeline 100 shown, the results of the liver / lesion detection level 130 of the AI ​​pipeline 100 include one or more contours (shapes) of the liver, and a detection map that identifies portions of medical imaging data elements corresponding to detected lesions 135 (e.g., a voxel-based map of liver lesions detected in input volume 105). The detection map is then input to the lesion segmentation level 140 of the AI ​​pipeline 100.

[0156] As mentioned earlier, lesion segmentation logic (e.g.) Figure 1 The lesion segmentation level (140) uses watershed techniques and corresponding ML / DL computer models to partition the detection map to generate image element partitions of the input volume of a medical image (slice). The liver lesion segmentation level also provides other mechanisms (such as one or more additional ML / DL computer models) that identify all contours corresponding to lesions present in the slice of the input volume based on image element partitioning, and perform the operation of identifying which contours correspond to the same lesion in three dimensions. The lesion segmentation level further provides multiple mechanisms (such as one or more additional ML / DL computer models) that aggregate relevant lesion contours to generate three-dimensional partitions of the lesion.

[0157] Lesion segmentation utilizes image elements of the lesion represented in the medical image and in-painting of non-liver tissue to focus on each lesion individually and perform active contour analysis. In this way, individual lesions can be identified and processed without bias due to other lesions in the medical image or due to portions of the image outside the liver. The result of lesion segmentation is a list of lesions with their corresponding shapes or contours within the input volume.

[0158] Figure 8 A block diagram depicts an overview of various aspects of a lesion segmentation process performed by lesion segmentation logic according to an illustrative embodiment. (See diagram.) Figure 8 The depicted lesion segmentation includes a mechanism for slice-based segmentation of two-dimensional detection (i.e., detection of lesions in two-dimensional slices) (box 810), connecting two-dimensional lesions along the z-axis (box 820), and slice-based refinement of the contour (box 830). Each of these boxes will be described in more detail below with reference to the following figures. Figure 8The segmentation process shown is implemented as a process for identifying all lesions in a given input volume during analysis and distinguishing lesions that are close to each other in an image (slice) of the input volume. For example, for the purpose of performing other downstream processing on detected lesions (such as during lesion classification), and for lesions in the output of a separately identified lesion list for use in downstream computational system operations (such as providing medical viewing applications, performing treatment recommendation operations, performing decision support operations, etc.), two lesions that appear to be merged in terms of pixels in one or more images may need to be identified as two different regions or different lesions.

[0159] As part of the sliced ​​2D image in frame 810, the mechanism of the illustrative embodiment uses existing watershed techniques to separate detection maps from previous lesion detection levels in the AI ​​pipeline (e.g., by...). Figure 1 The detection map (135) generated by the liver / lesion detection logic 130 of the AI ​​pipeline 100 is partitioned. The watershed algorithm requires defining seeds to perform mask partitioning. The watershed algorithm splits the mask into as many regions as there are seeds, such that each region has exactly one seed located approximately at its center, as shown below. Figure 10A and 10C As shown. In automatic segmentation, seeds within the mask can be obtained as local maxima of their distance maps (distances to the mask contour). However, this method is prone to noise and may result in too many seeds, leading to over-splitting of the mask. Therefore, we need to edit the partitions by reorganizing some regions of the region. Considering the empirical observation that most lesions are bubble-shaped, the guiding principle for region reorganization is to make the resulting new region approximately circular. For example, for Figure 10C The mask shown, this mechanism will merge the two regions identified by seeds 1051 and 1061 respectively, resulting in a new mask partition comprising only two roughly circular regions. Thus, for the lesions detected as defined in Figure 135, as in... Figure 9 The lesion shown on the left (described thereafter) can be divided into several bubble-like lesions, as in Figure 9 As shown on the right. They will be interpreted as cross-sections of 3D lesions on the slices.

[0160] Watershed segmentation is a region-based method originating from mathematical morphology. In watershed segmentation, an image is viewed as a topographical landscape with ridges and valleys. The elevation values ​​of the landscape are typically defined by the grayscale value of the corresponding pixel or its gradient magnitude, thus treating a two-dimensional image as a three-dimensional representation. The watershed transform decomposes the image into "catchment basins." For each local minimum, the catchment basin includes all points whose steepest descent path terminates at this minimum. The watershed separates the basins from each other. The watershed transform completely decomposes the image and assigns each pixel to a region or watershed.

[0161] Watershed segmentation requires selecting at least one label (referred to as a "seed" point) within each object in the image. Seed points can be selected manually. In one embodiment, seed points are selected through an automated process that takes into account application-specific knowledge of the object. Once the object is labeled, it can be grown using a morphological watershed transformation, which will be described in further detail below. Lesions typically have a "bubble" shape. Illustrative embodiments provide techniques for merging segmented regions of the watershed based on this assumption.

[0162] Subsequently, in block 820, the mechanism of the illustrative embodiment aggregates voxel partitions along the z-direction on each slice to produce a 3D output. Therefore, this mechanism must determine whether two sets of image elements (e.g., voxels) in different slices belong to the same lesion (i.e., whether they are aligned in 3D). This mechanism calculates measurements between lesions in adjacent slices based on the intersection and union of lesions, and applies a regression model to determine whether two lesions in adjacent slices are part of the same region. Each lesion can be considered as a set of voxels, and the mechanism determines the intersection of two lesions as the intersection of two sets of voxels, and the union of two lesions as the union of two sets of voxels.

[0163] This results in three-dimensional partitioning of the lesion; however, the contours may not fit the actual image well. Over-segmented lesions may exist. The illustrative embodiment proposes the use of active contours, which is a traditional framework for handling segmentation problems. This algorithm attempts to iteratively edit the contours to fit the image data better and better, while ensuring that they retain certain desired properties (such as shape smoothness). In block 830, the mechanism of the illustrative embodiment uses partitions obtained from the first level 810 and the second level 820 to initialize the active contours and focuses on one lesion at a time; otherwise, running active contours or random segmentation methods on adjacent lesions might cause them to merge into a single contour again, which is counterproductive because it essentially eliminates the benefits gained from the previous level of partitions. The mechanism focuses on one lesion and performs "patchwork" on lesion voxels and non-hepatic tissue in the vicinity of the focused lesion.

[0164] The linking of these three processing levels allows for processing that is not biased against other lesions in the image or pixels or lesions outside the liver.

[0165] Slice-based 2D inspection

[0166] Figure 9 The results of lesion detection and slice-based partitioning according to an illustrative embodiment are depicted. For example... Figure 9 As seen on the left, lesion area 910 was detected through the aforementioned AI pipeline process and can be obtained from the lesion detection logic (e.g., Figure 1 130 in the contour and detection map (e.g., Figure 1 The output of 135 is limited. For example... Figure 9 As shown on the right, according to an illustrative embodiment, Figure 8 The logic of block 810 in the diagram attempts to partition the region into three lesions 911, 912, and 913. The partitioning mechanism of the illustrative embodiment is based on existing watershed techniques, which operate to partition the detection map from previous lesion detection stages in an AI pipeline. Watershed algorithms are primarily used for image processing for segmentation purposes. The principle behind these known watershed algorithms is that a grayscale image can be viewed as a topographical surface, where high intensity represents peaks and hills, and low intensity represents valleys. The watershed technique begins by filling each isolated valley (local minimum) with water of different colors (markers). As the water rises, depending on the nearby peaks (gradients), water from different valleys of different colors will begin to merge. To avoid this, barriers are built at the locations where the water merges. The work of filling water and building barriers continues until all peaks are underwater, at which point the created barriers give the segmentation result. Again, watershed techniques are generally known, and therefore, a more detailed description is not provided herein. Any known technique for slicing and partitioning 2D images may be used without departing from the spirit and scope of the invention.

[0167] In the context of lesion segmentation, the empirical observation that most lesions are circular in shape strongly suggests that a segmentation that produces a set of circular regions is likely to be a good segmentation. However, as mentioned earlier, the quality of a watershed-type segmentation depends on the quality of the seed. In fact, an arbitrary set of seeds does not necessarily need to result in a set of circular regions. For example, Figure 10C The diagram illustrates a watershed partition caused by three seeds containing only one roughly circular region. The other two are not circular. However, their union is again roughly circular. This configuration is called over-splitting because the slanted split in the diagram divides the additional circular region into two smaller, non-circular regions. Therefore, an algorithm capable of correcting over-splitting is desired. The seed relabeling mechanism achieves this by merging several over-splitned regions to form a coarser partition containing only circular regions. For example, this mechanism targets… Figure 10C The partitioning decision merges the two regions identified by seeds 1051 and 1061 to form a new, more circular region.

[0168] The illustrative embodiment merges regions within a partition into a larger, more circular region that corresponds to a physical lesion. Partitioning divides a region into smaller regions, or, as described herein, partitioning divides a mask into smaller regions. In terms of contour, partitioning thus produces a set of smaller contours from a larger contour (see...). Figure 9 (From left to right)

[0169] The seed is obtained by extracting local maxima from a distance map computed from the input mask to the partitions. This map measures the Euclidean distance of each pixel to the mask contour. Depending on the topology of the input mask, the local maxima derived from this distance map can lead to over-segmented partitions using a watershed algorithm. In this case, the watershed is said to be over-segmented and tends to produce non-circular regions, which may be desirable in some applications but is not ideal for lesion segmentation. Figure 10C In the diagram, we show the synthetic input mask whose distance map has three local maxima. Thus, the watershed produces a partition containing three regions, only one of which is roughly circular (corresponding to seed 1071). The other two are not. The region with seed 1051 is only semi-circular. The seed relabeling mechanism then examines all seed pairs and determines that the two regions corresponding to seeds 1051 and 1061 should be merged together, forming a more perfect bubble. This operation results in a new partition containing only two regions, both roughly circular in shape.

[0170] A local maximum is a point that has the maximum distance from the contour compared to its immediate neighbors. Since a local maximum is a point and its distance to the contour is known, the mechanism of the illustrative embodiment can draw a circle centered on this point. The radius of the circle is the distance. For two local maximums, the mechanism can thus calculate the overlap of their corresponding circles. This is in... Figure 10A and Figure 10B Described in the text.

[0171] Seed relabeling determines whether to merge two regions as follows: If two regions whose associated seeds are directly adjacent, merging will occur; otherwise, the mechanism bases its decision on a hypothesis testing procedure. See, for example, [link to relevant documentation]. Figure 10A The illustrated example describes a case where the distance map produces two distinct local maxima, leading to the assumption that each maxima represents the center of a different circular lesion. Note that the distance map also allows the mechanism of the illustrative embodiment to inform how far the maxima are from the contour (boundary). This distance... Figure 10BThe outline is represented by a dashed line segment connecting the maximum value and the point on the contour. Therefore, if the assumption holds, the spatial extent of the two lesions can be inferred from the assumption that the lesions are approximately circular or "bubble-like". This allows the mechanism of the illustrative embodiment to be drawn as follows: Figure 10B The diagram shows two complete circles. The mechanism then measures the overlap of the two circles (e.g., using the classic Dice metric) and compares it to a predetermined threshold. If the overlap metric is greater than the threshold, the mechanism infers that the two bubbles overlap too much to be noticeable, and merging will occur. In other words, the mechanism of this illustrative embodiment then infers that the two local maxima correspond to two “centers” of the same lesion. However, in conventional watersheds, there is no such seed (i.e., maximum value) relabeling mechanism. Therefore, mask over-splitting often occurs.

[0172] Overlap can be measured in several ways. In one example embodiment, the mechanism uses the Dice coefficient. For example... Figure 10B The diagram shows two complete circles corresponding to two local maxima. This mechanism can calculate the Dice metric for these two circles. In this way, the mechanism can learn from the training dataset what optimal threshold to apply in practice, such that once the Dice metric exceeds that threshold, the two local maxima are actually the center of the same lesion.

[0173] Figure 10C and Figure 10D Another example of a lesion mask shape is provided, which is similar to Figure 10A and Figure 10B The difference lies in the circle formed by merging the two parts. Figure 10A China and Belgium in Figure 10C They are closer to each other. Because distance maps can be very sensitive to mask shape, in Figure 10C The example lesion mask shape contains three seeds. Following the above reasoning, the lesion splitting algorithm will... Figure 10C The lesion represented in the diagram splits into two separate lesions, but not into three separate lesions as would occur in the watershed technique without seed relabeling.

[0174] exist Figure 10C and Figure 10D In the text, seeds 1051 and 1061 represent ratios... Figure 10A and Figure 10BThe seeds described herein represent a more extreme case. Without the seed relabeling technique of the illustrative embodiment, a split (represented by slanted solid lines) would occur to separate them. However, using the seed relabeling mechanism of the illustrative embodiment, this undesirable result can be effectively avoided. Instead, since seed 1071 is far enough away from seeds 1051 and 1061, the same hypothesis testing process described above will help accept the hypothesis that seed 1071 corresponds to the center of a different bubble, thus leading to... Figure 10C and Figure 10D The vertical split is shown. Equivalently, this results in a different tag for seed 1071 than those assigned to seeds 1051 and 1061. However, similar to... Figure 10A and Figure 10B In the case described in the illustrative embodiment, the hypothesis testing procedure for the seed relabeling technique will determine that seeds 1051 and 1061 correspond to the same lesion.

[0175] Figure 11A This is a block diagram illustrating a mechanism for lesion splitting and relabeling according to an illustrative embodiment. Figure 11A As shown, the mechanism can be implemented as a computer model including one or more algorithms, machine learning computer models, etc., executed by one or more processors of one or more computing devices, and the one or more processors operate on the input volume of one or more medical image data structures, receiving a two-dimensional lesion mask 1101 and performing a distance transformation (box 1102) to generate a distance map 1111. The distance transformation (box 1102) is an operation performed on a binary mask that calculates the shortest distance to the mask contour (boundary) for each point in the lesion mask. The more points move towards the interior of the lesion mask, the further away from its contour (boundary). Thus, the distance transformation identifies the center point of the lesion mask (i.e., those points with a greater distance than other points). In one embodiment, the mechanism optionally performs Gaussian smoothing on the distance map 1111.

[0176] The mechanism then performs local maximum identification (box 1103) to generate a seed 1112. As mentioned above, these local maxima are points with the highest distance from the contour or boundary in Figure 1111. Based on the seed 1112, the mechanism performs a watershed technique (box 1104) to generate a watershed split lesion mask 1113. As mentioned above, this split lesion mask 1113 can be over-split, resulting in regions that do not conform to the assumed bubble shape of the lesion. Therefore, the mechanism performs seed relabeling (box 1120) based on the distances to Figure 1111, the seed 1112, and the split 2D lesion mask 1113 to generate an updated split lesion mask 1121. Seed relabeling is described below. Figure 11BFurther detailed description. The resulting updated split lesion mask 1121 will have regions that have been merged to form areas that more accurately conform to the bubble shape assumed for the lesion.

[0177] Figure 11B This is a block diagram illustrating a mechanism for seed relabeling according to an illustrative embodiment. Figure 11B As shown, the mechanism receives distance map 1111 and seeds 1112. This mechanism can be implemented as a computer model including one or more algorithms, a machine learning computer model, etc., executed by one or more processors of one or more computing devices and operating on the input volume of one or more medical image data structures. More specifically, the mechanism considers each pair of seeds (seed A and seed B) in seed 1112. The mechanism determines whether seed A and seed B are direct neighbors (box 1151). If seed A and seed B are direct neighbors, the mechanism assigns the same label to seed A and seed B (box 1155). In other words, seeds A and seed B are grouped to represent a single region.

[0178] As described below, in block 1151, if seed A and seed B are not direct neighbors, the mechanism performs spatial extent estimation (1152) based on distance map 1111 and determines the pairwise affinity of seed A and seed B. According to the illustrative embodiment, the spatial extent estimation assumes that the region is "bubble" shaped. Thus, the mechanism assumes that each seed represents a circle with a radius equal to the distance from the distance map.

[0179] The mechanism then calculates the overlap metric of the circles represented by seed A and seed B (box 1153). In one example embodiment, the mechanism uses the following dice metric:

[0180]

[0181] Where |A| represents the area of ​​the circle represented by seed A, and |B| represents the area of ​​the circle represented by seed B. Similarly, |A∩B| represents the area of ​​the intersection of A and B. In an alternative embodiment, this mechanism can calculate the overlap metric as follows:

[0182]

[0183] Where |A| represents the area of ​​the circle represented by seed A, |B| represents the area of ​​the circle represented by seed B, |A∩B| represents the area of ​​the intersection of A and B, and |A∪B| represents the area of ​​the union of A and B.

[0184] The mechanism determines whether the overlap metric is greater than a predetermined threshold (box 1154). In block 1154, if the overlap metric is greater than the threshold, the mechanism merges the corresponding regions in the split 2D lesion mask 1113 (box 1155).

[0185] If the affinity between two seeds is greater than a threshold, they are assigned the same label. Otherwise, at this level, it is unknown whether they should belong to the same group. This decision is left to the label propagation level. Figure 15 Box 1512 in the diagram, the label propagation level is the same module used in the z-axis connection, and the label propagation level will be described below.

[0186] When we have more than two seeds, repeat for all seed pairs before tag propagation. Figure 11B The same operation is performed, which produces seed groups. For example, there are cases where it is determined that seed pairs (a, b) and (b, c) belong to the same group, but seed pair (a, c) fails the test, such as... Figure 11B As shown. Tag propagation would then require placing a, b, and c in the same group (i.e., the regions corresponding to seeds a and c would still be merged). However, if seeds a, b, c, and d exist, and affinity calculations (performed on a total of six pairs) show that only (a, b) and (c, d) pass the test, then tag propagation will result in two groups containing (a, b) and (c, d), respectively. Therefore, if a seed pair fails the test, it means it's unknown whether they should be placed in the same group, rather than that they should belong to different groups.

[0187] For example, in Figure 10C In the given information, there are three seed pairs (1051-1061, 1051-1071, 1061-1071), and the mechanism should determine that seeds 1051 and 1061 should be assigned the same label (belonging to the same group). The label propagation step then clusters these three seeds into two groups: the first group contains only 1071, while the second group contains both 1051 and 1061.

[0188] Figure 12 This is a flowchart outlining an example operation for lesion splitting according to an illustrative embodiment. Figure 12 The operations outlined above can be found in the text above. Figures 11A-11B The mechanism described is used to execute it. For example... Figure 12As shown, the operation begins (step 1200), and the mechanism generates a distance map of the two-dimensional lesion mask (box 1101). As described above, this distance map can be generated by performing a distance transformation operation on the two-dimensional lesion mask and optionally performing Gaussian smoothing to remove noise. The mechanism then uses local maximum identification to generate groupings of data points (e.g., local maxima for each group) (step 1202). The mechanism performs lesion splitting based on local maxima to generate regions (step 1203). The mechanism then relabels the seeds based on pairwise similarity using the distance map (step 1204). The mechanism then merges the regions corresponding to seeds with the same label (step 1205). It should be understood that, as described above, due to the seed relabeling performed by the mechanism of the illustrative embodiment, the split lesion mask output in step 1205 does not have the over-splitting problem associated with watershed techniques due to mislabeling of data points associated with each lesion shape. The operation then ends (step 1206).

[0189] Z-axis connectivity of the lesion

[0190] The aforementioned lesion splitting and seed relabeling process can be performed on each two-dimensional image or slice of the input volume to generate an appropriately labeled lesion mask for each lesion represented in the corresponding two-dimensional image. However, the input volume represents a three-dimensional representation of the internal anatomy of a biological entity, and lesions that may appear to be associated with the same lesion when considered in three dimensions may actually be associated with different lesions. Therefore, in order to correctly identify individual lesions within a biological entity represented in three dimensions of the input volume, illustrative embodiments provide a mechanism for connecting two-dimensional lesions along the z-axis (i.e., in three dimensions).

[0191] The mechanism for performing the connection of two-dimensional lesions along the z-axis (referred to as the z-axis connection of lesions) involves executing a logistic regression model to determine the detection of three-dimensional z-axis lesions, based on the split lesion output generated by the mechanism described above. This mechanism connects two lesions in adjacent image slices. The two lesions are connected when the logistic regression model determines that they represent the same lesion. For example, for any two-dimensional lesions on adjacent image slices (i.e., slices whose z-axis coordinates are consecutively ordered along the z-axis in the set of three-dimensional tissues in the slices), as will be described below, this mechanism determines whether these two-dimensional lesions belong to the same three-dimensional lesion.

[0192] Figures 13A-13C This describes the process of z-axis connection for a lesion according to an illustrative embodiment. Figure 13A The lesion mask input was described. Figure 13B The lesion after cleavage in a slice is depicted, which can employ an improved lesion cleavage mechanism with relabeled markings from the previously described illustrative embodiments. Figures 13A-13BAs shown, slice 1310 has lesions 1311 and 1312, slice 1320 has lesion 1321, and slice 1330 has lesions 1331 and 1332. The z-connection of the lesion mechanism (i.e., the logistic regression model) is performed on the split lesion mask of each pair of adjacent slices in the input volume to compare each lesion in a given slice with each lesion in a pair of adjacent slices. For example, the z-connection of the lesion mechanism compares lesion 1311 (lesion A) in slice 1310 with lesion 1321 (lesion B) in slice 1320. For each comparison, the mechanism treats each lesion as a set of voxels and determines the intersection between lesion A (a set of voxels in lesion A) and lesion B (a set of voxels in lesion B) based on the size of lesion A and the size of lesion B. The z-connection of the lesion mechanism uses a logistic regression model to determine whether lesion A and lesion B are connected based on two overlap ratios, as follows:

[0193]

[0194] Where |A| represents the area of ​​the circle represented by seed A, |B| represents the area of ​​the circle represented by seed B, and |A∩B| represents the area of ​​the intersection of the circles represented by seed A and seed B. This mechanism uses these two ratios as input features to train a logistic regression model to determine the probability of connecting lesion A and lesion B. That is, as previously described above, using a machine learning process, the logistic regression model is trained on a volume of training images to generate predictions about the probability that, in each pair of slices in each training volume, a lesion in one slice is the same as or different from the lesion represented in the adjacent slice. This prediction is compared to a ground truth indication of whether the lesions are the same or different to generate a loss or error. The operating parameters (e.g., coefficients or weights) of the logistic regression model are then modified to reduce this loss or error until a predetermined number of training epochs have been performed or a predetermined stopping condition is met.

[0195] Logistic regression models are widely used to solve binary classification problems. However, in the context of this illustrative example, this logistic regression model predicts the probability that two cross-sections of a lesion are part of the same lesion. For this purpose, logistic regression uses two overlap ratios, r0 and r1, as previously mentioned. Specifically, the logistic model learns to linearly combine two features as follows:

[0196]

[0197] Where (C0, C1, b) are the operation parameters learned from the training volume via machine learning training operations. Let r0 and r1 be denoted by the symbols, and represent the minimum and maximum overlap ratios, respectively. The state of the operation parameters after training the logistic regression model can be represented as follows: During inference (i.e., after training the logistic regression model), when processing new input volumes of images (slices), a threshold t is set such that if and only if the relation When the prediction probability is established (i.e., the predicted probability is higher than a set threshold), the two sections are considered to belong to the same lesion.

[0198] There are two extreme cases. First, when the threshold t is set to 0, the z-linkage mechanism of the illustrative embodiment always determines that the lesions are the same lesions (i.e., these sections are connected). Then both the true positive rate and the false positive rate are 1. Second, when the threshold t is set to 1, the z-linkage mechanism will not recognize any sections of the lesions to be connected. In this case, both the true positive rate and the false positive rate will be 0. Therefore, the logistic regression model will determine whether a lesion section is associated with the same lesion across adjacent slices only when the threshold t is in the interval (0, 1). Using an ideal logistic regression model, the true positive rate equals 1 (all true connections are identified) and the false positive rate is 0 (zero false connections are performed).

[0199] Therefore, once the logistic regression model is trained, new slice pairs can be evaluated by calculating these ratios of the pairs and feeding them as input features into the trained logistic regression model to generate predictions for each of these pairs, and then considering lesions A and B as being associated with the same lesion in three dimensions if the predicted probability is equal to or greater than a predetermined threshold probability. Appropriate relabeling of lesions across slices can then be performed to appropriately associate lesions in two-dimensional slices with the same lesion representations in other adjacent slices, thereby identifying three-dimensional lesions within the input volume.

[0200] There exists a fundamental principle supporting two ratio input features for training a logistic regression model. For example, if lesions A and B are sufficiently different in size, then they cannot be part of the same lesion. Furthermore, if lesions A and B are disjoint (e.g., lesion 1312 in slice 1310 and lesion 1321 in slice 1320), then features r0 and r1 will have zero values. As described above, given two features r0 and r1, the logistic regression model performs regression and outputs a probability value between 0 and 1 representing the likelihood that lesions A and B are part of the same lesion.

[0201] Figure 13C A cross-sectional connection between slices according to an illustrative embodiment is depicted. For example... Figure 13C As shown, this mechanism determines that lesion 1311 in slice 1310 and lesion 1321 in slice 1321 are part of the same lesion by executing a trained logistic regression model of the illustrative embodiment. The trained logistic regression model predicts lesion commonality based on the overlap ratio as discussed above. The mechanism also determines that lesion 1321 in slice 1320 and lesion 1331 in slice 1330 are part of the same lesion in a similar manner. Thus, the mechanism propagates intersecting lesions along the z-axis and performs z-axis connection of the lesions.

[0202] Based on pairwise evaluation of the z-axis connectivity of slices in the input volume with respect to identifying lesions spanning two-dimensional slices, and by determining whether lesions are connected along the z-axis through a trained logistic regression model, lesion relabeling can be performed to ensure that the same lesion label is applied to every lesion mask present in each slice of the input volume (e.g., all lesion masks spanning a set of slices in the input volume). Where a lesion mask determined by the logistic regression model to be associated with the same lesion A can be relabeled to specify that they are part of the same lesion A. This can be performed for each lesion section in each slice of the input volume, thereby generating a three-dimensional association of lesion masks for one or more lesions present in the input volume. This information can then be used to represent or otherwise process lesions in three dimensions (such as in later downstream computational system operations), since all sections associated with the same lesion are correctly labeled in the input volume.

[0203] Figure 14A and Figure 14B The results of a trained logistic regression model based on an illustrative embodiment are presented. Figure 14A This illustrates the receiver operating characteristic (ROC) curves for the maximum overlap ratio (r0) + minimum overlap ratio (r1) and the maximum overlap ratio. The ROC curve is a graphical representation of the diagnostic capability of a binary classifier system as the discrimination threshold changes. The ROC curve is created by plotting the true positive rate (TPR) relative to the false positive rate (FPR) at various threshold settings. Figure 14B This illustrates the precision-recall curves for the maximum overlap ratio + minimum overlap ratio metric and the maximum overlap ratio metric. The precision-recall curve is a plot of precision (y-axis) and recall (x-axis) for different thresholds, much like an ROC curve, where precision is the score of relevant instances among retrieved instances, and recall (or sensitivity) is the score of the total number of relevant instances actually retrieved. As these graphs show, the two-feature logistic model outperforms its single-feature counterpart. These two features thus bring valuable information to this prediction task.

[0204] observe Figure 14A The curve representing the maximum overlap ratio (r0) + minimum overlap ratio (r1) shows that, with an appropriate threshold t, a trained logistic regression model can produce a true positive rate of approximately 95% at the cost of approximately 3% false positives. See also Figure 14B The plotted graph evaluates the trained logistic regression model in terms of accuracy and recall, and shows that both measures can achieve very good results with appropriate selection of the threshold t.

[0205] Figure 15 This is a flowchart outlining an example operation of a mechanism for connecting two-dimensional lesions along the z-axis, according to an illustrative embodiment. Figure 15 As shown, the operation begins (step 1500), and the mechanism selects a first image X from the input volume (step 1501) and selects a first lesion A in image X (step 1502). In some illustrative embodiments, the previously described splitting and relabeling mechanism can be used to process the image or slice in the input volume; however, this is not necessary. Instead, the mechanism of the illustrative embodiment involving z-axis connectivity of lesions can actually be performed with any input volume in which a lesion mask has already been identified.

[0206] Then, the z-axis connectivity mechanism of the illustrative embodiment selects a first lesion B in the adjacent image Y (step 1503). The mechanism then determines the intersection point between lesion A and lesion B for lesion A, and the intersection point between lesion A and lesion B for lesion B (step 1504). This mechanism generates a prediction or probability that lesion A and lesion B are the same lesion by applying a trained logistic regression model to the r0 and r1 features of the intersection point for lesion A and lesion B, and then compares this probability with a threshold probability, determining whether lesion A and lesion B belong to the same lesion based on these two intersection point values ​​(step 1505). Based on the determination result, sections of lesions in the image can be labeled or relabeled to indicate whether they are part of the same lesion.

[0207] The mechanism determines whether lesion B in image Y is the last lesion in image Y (step 1506). If lesion B is not the last lesion, the mechanism considers the next lesion B in the adjacent image Y (step 1507), and the operation returns to step 1504 to determine the intersection between lesion A and the new lesion B.

[0208] If lesion B is the last lesion in the adjacent slice or image Y at step 1506, then the mechanism determines whether lesion A is the last lesion in image X (step 1508). If lesion A is not the last lesion in image X, then the mechanism considers the next lesion A in image X (step 1509), and the operation returns to step 1502 to consider the first lesion B in the adjacent image Y.

[0209] If lesion A is the last lesion in image X at step 1508, the mechanism determines whether image X is the last image to be considered (step 1510). If image X is not the last image, the mechanism considers the next image X (step 1511), and the operation returns to step 1502 to consider the first lesion A in the new image X.

[0210] If image X is the last image to be considered at step 1510, the mechanism propagates intersecting lesions along the z-axis between images, where propagation means that labels associated with the same lesions identified through the above process are set to the same value to indicate that they are part of the same lesion (step 1512). This is performed for each individual lesion identified in the input volume, such that sections in each image associated with the same lesion are appropriately labeled, and thus, a three-dimensional representation of each lesion is generated by connecting the z-axis sections. The operation then ends (step 1513).

[0211] Contour Refinement

[0212] The above process produces accurate results in terms of the number and relative location of lesions, as well as in connecting lesions across two-dimensional space (within an image or slice) and three-dimensional space (across an image or slice in the input volume). However, lesion contours (boundaries) are not always well-defined and require improvement. Illustrative embodiments provide mechanisms for improving the accuracy of lesion contours. This additional mechanism can be employed in conjunction with the mechanisms described above as part of lesion segmentation, or it can be employed in other illustrative embodiments that do not require the specific lesion detection, lesion splitting and relabeling, and / or z-axis connection mechanisms described above.

[0213] Existing contouring algorithms work well only when there are lesions in the middle of an anatomical structure without surrounding lesions. However, they perform poorly when there are different situations that lead to the "leakage" problem, in which two or more adjacent lesions have initially distinct contours that merge into a single fully enclosing contour, thus completely negating the benefits of earlier 2D lesion mask splitting. In some cases, when lesions are near anatomical boundaries (e.g., liver boundaries), contouring algorithms distinguish between pixels of the anatomical structure relative to pixels of other anatomical structures (e.g., organs) in the image, rather than distinguishing one lesion from another, because contouring algorithms are best able to distinguish the pixels of these anatomical structures.

[0214] The mechanism of the illustrative embodiment repairs areas of no interest in an image or slice. Figure 16 This illustrates an example of two lesions with outlines in the same image, according to an illustrative embodiment. Figure 16 On the left, the active contour algorithm is used to determine the contours of the two lesions, 1611 and 1612. Active contour algorithms are a class of algorithms that iteratively evolve contours to better fit the image content.

[0215] According to this illustrative embodiment, the mechanism repairs non-liver tissue within contour 1612 and near but not within contour 1611, where repair means that pixel values ​​of pixels within and around contour 1612, as well as pixel values ​​of healthy tissue (non-pathological tissue) near contour 1611, are set to a specified value such that they all have the same value. For example, this value could be the average tissue value of a region identified as not associated with a lesion (i.e., healthy tissue with an anatomical structure, such as the liver).

[0216] The repair can be performed relative to a selected lesion contour 1611, such that the repair is applied to healthy tissue and other lesions (e.g., lesion 1612) in the image. In this way, when re-evaluating contour 1611, the contours and pixels associated with the selected lesion (e.g., 1611) are considered separately from other parts of the image. Contour 1611 can then be re-evaluated, and it can be determined whether the re-evaluation of contour 1611 results in an improved definition of contour 1611. That is, an initial determination of the contrast and variance between the pixels associated with the selected lesion contour 1611 and the pixels near the selected lesion contour 1611 can be generated. After calculating this contrast and variance before repair, repair can be performed against the selected lesion 1611, such that pixels associated with other lesion contours (e.g., 1612) and regions in the image representing the anatomical structures of healthy tissue are repaired using the average pixel intensity values ​​of healthy tissue.

[0217] The variance of a set of values ​​is determined as follows. Consider a set of voxels, containing, say, n voxels. First, calculate the arithmetic mean by summing their intensity values ​​and then dividing the sum by n. This is denoted as A. Second, square each of these voxel values ​​individually and then calculate the arithmetic mean. The result is denoted as B. The variance is then defined as B – A*A, that is, the difference between B and the square of A.

[0218] Therefore, the variance of the set of n values ​​{x1, ..., xn} is defined as follows:

[0219]

[0220] Calculate the variance between voxels inside and outside a given contour. Voxels inside the contour are those voxels enclosed by the contour, while voxels outside the contour are those voxels located outside the contour but within a predetermined distance from the contour.

[0221] This mechanism uses the active contour algorithm described above to recalculate the contour 1611 of the selected lesion after repair, and recalculates the contrast and / or variance of the new contour 1611 to determine whether these values ​​have been improved (higher contrast values ​​or lower variance values ​​inside and / or outside the lesion). If the contrast and variance have been improved, the newly calculated contour 1611 is retained as the contour of the corresponding lesion. This process can then be performed on lesion 1612, which is considered the selected lesion by subsequently repairing pixels associated with healthy tissue near lesion 1611 and contour 1612. In this way, each lesion is evaluated separately to generate its contour, thereby preventing lesions from leaking into each other.

[0222] The mechanism for calculating the lesion contour after repair can be based on the Chan-Vese segmentation algorithm, which is designed to segment objects without explicitly defined boundaries. This algorithm minimizes energy based on an iteratively evolving level set defined by the sum of differences corresponding to the average values ​​from outside the segmented region, the sum of differences corresponding to the average values ​​from inside the segmented region, and a weighted value depending on the length of the segmented region's boundary. Initialization is performed using a partitioned detection map (addressing local minima of energy).

[0223] Once the mechanism has segmented, it initializes the contour using previous estimates and determines whether the new contour is better (e.g., improving the contrast and variance of the contour). If the original contour is better, it is retained. If the new contour is better (e.g., improving the contrast and variance of the contour), the mechanism uses the new contour. In some illustrative embodiments, the mechanism determines which contour is better based on homogeneous regions and calculated variance. If the variance decreases both inside and outside the contour, the mechanism uses the new contour; otherwise, the mechanism uses the old contour. In another illustrative embodiment, the mechanism determines whether the contrast (the average value inside the contour versus the average value near the contour) is improved. Other techniques utilizing different measurements can be used to select between the old and new contours without departing from the spirit and scope of the illustrative embodiments.

[0224] Figure 17 This is a flowchart outlining example operations of a mechanism for slice-type contour refinement according to an illustrative embodiment. (e.g.) Figure 17 As shown, for a given contour (such as in the liver) in an image segmented to show a lesion, the operation begins (step 1700), and the mechanism determines a first contrast and variance of the initial contour (step 1701). The mechanism repairs lesion pixels (or three-dimensional voxels) near the lesion (step 1702). The mechanism then determines the contour around the lesion (step 1703). The mechanism then determines a second contrast and variance of the new contour (step 1704). The mechanism determines whether the second contrast and variance represent an improvement compared to the first contrast and variance (step 1705). If the second contrast and variance represent an improvement, the mechanism uses the updated contour to represent the lesion (step 1706). The operation then ends (step 1708).

[0225] If the second contrast and variance do not indicate improvement in step 1705, then the mechanism reverts to the initial contour (step 1707). The operation then ends (step 1708). This process can be repeated for each lesion identified in the input slice and / or input volume to recalculate the contour and improve the contour associated with each lesion present in the image / input volume.

[0226] False positive removal

[0227] After performing lesion segmentation to generate a list of lesions and their outlines, the AI ​​pipeline 100 performs a false positive level of processing 150 to remove incorrectly identified lesions from the lesion list. This false positive level 150 can take many forms to reduce the number of incorrectly identified lesions in the lesion list, for example, the outlines in Figure 1 and Figure 135 output by the liver / lesion detection logic 130, which are then merged by segmentation and relabeling performed in the lesion segmentation logic 140. The following description will illustrate novel false positive removal mechanisms that can be used to perform such false positive removal, but does not require this particular false positive removal. Furthermore, the false positive removal mechanisms described below can be used separately from the other mechanisms described above and can be applied to any list of objects identified in an image, wherein the illustrative embodiments specifically utilize this false positive removal of lesions in medical images. That is, the false positive removal mechanisms described in this section can be implemented separately and differently from the other mechanisms described above.

[0228] For illustrative purposes, it will be assumed that the false positive removal mechanism is implemented as part of AI pipeline 100 and as part of false positive removal logic 150 of AI pipeline 100. Thus, in false positive level 150, taking into account the three-dimensional nature of the input volume with z-axis connectivity and contour refinement of lesions as described above, the false positive removal mechanism described in this section operates on the lesion list generated by liver / lesion detection logic and lesion segmentation and relabeling. Figure 1 The list 148 is input to the false positive removal logic level 150, which processes list 148 in the manner described below and outputs a filtered or modified lesion list (in which misidentified lesions in the modified lesion list are minimized) to the lesion classification level 160. The lesion classification level then classifies the various lesions indicated in the modified lesion list.

[0229] That is, capturing all lesions in previous stages of the AI ​​pipeline 100 can result in an increased sensitivity setting that causes the AI ​​pipeline 100 to misidentify pixels that do not actually represent a lesion as part of a lesion. Therefore, false positives that should be removed may exist. False positive stage 150 includes logic that operates on a list of lesions and their outlines to remove false positives. It should be understood that such false positive removal must also balance the risk that, during examination (where the set of input volume levels is the opposite of the lesion level), the removal of false positives (if not performed appropriately) may result in lesions going undetected. This can be problematic because physicians and patients may not be aware of lesions requiring treatment. It should be understood that an examination can theoretically include several image volumes of the same patient. However, because some illustrative embodiments of the AI ​​pipeline that implements single-phase detection process only process an image of a single volume, it is assumed that this processing is performed with respect to a single volume. For clarity, “patient level” will be used hereafter instead of “examination level”, as this is of interest to the illustrative embodiments (whether the patient has a lesion). It should be understood that, in other illustrative embodiments, the operations described herein can be extended to the examination level, where multiple image volumes of the same patient can be evaluated.

[0230] For these illustrative embodiments, given the output of a previous stage (slice, mask, lesion, lesion and anatomical structure contour, etc.) of the AI ​​pipeline 100 as input 148 to the false positive removal stage 150, the false positive removal stage 150 operates at a highly specific operating point at the patient level (input volume level) to allow only a few patient-level false positives (normal patients / volumes detecting at least one lesion). This point can be retrieved from an analysis of patient-level subject operating characteristic (ROC) (patient-level sensitivity versus patient-level specificity). The highly specific operating point used here is referred to herein as the patient-level operating point OP. PATIENT Those volumes that produce at least some lesions, at the lesion level (here referred to as the lesion level operating point OP). LESION Use a more sensitive operating point at this location. The lesion-level operating point (OP) can be identified from the analysis of the lesion-level ROC curve (lesion sensitivity versus lesion specificity). LESION In order to maximize the number of lesions preserved.

[0231] These two operation points (i.e., OP) PATIENT and OP LESIONThis can be implemented in one or more trained ML / DL computer models. One or more trained ML / DL computer models are trained to classify the input volume and / or its list of lesions (the result of segmentation logic) as either true lesions or false lesions (i.e., true positives or false positives). The one or more trained ML / DL computer models can be implemented as binary classifiers, where the output indicates whether each lesion is a true positive or a false positive. The output set, including all lesions in the input lesion list, can be used to filter the lesion list to remove false positives. In one illustrative embodiment, the one or more trained ML / DL computer models first implement patient-level operation points to determine whether the classification result indicates any lesion in the lesion list is a true positive while filtering out false positives. If any true positives remain in the first filtered list of lesions after patient-level (input volume level) filtering, the remaining false positives (if any) are filtered out using lesion-level operation points. Thus, a filtered list of lesions that minimizes false positives is generated.

[0232] The implementation of an operation point can involve a single trained ML / DL computer model or multiple trained ML / DL computer models. For example, using a single trained ML / DL computer model, the operation point can be a setting of operation parameters for a dynamically switchable ML / DL computer model. For instance, using a patient-level operation point, input to the ML / DL computer model can be processed to generate a result indicating whether the lesion list includes true positives after classifying each lesion. If so, the operation point of the ML / DL computer model can be switched to a lesion-level operation point, and the input can be processed again, wherein each time a false positive is detected by the ML / DL computer model, it is removed from the final list of lesions output by the false positive removal stage. Alternatively, in some illustrative embodiments, two separate ML / DL computer models can be trained, one for the patient-level operation point and one for the lesion-level operation point, such that the result of the first ML / DL computer model indicating at least one true positive causes processing of the input by the second ML / DL computer model, and the false positives identified by both models are removed from the final list of lesions output by the false positive removal stage of the AI ​​pipeline.

[0233] Training multiple ML / DL computer models can involve machine learning training operations, in which the ML / DL computer models process training inputs, including image volumes and corresponding lists of lesions, where the lesion list includes lesion masks or contours, to generate a classification for each lesion in the image regarding whether it is a true positive or a false positive. The training inputs are further associated with ground truth information indicating whether the image contains a lesion, which can then be used to evaluate the output generated by the ML / DL computer models to determine loss or error, and then modify the operating parameters of the ML / DL computer models to reduce the determined loss / error. In this way, the ML / DL computer models learn input features representing true / false positive lesion detection. This machine learning can be performed with respect to each operation point (i.e., OP). PATIENT and OP LESION This allows the operating parameters of the ML / DL computer model to be learned while taking into account patient-level sensitivity / specificity and / or lesion-level sensitivity / specificity.

[0234] When classifying lesions as true or false positives, an input volume is considered positive if it contains at least one lesion (representing a patient at the "patient level"). An input volume is considered negative if it does not contain any lesions. To this end, a true positive is defined as a positive input volume (i.e., an input volume with at least one finding that is actually a lesion and is classified as such). A true negative is defined as a negative input volume (i.e., an input volume with no lesions and no finding classified as such). A false positive is defined as a negative input volume with no lesions, however, this input indicates a lesion in the finding (i.e., the AI ​​pipeline lists lesions when no lesions are present). A false negative is defined as a positive input volume with lesions, but the AI ​​pipeline does not indicate a lesion in the finding. The trained ML / DL computer model classifies the lesions in the input as true or false positives. False positives are filtered out from the output generated by false positive removal. False positive detection is performed at the patient level and the lesion level (i.e., two different operating points) with different sensitivity / specificity levels.

[0235] Two distinct operating points, one at the patient level and one at the lesion level, can be determined based on ROC curve analysis. The ROC curve can be calculated using ML / DL computer model validation data, which consists of several input volumes (e.g., several input volumes corresponding to different patient examinations) that may contain some lesions (between 0 and K lesions per examination). The input to the trained ML / DL computer model, or "classifier," consists of findings previously detected in the inputs, which are either actual lesions or false positives (e.g., the output of the lesion detection and segmentation levels in an AI pipeline). The first operating point (i.e., the patient-level operating point OP) is determined by these findings. PATIENT A true positive lesion is defined as retaining at least X% of those identified as true positives, meaning that almost all true positives are retained while some false positives are removed. The value of X can be set based on ROC curve analysis and can be any suitable value for a particular implementation. In one illustrative embodiment, the value of X is set to 98%, such that almost all true positives are retained while some false positives are removed.

[0236] Define the second operation point (i.e., the lesion-level operation point OP). LESION This makes the lesion sensitivity higher than that for the first operating point (i.e., the patient-level operating point OP). PATIENT The obtained lesion sensitivity is such that the specificity is higher than Y%, where Y depends on the actual performance of the trained ML / DL computer model. In one illustrative embodiment, Y is set to 30%. Examples of ROC curves for determining the operation point at the patient and lesion levels are shown in [reference to a specific example]. Figure 18A As shown in the image. Figure 18A As shown, the lesion level operation point is selected along the ROC curve of the lesion level, so that the lesion sensitivity is higher than the lesion sensitivity of the patient level operation point.

[0237] Figure 18B This is an example flowchart illustrating an operation for performing false positive removal based on patient and lesion level operation points, according to an illustrative embodiment. Figure 18BAs shown, the results of the segmented logic of the AI ​​pipeline are input 1810 to a first trained ML / DL computer model 1820 that implements the first operation point. Input 1810 includes an input volume (or image volume (VOI)) and a lesion list, which includes lesion mask or contour data specifying pixels or voxels corresponding to each lesion identified in the image data of the image volume, and labels associated with these pixels specifying the lesions' three-dimensional space (i.e., the outputs of segmentation, z-connection, and contour refinement previously described) corresponding to the input volume. The input can be represented as a set S. The first trained ML / DL computer model 1820 implements a patient-level operation point during its training to classify features extracted from the input using X% (e.g., 98%) of true positives retained in the resulting filtered lesion list generated by the classification of the trained ML / DL computer model 1820, with some false positives removed from the resulting list. The resulting list includes a subset S containing true positive lesions classified by the first ML / DL computer model 1820. + And a subset S- containing false-positive lesions classified by the first ML / DL computer model 1820.

[0238] The false positive removal logic further includes true positive assessment logic 1830, which determines whether the subset of true positives output by the first ML / DL computer model 1820 is empty. That is, the true positive assessment logic 1830 determines whether no element from S is classified as a true lesion by the first ML / DL computer model 1820. If the subset of true positives is empty, then the true positive assessment logic 1830 makes the subset of true positives S... + The output is a filtered list of lesions 1835 (i.e., lesions will not be identified in the lesion classification level output sent to the AI ​​pipeline). If the true positive evaluation logic 1830 determines a subset S of true positives... + If not empty, then execute the second ML / DL computer model 1840 on the input S, wherein the second ML / DL computer model 1840 implements the second operation point (i.e., the lesion level operation point OP) during its training. LESION It should be understood that, although two ML / DL computer models 1820 and 1840 are shown for ease of explanation, as mentioned above, these two operation points can be implemented in different sets of training operation parameters used to configure the same ML / DL computer model, such that the second ML / DL computer model can be the processing of input S with the same ML / DL computer model as 1820 but with different operation parameters corresponding to the second operation point.

[0239] The second ML / DL computer model 1840 processes the input with training operational parameters corresponding to the second operational point to generate lesion classifications again regarding whether they are true positives or false positives. The result is a subset S' containing the predicted lesions (true positives). + and a subset S' containing predicted false positives - The filtered list of lesions, 1845, is then output as a subset S'. + This effectively eliminates the problem in subset S' - The specified false positive.

[0240] Figure 18A and Figure 18B The example embodiments shown are described based on patient-level and lesion-level operation points. It should be understood that the mechanism for false positive removal can be implemented using operation points at various different levels. For example, similar operations can be performed at image volume level and voxel level operation points in a "voxel-based" false positive removal operation. Figure 18C This is an example flowchart of an operation for performing voxel-based false positive removal based on an illustrative embodiment of an input volume level and voxel level operation point. Figure 18C The operation in is similar to Figure 18B The operation is performed on voxels within the input set S. Using voxel-based false positive removal, the first operation point can again be a patient-level or input volume-level operation point, while the second operation point can be a voxel-level operation point OP. VOXEL In this context, true and false positives are assessed at the voxel level such that if a voxel indicates that it is associated with a lesion and it is actually associated with a lesion, it is a true positive; however, if a voxel is indicated to be associated with a lesion but it is not actually associated with a lesion, it is considered a false positive. Appropriate settings for the operating point can be regenerated based on the corresponding ROC curve, achieving a similar balance between sensitivity and specificity as described above.

[0241] It should also be understood that while the above illustrative embodiments of the false positive removal mechanism assume a single input volume from a patient examination, the illustrative embodiments can be applied to any grouping of one or more images (slices). For example, false positive removal can be applied to a single slice, a group of slices smaller than the input volume, or even multiple input volumes from the same examination.

[0242] Figure 19 This is a flowchart outlining an example operation of the false positive removal logic of an AI pipeline according to an illustrative embodiment. Figure 19As shown, the operation begins (step 1900) by receiving input S from a previous stage of the AI ​​pipeline, where the input may include, for example, an input image volume and a corresponding list of lesions including masks, contours, etc. (step 1910). This input is processed by a first trained ML / DL computer model trained to implement a first operation point (e.g., a relatively more specific and less sensitive patient-level operation point) to generate a first set of classifications for lesions including a subset of true positives and a subset of false positives (step 1920). It is determined whether the true positive subset is empty (step 1930). If the true positive subset is empty, the operation outputs the true positive subset as a filtered list of lesions (step 1940) and the operation terminates. If the true positive subset is not empty, the input S is processed by a second ML / DL computer model trained to implement a second operation point that is relatively more sensitive and less specific than the first operation point (step 1950). As described above, in some illustrative embodiments, the first and second ML / DL computer models may be the same model, but configured with different operating parameters corresponding to different training to achieve different operating points. The result of processing the second ML / DL computer model is a second group classification of lesions including a second true positive subset and a second false positive subset. The second true positive subset is then output as a filtered list of lesions (step 1960) and the operation terminates.

[0243] Example computer system environment

[0244] The illustrative embodiments can be utilized in many different types of data processing environments. To provide context for describing the specified elements and functions of the illustrative embodiments, the following is provided. Figure 20 and Figure 21 This serves as an example environment in which aspects of the illustrative embodiments can be implemented. It should be understood that... Figure 20 and Figure 21 This is merely an example and is not intended to assert or imply any limitation regarding the environment in which various aspects or embodiments of the invention may be implemented. Many modifications may be made to the depicted environment without departing from the spirit and scope of the invention.

[0245] Figure 20This diagram illustrates an illustrative embodiment of a cognitive system 2000 implementing a request processing pipeline 2008, which in some embodiments may be a question-and-answer (QA) pipeline, a treatment suggestion pipeline, a medical imaging enhancement pipeline, or any other pipeline based on artificial intelligence (AI) or cognitive computing that uses sophisticated AI mechanisms to process requests, approximating humans through processes related to the generated results, but through different computer-specified processes. For the purposes of this specification, it will be assumed that the request processing pipeline 2008 is implemented as a QA pipeline that operates on structured and / or unstructured requests in the form of input questions. An example of a question processing operation that can be used in conjunction with the principles described herein is described in U.S. Patent Application Publication No. 2011 / 0125734, the entire contents of which are incorporated herein by reference.

[0246] The cognitive system 2000 is implemented on one or more computing devices 2004A-D (including one or more processors and one or more memories, and potentially any other computing device elements known in the art, including buses, storage devices, communication interfaces, etc.) connected to the computer network 2002. For illustrative purposes only, Figure 20 A cognitive system 2000 is depicted implemented solely on computing device 2004A, but as described above, the cognitive system 2000 may be distributed across multiple computing devices (such as multiple computing devices 2004A-D). Network 2002 includes multiple computing devices 2004A-D, which can operate as server computing devices, and 2010-2012, which can operate as client computing devices, communicating with each other and with other devices or components via one or more wired and / or wireless data communication links, wherein each communication link includes one or more of a wire, router, switch, transmitter, receiver, etc. In some illustrative embodiments, the cognitive system 2000 and network 2002 implement their question processing and answer generation (QA) functions via the respective computing devices 2010-2012 of one or more cognitive system users. In other embodiments, cognitive system 2000 and network 2002 may provide other types of cognitive operations, including but not limited to request processing and cognitive response generation, which may take many different forms depending on the desired implementation (e.g., cognitive information retrieval, user training / instruction, cognitive evaluation of data, etc.). Other embodiments of cognitive system 2000 may be used with components, systems, subsystems, and / or devices other than those described herein.

[0247] Cognitive system 2000 is configured to implement a request processing pipeline 2008 that receives input from various sources. Requests may take the form of natural language questions, natural language requests for information, natural language requests for the execution of cognitive operations, etc. For example, cognitive system 2000 receives input from network 2002, a corpus or multiple corpora of electronic documents 2006, cognitive system users and / or other data and other possible input sources. In one embodiment, some or all of the inputs to cognitive system 2000 are routed through network 2002. Various computing devices 2004A-D on network 2002 include access points for content creators and cognitive system users. Some of the computing devices 2004A-D include corpora or multiple corpora for storing data 2006 (which are only for illustrative purposes). Figure 20 The device that stores the database (as a separate entity). Data from the 2006 corpus or portions of multiple corpora may also be provided on one or more other network-attached storage devices, in one or more databases, or... Figure 20 Other computing devices not explicitly shown. In various embodiments, network 2002 includes local network connectivity and remote connectivity, enabling cognitive system 2000 to operate in environments of any size (including local and global, such as the Internet).

[0248] In one embodiment, a content creator creates content in documents within a corpus or multiple corpora of data 2006 to be used as part of a corpus of data for cognitive system 2000. Documents include any files, text, articles, or data sources used in cognitive system 2000. A cognitive system user accesses cognitive system 2000 via a network connection to network 2002 or an Internet connection and inputs questions / requests to be answered / processed based on content from the corpus or multiple corpora of data 2006 into cognitive system 2000. In one embodiment, natural language is used to formulate the questions / requests. Cognitive system 2000 parses and interprets the questions / requests via pipeline 2008 and provides a response to the cognitive system user (e.g., cognitive system user 2010) containing one or more answers to the posed question, a response to the request, the result of processing the request, etc. In some embodiments, cognitive system 2000 provides a response to the user in a ranked list of candidate answers / responses, while in other illustrative embodiments, cognitive system 2000 provides a single final answer / response or a combination of a final answer / response with a ranked list of other candidate answers / responses.

[0249] The cognitive system 2000 implements a pipeline 2008, which includes multiple stages for processing input questions / requests based on information obtained from a corpus or multiple corpora of data 2006. The pipeline 2008 generates answers / responses to the input questions or requests based on the processing of the input questions / requests and the corpus or multiple corpora of data 2006.

[0250] In some illustrative embodiments, the cognitive system 2000 may be IBM Watson, available from International Business Machines, Inc., Armonk, New York. TM Cognitive systems, which are extended using the mechanisms of illustrative embodiments described below. As previously outlined, IBM Watson... TM The cognitive system pipeline receives input questions or requests, and then IBM Watson... TM The cognitive system's pipeline parses the input question or request to extract its key features, which are then used to formulate queries applied to the corpus 2006. Based on the queries applied to the corpus 2006, a set of hypotheses or candidate answers / responses to the input question / request is generated by examining portions of the corpus 2006 (hereinafter referred to as Corpus 2006) that contain potential valuable responses to the input question / response (hereinafter assumed to be the input question). IBM Watson TM The cognitive system pipeline 2008 then uses various inference algorithms to perform deep analysis on the language of the input question and the language used in each part of the corpus 2006 discovered during the application query.

[0251] Then, the scores obtained from different inference algorithms are weighted according to the statistical model, which summarizes the performance of IBM Watson. TM The pipeline 2008 of the cognitive system 2000 (in this example) has a confidence level regarding the evidence inferred from the question regarding potential candidate answers. This process is repeated for each candidate answer to generate a ranked list of candidate answers, which can then be presented to the user who submitted the input question (e.g., a user of client computing device 2010), or a final answer can be selected from the ranked list and presented to the user. (About IBM Watson) TM More information on the Cognitive Systems 2000 pipeline 2008 can be found, for example, on the IBM website, IBM Redbooks, etc. For example, regarding IBM Watson... TMInformation on the cognitive systems pipeline can be found in “Watson and Healthcare” by Yuan et al. in IBM developerWorks in 2011 and “The Era of Cognitive Systems: An Inside Look at IBM Watson and How it Works” by Rob High in IBM Redbooks in 2012.

[0252] As described above, while input to the cognitive system 2000 from client devices can be presented in the form of natural language questions, the illustrative embodiments are not limited thereto. Instead, input questions can actually be formatted or structured for analysis using structured and / or unstructured input (including but not limited to IBM Watson). TM The cognitive system 2000 uses natural language parsing and analysis mechanisms (such as those of cognitive systems) to parse and analyze any appropriate type of request to determine the basis for performing cognitive analysis and to provide the results of the cognitive analysis. For example, a physician, patient, etc., may send a request to the cognitive system 2000 via their client computing device 2010 for an operation based on a specific medical imaging (e.g., “identify liver lesions present in patient ABC”, “provide treatment recommendations for the patient”, or “identify changes in liver lesions in patient ABC”, etc.). According to the illustrative embodiment, such requests may be specifically targeted at cognitive computer operations that employ the lesion detection and classification mechanisms of the illustrative embodiment to provide a list of lesions, lesion outlines, lesion classifications, and outlines of anatomical structures of interest, upon which the cognitive system 2000 operates to provide cognitive computing output. For example, the request processing pipeline 2008 may process a request such as “identify liver lesions present in patient ABC” to parse the request and thereby identify the anatomical structure of interest as “liver”, with the specific input volume being the medical imaging volume of patient “ABC”, and the “lesion” in the anatomical structure being identified. Based on this analysis, a specific medical imaging volume corresponding to patient "ABC" can be retrieved from corpus 2006 and input into lesion detection and classification AI pipeline 2020. This pipeline operates on this input volume as previously described to identify a list of liver lesions. This list is output to cognitive computing system 2000 for further evaluation by request processing pipeline 2008 to generate medical imaging viewer application output, etc.

[0253] like Figure 20As shown, one or more of these computing devices (e.g., server 2004) can be specifically configured to implement an AI pipeline for lesion detection and classification 2020 (e.g., like... Figure 1 (AI pipeline 100 in the example). The configuration of the computing device may include providing dedicated hardware, firmware, etc., to facilitate the execution of the operations described herein with respect to the illustrative embodiments and the generation of output. The configuration of the computing device may also or alternatively include providing software applications stored in one or more storage devices and loaded into the memory of the computing device (such as server 2004), for causing one or more hardware processors of the computing device to execute the software applications, the software applications configuring the processors to perform operations and generate the output described herein with respect to the illustrative embodiments. Furthermore, any combination of dedicated hardware, firmware, software applications executed on hardware, etc., may be used without departing from the spirit and scope of the illustrative embodiments.

[0254] It should be understood that once a computing device is configured in one of these ways, it becomes a dedicated computing device specifically configured to implement the mechanisms of the illustrative embodiments, and not a general-purpose computing device. Furthermore, as described herein, the implementation of the mechanisms of the illustrative embodiments improves the functionality of the computing device and provides useful and specific results that facilitate automated lesion detection and classification in anatomical structures of interest, reducing errors and improving efficiency compared to manual processes.

[0255] As described above, the mechanisms of the illustrative embodiments utilize specially configured computing devices or data processing systems to perform operations for anatomical structure recognition, lesion detection, and classification. These computing devices or data processing systems may include various hardware elements that are specially configured, through hardware configurations, software configurations, or a combination of hardware and software configurations, to implement one or more of the systems / subsystems described herein. Figure 21 This is a block diagram of only one example data processing system in which various aspects of the illustrative embodiments may be implemented. Data processing system 2100 is a computer (such as...) Figure 20 Examples of servers (2004) in which computer-available code or instructions implementing various processes and aspects of illustrative embodiments of the invention can be located and / or executed to achieve the operation, output, and external effects of the illustrative embodiments described herein.

[0256] In the depicted example, the data processing system 2100 employs a hub architecture including a northbridge and memory controller hub (NB / MCH) 2102 and a southbridge and input / output (I / O) controller hub (SB / ICH) 2104. A processing unit 2106, main memory 2108, and a graphics processor 2110 are connected to the NB / MCH 2102. The graphics processor 2110 can be connected to the NB / MCH 2102 via an Accelerated Graphics Port (AGP).

[0257] In the depicted example, a local area network (LAN) adapter 2112 is connected to an SB / ICH 2104. An audio adapter 2116, a keyboard and mouse adapter 2120, a modem 2122, a read-only memory (ROM) 2124, a hard disk drive (HDD) 2126, a CD-ROM drive 2130, a universal serial bus (USB) port and other communication ports 2132, and a PCI / PCIe device 2134 are connected to the SB / ICH 2104 via buses 2138 and 2140. The PCI / PCIe device may include, for example, an Ethernet adapter, an add-in card, and a PC card for a notebook computer. PCI uses a card bus controller, while PCIe does not. The ROM 2124 may be, for example, a flash memory-based basic input / output system (BIOS).

[0258] HDD2126 and CD-ROM drive 2130 are connected to SB / ICH2104 via bus 2140. HDD2126 and CD-ROM drive 2130 can use interfaces such as Integrated Drive Electronics (IDE) or Serial Advanced Technology Accessory (SATA). Super I / O (SIO) device 2136 can be connected to SB / ICH2104.

[0259] The operating system runs on the processing unit 2106. The operating system coordinates and provides services to... Figure 21 The control of various components within the data processing system 2100. As a client, the operating system can be, for example... Commercially available operating systems. Object-oriented programming systems (such as Java). TM The programming system can run in conjunction with an operating system and provides Java execution on the data processing system 200. TM A program or application's call to the operating system.

[0260] As a server, the data processing system 2100 can, for example, run advanced interactive execution. Operating system or IBM eServer operating system TM Computer systems, Power-basedTM Computer systems with processors, etc. The data processing system 2100 may be a symmetric multiprocessor (SMP) system that includes multiple processors in the processing unit 2106. Alternatively, a single-processor system may be used.

[0261] The operating system, object-oriented programming system, and instructions for applications or programs reside on a storage device (such as HDD 2126) and can be loaded into main memory 2108 for execution by processing unit 2106. Processing for illustrative embodiments of the invention can be executed by processing unit 2106 using computer-usable program code, which may reside in memory (e.g., main memory 2108, ROM 2124) or, for example, in one or more peripheral devices 2126 and 2130.

[0262] Bus systems (such as) Figure 21 The bus 2138 or bus 2140 shown may include one or more buses. Of course, the bus system can be implemented using any type of communication structure or architecture that provides data transfer between different components or devices attached to that structure or architecture. Communication units (such as...) Figure 21 The modem 2122 or network adapter 2112 may include one or more means for transmitting and receiving data. The memory may be, for example, main memory 2108, ROM 2124, or something similar to... Figure 21 The cache was found in NB / MCH2102.

[0263] As described above, in some illustrative embodiments, the mechanism of the illustrative embodiments can be implemented as application-specific hardware, firmware, etc., application software stored in a storage device (such as HDD 2126) and loaded into a memory (such as main memory 2108), for execution by one or more hardware processors (such as processing unit 2106), etc. Thus, Figure 21 The computing device shown is specifically configured to implement the mechanisms of the illustrative embodiments and to perform operations and generate the outputs described herein in relation to the artificial intelligence pipeline for lesion detection and classification.

[0264] Those skilled in the art will understand that Figure 20 and Figure 21 The hardware in the system can vary depending on the implementation method. Besides... Figure 20 and Figure 21 The hardware described in the text, or its replacement Figure 20 and Figure 21The hardware described herein can be replaced with other internal hardware or peripheral devices (such as flash memory, equivalent non-volatile memory, or optical disc drives). Furthermore, without departing from the spirit and scope of the invention, the processing of the illustrative embodiments can be applied to multiprocessor data processing systems other than the SMP systems previously mentioned.

[0265] Furthermore, the data processing system 2100 can take the form of any of a number of different data processing systems, including client computing devices, server computing devices, tablet computers, laptop computers, telephones or other communication devices, personal digital assistants (PDAs), etc. In some illustrative examples, the data processing system 2100 may be a portable computing device configured with flash memory to provide non-volatile storage for storing, for example, operating system files and / or user-generated data. Essentially, the data processing system 2100 can be any known or subsequently developed data processing system without architectural limitations.

[0266] As described above, it should be understood that illustrative embodiments may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment that includes both hardware and software elements. In one example embodiment, the mechanism of the illustrative embodiment is implemented in software or program code, including but not limited to firmware, resident software, microcode, etc.

[0267] A data processing system suitable for storing and / or executing program code will include at least one processor, which is directly or indirectly coupled to memory elements via a communication bus (such as, for example, a system bus). Memory elements may include local memory used during the actual execution of the program code, mass storage devices, and cache memory that provides temporary storage for at least some of the program code to reduce the number of times code must be retrieved from the mass storage device during execution. Memory can be of various types, including but not limited to ROM, PROM, EPROM, EEPROM, DRAM, SRAM, flash memory, solid-state memory, etc.

[0268] Input / output (I / O) devices (including but not limited to keyboards, displays, pointing devices, etc.) can be coupled to the system directly or via intermediary wired or wireless I / O interfaces and / or controllers. I / O devices can take many different forms besides traditional keyboards, displays, pointing devices, etc., such as communication devices coupled via wired or wireless connections, including but not limited to smartphones, tablet computers, touchscreen devices, voice recognition devices, etc. Any known or hereafter developed I / O devices are intended to be within the scope of the illustrative embodiments.

[0269] Network adapters can also be coupled to the system to enable the data processing system to be coupled to other data processing systems or remote printers or storage devices via an intermediary private or public network. Modems, cable modems, and Ethernet cards are just some of the currently available types of network adapters for wired communications. Wireless communication-based network adapters can also be utilized, including but not limited to 802.11a / b / g / n wireless communication adapters, Bluetooth wireless adapters, etc. Any known or hereafter developed network adapters are intended to be within the spirit and scope of this invention.

[0270] The description of this invention is presented for purposes of illustration and description and is not intended to be exhaustive or limited to the forms disclosed herein. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. Embodiments were chosen and described in order to best explain the principles of the invention, its practical application, and to enable those skilled in the art to understand the invention with respect to various embodiments having various modifications suitable for the particular purpose contemplated. The terminology used herein was chosen to best explain the principles of the embodiments, their practical application, or technical improvements found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for seed relabeling for seeded slice-wise lesion segmentation of medical images, the method being executable in a data processing system comprising at least one processor and at least one memory including instructions executed by the at least one processor to implement a trained machine learning computer model for seed relabeling for seeded slice-wise lesion segmentation, wherein the trained machine learning computer model performs a method comprising: receiving a lesion mask of a three-dimensional medical image volume, wherein the lesion mask corresponds to detected lesions in the medical image volume, and wherein each detected lesion has a lesion contour; generating a distance map for a given two-dimensional slice in the medical image volume based on the lesion mask, wherein the distance map includes a distance to the lesion contour for each voxel of the given two-dimensional slice; performing local maximum identification to select a set of local maxima from the distance map such that each local maximum has a value greater than its immediate neighbor points; performing seed relabeling based on the distance map and the set of local maxima to generate a set of seeds, wherein each seed in the set of seeds represents a center of a different component of a lesion contour, wherein, performing seed relabeling comprises grouping first and second local maxima in response to determining that the first and second local maxima are immediate neighbors; and performing image segmentation on the lesion mask based on the set of seeds to form a split lesion mask.

2. The method of claim 1, wherein, generating the distance map comprises performing Gaussian smoothing on the distance map.

3. The method of claim 1, wherein, performing seed relabeling comprises: determining a circle centered at each local maximum, the circle having a radius equal to the circle’s corresponding distance in the distance map; computing an overlap measure of a first circle centered at a first local maximum and a second circle centered at a second local maximum; and grouping the first and second local maxima if the overlap measure is greater than a predetermined threshold.

4. The method of claim 3, wherein, the overlap measure is computed as follows: wherein, denotes the area of the first circle, denotes the area of the second circle, and denotes the area of the intersection of the first and second circles.

5. The method of claim 3, wherein, the overlap measure is computed as follows: wherein, denotes the area of the first circle, denotes the area of the second circle, denotes the area of the intersection of the first and second circles, and denotes the area of the union of the first and second circles.

6. The method of claim 1, wherein, performing image segmentation comprises performing a watershed algorithm on the lesion mask based on the set of local maxima to form an initial split lesion mask defining a first set of lesions.

7. The method of claim 6, wherein, performing image segmentation further comprises merging lesions in the first set of lesions based on results of the seed relabeling to form a revised split lesion mask.

8. A computer program product comprising a computer readable storage medium having computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to implement a trained machine learning computer model for seed relabeling for seeded slice-wise lesion segmentation, wherein the trained machine learning computer model performs the steps of the method according to any one of claims 1 to 7.

9. An apparatus for seed relabeling for seeded slice-wise lesion segmentation of medical images, comprising: a processor; and a memory. a memory coupled to the processor, wherein the memory comprises instructions that, when executed by the processor, cause the processor to implement a trained machine learning computer model for seed re-labeling for seed-based slice-wise lesion segmentation, wherein the trained machine learning computer model performs the steps of the method according to any one of claims 1 to 7.

10. A computer system comprising means for performing the steps of the method according to any one of claims 1 to 7.

Images