A method for determining the probability of the presence of at least one candidate lesion in at least one medical image.

Abstract

The present invention relates to a method implemented by computer means for determining the probability of the presence of at least one candidate lesion in at least one medical image, wherein the method is - A step of segmenting the image using a trained segmentation model to obtain at least one segmented region representing the location of at least one candidate lesion in the image, wherein the at least one segmented region is a connected component and is associated with the probability that the connected component is a lesion. - A step of detecting at least one candidate lesion on an image using a trained detection model to obtain at least one bounding box representing the location of a candidate lesion in the image, wherein the at least one bounding box is associated with the probability that the bounding box is a lesion. - A step of matching at least one connected component obtained by the segmentation model to the corresponding bounding box obtained by the detection model, - A step of predicting the probability that at least one connected component obtained by the segmentation model is a lesion, based on both the probability obtained by the segmentation model for the corresponding connected component and the probability obtained by the bounding box detection model that matches the corresponding connected component. This includes an inference stage.

Description

Technical Field

[0001] The present invention relates to a method implemented by computer means for determining the probability of the presence of at least one lesion candidate in at least one medical image.

Background Art

[0002] Over the past decade, many deep learning methods have been proposed for detecting cancerous lesions on medical images. On the other hand, semantic segmentation methods such as U-Net perform voxel-level classification and thus provide an accurate and detailed contour of the lesion to be detected. However, this approach often requires a post-processing step to aggregate voxel predictions into lesion-level predictions and is typically biased towards objects of large size. On the other hand, object detection methods such as the Feature Pyramid Network (FPN) propose to classify boxes within an image and can be applied to medical images to localize and characterize lesions. Box-based detection methods are less sensitive to large variations in the size of the target object but have a fundamental limitation in their ability to accurately describe the localization of lesions. In addition, depending on the imaging modality and clinical application under consideration, both frameworks may result in different levels of performance, thus giving rise to the problem of selecting an appropriate modeling approach.

[0003] Recent research has attempted to bridge the gap between semantic segmentation methods and object detection methods, among which mask R-CNN or Retina U-Net both propose a single architecture capable of segmenting and classifying lesions. To that end, they rely on a cost function that combines box regression, box classification, and segmentation loss. These models are capable of computing voxel-based predictions but are mainly optimized to generate bounding boxes and are evaluated accordingly, with the semantic segmentation component of the total loss acting as a regularization term.

Summary of the Invention

[0004] This specification proposes a method that aims to obtain the benefits of both semantic segmentation methods and box-based object detection methods. [Means for solving the problem]

[0005] For this purpose, this specification proposes a method, implemented by computer means, for determining the probability of the presence of at least one candidate lesion in at least one medical image, the method being (a) A step of segmenting the image using a trained segmentation model to obtain at least one segmented region representing the location of at least one candidate lesion in the image, wherein the at least one segmented region is a connected component and is associated with the probability that the connected component is a lesion; (b) A step of detecting at least one candidate lesion on an image using a trained detection model to obtain at least one bounding box representing the location of a candidate lesion in the image, wherein the at least one bounding box is associated with the probability that the bounding box is a lesion. (c) Matching at least one connected component obtained by the segmentation model to the corresponding bounding box obtained by the detection model, (d) A step of predicting the probability that at least one connected component obtained by the segmentation model is a lesion, also called a confidence score, based on both the probability obtained by the segmentation model for the corresponding connected component and the probability obtained by the bounding box detection model that matches the corresponding connected component. This includes an inference stage.

[0006] Step (a) may be performed concurrently with or before step (b).

[0007] A segmentation model is a type of deep learning model used to segment an image into different regions or categories. For example, when given a 3D image as input, a segmentation model is used to identify and segment different objects or regions within the 3D volume.

[0008] The output of a segmentation model is a mask or set of masks that shows the boundaries of segmented objects or regions in the input image. In the example of a 3D image, the output can also take the form of a 3D volume, with each voxel (volume pixel) having a binary value indicating whether or not it belongs to a segmented object.

[0009] The probabilities in the output indicate the model's confidence in predicting the segmentation of the input image. For each voxel in the output, the probability value indicates the likelihood that it belongs to a segmented object. A higher probability value indicates greater confidence in the model's segmentation prediction for that voxel.

[0010] In the context of segmentation models, connected components refer to regions of pixels or voxels in an image that are connected to one another and share common characteristics, in this case, belonging to the same class, such as a lesion class.

[0011] When a segmentation model is applied to an image, it generates a mask that assigns each pixel or voxel in the image to one of the segmented regions. Each segmented region corresponds to a connected component in the image, which can be further analyzed and processed individually.

[0012] Connected components can be identified and extracted using various algorithms, such as region growing, flood filling, or more commonly, connected component labeling. Once connected components are identified, additional processing can be applied individually to each of them to extract useful information or to perform further analysis.

[0013] A detection model is a type of deep learning model used to detect and locate objects, in this case lesions, within an image. For example, given a 3D image as input, a detection model is used to identify the presence of objects within a 3D volume and to position those objects in space by defining bounding boxes around them.

[0014] The output of the detection model is a set of bounding boxes indicating the location of objects in the input image. In the case of a 3D image, the output is a set of 3D bounding boxes, each representing a rectangular prism containing the detected object (in this case, a candidate lesion). Bounding boxes can be represented by their coordinates and dimensions (width, height, depth) in 3D space.

[0015] Each bounding box can be associated with a probability that represents the model's confidence in object detection predictions. For each bounding box in the output, the probability value indicates the likelihood that an object exists within that box. A higher probability value indicates greater confidence in the model's ability to detect and predict that object.

[0016] The image in question may be a 2D image or a 3D image.

[0017] A 2D image, also known as a two-dimensional image, is a flat image that has width and height dimensions but no depth. On the other hand, a 3D image, also known as a three-dimensional image, is an image that has width, height, and depth dimensions.

[0018] 2D images are composed of a grid of pixels arranged in two dimensions, while 3D images are composed of a grid of voxels arranged in three dimensions.

[0019] Medical images can be defined as visual representations of internal structures or functions, acquired using various imaging modalities. These images are generated using hardware and software systems that capture and process data, producing two-dimensional (2D) or three-dimensional (3D) images that aid in the diagnosis, treatment planning, and monitoring of various medical conditions. Medical images typically contain information about tissue density, composition, and function, and are interpreted by radiologists, physicians, or other healthcare professionals who have received specialized training in image analysis. Medical images can take many different forms, depending on the imaging modality used and the type of medical condition being investigated.

[0020] The medical image in question may be a CT scan image, such as a portal vein CT scan image.

[0021] CT scans, also known as computed tomography scans, are 3D images created by combining a series of X-ray images taken from different angles around the body. CT scans can provide detailed information about the internal structure and composition of organs, bones, and tissues.

[0022] The main difference between portal vein CT scan images and CT scan images lies in the focus of imaging. CT scan images are a type of medical imaging that uses a computed tomography (CT) scanner to produce detailed images of any part of the body. CT scans can be used to evaluate many different organs and tissues, such as the brain, chest, abdomen, and pelvis, and can be performed with or without contrast agents.

[0023] On the one hand, portal CT scan images focus particularly on the imaging of the portal venous system, including the veins that drain blood from the gastrointestinal tract, spleen, and pancreas to the liver. A portal CT scan is a type of CT scan specifically designed to image the portal venous system of the abdomen using a special protocol that optimizes the imaging of this system.

[0024] Portal CT scans can be useful for the diagnosis and monitoring of various conditions that affect lesions in the liver and pancreas, including tumors, infections, and inflammation. They can also be useful for evaluating the response to treatment and guiding further management of the condition. The portal venous system is an important component of the blood supply to the liver and other abdominal organs, and portal CT scans can provide useful information regarding its function and anatomical structure.

[0025] The medical image can be an MRI (magnetic resonance imaging) image.

[0026] MRI images are a type of medical imaging that uses strong magnetic fields and radio waves to create detailed images of the body. MRI is a non-invasive and safe imaging technique that can provide information about the internal structure of the body.

[0027] In medical imaging, a lesion refers to an abnormal or tissue area that appears different from the surrounding tissue in a medical image. Lesions can occur in any part of the body and can be caused by various conditions such as infections, inflammation, tumors, and injuries. [[ID=十七]]

[0028] The term "lesion" is not specific to a particular medical condition or imaging modality and can be used to refer to a wide range of abnormal findings within a medical image. Depending on the context, lesions in a medical image can, among other possible examples, be signs of benign or malignant tumors, infections, autoimmune diseases, or vascular abnormalities.

[0029] In the context of machine learning, the inference phase refers to the stage where a trained model is used to make predictions on new or previously unseen data. During this phase, the model takes in input data and generates outputs based on patterns learned from the training data.

[0030] In other words, the inference stage is the process of applying a trained model to real-world data in order to make predictions or decisions.

[0031] If the image contains multiple candidate lesions, the inference stage is as follows: - A segmentation step in which each of the segmented regions is a connected component and the probability that the connected component is a lesion is associated with the segmentation step, using a segmentation model trained to obtain multiple segmented regions representing the location of the lesion candidate in the image, and each of the segmented regions is a connected component and associated with the probability that the connected component is a lesion. - A step of detecting each or some of the candidate lesions on the image using a trained detection model to obtain multiple bounding boxes representing the location of candidate lesions in the image, wherein each of the bounding boxes is associated with the probability that the bounding box is a lesion. - A step of matching each connected component obtained by the segmentation model to one of the corresponding bounding boxes obtained by the detection model, - A step of predicting the probability that each connected component obtained by the segmentation model is a lesion, based on both the probability obtained by the segmentation model for the corresponding connected component and the probability obtained by the bounding box detection model that matches the corresponding connected component. It may include.

[0032] Different segmentation model architectures can be used. In fact, there are many different segmentation models that can be used for analyzing medical images, including the following: - U-Net: This is a convolutional neural network (CNN) architecture specifically for segmentation tasks in medical imaging. It uses a U-shaped architecture with skip connections to capture fine details and avoid the vanishing gradient problem. - DeepLabv3+: This is another CNN architecture that uses atlas convolution or extended convolution to capture features at different scales. It also incorporates a feature pyramid network to extract features from multiple scales. - Mask R-CNN: This is a CNN architecture that combines object detection and segmentation. It first identifies objects in an image, and then uses a separate network to segment them. - FCN: Fully Convolutional Network (FCN) is a class of CNNs specifically designed for pixel-level segmentation. It uses a series of convolutional layers to generate a segmentation map directly from the input image. - SegNet: This is an encoder-decoder architecture that uses pooling indices to upsample feature maps during decoding. It is designed to be memory efficient and has been successfully used in medical image segmentation tasks. - nnU-Net: This is a variation of U-Net that improves performance by using a nested architecture. It also incorporates data augmentation and ensemble to further improve segmentation accuracy. - PSPNet: Pyramid Scene Parsing Network is a CNN architecture that uses pyramid pooling modules to capture multiscale contextual information. It has been shown to perform well in segmentation tasks for both natural and medical images. - ResUNet++: This is a variation of U-Net that uses residual connections and dense skip connections to improve information flow and gradient propagation. It has been shown to achieve state-of-the-art performance in various medical image segmentation tasks.

[0033] These are just a few examples of the many segmentation models available for medical image analysis.

[0034] The segmentation model and / or detection model may be a U-net model.

[0035] U-Net is a type of convolutional neural network (CNN) architecture named after its U-shaped architecture, which consists of encoder and decoder components. The encoder component consists of convolutional layers that downsample an image to capture its features, while the decoder component consists of transposed convolutional layers that upsample the features to generate a segmentation map.

[0036] The U-Net architecture is designed to work well with limited training data and can produce accurate results even with small training sets.

[0037] This is achieved by using skip connections, which allow the decoder to access high-resolution features from the encoder, enabling the network to reconstruct fine details within the segmentation map.

[0038] This segmentation model could be the nnU-net model.

[0039] The nnU-Net model is an architecture for semantic segmentation tasks that uses a modified version of the U-Net architecture. Such a model is disclosed in [B8].

[0040] The layers of the nnU-Net model can vary depending on the specific implementation and task, but one example implementation may have the following layers: - Shrinkage path: The input image is first passed through a series of convolutional layers with small kernel sizes, increasing the number of feature maps while decreasing the spatial dimension of the image. These layers are often followed by max pooling layers to further reduce the spatial dimension of the feature maps. - Bottleneck: The bottleneck layer is the narrowest part of the network and consists of multiple convolutional layers with small kernel sizes. This layer helps capture the most prominent features of the input image while reducing the number of parameters in the network. - Upsampling Pass: The network's upsampling pass is used to upsample the feature map to the original size of the input image. This consists of a series of convolutional layers with a larger kernel size than the stenosis pass, followed by an upsampling layer to double the spatial dimension of the feature map. - Output Layer: The output layer of the network consists of a kernel-size 1 convolutional layer to generate the final segmentation mask. The output of this layer is typically passed through a softmax function to transform the output into a probability distribution across classes.

[0041] The softmax function is a commonly used activation function in the output layer of neural networks used for multi-class classification problems. The softmax function takes a vector of logits (scores) as input and normalizes them so that the sum of the scores equals 1. This generates a probability distribution across classes, where each element of the vector represents the probability that the input belongs to that class. In the context of semantic segmentation, the output of the softmax function represents the probability that each pixel belongs to a particular class. The class with the highest probability can then be assigned to each pixel to generate the final segmentation mask.

[0042] A segmentation mask is a binary image used to represent segmenting an input image into different regions or objects. Each pixel in the segmentation mask is assigned a value that corresponds to a specific class or object in the input image.

[0043] For example, in semantic segmentation, a segmentation mask assigns a unique label to each pixel in an image that indicates the class of object to which that pixel belongs.

[0044] Therefore, each pixel in the binary segmentation mask can be assigned either a value of 1 (lesion class) or 0 (no-lesion class) based on its predicted class probability.

[0045] During training, the nnU-Net model learns to predict the probability that each pixel belongs to each class based on the input image. The model then applies a threshold to the probability values ​​to generate a binary mask. Pixels with a probability above the threshold may be assigned to the lesion class (1), and pixels with a probability below the threshold are assigned to the no-lesion class (0).

[0046] According to this specification, the probabilities output by the activation function (e.g., softmax function) of the final layer of the segmentation model may be used, for example, as the output of step (a) of the method, i.e., before applying the threshold to obtain the segmentation mask.

[0047] The task of predicting the class label of an object along with its location using bounding boxes and probabilities is known as object detection. Several models can perform this task with high accuracy. Some known models for object detection include: - Faster R-CNN: This model uses a region proposal network to generate object proposals, and then uses a classifier to classify them. - YOLO (You Only Look Once): A real-time object detection model that processes the entire image in a single pass and simultaneously predicts class probabilities and bounding boxes. - SSD (Single Shot Detector): A model that uses a single convolutional network to predict object categories and bounding boxes. - RetinaNet: A model that uses a feature pyramid network to detect objects of different scales and assign appropriate scores to them.

[0048] According to this specification, the detection model may be an nn-Detection model.

[0049] nnDetection is a self-configurable framework for 3D (volume) medical object detection that can be applied to new medical datasets without manual intervention.

[0050] This model consists of a retina U-Net architecture that can automatically adjust its architecture and hyperparameters during training based on a specific task and dataset, without requiring manual tuning by the user.

[0051] nnDetection consists of three stages: data preprocessing, network architecture selection, and hyperparameter optimization. In the data preprocessing stage, input images are preprocessed to highlight features of interest and remove noise. In the network architecture selection stage, a validation set is used to evaluate candidate neural network architectures and select the architecture best suited to the specific medical imaging task. Finally, in the hyperparameter optimization stage, the hyperparameters of the selected neural network are optimized using a search algorithm to further improve the performance of the object detection model.

[0052] Such a model is disclosed in [B9].

[0053] In step (a), all pixels or voxels in the image that have a probability below a threshold can be removed from the segmented region.

[0054] The threshold may be equal to, for example, 0.1 for the probability that the value falls between 0 and 1.

[0055] If all pixels or voxels have a probability below the threshold, the output of the method according to this specification may indicate that no lesion candidate is identified.

[0056] In step (a), the probabilities of all pixels or voxels in each segmented region may be averaged, and the average value may be assigned as a single probability to the corresponding connected component.

[0057] In step (b), each bounding box containing fewer than a threshold number of pixels or voxels may be removed.

[0058] For example, the threshold number may be between 5 and 30, or it may be equal to, for example, 10.

[0059] In step (b), if multiple bounding boxes overlap, the overlapping bounding box with the highest probability may be selected.

[0060] In step (c), the matching is performed. - Determine the bounding box surrounding the linked component, - Select the bounding box obtained in step (b) that has the greatest overlap with the surrounding bounding box, for example, the largest intersection over union (IoU). This can be done by [method].

[0061] This matching method is sometimes called spatial matching.

[0062] In step (c), matching may be performed by selecting the bounding box obtained in step (b) that overlaps with the connected component and has the highest probability.

[0063] For that purpose, a bounding box surrounding the connected component can be determined. Then, the bounding box obtained in step (b) can be selected that has the highest probability of having an intersection over union with the surrounding bounding box that is strictly greater than 0.

[0064] This matching method is sometimes called maximum score matching because the score is the probability of the bounding box obtained in step (b).

[0065] In step (d), the prediction of the probability that the at least one segmented region is a lesion can be calculated via logistic regression, characterized by the probability obtained by the segmentation model for the corresponding connected component and the probability obtained by the bounding box detection model that matches the corresponding connected component.

[0066] Logistic regression is a type of regression analysis used in binary classification problems where the goal is to predict whether an input belongs to one of two classes (e.g., positive or negative). In the context of two different probabilities from two different models, logistic regression can be used to combine the predictions of the two models into a single output.

[0067] Assume there are two different models, each predicting the probability that an input belongs to one of two classes. Let these probabilities be p1 and p2, where p1 is the probability predicted by the first model and p2 is the probability predicted by the second model. To combine the predictions of the two models, we can use logistic regression with p1 and p2 as input features and a binary class label (positive or negative) as the output.

[0068] In logistic regression, the output is a probability value between 0 and 1 that represents the likelihood that the input belongs to the positive class. The logistic function (also known as the sigmoid function) is used to convert a linear combination of input features into a probability value. P(Y=1|X)=1 / (1+exp(-(b0+b1*p1+b2*p2))) In the formula, P(Y=1|X) is the probability that the input belongs to the positive class, p1 and p2 are input features, b0, b1, and b2 are coefficients of the logistic regression model learned during training, and exp is the exponential function.

[0069] During training, the logistic regression model learns the optimal values ​​for the coefficients b0, b1, and b2 by minimizing a loss function, such as a binary cross-entropy loss. Once the model is trained, it can be used to predict the class label (positive or negative) of a new input by calculating probabilities using the logistic function and applying a threshold (e.g., 0.5) to the probability values. If the probability is greater than the threshold, the input is classified as positive; otherwise, it is classified as negative.

[0070] If, in step (a), the first model outputs a probability map that contains no connected components, but the second model in step (b) outputs a bounding box with associated probabilities, then the final contour of the candidate lesion may be a box. The associated probabilities can then be a linear combination of p1 and p2 calculated during step (d), where p1 is the probability of the model from step (a) and is equal to 0, and p2 is the probability of the model from step (b).

[0071] In step (d), the prediction of the probability that at least one segmented region is a lesion can be calculated via logistic regression using complementary variables as additional features, such as demographics (age, sex, ...), clinical variables (history of cancer, history of abdominal pain, ...), biological variables (serum PSA concentration, bilirubin, ...), genomics (RNA-seq profile, presence of gene mutations, ...), radiomics (lesion volume, mean lesion HU, ...), or any other relevant variables.

[0072] This specification also relates to computer programs that, when executed by a processor, include instructions for carrying out the methods described above.

[0073] This specification also relates to a non-temporary computer-readable recording medium on which a program is recorded, the program being for carrying out the methods described above when the program is executed by a processor.

[0074] Memory for storing at least the instructions of a computer program is known as program memory or instruction memory. This type of memory can store the instructions necessary for the processor to execute a program, along with any associated data.

[0075] For example, several types of memory may be used, including read-only memory (ROM), flash memory, electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), cache memory, and virtual memory.

[0076] The term "non-temporary" indicates that data stored on the medium is retained even after the medium is removed from the computer system or the power is turned off.

[0077] Examples of computer-readable non-temporary storage media include hard disk drives, solid-state drives, optical discs (such as CDs or DVDs), USB drives, and memory cards. These storage devices are commonly used to store software programs, data files, and other digital content that can be accessed and processed by computer systems.

[0078] In contrast, temporary media only store data temporarily and cannot retain the data once removed from the computer system or powered off. Examples of temporary media include computer memory (such as RAM) and cache memory.

[0079] This specification also states, - An input interface for receiving at least one medical image, - Memory for storing at least the instructions of the aforementioned computer program, - A processor that accesses memory to read the instruction and then performs the method described above, - An output interface to provide information based on the prediction in step (d) and This also relates to computer devices, including [specific devices].

[0080] An input interface for receiving images is a hardware or software component that allows a user or system to input images into a computer or other digital device for processing or analysis.

[0081] The input interface may include the following: - File Upload: This is a software interface that allows users to upload image files from their local device to a remote server or application. This is a common method for transferring images to cloud-based services or web applications. - Network protocol: This is a software interface that allows devices to communicate with each other over a network. Images can be sent between devices using network protocols such as TCP / IP or HTTP.

[0082] The input interface may be an interface between a medical imaging machine and a computer. The interface may include a data transfer protocol that enables the transmission of images from the imaging machine to the computer for processing and analysis.

[0083] Some common software or hardware components of medical imaging input interfaces may include the following: - DICOM (Digital Imaging and Communications in Medicine) protocol: This is a widely used protocol for transmitting medical imaging data, including images, annotations, and patient information. DICOM is a medical imaging standard that enables the transfer of images between different devices and systems, regardless of vendor or technology. - PACS (Picture Archiving and Communication System): This is a system for storing, retrieving, and distributing medical images and related data. A PACS may include software for acquiring images from imaging machines, transmitting images over a network, and storing images in a central database. - Network Interface: Medical imaging machines may have a built-in network interface such as Ethernet or Wi-Fi, allowing images to be transmitted directly to a computer or server. Alternatively, images can be transferred via removable media such as USB drives or CDs.

[0084] The output interface may include a display such as a monitor, projector, mobile device screen, or virtual reality headset screen.

[0085] Other features, details, and benefits are shown in the detailed description and diagrams below. [Brief explanation of the drawing]

[0086] [Figure 1] This figure schematically illustrates an example of a computer device according to this specification. [Figure 2] This figure outlines the proposed methods for voxel-based semantic segmentation model prediction and box-based object detection model prediction. [Figure 3] This figure shows the precision-recall and sensitivity-specificity curves calculated on the test set for two baseline methods (nnU-Net for semantic segmentation and nnDetection for object detection) and two proposed ensemble approaches. [Figure 4]This figure shows a qualitative comparison between the proposed ensemble and a standalone baseline method, with the raw output of nnU-Net (first column), the raw output of nnDetection (second column), and the proposed ensemble (last column). [Modes for carrying out the invention]

[0087] The attached drawings contain meaningful colors. This application will be published in black and white, but a color version of the attached drawings has been submitted to the government.

[0088] Figure 1 schematically shows an example of a computer device 1 according to the present invention. This computer device 1 is - Input interface 2, - Memory 3 for storing at least the instructions of a computer program, - A processor 4 that accesses memory 3 to read the instruction and performs the method described herein, - Output interface 5 and It is equipped with.

[0089] The method may include a training phase and an inference phase.

[0090] Prior to the training phase, data may be collected and processed to train the segmentation model and the detection model, respectively.

[0091] Image data This section describes an example of such data collection, and different datasets are utilized for three exemplary clinical applications considered: the detection of lesions in the patient's liver, pancreas, and prostate.

[0092] liver An internal database of 1975 portal-phase computed tomography (CT) scans collected from multiple healthcare institutions is divided into 80% / 20% ratios to form training and validation sets. The 1975 cases included patients with a variety of liver lesions, ranging from benign cysts or granulomas to malignant hepatocellular carcinoma or metastases. By aggregating CT scans from the publicly available LiTS[B1], IRCAD[B2], and CHAOS[B3] databases, a separate test set of 65 cases is constructed, consisting of 45 liver cancer patients and 20 healthy liver donor candidates. Twenty LiTS cases are discarded due to corrupted image metadata. Furthermore, to focus evaluation on the most difficult targets to detect, only cases containing lesions less than 2 cm in diameter may be included in the test set. Manual volume lesion segmentation maps are available for all collected images.

[0093] pancreas A total of 2134 portal vein CT scans from an internal database are used for training. Images are collected from multiple manufacturers and healthcare institutions. Of the 2134 patients, 1692 had pancreatic lesions and 442 were healthy controls. Each scan was reviewed by a radiologist, and the pancreas, and lesions where visible, were systematically contoured. The data is divided into training and validation sets of 1707 and 427 cases, respectively. Independent test datasets are constructed relying on publicly available data from the Medical Decathlon Challenge [B4] and The Cancer Imaging Archive [B5]. This yielded a total of 361 independent test cases, of which 281 had pancreatic lesions segmented by radiologists and 80 were healthy control subjects.

[0094] prostate A total of 1658 cases will be used for training, validation, and testing from the publicly available PI-CAI[B6] and Prostate158[B7] databases. The study is guaranteed to be multicenter and multi-maker. Available input MRI modalities are T2-weighted (T2w) images, maximum b-value diffusion-weighted imaging (DWI), and apparent diffusion coefficient (ADC) maps. Of the 1658 cases, 509 are positive patients (425 from PI-CAI and 84 from Prostate158), i.e., patients with at least one clinically significant lesion with ISUP ≥ 2. The other cases are negative, i.e., benign tissue or low-grade cancer. The data will be split into training, validation, and test sets of 1398, 100, and 160 cases, respectively.

[0095] Training Stage - Ensemble Method An example of a training phase involves different steps, as shown in Figure 1. During training, ensemble techniques are employed to combine multiple models to improve predictive performance. The idea behind such techniques is that combining multiple models can lead to more accurate and robust predictions than any single model alone.

[0096] Training voxel-based semantic segmentation models For each clinical application (liver, pancreas, prostate), a deep convolutional network that segments the organ of interest and potential lesions is learned during the training phase from the aforementioned training data or training set. Different input modalities are used depending on the clinical task, such as portal vein CT scans for liver and pancreatic cancers, and T2w (T2-weighted), ADC (apparent diffusion coefficient), and DWI (diffusion-weighted imaging) MRI modalities for prostate cancer.

[0097] The segmentation model may be a convolutional model, such as a U-net model. The final layer of the segmentation model may include a softmax activation function and thresholding to provide a segmentation mask.

[0098] As described above, in a segmentation model with a softmax activation function in the final layer, the model's output is a probability distribution across different classes for each pixel in the input image. Thresholding is commonly applied to obtain a segmentation mask from the output probabilities. The threshold is a value between 0 and 1 and determines which class each pixel should be assigned to. For example, if the threshold is set to 0.5, any pixel with a probability of 0.5 or greater for a particular class will be assigned to that class, and any pixel with a probability of less than 0.5 for all classes will be considered background.

[0099] The resulting segmentation mask is a binary image in which each pixel is assigned either to a class or to the background. The threshold can be adjusted to achieve a balance between precision and recall, depending on the specific requirements of the segmentation task.

[0100] It should be noted that the choice of activation function in the final layer differs depending on the specific problem being addressed. Softmax is commonly used for multi-class segmentation tasks. However, other activation functions such as sigmoid or tanh may be used for binary segmentation tasks.

[0101] The raw predictive softmax map (probabilities output by the softmax activation function before applying thresholds to obtain a segmentation mask) is post-processed into a probabilistic lesion detection map using the following approach. (i) Set the probability of voxels with a probability below a threshold, for example, 0.1 or less, to 0, and obtain a probability map. The probability map of voxels is a 3D representation of the volume of interest to which each voxel is assigned a probability value. (ii) Decompose the resulting probability map into connected components. (iii) Assign a single lesion level probability to each connected component by averaging the probabilities of all voxels in that connected component.

[0102] Training a box-based object detection model Similar to semantic segmentation models, object detection models are trained for each clinical application. Each detection network receives the same input as the segmentation model. A single output class is defined for target lesions. The detection algorithm returns scored bounding boxes around suspicious areas in the image. Small boxes with fewer than 10 voxels may be removed.

[0103] Spatial and probabilistic ensemble Voxel-based and box-based predictions were ensembled to generate a final detection map in which each detected lesion was defined as a connected component with a single associated probability value.

[0104] This is done in the following two steps. (i) Matching between lesions detected by semantic segmentation models and object detection models, (ii) A combination of the probability of the segmentation model and detection model for the matched lesion.

[0105] Two matching criteria are defined and tested between the connected components obtained by the segmentation model and the detection boxes obtained by the detection model. - Spatial matching: For each connected component, the most fitting bounding box that encloses the connected component is determined, and the bounding box predicted by the detection model that has the maximum overlap (i.e., has the maximum IoU) with the bounding box of the connected component is identified. - Maximum score matching: For each connected component, identify all overlapping boxes and associate them with the box that has the highest predicted probability of lesion.

[0106] After the linked components and box predictions are associated in pairs, the probabilities of the corresponding lesions are combined using a calibrated two-parameter logistic regression model. For each clinical application, the logistic regression model is calibrated (i.e., trained) on a validation set.

[0107] Evaluation of ensemble techniques The evaluation aims to assess the performance of the ensemble technique in order to determine how well the model generalizes to new, unseen data.

[0108] The evaluation involves measuring performance against a set of metrics related to the problem being solved. The evaluation is performed using a separate validation dataset that was not used during training (see validation as defined above).

[0109] Baseline semantic segmentation models and object detection models Lesion segmentation and detection are performed using nnU-Net[B8] and nn-Detection[B9], respectively. Both methods provide a standardized framework for reliably designing and training networks through robust data preprocessing, optimal hyperparameter selection, and large-scale data augmentation. These approaches are adapted to different imaging modalities (CT, magnetic resonance imaging: MRI).

[0110] Metrics The proposed ensemble method was evaluated in patient-level diagnosis and lesion-level detection. Patient-level performance was assessed using the Area Under Receiver Operating Characteristic (AUC-ROC) metric. Lesion-level detection performance was assessed using the Mean Precision (AP) metric. For comparison, nnU-Net and nnDetection were also evaluated.

[0111] Hit criteria between predicted lesion and true value Each predicted lesion is considered true positive if its 3D overlap with the true lesion has an IoU ≥ 0.1. Otherwise, the detected lesion is considered false positive. If multiple detected lesions match the same true lesion, only the lesion with the largest overlap is retained, and the other detected lesions are discarded.

[0112] result The prediction performance is reported in Table 1 below.

[0113] [Table 1] Table 1: Comparison of the proposed method with standalone baseline models (nnU-Net for semantic segmentation and nnDetection for detection). Patient-level ROC-AUC and lesion-level mean precision (AP) are shown. "MS Ensembled" represents the maximum score matching after logistic regression ensemble, and "IoU Ensembled" represents the spatial matching after logistic regression ensemble. The best result in each column is shown in bold, and the runner-up is underlined. ROC: Receiver Operating Characteristic. AUC: Area under the curve.

[0114] Regarding lesion detection performance, spatial ensemble (IoU ensemble) yielded better AP compared to nnU-Net and nnDetection in three organs, with an increase of up to 7.5 points for prostate lesions.

[0115] Sensitivity-specificity curves are shown in Figure 3 for each test set and approach. While maximum score ensemble (MS ensemble) only improved the AUC by 1.6 points for liver cancer, the best operating point, defined as the best trade-off between sensitivity and specificity (bACC in Figure 3), was systematically obtained through the ensemble approach.

[0116] Ensembling resulted in a particularly significant increase in the case of prostate cancer. Prostate lesions visible on MRI are especially difficult for experts to detect. In this situation, the detection of ambiguous lesions may have benefited from the fusion of semantic and detection networks. Improvements in lesion-level performance after ensembling were also observed in liver and pancreatic cancer.

[0117] The precision-recall curve and sensitivity-specificity curve are shown in Figure 3.

[0118] Figure 4 provides a qualitative comparison of the proposed approach with the baseline single method. Two healthy cases (one liver, one prostate) are shown, and in both cases, nnU-net predicted a relatively high probability, while nnDetection proposed a lower probability. Thus, ensemble is more beneficial than relying solely on the output of nnU-Net. Two lesion cases (one pancreas, one prostate) are also shown. For the pancreas, nnU-Net predicted a relatively low probability (p≒0.5), while nnDetection proposed a high probability (p≒0.9). Ensemble increased the confidence associated with the detected lesion. In the prostate case, it can be noted that nnDetection proposed two boxes very close to each other around the lesion in nnU-Net, one with a probability p≒0.6 and the other with p≒0.4. In this case, the final ensemble benefited from a maximum score matching type rather than spatial matching.

[0119] (References) TIFF2026521454000003.tif237166 [Explanation of Symbols]

[0120] 1. Computer devices 2 Input Interfaces 3 memory 4 processors 5 Output Interfaces

Claims

1. A method performed by computer means for determining the probability of the presence of at least one candidate lesion in at least one medical image, wherein the method is (a) A step of segmenting the at least one lesion candidate on the image using a trained segmentation model to obtain at least one segmented region representing the location of the at least one lesion candidate in the image, wherein the at least one segmented region is a connected component and is associated with the probability that the connected component is a lesion, (b) A step of detecting the at least one lesion candidate on the image using a trained detection model to obtain at least one bounding box representing the location of the lesion candidate in the image, wherein the at least one bounding box is associated with the probability that the bounding box is a lesion, (c) Matching the at least one connected component obtained by the segmentation model to the corresponding bounding box obtained by the detection model, (d) A step of predicting the probability that at least one connected component obtained by the segmentation model is a lesion, based on both the probability obtained by the segmentation model for the corresponding connected component and the probability obtained by the detection model for the bounding box that matches the corresponding connected component. A method that includes an inference stage.

2. The method according to claim 1, wherein the segmentation model and / or the detection model is a U-net model.

3. The method according to claim 1 or 2, wherein in step (a), all pixels or voxels of the image having a probability of being less than a threshold are removed from the segmented region.

4. The method according to any one of claims 1 to 3, wherein in step (a), the probabilities of all pixels or voxels in each segmented region are averaged, and the average is assigned as a single probability to the corresponding connected component.

5. The method according to any one of claims 1 to 4, wherein in step (b), each bounding box containing fewer than a threshold number of pixels or voxels is removed.

6. The method according to any one of claims 1 to 5, wherein in step (b), if multiple bounding boxes overlap, the bounding box with the highest probability is selected.

7. In step (c), the matching occurs. - Determine the bounding box surrounding the aforementioned connecting component, - Select the bounding box obtained in step (b) that has the largest intersection over union with the surrounding bounding box. The method according to any one of claims 1 to 6, as performed by...

8. The method according to any one of claims 1 to 6, wherein matching is performed in step (c) by selecting the bounding box obtained in step (b) that overlaps with the connected component and has the highest probability.

9. The method according to any one of claims 1 to 8, wherein in step (d), the prediction of the probability that the at least one segmented region is a lesion is calculated via logistic regression, characterized in that it uses the probability obtained by the segmentation model for the corresponding connected component or segmented region and the probability obtained by the detection model for the bounding box matching the corresponding connected component.

10. The method according to any one of claims 1 to 9, wherein in step (d), the prediction of the probability that the at least one segmented region is a lesion is calculated via logistic regression using complementary variables such as demographics, clinical variables, biological variables, genomics, and radiomics, as additional features.

11. A computer program, which, when executed by a processor, includes instructions for carrying out the method described in any one of claims 1 to 10.

12. A non-temporary computer-readable recording medium on which a program is recorded, wherein the program is for carrying out the method according to any one of claims 1 to 10 when the program is executed by a processor.

13. - An input interface (2) for receiving at least one medical image, - At least a memory (3) for storing instructions for the computer program described in claim 11, - A processor (4) that accesses the memory (3) to read the instruction and then performs the method according to any one of claims 1 to 10, - An output interface (5) for providing information based on the prediction in step (d) and A computer device (1) comprising:

Images