Method and system for computer-aided diagnosis using an ensemble of 2D and 3D neural networks based on medical images

Abstract

A method (100) for generating a machine learning model for characterizing a plurality of regions of interest (ROIs) based on a plurality of 3D medical images and a related method (200) for characterizing a region of interest (ROI) based on at least one 3D medical image are provided. The methods proposed herein aim to provide complementary strategies that enable the classification of ROIs from 3D medical images, which can obtain the advantageous complementary benefits of both 2D CNN and 3D CNN to improve the accuracy of prediction. More precisely, the present disclosure proposes a 2D model that complements the 3D model such that the diagnostic sensitivity / specificity is improved by leveraging complementary concepts.

Description

Technical Field

[0001] The present disclosure is broadly related to methods and systems for computer-aided diagnosis (CAD) based on three-dimensional medical images.

[0002] More particularly, the present invention relates to a machine learning model for characterizing regions of interest (ROIs) in three-dimensional medical images.

Background Art

[0003] Medical images have made it possible to diagnose internal organ diseases without performing invasive procedures. In particular, in 1967, Godfrey Hounsfield conceived of the computed tomography method that enables us today to take 3D medical images of a patient's body. Since then, 3D medical images have been widely used by clinicians to diagnose diseases such as cancer.

[0004] The problem with these 3D images is that they are usually voluminous and require a long time for radiologists to read. For this reason, computer-aided diagnosis (CAD) has become important, particularly in screening programs such as the US National Lung Screening Trial, to support the improvement of radiologists' productivity and diagnostic ability in image interpretation. In the 2012 ImageNet Large Scale Visual Recognition Challenge, deep neural networks outperformed all other conventional models. As a result, it has been recognized that deep learning is far more efficient than the methods used at that time, bringing about a revolution in the image processing community.

Summary of the Invention

Problems to be Solved by the Invention

[0005] The approach based on 2D CNN is generally known to be insufficient for estimating the malignancy of pulmonary nodules because the application of 2D CNN (CADx) for nodule characterization is limited to the 2D space and the incorporation of important contextual and volumetric information is insufficient. To overcome such drawbacks, in U.S. Patent Application Publication No. 2021 / 0225511A1 entitled "Methods and Systems for Improving Cancer Detection Using Deep Learning" and U.S. Patent Application Publication No. 2020 / 0160997A1 entitled "Methods for Detecting and Diagnosing Lung and Pancreatic Cancer from Image Scans", a fully 3D deep neural network architecture designed to detect pulmonary nodules to evaluate the 3D characteristics of nodule shape has been proposed.

[0006] However, 3D CNN has limitations that are much less influential in the case of 2D CNN. For example, 3D CNN requires a much larger dataset (where 2D samples are aggregated) to process and train the neural network, increasing the likelihood of overfitting and inhibiting the neural network from having a sufficiently deep architecture.

[0007] To overcome such drawbacks, in "Venkadesh" (see "Deep Learning for Risk Assessment of Pulmonary Nodules Detected at Low-Dose Screening CT" by Venkadesh et al., published in Radiology, 300(2):438 - 447, August 2021), it has been proposed to use an ensemble of individually trained 2D and 3D CNNs to analyze multiple patches of nodules and characterize the malignancy of lung lesions using a deep model.

[0008] However, in Venkadesh's (2D model and 3D model) combined strategy, the 2D model is composed by modeling a number of independent 2D networks that take different 2D rotation views of each ROI as input (see "Lung nodule detection in CT images: Reduction of false positives using multi-view convolutional networks" by Setio et al., published in IEEE Trans Med Imaging, 35(5):1160 - 1169, May 2016). Medical images generally have high anisotropy, which means that the voxel intervals along the x-axis, y-axis, and z-axis are different from each other. Therefore, as proposed by Venkadesh, when obtaining different rotation views for each ROI, different resolutions occur along different axes, and it is necessary to correct this by a preliminary resampling operation. The main drawback of the resampling operation is that it is based on assumptions about how the images between slices look, and these assumptions may be incorrect and not faithful to the original data.

[0009] Furthermore, it should be noted that in Venkadesh's solution, there are not only problems where information is changed by the resampling operation in the case of particularly small findings (such as initial malignant lung nodules), but also problems where information is lost due to the ROI being partially covered and the relevant information of the ROI required for classification being filtered.

[0010] In fact, since each 2D model of Venkadesh (or Setio) is trained in only one direction (only one rotation view), the effective and available sample size is further divided. Therefore, the number of images in the training dataset is quite limited, an invariant representation of the findings cannot be obtained, and the generalization ability of the model is quite low from a design perspective.

[0011] A desirable approach would be to predict the diagnostic score by continuous analysis of a neural network that represents the information in an invariant way based on as comprehensive information as possible from 3D images.

[0012] Therefore, there is a need for a method for computer-aided diagnosis (CAD) based on three-dimensional medical images that improves the sensitivity / specificity of diagnosis and / or the "area under the curve" (AUC) of the "receiver operating characteristic" (ROC) score.

[0013] More specifically, providing a robust prediction and / or improving the AUC-ROC score of the prediction and / or covering each nodule throughout and / or obtaining an invariant representation of the nodule and / or considering both small and large nodules, a method for computer-aided diagnosis based on three-dimensional medical images is needed. **Means for Solving the Problem**

[0014] According to a first aspect, a method for generating a machine learning model that characterizes a plurality of regions of interest (ROIs) based on a plurality of 3D medical images is disclosed. The method includes: providing the plurality of 3D medical images; localizing a plurality of ROIs within the plurality of 3D medical images and, for each ROI, defining a 3D bounding box by a set of six end coordinates (Xmin, Xmax, Ymin, Ymax, Zmin, Zmax); On the one hand, for each defined 3D bounding box, a 2D preprocessing step for extracting a plurality of Ki 2D patches from each ROI included in the 3D bounding box; training a 2D machine learning model based on the extracted 2D patches; On the other hand, for each defined 3D bounding box, a 3D preprocessing step for extracting at least one 3D patch from each ROI included in the 3D bounding box; Training a 3D machine learning model for characterizing the plurality of ROIs based on the 3D patches extracted for the plurality of ROIs, The 2D preprocessing step includes at least extracting a plurality of Ki 2D patches as consecutive parallel slices with respect to a single predetermined plane, The 2D machine learning model is configured to be trained to characterize a plurality of ROIs based on a plurality of 2D patches extracted for the plurality of ROIs.

[0015] In some examples, each of the plurality of Ki 2D patches is obtained without resampling.

[0016] In some examples, the single predetermined plane is parallel to one of the three planes of the sagittal plane, the coronal plane, and the axial plane, preferably parallel to the plane with the highest resolution, and more preferably parallel to the axial plane.

[0017] In some examples, the 2D preprocessing step further includes, before the step of extracting the plurality of Ki 2D patches, calculating the coordinates of the center of each ROI in the axial view of each bounding box (Xmin, Xmax, Ymin, Ymax, Zmin, Zmax), where each center is defined by the coordinates ((Xmin + Xmax) / 2, (Ymin + Ymax) / 2) for all Z within, for example, [Zmin, Zmax], to extract 2D ROI coordinates. The step of extracting the plurality of Ki 2D patches may include, for example, continuously cutting out a plurality of 2D patches of dimensions P×Q, where P = Q = 64, along the z-axis.

[0018] In some examples, the 2D preprocessing step further includes a step of normalizing the 2D voxel values, and this normalization is performed before or after the step of extracting the plurality of Ki 2D patches.

[0019] In some examples, the 3D preprocessing step is, including the step of extracting at least one 3D patch of each ROI, before the step of extracting at least one 3D patch of each ROI, further including the step of extracting 3D ROI coordinates by calculating the coordinates of the center of each ROI, where the center of each ROI is defined, for example, by the coordinates ((Xmin + Xmax) / 2, (Ymin + Ymax) / 2, (Zmin + Zmax) / 2). The step of extracting at least one 3D patch may include the step of cutting out a 3D patch of dimensions P×Q×R, for example, where P = Q = R = 64, around the center coordinates of each ROI. Alternatively, the 3D preprocessing step may further include the step of normalizing the 3D voxel values, and this normalization is performed before or after the extraction step. Further, the 3D preprocessing step may further include the step of resampling the 3D volume of each 3D image in which the ROI is located before the step of extracting the at least one 3D patch.

[0020] In some examples, the 2D training step includes the initialization step of the 2D machine learning model and the continued training step of the 2D machine learning model. The 2D machine learning model is designed to calculate a diagnostic score based on one 2D patch, and the initialization step may include the step of initializing the weights of the 2D machine learning model based on at least one of a random determination of the weights and predetermined weights. Further, or alternatively, the continued training step i. For each randomly selected mini - batch of all the extracted patches of a plurality of ROIs, calculating a diagnostic score using the 2D machine learning model; ii. For each pre - calculated diagnostic score, calculating an error estimate made by the 2D machine learning model; iii. Updating the weights using a gradient - based method. iv. Repeating steps i. to iii. until a defined stopping condition is reached, thereby achieving a trained 2D machine learning model with optimized weights, may be included.

[0021] In some examples, the 3D training step includes an initialization step of the 3D machine learning model, and a continued training step of the 3D machine learning model. The 3D machine learning model is designed to calculate a diagnostic score based on one 3D patch, and the initialization step may include initializing the weights of the 3D machine learning model based on at least one of a random determination of the weights and predetermined weights. Further, or alternatively, the continued training step includes i. For each randomly selected mini - batch of all the extracted patches of a plurality of ROIs, calculating a diagnostic score using the 3D machine learning model; ii. For each pre - calculated diagnostic score, calculating an error estimate made by the 3D machine learning model; iii. Updating the weights using a gradient - based method; iv. Repeating steps i. to iii. until a defined stopping condition is reached, thereby achieving a trained 3D machine learning model with optimized weights, may be included.

[0022] In some examples, each training step may further be performed on morphological features and / or radiomics features. The morphological features may include at least baseline features. The baseline features are the volume of the at least one anatomical structure, the elongation of the at least one anatomical structure, the minimum size and / or maximum size of the oriented bounding box of the at least one anatomical structure, and the HPI average value of the at least one anatomical structure may include at least one of them.

[0023] In some examples, the method according to the first aspect of the present invention may further include the step of providing clinical information for each patient, and each training step is further performed on the clinical information of the patient.

[0024] According to a second aspect, a method for characterizing a region of interest (ROI) based on at least one 3D medical image is disclosed, the method comprising providing the at least one 3D medical image, and localizing one ROI in the at least one 3D medical image as a candidate ROI, and defining one candidate 3D bounding box for the candidate ROI by a set of six end coordinates (Xmin, Xmax, Ymin, Ymax, Zmin, Zmax), On the one hand, for the candidate 3D bounding box, a 2D preprocessing step for extracting a plurality of candidate 2D patches from the candidate ROI included in the candidate 3D bounding box, and executing a trained 2D machine learning model configured to characterize the candidate ROI, On the other hand, for the candidate 3D bounding box, a 3D preprocessing step for extracting at least one candidate 3D patch from the candidate ROI included in the candidate 3D bounding box, and executing a trained 3D machine learning model configured to characterize the candidate ROI based on the at least one extracted candidate 3D patch, and ensembling a plurality of 2D diagnostic scores (output by executing the 2D machine learning model) provided by the trained 2D machine learning model and a plurality of 3D diagnostic scores (output by executing the 3D machine learning model) provided by the trained 3D machine learning model for each candidate ROI to output one overall score. The 2D preprocessing step includes at least the step of extracting a plurality of K candidate 2D patches as continuous parallel slices with respect to a single predetermined plane. Based on the plurality of extracted candidate 2D patches, the step of executing a trained 2D machine learning model is performed.

[0025] The scoring method achieved by executing the method introduced above takes advantage of the fact that it is based on the extracted 2D slices that are faithful to the original data, and thus does not fail to reflect the facts in this regard.

[0026] Therefore, a method for computer-aided diagnosis based on 3D medical images is proposed, which improves the diagnostic sensitivity / specificity and / or the "area under the curve" (AUC) of the "receiver operating characteristic" (ROC) score.

[0027] More specifically, therefore, providing a robust prediction and / or covering each ROI across the board and / or considering both small-scale ROIs and large-scale ROIs, a method for computer-aided diagnosis based on 3D medical images is proposed.

[0028] In some examples, the trained 2D and 3D machine learning models are trained by executing a method for generating a machine learning model according to the first aspect of the present invention.

[0029] In some examples, the single predetermined plane is parallel to one of the three planes of the sagittal plane, the coronal plane, and the axial plane, preferably parallel to the plane with the highest resolution, and more preferably parallel to the axial plane.

[0030] In some examples, the 2D preprocessing step further includes, before the step of extracting the plurality of K candidate 2D patches, calculating, for each center, the coordinates of the center of the candidate ROI in the axial view of the candidate 3D bounding box (Xmin, Xmax, Ymin, Ymax, Zmin, Zmax) defined by the coordinates ((Xmin + Xmax) / 2, (Ymin + Ymax) / 2) for all Z within, for example, [Zmin, Zmax], thereby extracting the 2D ROI coordinates. The step of extracting the plurality of K candidate 2D patches may include, for example, continuously cutting out a plurality of K candidate 2D patches of dimension P×Q, where P = Q = 64, along the Z axis. Thus, the plurality of Ki 2D patches may include or be composed of a plurality of P×Q 2D patches for each 3D bounding box, preferably, each plurality of P×Q 2D patches is composed of Ki = Zmax - Zmin patches.

[0031] In some examples, the 2D preprocessing step further includes a step of normalizing the 2D voxel values, and this normalization is performed before or after the step of extracting the plurality of K candidate 2D patches.

[0032] In some examples, the 3D preprocessing step includes a step of extracting at least one candidate 3D patch of the candidate ROI Before the step of extracting at least one candidate 3D patch of the candidate ROI, the step further includes extracting 3D ROI coordinates by calculating the coordinates of the center of the candidate ROI, which is defined, for example, by the coordinates ((Xmin + Xmax) / 2, (Ymin + Ymax) / 2, (Zmin + Zmax) / 2). The step of extracting at least one candidate 3D patch may include the step of cropping a candidate 3D patch of dimensions P×Q×R, for example where P = Q = R = 64, around the center coordinates of the candidate ROI. Further, or alternatively, the 3D preprocessing step may further include a step of normalizing the 3D voxel values, which is performed before or after the step of extracting at least one candidate 3D patch of the candidate ROI. Further, or alternatively, the 3D preprocessing step may further include the step of resampling the 3D volume of each 3D image in which the candidate ROI is located before the step of extracting at least one candidate 3D patch of the candidate ROI.

[0033] In some examples, the step of running the trained 2D machine learning model is performed N times based on a plurality of K extracted candidate 2D patches to calculate K candidate diagnostic scores for each of the candidate ROIs each time, thereby achieving K candidate 2D diagnostic scores for each of the N series.

[0034] In some examples, the step of running the trained 3D machine learning model is performed M times based on at least one extracted candidate 3D patch to calculate at least one candidate diagnostic score for each of the candidate ROIs each time, thereby achieving at least one candidate 3D diagnostic score for each of the M series.

[0035] The step of ensembling is The step of pooling N series of at least one candidate 2D diagnostic score for each ROI, which is calculated N times for the plurality of K extracted candidate 2D patches, by max pooling may be included. The step of max pooling includes generating one maximum candidate 2D diagnostic score Sj by selecting, for each series j of the N series of the K candidate 2D diagnostic scores, the diagnostic score having the maximum value from the K candidate 2D diagnostic scores, thereby achieving one candidate maximum 2D diagnostic score for each of the N series. The trained 2D machine learning models used for each series j of the N series are different from each other and preferably have different weights. The step of pooling M series of at least one candidate 3D diagnostic score for each ROI, which is calculated M times for the at least one extracted candidate 3D patch, may be included. Furthermore, the step of ensemble may further include the step of outputting one overall score for each candidate ROI by pooling the N 2D diagnostic scores and the M 3D diagnostic scores together.

[0036] In some examples, the method according to the second aspect of the present invention may further include the step of extracting morphological features of the ROI for the candidate ROI by using a morphological radiomics feature extractor based on a 3D medical image. The morphological features may include baseline features. The baseline features may include the volume of at least one anatomical structure, the elongation of at least one anatomical structure, the minimum size and / or maximum size of the oriented bounding box of at least one anatomical structure, and the HPI average value of at least one anatomical structure and may include at least one of them.

[0037] According to any one of the first aspect or the second aspect of the present invention, the three-dimensional medical image may include a computed tomography (CT) image or a magnetic resonance image (MRI) image.

[0038] A further aspect of the present disclosure relates to a non - transitory computer - readable medium that stores instructions which, when executed by at least one digital processing device, execute at least the steps of the method according to the first aspect of the present invention and / or the steps of the method according to the second aspect of the present invention.

[0039] A further aspect of the present disclosure relates to a computer program product that includes instructions which, when executed by at least one digital processing device, execute at least the steps of the method according to the first aspect of the present invention and / or the steps of the method according to the second aspect of the present invention.

[0040] Other objects, features and advantages of the invention(s) disclosed herein, as well as various embodiments / aspects thereof, may become apparent in light of the following description of some exemplary embodiments.

Brief Description of the Drawings

[0041] Embodiments of the present disclosure are referred to in detail, non - limiting examples of which may be shown in the accompanying drawings. Some of the figures may be in the form of diagrams. For ease of understanding, some elements in the figures may be exaggerated and other elements may be omitted.

[0042] Text (such as legends, notes, reference numbers, etc.) displayed on the drawings is incorporated herein by reference.

[0043]

Figure 1

Figure 2

Figure 3

Figure 4

Modes for Carrying Out the Invention

[0044] "Findings" can be defined as all lesions, sclerosis, nodules, malignant or benign, and anything of clinical interest that can be useful for diagnosis.

[0045] The region of interest (ROI) can be defined as a subset of voxels of a 3D image that covers a specific type of tissue, such as a sub - part of the 3D image, more precisely, an organ, a lesion, sclerosis, nodules, malignant or benign, and anything of clinical interest that can be useful for diagnosis. If the entire image is of interest and is considered relevant to the diagnosis, the ROI can cover the entire 3D image. In the context of the present disclosure, "ROI" can be replaced with the term "findings".

[0046] The bounding box can be defined as the smallest rectangular parallelepiped that contains the ROI. The bounding box can be composed of all the voxels of the 3D image. In the context of the present disclosure, one bounding box contains one ROI. In other words, one ROI corresponds to at least one bounding box.

[0047] Axial planes can be defined as cross - sectional planes perpendicular to the long axis of the body, as in the case of CT scans and MRI scans.

[0048] In the context of the present disclosure, consecutive patches or slices mean that they are adjacent to each other in order. As described above, 2D CNNs and 3D CNNs have different limitations. As a result, the architectures, and even the representations, encoded by 2D and 3D networks for a particular problem have to be thought of in quite different ways.

[0049] However, simply designing a system for any of the models (2D and 3D) and combining 2D CNN and 3D CNN would be insufficient to solve the drawbacks of both models. For example, in Venkadesh's solution for combining 2D and 3D models, it is unable to address the limitations of 3D CNN by the 2D model. Instead, further problems occur, such as the loss and alteration of information caused by its patch extraction method that necessarily includes the resampling operation used in the 3D model, as well as the limitation of the sample size for training the 2D model.

[0050] The disclosure proposed herein aims to provide a complementary strategy that enables the classification of ROIs from 3D medical images and can improve the accuracy of prediction by leveraging the advantages and complementarity of both 2D and 3D CNNs.

[0051] More precisely, the present disclosure proposes a 2D model that complements the 3D model such that the diagnostic sensitivity / specificity is improved by utilizing complementary concepts in at least the following aspects.

[0052] i) 2D preprocessing and 3D preprocessing Generally, the 3D preprocessing module has more information, but due to the above reasons, especially the artificial assumptions made during the resampling operation (by interpolation operation), its accuracy is compromised. Conversely, the 2D preprocessing module has less information but is more faithful to the original data. However, in some 2D models such as Venkadesh and Setio, considering their extraction methods of 2D views, a resampling operation is required during preprocessing, so the fidelity may be compromised by the resampling operation and partial information regarding 3D images.

[0053] The present disclosure enables obtaining as comprehensive information as possible from 3D images through continuous analysis by a neural network, and the neural network represents such information in an invariant manner by extracting, for each ROI in a predetermined plane having a better resolution, a plurality of consecutive parallel slices by means of its 2D model. In this extraction used in the 2D model, no resampling operation is required. Further, in this solution, the entire nodule and nodules of both large and small sizes are considered.

[0054] ii) Diagnosis by a trained 2D model The extraction method proposed by the present disclosure enables extracting a plurality of K 2D patches for one ROI for diagnosis, and when these plurality of K 2D patches are input into a trained 2D model, K diagnostic scores are obtained. As a result, the highest score can be selected from among the K scores, and this highest score is likely to be a cancer (a nodule with the highest risk of malignancy).

[0055] iii) Ensemble of 2D model and 3D model for calculating the overall score, pooling of all scores calculated from the 2D model and 3D model for calculating the overall score Assuming that a low-value score "0" indicates a low risk and a high-value score "1" indicates a high risk, a medium score (e.g., 0.4 - 0.6) indicates that there is uncertainty in the prediction. When there are many high or low values and some intermediate values, in order to reduce uncertainty while emphasizing high or low values, the present invention proposes pooling all scores. However, the pooling method is not limited to averaging the scores, and it may also be beneficial to use other methods for deriving one score from 2D scores and 3D scores (e.g., XGBoost, weighted average, etc.).

[0056] Therefore, the method of the present invention that trains machine learning to characterize ROI and uses a trained machine learning model for the same purpose is advantageous in that it is important to constitute a method of selecting 2D images as 2D model inputs to a 3D volume and a subsequent 2D network structure for further aggregating them into a single volume prediction in order to obtain accurate results while being completely open.

[0057] More specifically, the advantages of the present disclosure are to provide a prediction with robustness and / or to improve the AUC-ROC score of the prediction and / or to cover each ROI across the whole and / or to obtain an invariant representation of nodules and / or it may be to consider both small-scale ROIs and large-scale ROIs.

[0058] FIG. 1 shows a flowchart of an embodiment of the first aspect of the present invention.

[0059] According to a first aspect of the present invention, the present invention provides a method 100 for generating a machine learning model that characterizes a plurality of regions of interest (ROIs) based on a plurality of 3D medical images.

[0060] The application of ROI characterization can be the classification of the severity of a disease, the determination of whether an ROI is benign or malignant based on a decisive score or other factors, or the determination of a pathological state. Although the application of the present disclosure is described in the context of calculating a diagnostic score (such as a malignancy score), it will be recognized that it can be extended to applications other than diagnostic scores, such as the classification of the severity of a disease.

[0061] Although the present application is disclosed in the context of liver fibrosis and pulmonary nodules, it should be recognized that the diseases in the context of the present disclosure can occur in defined anatomical structures and are not limited to tumors, whether cancerous or not, and are not limited to the liver or the lungs.

[0062] Method 100 includes step 105 of providing the plurality of 3D medical images, step 107 of localizing a plurality of ROIs within the plurality of 3D medical images, and for each ROI, step 108 of defining a 3D bounding box by a set of six end coordinates (Xmin, Xmax, Ymin, Ymax, Zmin, Zmax).

[0063] The 3D medical images provided in step 105 may include computed tomography (CT) images or magnetic resonance imaging (MRI) images.

[0064] In the context of the present disclosure, one 3D image may include zero, one, or a plurality of ROIs. When a plurality of 3D medical images are provided, a plurality of ROIs may be localized in step 107.

[0065] The ROI localization step 107 may be implemented automatically, for example, by CADe, or manually, for example, by a clinician.

[0066] In some examples, the ROI localization step 107 may include the step of segmenting at least one identified anatomical structure of a patient in each of the plurality of 3D medical images and the step of defining an implied mask of the at least one anatomical structure as the ROI.

[0067] In some examples, the ROI localization step 107 may be advantageously limited to the at least one determined anatomical structure. The method 100 may further include a step of determining whether the localized ROI is within the at least one determined anatomical structure. It may be contemplated to calculate the ratio between the volume of the intersection of the bounding box and the lung mask and the volume of the bounding box to determine whether the localized ROI is inside the at least one determined anatomical structure. In some examples, when the ratio is 0.5 or more, the localized ROI is considered to be inside the determined anatomical structure. The ratio may vary depending on the anatomical structure, the application of the localization, and other factors. In some examples, the method may further include a step of retaining only the localized ROIs determined to be inside the determined anatomical structure.

[0068] On the other hand, the method 100 further includes for each defined 3D bounding box, a 2D preprocessing step 110 for extracting a plurality of 2D patches from the ROIs included in the 3D bounding box, and a step 130 of training a 2D machine learning model based on the plurality of extracted 2D patches.

[0069] On the other hand, the method 100 further includes for each 3D bounding box, a 3D preprocessing step 120 for extracting at least one 3D patch from each ROI included in the 3D bounding box, and a step 150 of training a 3D machine learning model for characterizing the plurality of ROIs based on the 3D patches extracted for the plurality of ROIs.

[0070] As an example, FIG. 4 shows the ROI within the bounding box. In the context of the present disclosure, each bounding box contains one ROI. For ease of understanding, the bounding box is shown as a set of six end coordinates (Xmin, Xmax, Ymin, Ymax, Zmin, Zmax). It should be recognized that other coordinates can be used as long as they are applicable to the 3D X, Y, Z coordinate system.

[0071] Advantageously, for each defined 3D bounding box, method 100 includes step 115 where a 2D preprocessing step 110 extracts a plurality of Ki 2D patches (for each ROI i) as consecutive parallel slices with respect to at least a single predetermined plane.

[0072] Each 2D patch of the plurality of Ki 2D patches is obtained without a resampling operation. According to this method, the extracted 2D patches can reflect the facts without only covering the integrity of the original data but also providing information changed by the resampling operation.

[0073] Preferably, the single predetermined plane is parallel to one of the three planes: the sagittal plane, the coronal plane, and the axial plane. More preferably, the single predetermined plane is parallel to the plane with the best resolution. Even more preferably, the single predetermined plane is parallel to the axial plane.

[0074] FIG. 3 shows the anatomical planes and the body axes (X-axis, Y-axis, Z-axis).

[0075] Referring to FIG. 3, the sagittal plane (X-plane), the coronal plane (Y-plane), and the transverse plane (Z-plane) are shown. Each of these three planes contains two of the three coordinate axes (X, Y, Z) of the three-dimensional coordinate system. For example, the plane containing the X-axis and the Y-axis is the axial plane called the XY plane (not labeled in the figure) or the Z-plane. As shown in FIG. 3, the axial plane is perpendicular to the long axis (Z-axis), i.e., the backbone of the body (head, neck, torso, tail). In the context of the present disclosure, the axial plane is interchangeable with the transverse plane. The sagittal plane runs from front to back, dividing the body into right and left sides. The coronal plane runs from side to side, dividing the body into front and back parts. In some examples, the 2D slices are extracted with respect to the same predetermined plane. In still other examples, the same predetermined plane is parallel to the axial plane at the highest resolution. The ROI is extracted in a plane at the best resolution, but in the context of the present disclosure, it can be understood that the ROI can be better represented by an axially viewed patch.

[0076] This approach enables consideration of the morphological features of the nodules (ROIs), which can be particularly advantageous in the case of small spiculated nodules. Furthermore, this approach enables consideration of the size of the nodules and can cover the entire nodule by extraction, which can be particularly advantageous in the case of large nodules.

[0077] According to this method, the extracted 2D patches not only cover the integrity of the original data, but since the extracted 2D patches are obtained without a resampling operation, they can also reflect the facts. By extracting 3D medical images used for training 2D machine learning models as comprehensively and faithfully as possible, the AUC-ROC score of the model is improved.

[0078] The 2D preprocessing step 110 may further include, before the step 115 of extracting the plurality of Ki 2D patches, calculating the coordinates of the center of each ROI in the axial view of each bounding box (Xmin, Xmax, Ymin, Ymax, Zmin, Zmax), where each center is defined by the coordinates ((Xmin + Xmax) / 2, (Ymin + Ymax) / 2) for all Z within, for example, [Zmin, Zmax], to extract the 2D ROI coordinates. Thus, for each ROI, the plurality of Ki 2D patches can be configured as Ki = Zmax - Zmin + 1 patches.

[0079] More specifically, the step 115 of extracting the plurality of Ki 2D patches may include, for example, the step of continuously cutting out a plurality of 2D patches of dimensions P×Q with P = Q = 64 along the Z-axis. Therefore, the plurality of Ki 2D patches are each composed of a plurality of Ki 2D patches of dimensions P×Q for each 3D bounding box.

[0080] Optionally, the 2D preprocessing step 110 may further include a first step of normalizing the 2D voxel values. This first normalization step may be performed before or after the step 115 of extracting the plurality of Ki 2D patches, and / or may be related to the normalization of the luminance attenuation (HU) of the relevant pixels or voxels. More specifically, the first step of normalizing the 2D voxel values may include truncating the voxel (or pixel, since we are talking about 2D patches) values between two defined minimum and maximum values, and then applying a defined affine function to the values of the voxels of the 2D patch for transformation. If step 115 is performed before the first normalization step, step 112 may only be applied to the 3D image form from which the Ki 2D patches are extracted in step 115. If the first normalization step is performed before step 115, it is necessary to normalize the luminance attenuation (HU) of all 2D patches.

[0081] Therefore, following the step 115 of extracting the plurality of Ki 2D patches, a plurality of normalized Ki 2D patches may be obtained, which may be used in the 2D training step 130 of the 2D machine learning model.

[0082] As described above, more advantageously, the method 100 trains a 2D machine learning model based on the plurality of Ki 2D patches extracted for a plurality of ROIs, that is, trains a 2D machine learning model using the plurality of 2D patches. According to this method, the sampling size is increased, thereby improving the accuracy of the trained machine learning model and also improving the robustness of the model. In particular, the synergistic effect of the combination with a specific method in which the 2D preprocessing step 110 is executed results in a further improvement in the accuracy of the trained model.

[0083] The 2D training step 130 may include the initialization step 135 of the 2D machine learning model, and the continued training step 140 of the 2D machine learning model.

[0084] In some examples, the initialization step 135 may include initializing the weights of the 2D machine learning model based on at least one of a random determination of the weights and predetermined weights.

[0085] The continued training step 140 includes i. For each randomly selected mini-batch of all the extracted patches of a plurality of ROIs, calculating a diagnostic score using the 2D machine learning model; ii. For each pre-calculated diagnostic score, calculating an error estimate performed by the 2D machine learning model; iii. Updating the weights using a gradient-based method; iv. Repeating the steps of calculating the diagnostic score, calculating the error estimate, and updating the weights until a defined stopping condition is reached. When the defined stopping condition is reached, the training step ends.

[0086] In some examples, the stopping condition may be reached in at least one of the following situations, but this list is not exhaustive. 1) After a certain number of trainings, 2) After reaching a certain performance of the model, 3) If the performance does not change after a certain number of iterations.

[0087] In this way, a trained 2D machine learning model with optimized weights is achieved.

[0088] To perform the continuous training step 140, more specifically, it can use the mini-batch gradient descent method, which is a variant of the gradient descent algorithm that divides the training dataset into small batches used to calculate the model error and update the model coefficients. In some examples, the continuous training of the 2D machine learning model can be based on mini-batches.

[0089] Note that among the known gradient-based methods, Adam, RMSprop, and SGD are methods that can be used for each step of weight update described herein.

[0090] In parallel with the method of achieving a trained 2D machine learning model with optimized weights, a method of achieving a trained 3D machine learning model with optimized weights will be described below.

[0091] The aforementioned 3D preprocessing step 120 for extracting at least one 3D patch from each ROI included in the 3D bounding box includes a step 125 of extracting at least one 3D patch of each ROI, and may further include a step 121 of extracting 3D ROI coordinates by calculating the coordinates of the center of each ROI before the step 125 of extracting at least one 3D patch of each ROI. The center of each ROI can be defined, for example, by the coordinates ((Xmin + Xmax) / 2, (Ymin + Ymax) / 2, (Zmin + Zmax) / 2).

[0092] More specifically, step 125 of extracting at least one 3D patch may include a step of cutting out a 3D patch of dimensions P×Q×R, for example P = Q = R = 64, around the center coordinates of each ROI.

[0093] Similar to the 2D preprocessing step 110, the 3D preprocessing step 120 may further include a 3D voxel value normalization step 123.

[0094] The 3D voxel value normalization step 123 may be executed before or after the extraction step 125. More specifically, the 3D voxel value normalization step 123 may include clipping of voxel values between two defined minimum and maximum values, followed by transformation by applying a defined affine function to the voxel values of the 3D patch.

[0095] In contrast to the 2D preprocessing step 110, the 3D preprocessing step 120 may include a step 122 of resampling the 3D volume of each 3D image where the ROI was localized in step 107.

[0096] In some examples, step 122 may be executed before the step 125 of extracting the at least one 3D patch. As described above, the step 122 of resampling the 3D volume may be required to solve problems caused by potential image anisotropy and may be configured to achieve equidistant spacing. In some examples, a resolution of 0.65mm×0.65mm×0.65mm may be achieved for the X-axis, Y-axis, and Z-axis respectively.

[0097] To increase the sample size for 3D machine learning training, the cut-out 3D patch may be further processed to generate a plurality of 3D patches. In some examples, multiple patches can be cut out for one ROI. In one example, the center of the 3D patch can be slightly shifted. In another example, the image can be rotated.

[0098] Accordingly, following step 125 of extracting at least one 3D patch, for a plurality of located ROIs, a plurality of normalized and resampled 3D patches can be achieved, and the plurality of normalized and resampled 3D patches are usable for the 3D training step 150 of the 3D machine learning model.

[0099] Similar to the 2D training step 130 of the 2D machine learning model, the 3D training step 150 of the 3D machine learning model may include an initialization step 155 of the 3D machine learning model and a continued training step 160 of the 3D machine learning model.

[0100] In some examples, the initialization step 155 may include the step of initializing the weights of the 3D machine learning model based on the random determination of the weights or based on predetermined weights.

[0101] The continued training step 160 i. For each randomly selected mini-batch of all the extracted patches of a plurality of ROIs, calculating a diagnostic score using the 3D machine learning model; ii. For each pre-calculated diagnostic score, calculating an error estimate performed by the 3D machine learning model; iii. Updating the weights using a gradient-based method; iv. Repeating the steps of calculating the diagnostic score, calculating the error estimate, and updating the weights until a defined stop condition is reached. Thereby, a trained 3D machine learning model with optimized weights is achieved.

[0102] In the context of the present disclosure, it should be understood that the examples and detailed descriptions provided above with respect to the continuous training step 140 of the 2D machine learning model are also applicable to the continuous training step 160 of the 3D machine learning model.

[0103] At this stage, therefore, as a result of method 100 for generating the machine learning model, a trained 2D machine learning model with optimized weights and a trained 3D machine learning model with optimized weights are achieved.

[0104] In some examples, method 100 may further include performing at least one of training step 130 and training step 150, which is performed based on morphological features and / or radiomics features by extracting morphological features of the ROI (based on the 2D and / or 3D patches) and using a morphological radiomics feature extractor. The morphological features include at least baseline features, and the baseline features are, for example, the volume of the at least one anatomical structure, the elongation of the at least one anatomical structure, the minimum size and / or maximum size of the oriented bounding box of the at least one anatomical structure, and the HPI average value of the at least one anatomical structure including at least one of them.

[0105] Method 100 may further include providing clinical information regarding each patient, whereby training step 130 and / or 150 may be further performed with respect to the clinical information of the patient.

[0106] By using supplementary features such as morphological features and / or radiomics features to train at least one of the 2D model and the 3D model, it is possible to enhance the robustness of the model.

[0107] It should be appreciated that the 2D machine learning models and / or 3D machine learning models according to the present disclosure may be based on deep neural networks, which may be convolutional neural networks, and vice versa.

[0108] 2 shows a flowchart of an embodiment of a second aspect of the present invention, according to which the present invention provides a method 200 for characterizing a region of interest (ROI) based on at least one 3D medical image by applying a trained machine learning model.

[0109] Method 200 is providing 205 said at least one 3D medical image; a step 207 of localizing one ROI in said at least one 3D medical image as a candidate ROI; Includes.

[0110] Steps 205 and 207 may be further elaborated in the same manner as steps 105 and 107 detailed above.

[0111] In some examples, one ROI is localized each time for characterization. The localization step 207 can be repeated iteratively to localize one new ROI each time for characterization.

[0112] The method 200 further comprises: For the candidate ROI, step 208 includes defining one candidate 3D bounding box by one set of six edge coordinates (Xmin, Xmax, Ymin, Ymax, Zmin, Zmax).

[0113] Meanwhile, the method 200 further comprises: a 2D pre-processing step 210 for extracting, for a candidate 3D bounding box, a plurality of candidate 2D patches from the candidate ROI contained in the candidate 3D bounding box; Including step 230 of executing a trained 2D machine learning model configured to characterize one ROI, which is a candidate ROI.

[0114] The trained 2D machine learning model can be trained by executing method 100 for generating a machine learning model according to the first aspect of the present invention. On the other hand, method 200 further includes For the candidate 3D bounding box, a 3D preprocessing step 220 for extracting at least one candidate 3D patch from the candidate ROI included in the candidate 3D bounding box, and Based on the at least one extracted candidate 3D patch, including step 250 of executing a trained 3D machine learning model configured to characterize one ROI, which is a candidate ROI here.

[0115] The trained 3D machine learning model can be trained by executing method 100 for generating a machine learning model according to the first aspect of the present invention.

[0116] Advantageously, in method 200, the 2D preprocessing step 210 includes at least step 215 of extracting a plurality of K candidate 2D patches as continuous parallel slices with respect to a single predetermined plane.

[0117] Each 2D patch of the plurality of K 2D patches is obtained without resampling. According to this method, the extracted 2D patches can not only cover the integrity of the original data, but also reflect the facts without providing information changed by the resampling operation.

[0118] Similar to method 100, a single predetermined plane is parallel to one of the three planes: the sagittal plane, the coronal plane, and the axial plane. More preferably, the single predetermined plane is parallel to the plane with the highest resolution. Even more preferably, the single predetermined plane is parallel to the axial plane. Therefore, the candidate ROI can be better reflected by the patches viewed axially. According to this method, not only can small (spiny) ROIs be accurately diagnosed considering the morphological characteristics of the ROI, but large ROIs can also be accurately diagnosed considering the size of the ROI.

[0119] Therefore, not only does the extracted 2D patch cover the integrity of the original data, but since the extracted 2D patch is obtained without a resampling operation, it can also reflect the facts. By extracting the 3D medical image input into the machine learning model to characterize the ROI as comprehensively and faithfully as possible, the accuracy of characterization (prediction) is improved.

[0120] The 2D preprocessing step 210 may further include a step of extracting 2D ROI coordinates by calculating the coordinates of the center of the candidate ROI in the axial view of the candidate 3D bounding box (Xmin, Xmax, Ymin, Ymax, Zmin, Zmax) before the step 215 of extracting the plurality of K candidate 2D patches. Each center can be defined, for example, by the coordinates ((Xmin + Xmax) / 2, (Ymin + Ymax) / 2) for all Z within [Zmin, Zmax]. Further, the step 215 of extracting the plurality of K candidate 2D patches may include a step of continuously cutting out a plurality of K candidate 2D patches of size P×Q with, for example, P = Q = 64 along the Z-axis.

[0121] Optionally, the 2D preprocessing step 210 may further include a second step of normalizing the 2D voxel values. This second normalization step may be performed before or after the step 215 of extracting the plurality of K candidate 2D patches, and / or may be related to the normalization of the luminance attenuation (HU) of the related pixels or voxels. More specifically, the second step of normalizing the 2D voxel values may include truncating the voxel (or pixel, since we are talking about 2D patches) values between two defined minimum and maximum values, and subsequent transformation by applying a defined affine function to the values of the voxels of the candidate 2D patches. If step 215 is performed before the second normalization step, the second normalization step may only be applied to the candidate 3D image form in which the K candidate 2D patches are extracted in step 125. If the second normalization step is performed before step 215, it is necessary to normalize the luminance attenuation (HU) of all candidate 2D patches.

[0122] Accordingly, following the step 215 of extracting the plurality of K candidate 2D patches, a plurality of normalized and resampled candidate 2D patches may be achieved, and the normalized and resampled candidate 2D patches are usable for executing 230 the trained 2D machine learning model. The aforementioned 3D preprocessing step 220 for extracting at least one candidate 3D patch from the candidate ROI is may include a step 225 of extracting at least one candidate 3D patch of the candidate ROI, Before the step 225 of extracting at least one candidate 3D patch of the candidate ROI, it may further include a step of extracting 3D ROI coordinates by calculating the coordinates of the center of the candidate ROI. The center of the candidate ROI may be defined, for example, by the coordinates ((Xmin + Xmax) / 2, (Ymin + Ymax) / 2, (Zmin + Zmax) / 2).

[0123] More specifically, the step 225 of extracting at least one candidate 3D patch may include a step of cropping a candidate 3D patch of dimensions P×Q×R, where for example P = Q = R = 64, around the center coordinates of the candidate ROI.

[0124] Similar to the 3D preprocessing step 120 according to the first aspect of the present invention, the 3D preprocessing step 220 according to the second aspect of the present invention may further include a step 223 of normalizing the 3D voxel values of the at least one 3D image in which the candidate ROI was localized in step 207 and / or a step of resampling the 3D volume.

[0125] The step 223 of normalizing the 3D voxel values may be executed before or after the extraction step 225. More specifically, the step 223 of normalizing the 3D voxel values may include clipping the voxel values between two defined minimum and maximum values, and then applying a defined affine function to the values of the voxels of the candidate 3D patch for transformation.

[0126] The step of resampling the 3D volume of the at least one 3D image in which the candidate ROI was localized in step 207 may be executed before the step 225 of extracting the at least one candidate 3D patch. The resampling step 122 of the 3D volume may be configured to achieve an isometric interval and / or a resolution of 0.65 mm×0.65 mm×0.65 mm.

[0127] Therefore, following the step 225 of extracting at least one candidate 3D patch, a normalized and resampled candidate 3D patch may be achieved, and the normalized and resampled candidate 3D patch can be used in the step 250 of executing the trained 3D machine learning model.

[0128] Furthermore, the method 200 also includes Step 270 of ensembling a plurality of 2D diagnostic scores provided by executing the trained 2D machine learning model and a plurality of 3D diagnostic scores provided by executing the trained 3D machine learning model to output one overall score for each of the candidate ROIs is included.

[0129] Next, step 270 of ensembling potentially includes step 271 of pooling N series of at least one candidate 2D diagnostic score of each ROI, calculated N times for the plurality of K extracted candidate 2D patches, by max pooling; and step 272 of pooling M series of at least one candidate 3D diagnostic score of each ROI, calculated M times for the at least one extracted candidate 3D patch, thereby utilizing multiple executions of the trained 2D and 3D machine learning models. In some examples, the step of max pooling further includes for each series j of the N series of the K candidate 2D diagnostic scores, generating one maximum candidate 2D diagnostic score Sj by selecting the diagnostic score having the maximum value from the K candidate 2D diagnostic scores, thereby achieving one candidate maximum 2D diagnostic score for each series of the N series, and the trained 2D machine learning models used for each series j of the N series are different from each other and preferably have different weights.

[0130] It is advantageous to have a plurality of K 2D patches extracted and input to the 2D machine learning model, which enables obtaining a number of outputs such as diagnostic scores, and the highest score corresponding to the ROI of greatest interest for diagnosis can be selected (max pooling). For example, the highest score may mean that in a particular case, the ROI is likely to be diagnosed as a malignant tumor or cancer.

[0131] Using different 2D machine models with different weights is even more advantageous because it can improve the robustness of the prediction.

[0132] The step 270 of ensembling may further include a step 275 of outputting one overall score for each candidate ROI by pooling together the N 2D diagnostic scores of step 271 and the M 3D diagnostic scores of step 272. Thus, one overall score is achieved for each candidate ROI.

[0133] As described above, pooling is particularly advantageous in the context of the present disclosure as it allows for emphasizing high or low values while reducing uncertainty. However, the pooling method is not limited to averaging the scores, as it may also be beneficial to use other methods (e.g., XGBoost, weighted average, etc.) for deriving one score from the 2D and 3D scores.

[0134] To further enhance the robustness of method 200, the latter may further include a step 260 of extracting morphological features of the candidate ROI for the candidate ROI, based on the 3D medical image, using a morphological radiomics feature extractor. Next, the step 270 of ensembling may include ensembling the N 2D diagnostic scores from the 2D model, the M 3D diagnostic scores from the 3D model, and at least one of the extracted morphological and / or radiomics features 260. The morphological features relevant here may be detailed in the same way as those detailed above in relation to method 100 according to the first aspect of the invention.

[0135] Using morphological features and / or radiomics features in addition to 2D and 3D diagnostic scores to characterize candidate ROIs will further strengthen the robustness of the prediction.

[0136] The embodiments and aspects disclosed in detail herein are not limiting, but may be described and illustrated in conjunction with systems, devices, and methods that are exemplary and explanatory. Specific configurations and details may be shown so as to enable understanding of the disclosure. However, it should be apparent to those skilled in the art that the disclosure may be practiced without some of the specific details presented herein. Further, for clarity of explanation, some of the well-known steps and components may be presented only in general terms or may be omitted.

[0137] Some procedures may be presented and described in a series of (consecutive) steps. The order of the steps is exemplary, and the steps may be executed in an order different from the presented order, some of the described steps may be omitted, some additional steps may be omitted from the order, and it should be understood that they may be described elsewhere.

[0138] References may be made to the disclosures of prior patents, publications, and applications. Some of the text and drawings from these sources may be presented herein, but may be modified, edited, or commented on to better harmonize with the disclosure of this application. No citation or identification of any reference should be construed as an admission that such reference is available as prior art to the disclosure.

[0139] The methods described herein can be executed on any form of computer(s). These components can be executed as a dedicated application or in a client-server architecture including a web-based architecture, and can include functional programs, code, and code segments. Any computer can include a processor, a memory for storing and executing program data, a persistent storage such as a disk drive, a communication port for handling communication with external devices, and a user interface device including a display, keyboard, mouse, etc. When some software or algorithms are included, they can be stored as program instructions executable by a processor or computer-readable code on a computer-readable medium such as a read-only memory (ROM), random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. The computer-readable recording medium can also have the computer-readable code stored in a distributed form and be distributed on a computer system connected to a network so as to be executed. This medium can be read by a computer, stored in the memory, and executed by the processor.

[0140] For the purpose of facilitating understanding of the principles of the various embodiments of the present disclosure, reference is made to the preferred embodiments shown in the drawings, and a specific language is used to describe this embodiment. However, it is not intended that the scope of the disclosure be limited by this specific language, and the disclosure should be construed to include all embodiments that would occur to those of ordinary skill in the art.

[0141] The present disclosure may be described from the perspective of components of functional blocks and various processing steps. Such functional blocks may be implemented by any number of hardware and / or software components configured to perform a particular function. For example, the present disclosure may employ various integrated circuit components such as memory elements, processing elements, logic elements, look-up tables, etc., that may execute various functions under the control of one or more microprocessors or other control devices. Similarly, where elements of the present disclosure are executed using software programming or software elements, the present disclosure may be implemented using any programming language or scripting language such as C, C++, Java, Python, assembler, etc., and various algorithms may be implemented using any combination of data structures, objects, processes, routines, or other programming elements. Functional aspects may be implemented by algorithms executed on one or more processors. Further, the present disclosure may employ any number of conventional techniques for the configuration of electronic devices, signal processing and / or control, data processing, and the like.

Claims

1. A method (100) for generating a neural network to characterize multiple regions of interest (ROIs) based on multiple 3D medical images, The steps include providing the aforementioned multiple 3D medical images (105), The process includes the steps of: (107) locating multiple ROIs within the multiple 3D medical images, and (108) defining a 3D bounding box for each ROI using a set of six end coordinates (Xmin, Xmax, Ymin, Ymax, Zmin, Zmax); on the other hand, For each defined 3D bounding box, a 2D preprocessing step (110) is performed to extract a plurality of Ki 2D patches from each ROI contained within the 3D bounding box, The step (130) includes training a 2D neural network based on the extracted 2D patches. On the other hand, For each defined 3D bounding box, a 3D preprocessing step (120) for extracting at least one 3D patch from each ROI contained within the 3D bounding box, A method comprising the step (150) of training a 3D neural network to characterize the plurality of ROIs based on the extracted 3D patches, The 2D preprocessing step (110) includes at least the step (115) of extracting a plurality of Ki 2D patches as continuous parallel slices with respect to a single predetermined plane, A method wherein the 2D neural network is configured to be trained to characterize the plurality of ROIs based on the plurality of extracted 2D patches for the plurality of ROIs.

2. The method according to claim 1 (100), wherein the single predetermined plane is parallel to one of three planes: the sagittal plane, the coronal plane, and the axial plane, preferably parallel to the plane with the highest resolution, and more preferably parallel to the axial plane.

3. The method according to claim 1 or 2 (100), wherein the 2D preprocessing step (110) further includes, before the step (115) of extracting the plurality of Ki 2D patches, the step of extracting 2D ROI coordinates by calculating the coordinates of the center of each ROI in the axial view of each bounding box (Xmin, Xmax, Ymin, Ymax, Zmin, Zmax), where each center is defined by the coordinates ((Xmin + Xmax) / 2, (Ymin + Ymax) / 2) for all Z in [Zmin, Zmax].

4. The method according to claim 3 (100), wherein the step of extracting a plurality of Ki 2D patches (115) includes the step of continuously cutting a plurality of 2D patches of dimension P × Q along the z axis, for example, P = Q = 64.

5. The method according to claim 1 or 2 (100), wherein the 2D preprocessing step (110) further includes a first step of normalizing 2D voxel values, which is performed before or after the extraction step (115) of the plurality of Ki 2D patches.

6. The aforementioned 3D preprocessing step (120) is, The step (125) includes extracting at least one 3D patch from each ROI, The method according to claim 1 or 2 (100), further comprising the step (121) of extracting 3D ROI coordinates by calculating the coordinates of the center of each ROI, where the center of each ROI is defined, for example, by coordinates ((Xmin + Xmax) / 2, (Ymin + Ymax) / 2, (Zmin + Zmax) / 2), prior to the step (125) of extracting at least one 3D patch of each ROI.

7. The method according to claim 1 or 2 (100), wherein the step of extracting at least one 3D patch (125) includes the step of cutting out a 3D patch of dimensions P × Q × R around the central coordinates of each ROI, for example, P = Q = R = 64.

8. The method according to claim 5 (100), wherein the 3D preprocessing step (120) further includes a step (122) of resampling the 3D volume of each 3D image in which the ROI is localized (107), prior to the step (125) of extracting the at least one 3D patch.

9. The aforementioned 2D training step (130) is, The initialization step of the 2D neural network (135), The process includes a continuous training step (140) of the aforementioned 2D neural network, The 2D neural network is designed to calculate a diagnostic score based on a single 2D patch, and the initialization step (135) includes the step of initializing the weights of the 2D neural network based on a random determination of the weights and at least one of predetermined weights, The aforementioned continuous training step (140) is i. A step of calculating a diagnostic score using a 2D neural network for each randomly selected minibatch of all extracted patches of the plurality of ROIs, ii. A step of calculating the error estimation performed by the 2D neural network for each pre-calculated diagnostic score, iii. A step of updating the weights using a gradient-based method, iv. The method according to claim 1 or 2 (100), comprising the step of repeating steps i to iii until a predefined stopping condition is reached, thereby achieving a trained 2D neural network with optimized weights.

10. The aforementioned 3D training step (150) is, The initialization step of the 3D neural network (155), The process includes a continuous training step (160) of the 3D neural network, The 3D neural network is designed to calculate a diagnostic score based on a single 3D patch, and the initialization step (155) includes the step of initializing the weights of the 3D neural network based on a random determination of the weights and at least one of predetermined weights, The aforementioned continuous training step (160) is i. For each randomly selected minibatch of all the extracted patches of the plurality of ROIs, a step of calculating a diagnostic score using a 3D neural network, ii. A step of calculating the error estimation performed by the 3D neural network for each pre-calculated diagnostic score, iii. A step of updating the weights using a gradient-based method, iv. The method according to claim 1 or 2 (100), comprising the step of repeating steps i to iii until a predefined stopping condition is reached, thereby achieving a trained 3D neural network with optimized weights.

11. A method (200) for characterizing a region of interest ROI based on at least one 3D medical image, The steps include providing at least one 3D medical image (205), The steps include: (207) positioning one ROI as a candidate ROI in at least one 3D medical image, The steps include (208) defining a candidate 3D bounding box for a candidate ROI using a set of six endpoint coordinates (Xmin, Xmax, Ymin, Ymax, Zmin, Zmax), on the other hand, With respect to the candidate 3D bounding box, a 2D preprocessing step (210) is performed to extract a plurality of candidate 2D patches from the candidate ROI included in the candidate 3D bounding box. The steps include (230) running a trained 2D neural network configured to characterize the candidate ROI, On the other hand, With respect to the candidate 3D bounding box, a 3D preprocessing step (220) is performed to extract at least one candidate 3D patch from the candidate ROIs included in the candidate 3D bounding box. Step (250) of running a trained 3D neural network configured to characterize the candidate ROI based on at least one extracted candidate 3D patch, A method comprising the step (270) of ensembling a plurality of 2D diagnostic scores provided by the trained 2D neural network and a plurality of 3D diagnostic scores provided by the trained 3D neural network to output one overall score for each candidate ROI, The 2D preprocessing step (210) includes at least the step (215) of extracting a plurality of K candidate 2D patches as continuous parallel slices with respect to a single predetermined plane, A method comprising the step (230) of running the trained 2D neural network based on the extracted candidate 2D patches.

12. The method according to claim 11 (200), wherein the trained 2D neural network and 3D neural network are trained by performing the method for generating a neural network according to claim 1 (100).

13. The method according to claim 11 or 12 (200), wherein the single predetermined plane is parallel to one of three planes: the sagittal plane, the coronal plane, and the axial plane, preferably parallel to the plane with the highest resolution, and more preferably parallel to the axial plane.

14. The method according to claim 11 or 12 (200), wherein the 2D preprocessing step (210) further includes, before the step (215) of extracting the plurality of K candidate 2D patches, the step of extracting 2D ROI coordinates by calculating the coordinates of the center of the candidate ROI in the axial view of the candidate 3D bounding box (Xmin, Xmax, Ymin, Ymax, Zmin, Zmax), where each center is defined by the coordinates ((Xmin + Xmax) / 2, (Ymin + Ymax) / 2) for all Z in [Zmin, Zmax].

15. The method according to claim 14 (200), wherein the step of extracting a plurality of K candidate 2D patches (215) includes the step of continuously cutting out a plurality of K candidate 2D patches of dimension P × Q along the z axis, for example, P = Q = 64.

16. The method according to claim 11 or 12 (200), wherein the 2D preprocessing step (210) further includes a second step of normalizing 2D voxel values, which is performed before or after the step of extracting a plurality of K candidate 2D patches (215).

17. The aforementioned 3D preprocessing step (220) is, The step (225) includes extracting at least one candidate 3D patch of the candidate ROI, The method according to claim 11 or 12 (200), further comprising the step of extracting 3D ROI coordinates by calculating the coordinates of the center of the candidate ROI, for example, ((Xmin + Xmax) / 2, (Ymin + Ymax) / 2, (Zmin + Zmax) / 2), prior to the step of extracting at least one candidate 3D patch of the candidate ROI (225).

18. The method according to claim 17 (200), wherein the step of extracting at least one candidate 3D patch (225) includes the step of cutting out a candidate 3D patch of dimensions P × Q × R around the center coordinates of the candidate ROI, for example, P = Q = R = 64.

19. The method according to claim 17 (200), wherein the 3D preprocessing step (220) further includes a step of resampling the 3D volume of each 3D image in which the candidate ROI is located (207), prior to the step of extracting at least one candidate 3D patch of the candidate ROI (225).

20. The method according to claim 11 or 12 (200), wherein the step of running the trained 2D neural network (230) is performed N times based on the plurality of K extracted candidate 2D patches to compute K candidate diagnostic scores for one candidate ROI each time, thereby achieving K candidate 2D diagnostic scores for each of the N sequences.

21. The method according to claim 11 or 12 (200), wherein the step of running the trained 3D neural network (250) is performed M times based on at least one extracted candidate 3D patch to compute at least one candidate diagnostic score for one candidate ROI each time, thereby achieving at least one candidate 3D diagnostic score for each of the M sequences.

22. The aforementioned ensemble step (270) is, Step (271) of pooling N sequences of at least one candidate 2D diagnostic score for each ROI, calculated N times for the plurality of K extracted candidate 2D patches, by max pooling, wherein the max pooling step includes generating one maximum candidate 2D diagnostic score Sj by selecting the diagnostic score having the maximum value from the K candidate 2D diagnostic scores for each sequence j of the N sequences of candidate 2D diagnostic scores, thereby achieving one candidate maximum 2D diagnostic score for each sequence for the N sequences, and the trained 2D neural networks used for each sequence j of the N sequences are different from each other, preferably having different weights, The method according to claim 20 (200), comprising the step (272) of pooling the M-sequence of at least one candidate 3D diagnostic score for each ROI calculated for the at least one extracted candidate 3D patch.

23. The method (200) of claim 22, wherein the ensemble step (270) further includes step (275) outputting one overall score for each candidate ROI by pooling together the N 2D diagnostic scores from step (271) and the M 3D diagnostic scores from step (272).

24. The method according to claim 1 or 2 (100), wherein the 3D medical image includes at least one of computed tomography (CT) images and magnetic resonance imaging (MRI) images.

25. The method according to claim 11 or 12 (200), wherein the 3D medical image includes at least one of computed tomography (CT) images and magnetic resonance imaging (MRI) images.

26. A computer program product comprising instructions that, when executed by at least one digital processing device, perform a step (100) of the method according to claim 1 or 2.

27. ​​A computer program product comprising instructions that, when executed by at least one digital processing device, perform a step (200) of the method according to claim 11 or 12.

Images