Method and system for computer-aided diagnosis of integrated 2D and 3D neural networks based on medical images

Abstract

A method (100) for generating a machine learning model for characterizing a plurality of regions of interest (ROI) based on a plurality of 3D medical images and an associated method (200) for characterizing a region of interest (ROI) based on at least one 3D medical image are provided. The method proposed herein aims to provide a complementary strategy to enable classification of ROIs from 3D medical images, which can take advantage of the advantages and complementarity of 2D and 3D CNNs to improve the accuracy of prediction. More specifically, the present disclosure proposes the sensitivity / specificity of a 2D model that supplements a 3D model, thereby diagnosing by utilizing the supplemental concept.

Description

Technical Field

[0001] The present disclosure relates broadly to methods and systems for computer-aided diagnosis (CAD) based on three-dimensional medical images.

[0002] The present invention more particularly relates to machine learning models for characterizing regions of interest (ROIs) in three-dimensional medical images. Background Art

[0003] Medical imaging technology has made it possible to diagnose diseases of internal organs without any invasive procedures. In particular, in 1967, Godfrey Hounsfield proposed the concept of computed tomography (CT), which has enabled us to obtain 3D medical images of the inside of a patient's body today. Since then, 3D medical imaging has been widely used to help clinicians diagnose diseases such as cancer.

[0004] The problem with these 3D images is that they are usually bulky and take a lot of time for radiologists to read. Therefore, computer-aided diagnosis (CAD) has become important to help improve radiologists' productivity and diagnostic performance in image interpretation, especially in the context of screening programs like the National Lung Screening Trial. In 2012, a deep neural network surpassed all traditional models in the ImageNet Large Scale Visual Recognition Challenge, which triggered a revolution in the field of image processing because people had to admit that deep learning was much more efficient than the methods used at the time.

[0005] It is known that 2D convolutional neural network (CNN) based methods are generally insufficient in assessing the malignancy of lung nodules because 2D CNNs are limited to 2D space when applied to nodule characterization (CADx) and cannot fully consider important contextual and volumetric information. To overcome such shortcomings, in patent publication No. US 2021 / 0225511 A1 entitled "Method and system for improving cancer detection using deep learning" and in patent publication No. US 2020 / 0160997 A1 entitled "Method for detection and diagnosis of lung and pancreatic cancers from imaging scans", a fully 3D deep neural network architecture is proposed, which is designed to detect lung nodules in order to understand the 3D nature of the nodule shape.

[0006] However, 3D CNNs also have some limitations that 2D CNNs are less affected by. For example, 3D CNNs require much larger datasets to process and train the neural network (where 2D samples are clustered) and are more likely to overfit, limiting the neural network to have sufficiently deep architectures.

[0007] To overcome such shortcomings, Venkadesh et al. proposed using an ensemble of independently trained 2D and 3D CNNs to analyze multiple nodule patches and use deep models to characterize the malignancy of lung lesions (see Venkadesh et al., “Deep Learning for Malignancy Risk Estimation of Pulmonary Nodules Detected at Low-Dose Screening CT,” Radiology, Aug 2021;300(2):438-447).

[0008] However, in the Vankadesh combination strategy (of 2D and 3D models), the 2D model is configured by modeling multiple independent 2D networks whose inputs are different 2D rotated views of each ROI (see Setio et al., “Pulmonary nodule detection in CT images: false positive reduction using multi-view convolutional networks”, IEEE Trans Med Imaging, 2016 May;35(5):1160-1169). Since medical images are usually highly anisotropic, meaning that the voxel spacing along the x-, y-, and z-axes is different from each other, taking different rotated views for each ROI as proposed in Venkadesh will therefore result in different resolutions along different axes, which must be compensated by a preliminary resampling operation. The main disadvantage of the resampling operation is that it is based on assumptions about the image appearance between slices, which may be wrong and inconsistent with the original data.

[0009] Furthermore, it is important to note that Venkadesh’s solution not only changes the information due to the resampling operation, especially in the case of small findings (such as early malignant lung nodules), but also causes information loss due to partial coverage of the ROI, thereby filtering out relevant information of the ROI used for classification.

[0010] In fact, since each 2D model of Venkadesh (or Setio) is trained only on a single orientation (a single rotated view), the effective available sample size is further divided, making the training dataset a rather limited number of images, failing to obtain the discovered invariant representations and yielding rather poor model generalization from a design perspective.

[0011] An ideal approach would be to predict a diagnostic score based on as comprehensive information as possible from the 3D images through continuous analysis by a neural network that would represent this information in an invariant manner.

[0012] Therefore, there is a need for a method for computer-aided diagnosis (CAD) based on three-dimensional medical images that improves the sensitivity / specificity of diagnosis and / or the "area under the curve" (AUC) of the "receiver characteristic operator" (ROC) score.

[0013] More specifically, there is a need for a method for computer-aided diagnosis based on three-dimensional medical images, which method: - provide robust forecasts, and / or - Improve the AUC-ROC score of the prediction, and / or - Cover each nodule as a whole, and / or - obtain an invariant representation of the nodule, and / or -Consider small and large nodules. Summary of the invention

[0014] According to a first aspect, a method for generating a machine learning model for characterizing a plurality of regions of interest (ROIs) based on a plurality of 3D medical images is disclosed, comprising the following steps: - Provides multiple 3D medical images, and - localizing multiple ROIs in multiple 3D medical images, and for each ROI, defining a 3D bounding box by a set of 6 extremal coordinates (Xmin, Xmax, Ymin, Ymax, Zmin, Zmax), and on the one hand: - for each defined 3D bounding box, perform 2D pre-processing to extract a number of Ki 2D patches from each ROI included in the 3D bounding box, and - Train a 2D machine learning model based on the extracted 2D patches, on the other hand: - for each defined 3D bounding box, performing 3D pre-processing to extract at least one 3D tile from each ROI included in the 3D bounding box, and - a 3D machine learning model for characterizing the multiple ROIs, trained based on the extracted 3D patches for the multiple ROIs, wherein the step of 2D preprocessing comprises extracting at least a plurality of Ki 2D tiles as continuous and parallel slices relative to a single predetermined plane, and The 2D machine learning model is configured to be trained based on the extracted multiple 2D patches for multiple ROIs to characterize the multiple ROIs.

[0015] In some examples, each 2D tile in the plurality of Ki 2D tiles is obtained without any resampling.

[0016] In some examples, the single predetermined plane is parallel to one of the following three planes: sagittal plane, coronal plane, and axial plane, and is preferably parallel to the plane with the best resolution, and is further preferably parallel to the axial plane.

[0017] In some examples, the step of 2D preprocessing further includes: before the step of extracting a plurality of Ki 2DKi 2D tiles, a step of extracting 2D ROI coordinates by calculating the coordinates (Xmin, Xmax, Ymin, Ymax, Zmin, Zmax) of the center of each ROI in the axial view of each bounding box, wherein for all Z in [Zmin, Zmax], each center is defined, for example, by coordinates ((Xmin+Xmax) / 2, (Ymin+Ymax) / 2). The step of extracting a plurality of Ki 2D tiles may include continuously cropping a plurality of 2D tiles of size P×Q, for example, P=Q=64, along the z-axis.

[0018] In some examples, the step of 2D preprocessing further includes a step of 2D voxel value normalization, which may be implemented before or after the step of extracting a plurality of Ki 2D tiles.

[0019] In some examples, 3D preprocessing steps: - comprising the step of extracting at least one 3D patch of each ROI, and -Before the step of extracting at least one 3D tile of each ROI, it also includes a step of extracting 3D ROI coordinates by calculating the center coordinates of each ROI, wherein each ROI center is defined by, for example, coordinates ((Xmin+Xmax) / 2, (Ymin+Ymax) / 2, (Zmin+Zmax) / 2). The step of extracting at least one 3D tile may include cropping a 3D tile of size P×Q×R, for example, P=Q=R=64, around the center coordinates of each ROI. Otherwise, the 3D preprocessing step may also include a step of normalizing 3D voxel values, which may be implemented before or after the extraction step. In addition, the 3D preprocessing step may also include: before the step of extracting at least one 3D tile, a step of 3D volume resampling each 3D image, wherein the ROI has been localized.

[0020] In some examples, the steps of 2D training include: - initialization step for the 2D machine learning model, and -Continue training steps for 2D machine learning models.

[0021] In the case where the 2D machine learning model is designed to calculate the diagnostic score based on one 2D image patch, the initialization step may include initializing the weights of the 2D machine learning model based on at least one of a random determination of the weights and a predetermined weight. Additionally or alternatively, the continuing training step may include: i. for each mini-batch of all extracted patches of multiple ROIs, a step of using a 2D machine learning model to calculate a diagnostic score, wherein the mini-batch is randomly selected, ii. For each previously calculated diagnostic score, a calculation step of the error estimate performed by the 2D machine learning model, iii. Steps to update weights using gradient-based methods, iv. Repeat steps i-iii until a predefined stop condition is reached, This results in a trained 2D machine learning model with optimized weights.

[0022] In some examples, the 3D training step includes: - initialization step of the 3D machine learning model, and -Continue training steps of 3D machine learning model.

[0023] In the case where the 3D machine learning model is designed to calculate the diagnostic score based on one 3D image patch, the initialization step may include initializing the weights of the 3D machine learning model based on at least one of a random determination of the weights and a predetermined weight. Additionally or alternatively, the continuing training step may include: i. for each mini-batch of all extracted patches of the plurality of ROIs, a step of calculating a diagnostic score using a 3D machine learning model, wherein the mini-batches are randomly selected, ii. For each previously calculated diagnostic score, a calculation step of the error estimate performed by the 3D machine learning model, iii. Steps to update weights using gradient-based methods, iv. Repeat steps i-iii until a predefined stop condition is reached, This results in a trained 3D machine learning model with optimized weights.

[0024] In some examples, each training step may also be performed on morphological and / or radiomic features. The morphological features may include at least baseline features. The baseline features may include at least one of the following features: - the volume of at least one anatomical structure, - elongation of at least one anatomical structure, - a minimum and / or maximum size of an oriented bounding box of at least one anatomical structure, and - The mean HPI value of at least one anatomical structure.

[0025] In some examples, the method according to the first aspect of the present invention may further include: - provide clinical information about each patient, and Each training step is also performed on the patient's clinical information.

[0026] According to a second aspect, a method for characterizing a region of interest ROI based on at least one 3D medical image is disclosed, the method comprising the following steps: - providing at least one 3D medical image, and - localizing a ROI as a candidate ROI in the at least one 3D medical image, and - For a candidate ROI, define a candidate 3D bounding box by a set of 6 extrema coordinates (Xmin, Xmax, Ymin, Ymax, Zmin, Zmax), and on the one hand: - for a candidate 3D bounding box, performing 2D preprocessing to extract a plurality of candidate 2D patches from a candidate ROI included in the candidate 3D bounding box, and - implementing a trained 2D machine learning model configured to characterize candidate ROIs, on the other hand: - for a candidate 3D bounding box, performing 3D preprocessing to extract at least one candidate 3D tile from a candidate ROI included in the candidate 3D bounding box, and - implementing a trained 3D machine learning model configured to characterize a candidate ROI based on at least one extracted candidate 3D tile, and then - integrating a plurality of 2D diagnostic scores provided by a trained 2D machine learning model (by implementing the output of the 2D machine learning model) and a plurality of 3D diagnostic scores provided by a trained 3D machine learning model (by implementing the output of the 3D machine learning model) to output an overall score for the candidate ROI, wherein the step of 2D preprocessing comprises extracting at least a plurality of candidate K 2D tiles as continuous and parallel slices relative to a single predetermined plane, and Among them, the step of implementing the trained 2D machine learning model is performed based on the extracted multiple candidate 2D tiles.

[0027] The scoring achieved by implementing the method presented above exploits the fact that the method is based on extracted 2D slices that are faithful to the original data and therefore do not deviate from reality in this respect.

[0028] Therefore, a method for computer-aided diagnosis based on three-dimensional medical images is proposed, which improves the sensitivity / specificity of diagnosis and / or the "area under the curve" (AUC) of the "receiver characteristic operator" (ROC) score.

[0029] More specifically, a method for computer-aided diagnosis based on three-dimensional medical images is therefore proposed, the method: - provide robust forecasts, and / or - Cover each ROI as a whole, and / or -Consider small ROI and large ROI.

[0030] In some examples, the trained 2D and 3D machine learning models have been trained by implementing the method for generating a machine learning model according to the first aspect of the invention.

[0031] In some examples, the single predetermined plane is parallel to one of the following three planes: sagittal plane, coronal plane, and axial plane, and is preferably parallel to the plane with the best resolution, and is further preferably parallel to the axial plane.

[0032] In some examples, the 2D preprocessing step further includes: before the step of extracting a plurality of candidate K 2D tiles, a step of extracting 2D ROI coordinates by calculating the coordinates (Xmin, Xmax, Ymin, Ymax, Zmin, Zmax) of the center of the candidate ROI in the axial view of the candidate 3D bounding box, wherein for all Z in [Zmin, Zmax], each center is defined, for example, by coordinates ((Xmin+Xmax) / 2, (Ymin+Ymax) / 2). The step of extracting a plurality of candidate K 2D tiles may include continuously cropping a plurality of candidate K 2D tiles of size P×Q, for example, P=Q=64, along the z-axis. Thus, the plurality of Ki 2D tiles may be composed or constituted of a plurality of P×Q 2D tiles for each 3D bounding box, preferably, each of the plurality of P×Q 2D tiles is constituted by a Ki=Zmax-Zmin tile.

[0033] In some examples, the 2D pre-processing step further includes a step of normalizing 2D voxel values, which may be implemented before or after the step of extracting a plurality of candidate K 2D tiles.

[0034] In some examples, 3D preprocessing steps: - comprising the step of extracting at least one candidate 3D tile of the candidate ROI, and -Before the step of extracting at least one candidate 3D tile of the candidate ROI, it also includes a step of extracting 3D ROI coordinates by calculating the center coordinates of the candidate ROI, wherein the center of the candidate ROI is defined by, for example, coordinates ((Xmin+Xmax) / 2, (Ymin+Ymax) / 2, (Zmin+Zmax) / 2). The step of extracting at least one candidate 3D tile may include cropping a candidate 3D tile of size P×Q×R, for example, P=Q=R=64, around the center coordinates of the candidate ROI. Additionally or alternatively, the step of 3D preprocessing may also include a step of normalizing 3D voxel values, which may be implemented before or after the step of extracting at least one candidate 3D tile of the candidate ROI. Additionally or alternatively, the step of 3D preprocessing may also include: before the step of extracting at least one candidate 3D tile of the candidate ROI, performing 3D volume resampling on each 3D image in which the candidate ROI has been localized.

[0035] In some examples, the step of implementing the trained 2D machine learning model is performed N times based on multiple extracted candidate K 2D patches to calculate K candidate diagnostic scores for one candidate ROI each time, thereby providing candidate K 2D diagnostic scores for each series in the N series.

[0036] In some examples, the step of implementing the trained 3D machine learning model is performed M times based on at least one extracted candidate 3D tile to calculate at least one candidate diagnosis score for one candidate ROI each time, thereby providing at least one candidate 3D diagnosis score for each series in the M series.

[0037] Integration steps may include: - a step of pooling at least one candidate 2D diagnostic score for each ROI of N series of the N plurality of extracted candidate K 2D tiles by maximum pooling, wherein the step of maximum pooling comprises: for each series j of the N series of candidate K 2D diagnostic scores, generating a maximum candidate 2D diagnostic score Sj by selecting the diagnostic score with the maximum value from the candidate K 2D diagnostic scores, thereby achieving a candidate maximum 2D diagnostic score for each series of the N series, wherein the trained 2D machine learning models for each series j of the N series are different from each other, preferably with different weights, and - A pooling step for calculating M series of at least one candidate 3D diagnostic score for each ROI of M times at least one extracted candidate 3D tile.

[0038] - In addition, the integration step may further include a step of outputting a total score for each candidate ROI by integrating the N 2D diagnostic scores and the M 3D diagnostic scores together.

[0039] In some examples, the method according to the second aspect of the present invention may further include the following steps: for a candidate ROI, extracting morphological features of the ROI based on the 3D medical image and by using a morphological-radiological feature extractor. The morphological features may include baseline features. The baseline features may include at least one of the following: - the volume of at least one anatomical structure, - elongation of at least one anatomical structure, - a minimum and / or maximum size of an oriented bounding box of at least one anatomical structure, and - The mean HPI value of at least one anatomical structure.

[0040] According to the first aspect or the second aspect of the present invention, the three-dimensional medical image may include a computed tomography (CT) image or a magnetic resonance imaging (MRI) image.

[0041] Another aspect of the present disclosure relates to a non-transitory computer-readable medium storing instructions which, when implemented by at least one digital processing device, perform at least the steps of the method according to the first aspect of the present invention and / or the steps of the method according to the second aspect of the present invention.

[0042] Another aspect of the present disclosure relates to a computer program product comprising instructions which, when implemented by at least one digital processing device, perform at least the steps of the method according to the first aspect of the invention and / or the steps of the method according to the second aspect of the invention.

[0043] Other objects, features and advantages of the present invention disclosed herein and its various embodiments / aspects may become apparent from the following description of some exemplary embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Reference will be made in detail to embodiments of the present disclosure, non-limiting examples of which may be shown in the figures of the accompanying drawings. Some of the figures may be in the form of diagrams. For clarity of illustration, some elements in the figures may be exaggerated; other elements may be omitted.

[0045] Any text (legends, notes, reference numerals, etc.) appearing on the accompanying drawings is incorporated herein by reference.

[0046] Figure 1 are flow charts of some examples of methods according to the first aspect of the present disclosure.

[0047] Figure 2 are flow charts of some examples of methods according to the second aspect of the present disclosure.

[0048] Figure 3 The anatomical planes and body axes (X, Y, Z) are shown.

[0049] Figure 4 An example of a nodule within a bounding box is shown. DETAILED DESCRIPTION

[0050] “Findings” can be defined as lesions, consolidations, nodules, malignant or benign, and anything clinically significant that may aid in the diagnosis.

[0051] A region of interest (ROI) can be defined as a sub-portion of a 3D image, more specifically, a subset of voxels of a 3D image covering a specific type of tissue, such as an organ, lesion, consolidation, nodule, malignant or benign, and any clinically significant tissue that may aid in diagnosis. If the entire image is considered meaningful and relevant to the diagnosis, the ROI may cover the entire 3D image. In the context of the present disclosure, "ROI" is interchangeable with the term "finding".

[0052] The bounding box can be defined as the smallest rectangular cuboid including the ROI. The bounding box can include all voxels of the 3D image. In the context of the present disclosure, a bounding box contains one ROI. In other words, one ROI corresponds to at least one bounding box.

[0053] The axial plane can be defined as a transverse plane at right angles to the long axis of the body, such as the transverse plane in CT scans and MRI scans.

[0054] In the context of the present disclosure, consecutive tiles or slices mean that they are successively close to or adjacent to each other.

[0055] As mentioned above, 2D CNNs and 3D CNNs have different limitations. Therefore, the architectures and other representations used to encode a given problem in 2D and 3D networks must be conceived in very different ways.

[0056] However, the fact that 2D CNN and 3D CNN are combined by designing a system for either model (2D and 3D) will not be enough to address the shortcomings of both models. For example, Venkadesh’s solution of combining 2D and 3D models still fails to address the limitations in 3D CNNS by its 2D model, but introduces further problems such as the loss and alteration of information caused by its patch extraction method, which inevitably includes the resampling operation used in the 3D model and the limited sample size used to train the 2D model.

[0057] The present disclosure aims to provide complementary strategies to achieve classification of ROIs from 3D medical images, which can exploit the advantages and complementarity of 2D and 3D CNNs to improve the accuracy of prediction.

[0058] More specifically, the present disclosure proposes a 2D model that complements the 3D model, such that the sensitivity / specificity of the diagnosis is improved by utilizing the complementary concepts in at least the following aspects: i) 2D preprocessing and 3D preprocessing Typically, 3D pre-processing modules have more information, but accuracy is compromised due to the reasons mentioned above, and in particular due to artificial assumptions made during resampling (through interpolation operations).

[0059] Unlike its counterpart, the 2D preprocessing module is less informative but more faithful to the original data. However, in some 2D models, such as the one in Venkadesh and Setio, since resampling will be required during the preprocessing of the extraction method given a 2D view, it may suffer from the included faithfulness due to the resampling and partial information about the 3D image.

[0060] The present disclosure allows obtaining as comprehensive information as possible from 3D images through continuous analysis of a neural network, which will represent this information in an invariant way due to its 2D model, and in particular by extracting multiple continuous and parallel slices for each ROI in a predetermined plane with better resolution. In the current extraction used in the 2D model, no resampling will be required. In addition, the present solution takes into account the entire nodule as well as small and large sized nodules.

[0061] ii) Using trained 2D models for diagnosis The extraction method proposed in the present disclosure allows for extracting multiple K 2D patches for one ROI for diagnosis, and inputting the multiple K 2D patches into a trained 2D model will produce K diagnostic scores, so that the highest score can be selected among the K scores, where the highest score is more likely to be cancer (the nodule with the highest risk of malignancy).

[0062] iii) Integrate 2D and 3D models to calculate the total score: Pool all scores calculated from 2D and 3D models to calculate the total score.

[0063] Assuming that a low score of "0" indicates low risk and a high score of "1" indicates high risk, a mid-range score (e.g., 0.4-0.6) will indicate uncertainty in the prediction. In the presence of many high or low values ​​and some intermediate values, in order to highlight the high or low values ​​while reducing uncertainty, the present invention proposes pooling all scores. However, the pooling method is not limited to averaging the scores, as it is also advantageous to use other methods to derive a score from the 2D and 3D scores (e.g., XGBoost, weighted average, etc.).

[0064] Therefore, the claimed method of training machine learning for characterizing ROIs and using the trained machine learning model for the same purpose by configuring how to select 2D images as 2D model inputs in the 3D volume and subsequent 2D network structure to further aggregate them into a single volume prediction remains completely open but is also critical to obtaining accurate results.

[0065] More specifically, the advantages of the present disclosure may be: - provide robust forecasts, and / or - Improve the AUC-ROC score of the prediction, and / or - Cover each ROI as a whole, and / or - obtain an invariant representation of the nodule, and / or -Consider small ROI and large ROI.

[0066] Figure 1 A flow chart illustrating an embodiment of the first aspect of the present invention is shown.

[0067] According to its first aspect, the present invention provides a method 100 for generating a machine learning model for characterizing a plurality of regions of interest (ROIs) based on a plurality of 3D medical images.

[0068] Applications of characterizing a ROI may be classification of disease severity, making a decision whether the ROI is benign or malignant based on a certainty score or other factors, or making a decision of a pathological state.

[0069] While applications of the present disclosure are discussed in the context of calculating diagnostic scores, such as malignancy scores, it should be understood that applications may be extended to those other than diagnostic scores, such as classification of disease severity.

[0070] It should be understood that while the present application is disclosed in the context of liver fibrosis and lung nodules, the diseases in the context of the present disclosure may manifest in defined anatomical structures and may not be limited to tumors, whether cancerous or not, nor to the liver or lungs.

[0071] The method 100 comprises the following steps: - Provides more than 105 3D medical images, and - Localizing 107 a plurality of ROIs in a plurality of 3D medical images and for each ROI defining 108 a 3D bounding box by a set of 6 extremal coordinates (Xmin, Xmax, Ymin, Ymax, Zmin, Zmax).

[0072] The 3D medical image provided 105 may include a computed tomography (CT) image or a magnetic resonance imaging (MRI) image.

[0073] In the context of the present disclosure, one 3D image may contain 0, 1 or multiple ROIs. In case multiple 3D medical images are provided, multiple ROIs may be localized 107 .

[0074] The localization 107 of the ROI may be performed automatically, for example, by CADe, or manually, for example, by a clinician.

[0075] In some examples, localizing 107 the ROI may include segmenting at least one determined anatomical structure of the patient in each of the plurality of 3D medical images, and defining an implicit mask of the at least one anatomical structure as the ROI.

[0076] In some examples, the localization 107 of the ROI can advantageously be limited to at least one determined anatomical structure. The method 100 may also include a step of determining whether the localized ROI is inside the at least one determined anatomical structure. In order to determine whether the localized ROI is inside the at least one determined anatomical structure, it is conceivable to calculate the ratio between the volume of the intersection of the bounding box and the lung mask and the volume of the bounding box. In some examples, if the ratio is equal to or greater than 0.5, the localized ROI is considered to be inside the determined anatomical structure. The ratio may vary depending on the anatomical structure, the application of the localization, and other factors. In some examples, the method may also include the step of keeping only the localized ROI that has been determined to be inside the determined anatomical structure.

[0077] In one aspect, the method 100 further comprises the following steps: - for each defined 3D bounding box, performing 2D pre-processing 110 to extract a plurality of 2D tiles from the ROI included in the 3D bounding box, and -Training 130 a 2D machine learning model based on the extracted multiple 2D tiles.

[0078] On the other hand, the method 100 further comprises the following steps: - for each 3D bounding box, performing 3D pre-processing 120 to extract at least one 3D tile from each ROI included in the 3D bounding box, and - Training 150 a 3D machine learning model for characterizing the plurality of ROIs based on the extracted 3D patches for the plurality of ROIs.

[0079] For illustrative purposes, Figure 4, ROIs in bounding boxes are shown in FIG. In the context of the present disclosure, each bounding box contains one ROI. For ease of understanding, a bounding box is represented by a set of 6 extreme coordinates (Xmin, Xmax, Ymin, Ymax, Zmin, Zmax). It should be understood that other coordinates may also be used as long as they are suitable for the 3D X, Y, Z coordinate system.

[0080] Advantageously, the method 100 is such that, for each defined 3D bounding box, the step of 2D pre-processing 110 comprises at least extracting 115 a plurality of Ki 2D tiles (for each ROIi) as consecutive and parallel slices relative to a single predetermined plane.

[0081] Each 2D tile in the plurality of Ki 2D tiles is obtained without any resampling.

[0082] In this way, the extracted 2D tiles can not only cover the integrity of the original data but also reflect the reality without providing information about the changes caused by the resampling operation.

[0083] Preferably, the single predetermined plane is parallel to one of the following three planes: sagittal plane, coronal plane and axial plane. More preferably, the single predetermined plane is parallel to the plane with the best resolution. Further preferably, the single predetermined plane is parallel to the axial plane.

[0084] Figure 3 The anatomical planes and body axes (X, Y, Z) are shown.

[0085] refer to Figure 3 , which shows the sagittal plane (X plane), the coronal plane (Y plane) and the transverse plane (Z plane). Each of these three planes contains two of the three coordinate axes (X, Y, Z) in the three-dimensional coordinate system. For example, the plane containing the X axis and the Y axis is an axial plane called the XY plane (not labeled in the figure) or the Z plane.

[0086] like Figure 3 As shown, the axial plane is perpendicular to the long axis (Z axis), which is the spinal column of the body (head, neck, trunk, tail).

[0087] In the context of the present disclosure, the axial plane may be interchanged with the transverse plane.

[0088] The sagittal plane runs from front to back, dividing the body into right and left parts. The coronal plane runs from side to side, dividing the body into front and back parts.

[0089] In some examples, the 2D slices are extracted relative to the same predetermined plane. In some other examples, the same predetermined plane is parallel to the axial plane with the best resolution. The ROI is extracted in the plane with the best resolution, and it can be understood in the context of the present disclosure that the ROI can be better depicted by viewing the image block axially.

[0090] This method can take into account the morphological properties of the nodule (ROI) and may be particularly advantageous for small, spiky nodules.

[0091] Furthermore, this method allows taking nodule size into account and may be particularly advantageous in case of large nodules, as the extraction can cover the entire nodule.

[0092] In this way, the extracted 2D patches not only cover the integrity of the original data, but also the extracted 2D patches can reflect the reality because they are obtained without any resampling. The most comprehensive and realistic extraction of 3D medical images to be used for training 2D machine learning models helps to improve the AUC-ROC score of the model.

[0093] The 2D pre-processing step 110 may further comprise, before the step of extracting 115 a plurality of Ki 2D tiles, a step of extracting 2D ROI coordinates by calculating the coordinates (Xmin, Xmax, Ymin, Ymax, Zmin, Zmax) of the center of each ROI in the axial view of each bounding box, wherein for all Z in [Zmin, Zmax], each center is defined, for example, by the coordinates ((Xmin+Xmax) / 2, (Ymin+Ymax) / 2). Thus, for each ROI, each plurality of Ki 2D tiles may comprise Ki=Zmax-Zmin+1 tiles.

[0094] More specifically, the step of extracting 115 a plurality of Ki 2D tiles may include successively cropping a plurality of 2D tiles of size P×Q along the Z axis, for example, P=Q=64.

[0095] Therefore, multiple Ki 2D tiles are composed or constructed from multiple Ki 2D tiles of size P×Q for each 3D bounding box.

[0096] If necessary, the 2D pre-processing step 110 may also include a first step of 2D voxel value normalization. This first normalization step may be implemented before or after the step of extracting 115 a plurality of Ki 2D tiles and / or may involve normalization of the brightness attenuation (HU) of the relevant pixels or voxels. More specifically, the first step of 2D voxel value normalization may involve the interception of the values ​​of the voxels (or pixels, since we are talking about 2D tiles) between two predefined minimum and maximum values ​​and then transforming them by applying a predefined affine function to the values ​​of the voxels of the 2D tiles. If step 115 is implemented before the first normalization step, step 112 only needs to be applied to the 3D image form of the Ki 2D tiles to be extracted 115. If the first normalization step is implemented before step 115, the brightness attenuation (HU) on all 2D tiles needs to be normalized.

[0097] Therefore, after the step of extracting 115 a plurality of Ki 2D patches, a plurality of normalized Ki 2D patches may be achieved, which may be used for 2D training 130 of a 2D machine learning model.

[0098] As previously described and further advantageously, method 100 enables training a 2D machine learning model based on a plurality of extracted Ki2D patches for a plurality of ROIs, i.e., training a 2D machine learning model using a plurality of 2D patches.

[0099] In this way, the sampling size is increased, thereby improving the accuracy of the trained machine learning model and improving the robustness of the model, especially the synergistic effect of combining with a specific way of performing the 2D preprocessing 110 step, thereby further improving the accuracy of the trained model.

[0100] The 2D training step 130 may include: - an initialization step 135 of the 2D machine learning model, and -Continue training step 140 of the 2D machine learning model.

[0101] In some examples, initialization step 135 may include initializing weights of the 2D machine learning model based on at least one of randomly determined weights and predetermined weights.

[0102] The continuing training step 140 may include: i. for each mini-batch of all extracted patches of multiple ROIs, a step of using a 2D machine learning model to calculate a diagnostic score, wherein the mini-batch is randomly selected, ii. For each previously calculated diagnostic score, a calculation step of the error estimate performed by the 2D machine learning model, iii. Steps to update weights using gradient-based methods, iv. Repeat the steps of calculating the diagnostic score, calculating the error estimate, and updating the weights until a predefined stopping condition is reached.

[0103] When a predefined stopping condition is reached, the training is terminated.

[0104] In some examples, the stopping condition may be reached in at least one of the following cases, which list is non-exhaustive: 1) After a certain number of trainings; 2) After reaching a certain performance of the model; 3) If the performance does not change after some iterations.

[0105] Hence, a trained 2D machine learning model with optimized weights is achieved.

[0106] To implement the continued training step 140, it can more specifically use mini-batch gradient descent, which is a variation of the gradient descent algorithm that divides the training data set into mini-batches for calculating model errors and updating model coefficients. In some examples, continued training of the 2D machine learning model can be based on mini-batches.

[0107] It should be noted that among the known gradient-based methods, Adam, RMSprop, and SGD are methods that can be used for each step of updating the weights described herein.

[0108] The following describes how to implement a trained 3D machine learning model with optimized weights in parallel with the way a trained 2D machine learning model with optimized weights is implemented.

[0109] In order to extract at least one 3D tile from each ROI included in the 3D bounding box, the above steps of 3D pre-processing 120 may: - comprising the step of extracting 125 at least one 3D tile of each ROI, and - Before the step of extracting 125 at least one 3D tile of each ROI, a step of extracting 121 3D ROI coordinates by calculating the center coordinates of each ROI. Each ROI center can be defined by coordinates ((Xmin+Xmax) / 2, (Ymin+Ymax) / 2, (Zmin+Zmax) / 2), for example.

[0110] More specifically, the step of extracting 125 at least one 3D tile may include the step of cropping the 3D tile with a size of P×Q×R around the central coordinates of each ROI, for example, P=Q=R=64.

[0111] For the 2D pre-processing step 110 , the 3D pre-processing step 120 may further include a step of 3D voxel value normalization 123 .

[0112] The step of 3D voxel value normalization 123 may be implemented before or after the step of extracting 125. More specifically, the step of 3D voxel value normalization 123 may involve intercepting the value of the voxel between two predefined minimum and maximum values, and then transforming the value of the voxel of the 3D tile by applying a predefined affine function to the voxel.

[0113] In contrast to the 2D pre-processing step 110 , the 3D pre-processing step 120 may comprise a step of 3D volume resampling 122 of each 3D image, in which the ROI has been localized 107 .

[0114] In some examples, step 122 may be implemented before the step of extracting 125 at least one 3D tile.

[0115] As previously described, the step of 3D volume resampling 122 may be required to address issues caused by potential imaging anisotropy and may be configured to achieve equidistant spacing. In some examples, a resolution of 0.65 mm x 0.65 mm x 0.65 mm may be achieved in the X, Y, and Z axes, respectively.

[0116] To increase the sample size for training 3D machine learning, the cropped 3D tile can be further processed to generate more than one 3D tile. In some examples, we can crop multiple tiles for one ROI. In one example, the center of the 3D tile can be slightly shifted. In another example, the image can be rotated.

[0117] Therefore, after the step of extracting 125 at least one 3D tile, multiple normalized and resampled 3D tiles may be achieved for multiple localized ROIs, wherein the multiple normalized and resampled 3D tiles may be used for 3D training 150 of a 3D machine learning model.

[0118] For 2D training 130 of a 2D machine learning model, 3D training 150 of a 3D machine learning model may include: - an initialization step 155 of the 3D machine learning model, and -Continue training step 160 of the 3D machine learning model.

[0119] In some examples, initialization step 155 may include initializing weights of the 3D machine learning model based on a random determination of weights or based on predetermined weights.

[0120] The continuing training step 160 may include: i. for each mini-batch of all extracted patches of the plurality of ROIs, a step of calculating a diagnostic score using a 3D machine learning model, wherein the mini-batches are randomly selected, ii. For each previously calculated diagnostic score, a calculation step of the error estimate performed by the 3D machine learning model, iii. Steps to update weights using gradient-based methods, iv. Repeat the steps of calculating the diagnostic score, calculating the error estimate, and updating the weights until a predefined stopping condition is reached.

[0121] Hence, a trained 3D machine learning model with optimized weights is achieved.

[0122] It should be understood that in the context of the present disclosure, the examples and descriptions provided above regarding the continued training of step 140 of the 2D machine learning model are also applicable to the continued training of step 160 of the 3D machine learning model.

[0123] At this stage, as a result of the method 100 for generating a machine learning model, a trained 2D machine learning model with optimized weights and a trained 3D machine learning model with optimized weights are thus achieved.

[0124] In some examples, method 100 may include extracting morphological features of the ROI (based on 2D and / or 3D tiles), and further performing at least one of training step 130 and training step 150 on the morphological and / or radiomic features by using a morphological-radiomics feature extractor. The morphological features include at least baseline features, wherein the baseline features include, for example, at least one of the following: - the volume of at least one anatomical structure, - elongation of at least one anatomical structure, - a minimum and / or maximum size of an oriented bounding box of at least one anatomical structure, and - The mean HPI value of at least one anatomical structure.

[0125] The method 100 may further include the step of providing clinical information about each patient so that the training steps 130 and / or 150 may be further performed on the patient clinical information.

[0126] Using supplementary features such as morphological and / or radiomic features to train at least one of the 2D model and the 3D model may improve the robustness of the model.

[0127] It should be understood that the 2D machine learning model and / or the 3D machine learning model according to the present disclosure may be based on a deep neural network, which may be a convolutional neural network, and vice versa.

[0128] Figure 2 A flow chart illustrating an embodiment of the second aspect of the invention is shown.

[0129] According to its second aspect, the present invention provides a method 200 for characterizing a region of interest ROI based on at least one 3D medical image by applying a trained machine learning model.

[0130] The method 200 comprises the following steps: - providing 205 at least one 3D medical image, and - Localizing 207 a ROI as a candidate ROI in at least one 3D medical image.

[0131] Steps 205 and 207 may be further described in detail as the above-described detailed steps 105 and 107 .

[0132] In some examples, one ROI is localized at a time for characterization. The localizing step 207 may be repeated iteratively to localize a new ROI each time for characterization.

[0133] The method 200 further includes the following steps: - For the candidate ROI, define 208 a candidate 3D bounding box by a set of 6 extrema coordinates (Xmin, Xmax, Ymin, Ymax, Zmin, Zmax).

[0134] In one aspect, the method 200 further comprises the following steps: - for a candidate 3D bounding box, performing 2D preprocessing 210 to extract a plurality of candidate 2D tiles from a candidate ROI included in the candidate 3D bounding box, and - Implementing 230 a trained 2D machine learning model configured for characterizing a ROI (here a candidate ROI).

[0135] The trained 2D machine learning model can be trained by implementing the method 100 for generating a machine learning model according to the first aspect of the present invention. On the other hand, the method 200 further includes the following steps: - for a candidate 3D bounding box, performing 3D preprocessing 220 to extract at least one candidate 3D tile from a candidate ROI included in the candidate 3D bounding box, and - implementing 250 a trained 3D machine learning model configured for characterizing a ROI (here a candidate ROI) based on at least one extracted candidate 3D tile.

[0136] The trained 3D machine learning model may be trained by implementing the method 100 for generating a machine learning model according to the first aspect of the present invention.

[0137] Advantageously, the method 200 is such that the step of 2D pre-processing 210 comprises at least: extracting 215 a plurality of candidate K 2D tiles as continuous and parallel slices relative to a single predetermined plane.

[0138] Each 2D tile in a plurality of K 2D tiles is obtained without any resampling.

[0139] In this way, the extracted 2D tiles can not only cover the integrity of the original data but also reflect the reality without providing information about the changes caused by the resampling operation.

[0140] As in method 100, the single predetermined plane is parallel to one of the following three planes: sagittal plane, coronal plane and axial plane. More preferably, the single predetermined plane is parallel to the plane with the best resolution. Further preferably, the single predetermined plane is parallel to the axial plane. Therefore, the candidate ROI can be better depicted by the axial observation block.

[0141] In this way, not only small (needle-shaped) ROIs can be accurately diagnosed by considering the morphological properties of the ROI, but also large ROIs can be accurately diagnosed by considering the size of the ROI.

[0142] Therefore, the extracted 2D patches not only cover the integrity of the original data, but also reflect the reality due to the absence of any resampling. The 3D medical images to be input to the machine learning model for characterizing the ROI are extracted as comprehensively and faithfully as possible to improve the accuracy of the characterization (prediction).

[0143] The step of 2D preprocessing 210 may further include: before the step of extracting 215 a plurality of candidate K 2D tiles, a step of extracting 2D ROI coordinates by calculating the coordinates (Xmin, Xmax, Ymin, Ymax, Zmin, Zmax) of the center of the candidate ROI in the axial view of the candidate 3D bounding box. For all Z in the interval [Zmin, Zmax], each center may be defined, for example, by coordinates ((Xmin+Xmax) / 2, (Ymin+Ymax) / 2). In addition, the step of extracting 215 a plurality of candidate K 2D tiles may include continuously cropping a plurality of candidate K 2D tiles of size P×Q along the z-axis, for example, P=Q=64.

[0144] If desired, the 2D pre-processing step 210 may also include a second step of 2D voxel value normalization. This second normalization step may be implemented before or after the step of extracting 215 a plurality of candidate K 2D tiles, and / or may involve normalization of the brightness attenuation (HU) of the relevant pixels or voxels. More specifically, the second step of 2D voxel value normalization may involve the interception of the values ​​of the voxels (or pixels, since we said 2D tiles) between two predefined minimum and maximum values, and then transforming them by applying a predefined affine function to the values ​​of the voxels of the candidate 2D tiles. If step 215 is implemented before the second normalization step, the second normalization step only needs to be applied to the candidate 3D image from which the candidate K 2D tiles are to be extracted 125. If the second normalization step is implemented before step 215, the brightness attenuation (HU) on all candidate 2D tiles needs to be normalized.

[0145] Therefore, after the step of extracting 215 a plurality of candidate K 2D tiles, a plurality of normalized and resampled candidate 2D tiles may be implemented, wherein the normalized and resampled candidate 2D tiles may be used to implement 230 a trained 2D machine learning model. In order to extract at least one candidate 3D tile from the candidate ROI, the above step of 3D preprocessing 220 may: - comprising the step of extracting 225 at least one candidate 3D tile of the candidate ROI, and - Before the step of extracting 225 at least one candidate 3D tile of the candidate ROI, the step of extracting 3D ROI coordinates by calculating the center coordinates of the candidate ROI is also included. The center of the candidate ROI may be defined by coordinates ((Xmin+Xmax) / 2, (Ymin+Ymax) / 2, (Zmin+Zmax) / 2), for example.

[0146] More specifically, the step of extracting 225 at least one candidate 3D tile may include cropping the candidate 3D tile with a size of P×Q×R, for example, P=Q=R=64, around the center coordinates of the candidate ROI.

[0147] For the 3D preprocessing step 120 according to the first aspect of the invention, the 3D preprocessing step 220 according to the second aspect of the invention may further include a 3D voxel value normalization step 223 and / or a 3D volume resampling step of at least one 3D image, wherein the candidate ROI has been localized 207 .

[0148] The step of 3D voxel value normalization 223 may be implemented before or after the extraction step 225. More specifically, the step of 3D voxel value normalization 223 may involve intercepting the value of the voxel between two predefined minimum and maximum values, and then transforming the value of the voxel of the candidate 3D tile by applying a predefined affine function to the voxel.

[0149] The step of 3D volume resampling of at least one 3D image in which the candidate ROI has been localized 207 may be implemented before the step of extracting 225 at least one candidate 3D tile. The step of 3D volume resampling 122 may be configured to achieve equidistant spacing and / or a resolution of 0.65 mm x 0.65 mm x 0.65 mm.

[0150] Therefore, after the step of extracting 225 at least one candidate 3D tile, a normalized and resampled candidate 3D tile may be implemented, wherein the normalized and resampled candidate 3D tile may be used to implement 250 a trained 3D machine learning model.

[0151] Then, the method 200 comprises the following steps: -Integrating 270 multiple 2D diagnostic scores provided by implementing a trained 2D machine learning model and multiple 3D diagnostic scores provided by implementing a trained 3D machine learning model so that each candidate ROI outputs a total score.

[0152] The integration 270 step then takes advantage of multiple implementations of the trained 2D and 3D machine learning models, potentially including: - a step of pooling 271 by max pooling for computing N series of at least one candidate 2D diagnostic score for each ROI of the N-fold plurality of extracted candidate K 2D patches, and - Pooling 272 step for calculating M series of at least one candidate 3D diagnosis score for each ROI of M times at least one extracted candidate 3D tile.

[0153] In some examples, the max pooling step also includes: -For each series j of N series of candidate K 2D diagnostic scores, a step of generating a maximum candidate 2D diagnostic score Sj by selecting the diagnostic score with the maximum value from the candidate K 2D diagnostic scores, thereby achieving a candidate maximum 2D diagnostic score for each series of N series, and wherein the trained 2D machine learning models for each series j of the N series are different from each other, preferably with different weights.

[0154] Advantageously, multiple K 2D patches are extracted and fed into a 2D machine learning model, which allows obtaining multiple outputs, such as diagnostic scores, such that the highest score corresponding to the ROI of highest interest for diagnosis can be selected (max pooling). For example, the highest score may mean that the ROI has a higher probability of being diagnosed as malignant or cancerous in a particular case.

[0155] Using different 2D machine models with different weights is also advantageous as this can improve the robustness of the predictions.

[0156] The integration 270 step may also include the step of outputting 275 a total score for each candidate ROI by pooling together the N 2D diagnostic scores of step 271 and the M 3D diagnostic scores of step 272. Thus, a total score for each candidate ROI is achieved.

[0157] As mentioned above, pooling is particularly advantageous in the context of the present disclosure because it allows highlighting high or low values ​​while reducing uncertainty. However, pooling methods are not limited to averaging scores, as using other methods to derive a score from 2D and 3D scores (e.g., XGBoost, weighted average, etc.) is also advantageous.

[0158] In order to enhance the robustness of the method 200 again, the step may further include the following steps: for the candidate ROI, based on the 3D medical image and by using a morphological-radiological feature extractor, extract 260 the morphological features of the candidate ROI. Then, the integration 270 step may include integrating the N 2D diagnostic scores from the 2D model, the M 3D diagnostic scores from the 3D model, and at least one of the morphological and / or radiological features extracted 260. The morphological features involved here may be described in detail in the same manner as described in detail above with respect to the method 100 according to the first aspect of the present invention.

[0159] Using morphological and / or radiomic features in addition to 2D and 3D diagnostic scores to characterize candidate ROIs will further enhance the robustness of the prediction.

[0160] The embodiments and aspects disclosed in detail herein may be described and elaborated in conjunction with systems, devices and methods intended to be exemplary and illustrative, but not limiting in scope. Specific configurations and details may be elaborated to provide an understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without some of the specific details presented herein. In addition, for purposes of clarity, some well-known steps or components may only be generally described or even omitted.

[0161] Some processes may be presented and described as a series of (sequential) steps. It should be understood that the order of the steps is exemplary, and the steps may be performed in an order different from that presented, some steps described may be omitted, and some additional steps may be omitted from the sequence and may be described elsewhere.

[0162] Reference may be made to the disclosures of existing patents, publications, and applications. Some text and drawings from these sources may be presented herein, but may be modified, edited, or commented upon to more smoothly merge with the disclosure of the present application. The citation or identification of any reference should not be construed as an admission that the reference may be used as prior art for the present disclosure.

[0163] The method described herein can be implemented on one or more computers of any form. Its components can be implemented as a dedicated application or in a client-server architecture including a web-based architecture, and can include functional programs, codes, and code segments. Any computer can include a processor, a memory for storing program data and executing program data, a permanent memory such as a disk drive, a communication port for processing communication with an external device, and a user interface device, including a display, a keyboard, a mouse, etc. When it comes to some software or algorithms, they can be used as program instructions that can be executed on a processor or computer-readable codes stored on a computer-readable medium, such as a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device. The computer-readable recording medium can also be distributed on a network-coupled computer system so that the computer-readable code is stored and executed in a distributed manner. The medium can be read by a computer, stored in a memory, and executed by a processor.

[0164] In order to promote the understanding of the principles of the various embodiments of the present disclosure, reference has been made to the preferred embodiments shown in the drawings, and specific language has been used to describe the embodiments. However, this specific language is not intended to limit the scope of the present disclosure, and the present disclosure should be interpreted as covering all embodiments that would normally occur to one of ordinary skill in the art.

[0165] The present disclosure can be described according to function block components and various processing steps.Such function blocks can be implemented by any number of hardware and / or software components configured to perform a specified function.For example, the present disclosure can adopt various integrated circuit components, such as memory elements, processing elements, logic elements, lookup tables, etc., which can perform various functions under the control of one or more microprocessors or other control devices.Similarly, in the case of using software programming or software elements to implement the elements of the present disclosure, the present disclosure can be implemented with any programming or scripting language (such as C, C++, Java, Python, assembly language, etc.), wherein various algorithms are implemented with any combination of data structures, objects, processes, routines or other programming elements.Functional aspects can be implemented in algorithms executed on one or more processors.In addition, the present disclosure can adopt any number of conventional techniques for electronic configuration, signal processing and / or control, data processing, etc.

Claims

1. A method (100) for generating a neural network for characterizing a plurality of regions of interest (ROIs) based on a plurality of 3D medical images, comprising the following steps: - providing (105) said plurality of 3D medical images, and - localizing (107) a plurality of ROIs in the plurality of 3D medical images and, for each ROI, defining (108) a 3D bounding box by a set of 6 extremal coordinates (Xmin, Xmax, Ymin, Ymax, Zmin, Zmax), and on the one hand: - for each defined 3D bounding box, performing 2D pre-processing (110) in order to extract a plurality of Ki 2D patches from each ROI included in said 3D bounding box, and - training (130) a 2D neural network based on said extracted 2D patches, on the other hand: - for each defined 3D bounding box, performing 3D pre-processing (120) in order to extract at least one 3D tile from each ROI included in said 3D bounding box, and - training (150) a 3D neural network for characterizing the plurality of ROIs based on the extracted 3D patches for the plurality of ROIs, wherein the step of 2D preprocessing (110) comprises at least extracting (115) a plurality of Ki 2D tiles as continuous and parallel slices relative to a single predetermined plane, and The 2D neural network is configured to be trained to characterize the plurality of ROIs based on the extracted plurality of 2D patches for the plurality of ROIs.

2. The method (100) according to claim 1, wherein: The single predetermined plane is parallel to one of the following three planes: the sagittal plane, the coronal plane and the axial plane, and is preferably parallel to the plane with the best resolution, and is further preferably parallel to the axial plane.

3. The method (100) according to any one of claims 1 or 2, wherein: The step of 2D pre-processing (110) further comprises, before the step of extracting (115) a plurality of Ki 2D tiles, a step of extracting 2D ROI coordinates by calculating the coordinates (Xmin, Xmax, Ymin, Ymax, Zmin, Zmax) of the center of each ROI in the axial view of each bounding box, wherein for all Z in [Zmin, Zmax], each center is defined, for example, by coordinates ((Xmin+Xmax) / 2, (Ymin+Ymax) / 2).

4. The method (100) according to claim 3, wherein: The step of extracting (115) a plurality of Ki 2D tiles comprises successively cropping a plurality of 2D tiles of size P×Q along the z-axis, for example P=Q=64.

5. The method (100) according to any one of the preceding claims, wherein: The step of 2D pre-processing (110) also comprises a first step of 2D voxel value normalization, which is implemented before or after the step of extracting (115) a plurality of Ki 2D tiles.

6. The method (100) according to any one of the preceding claims, wherein: The 3D preprocessing step (120): - comprising the step of extracting (125) at least one 3D tile of each ROI, and - Before the step of extracting (125) at least one 3D tile of each ROI, it also includes the step of extracting (121) 3D ROI coordinates by calculating the center coordinates of each ROI, wherein each ROI center is defined by coordinates ((Xmin+Xmax) / 2, (Ymin+Ymax) / 2, (Zmin+Zmax) / 2) for example.

7. The method (100) according to any one of the preceding claims, wherein: The step of extracting (125) at least one 3D tile comprises cropping the 3D tile of size P×Q×R around the center coordinates of each ROI, for example P=Q=R=64.

8. The method (100) according to any one of claims 5 to 7, wherein: The step of 3D pre-processing (120) further comprises: before the step of extracting (125) at least one 3D tile, a step of 3D volume resampling (122) of each 3D image, wherein the ROI has been localized (107).

9. The method (100) according to any one of the preceding claims, wherein: The steps of 2D training (130) include: - a step of initializing the 2D neural network (135), and - a step of continuing training of the 2D neural network (140), Wherein, in the case where the 2D neural network is designed to calculate a diagnosis score based on a 2D image patch, the initialization step (135) includes initializing the weights of the 2D neural network based on at least one of a randomly determined weight and a predetermined weight, and The continuing training step (140) includes: i. for each mini-batch of all extracted patches of the plurality of ROIs, a step of calculating a diagnostic score using a 2D neural network, wherein the mini-batch is randomly selected, ii. a step of calculating an error estimate by the 2D neural network for each previously calculated diagnostic score, iii. a step of updating said weights using a gradient-based method, iv. Repeat steps i to iii until a predefined stop condition is reached, This results in a trained 2D neural network with optimized weights.

10. The method (100) according to any one of claims 1 to 9, wherein: The 3D training step (150) includes: - a step of initializing the 3D neural network (155), and - a step of continuing training of the 3D neural network (160), Wherein, in the case where the 3D neural network is designed to calculate a diagnosis score based on a 3D image patch, the initialization step (155) includes initializing the weights of the 3D neural network based on at least one of a randomly determined weight and a predetermined weight, and The continuing training step (160) includes: i. for each mini-batch of all extracted patches of the plurality of ROIs, a step of calculating a diagnostic score using a 3D neural network, wherein the mini-batch is randomly selected, ii. for each previously calculated diagnostic score, a step of calculating an error estimate by the 3D neural network, iii. a step of updating said weights using a gradient-based method, iv. Repeat steps i to iii until a predefined stop condition is reached, This results in a trained 3D neural network with optimized weights.

11. A method (200) for characterizing a region of interest (ROI) based on at least one 3D medical image, comprising the following steps: - providing (205) the at least one 3D medical image, and - localizing (207) a ROI as a candidate ROI in the at least one 3D medical image, and - for the candidate ROI, defining (208) a candidate 3D bounding box by a set of 6 extrema coordinates (Xmin, Xmax, Ymin, Ymax, Zmin, Zmax), and on the one hand: - for the candidate 3D bounding box, performing 2D preprocessing (210) to extract a plurality of candidate 2D patches from the candidate ROI included in the candidate 3D bounding box, and - implementing (230) a trained 2D neural network configured to characterize said candidate ROI, on the other hand: - for the candidate 3D bounding box, performing 3D preprocessing (220) to extract at least one candidate 3D tile from the candidate ROI included in the candidate 3D bounding box, and - implementing (250) a trained 3D neural network configured to characterize the candidate ROI based on at least one extracted candidate 3D patch, and then - integrating (270) a plurality of 2D diagnostic scores provided by the trained 2D neural network and a plurality of 3D diagnostic scores provided by the trained 3D neural network, so as to output a total score for each of the candidate ROIs, wherein the step of 2D preprocessing (210) comprises at least extracting (215) a plurality of candidate K 2D tiles as continuous and parallel slices relative to a single predetermined plane, and Therein, the step of implementing (230) the trained 2D neural network is performed based on the extracted plurality of candidate 2D tiles.

12. The method (200) according to claim 11, wherein: The trained 2D and the trained 3D neural networks have been trained by implementing the method (100) for generating a neural network according to any one of claims 1-10.

13. The method (200) according to any one of claims 11 and 12, wherein: The single predetermined plane is parallel to one of the following three planes: the sagittal plane, the coronal plane and the axial plane, and is preferably parallel to the plane with the best resolution, and is further preferably parallel to the axial plane.

14. The method (200) according to any one of claims 11 to 13, wherein: The step of 2D pre-processing (210) further comprises, before the step of extracting (215) a plurality of candidate K 2D patches, a step of extracting 2D ROI coordinates by calculating the coordinates (Xmin, Xmax, Ymin, Ymax, Zmin, Zmax) of the center of the candidate ROI in an axial view of the candidate 3D bounding box, wherein for all Z in [Zmin, Zmax], each center is defined, for example, by coordinates ((Xmin+Xmax) / 2, (Ymin+Ymax) / 2).

15. The method (200) according to claim 14, wherein: The step of extracting (215) a plurality of candidate K 2D tiles comprises successively cropping a plurality of K candidate 2D tiles of size P×Q, for example P=Q=64, along the z-axis.

16. The method (200) according to any one of claims 11 to 14, wherein: The step of 2D pre-processing (210) further comprises a second step of 2D voxel value normalization, which is implemented before or after the step of extracting (215) a plurality of candidate K 2D tiles.

17. The method (200) according to any one of claims 11 to 16, wherein: The 3D pre-processing step (220): - comprising the step of extracting (225) at least one candidate 3D tile of said candidate ROI, and - before the step of extracting (225) at least one candidate 3D tile of the candidate ROI, further comprising the step of extracting 3D ROI coordinates by calculating the center coordinates of the candidate ROI, wherein the candidate ROI center is defined by coordinates ((Xmin+Xmax) / 2, (Ymin+Ymax) / 2, (Zmin+Zmax) / 2), for example.

18. The method (200) according to claim 17, wherein: The step of extracting (225) at least one candidate 3D tile comprises cropping the candidate 3D tile of size P×Q×R around the center coordinates of the candidate ROI, for example, P=Q=R=64.

19. The method (200) according to any one of claims 17 and 18, wherein: The step of 3D pre-processing (220) further comprises: performing 3D volume resampling on each 3D image before the step of extracting (225) at least one candidate 3D tile of the candidate ROI, wherein the candidate ROI has been localized (207).

20. The method (200) according to any one of claims 11 to 19, wherein: The step of implementing (230) the trained 2D neural network is performed N times based on the plurality of extracted candidate K 2D patches to calculate K candidate diagnostic scores for one candidate ROI at a time, thereby implementing a series of candidate K 2D diagnostic scores for each of the N series.

21. The method (200) according to any one of claims 11 to 20, wherein: The step of implementing (250) the trained 3D neural network is performed M times based on at least one extracted candidate 3D image block to calculate at least one candidate diagnosis score for one candidate ROI each time, thereby implementing at least one candidate 3D diagnosis score for each series of M series.

22. The method (200) according to any one of claims 20 and 21, wherein: The step of integrating (270) includes: - a step of pooling (271) by maximum pooling for calculating at least one candidate 2D diagnostic score for N series of each ROI of the N plurality of extracted candidate K 2D tiles, wherein the step of maximum pooling comprises: for each series j of the N series of candidate K 2D diagnostic scores, generating a candidate maximum 2D diagnostic score Sj by selecting the diagnostic score with the maximum value from the candidate K 2D diagnostic scores, thereby achieving a candidate maximum 2D diagnostic score for each series of N series, wherein the trained 2D neural networks for each series j of the N series are different from each other, preferably have different weights, and - Pooling (272) step for calculating at least one candidate 3D diagnosis score of said M series for each ROI of said at least one extracted candidate 3D tile.

23. The method (200) of claim 22, wherein: The step of integrating (270) further includes the step of outputting (275) a total score for each of the candidate ROIs by pooling together the N 2D diagnostic scores of step (271) and the M 3D diagnostic scores of step (272).

24. The method (100) according to any one of claims 1-10 or the method (200) according to any one of claims 11-23, wherein: The 3D medical image includes at least one of: a computed tomography (CT) image and a magnetic resonance imaging (MRI) image.

25. A computer program product comprising instructions which, when implemented by at least one digital processing device, perform at least one of the steps of the method (100) according to claims 1-10 and the steps of the method (200) according to claims 11-23.

Images