Improved depiction of image-level annotations for precise training of medical image segmentation models

Abstract

The present disclosure provides an improved depiction of image-level annotations for precise training of a medical image segmentation model. Methods and systems for refining annotations in medical images are presented. The process begins from a medical image that receives an initial coarse annotation followed by cropping the image to focus on a specified region of interest. The pixel intensity distribution within this region is subjected to an analysis to detect different patterns and determine an optimal threshold. Using these thresholds, the image is segmented and then refined by an iterative expectation maximization algorithm. Similarity matching with a reference structure is performed using structural similarity index calculations, thereby ensuring accuracy of annotations. According to the innovative method, the conventional image processing technology and the deep learning method are fused, so that refined voxel-level annotation is realized from rough image-level indication. According to the method, the accuracy of annotation is greatly improved, the manual labor is reduced, the performance of a machine learning model is enhanced, and the use of computing resources is optimized.

Description

Technical Field

[0001] The present invention relates to the field of medical imaging and more specifically to a method for refining image annotations for preparing these images as training data for machine learning models. The aim is to improve the training of medical image segmentation models using deep learning. The proposed solution can also be used as a standalone tool to refine the standard designation of regions of interest (ROIs) on annotated images so that the annotations better fit the target region, such as a pathology. This also facilitates the proper analysis of the annotated medical images. Background Art

[0002] In recent years, the technology of artificial intelligence has undergone significant progress. One of the most promising areas of use is the application of this technology to medical image processing. This transformation has led to the automation of many tasks that were previously manual, that is, performed by skilled humans (from reading to annotation or even diagnosis), resulting in more efficient processes. However, for the success of machine learning methods and deep learning methods in this field, accurate training data is required, that is, the availability of medical images with accurate annotations as reference and test data for training models is crucial.

[0003] Traditionally, the gold standard method for producing these reference annotations is manual annotation, that is, annotations made by skilled people (technicians or medical practitioners). Although it may be considered the most accurate, manual annotation has its own set of challenges. First, it requires the participation of trained professionals such as technicians, radiologists, or pathologists, who bring their expertise to the task. However, this manual process is inherently time-consuming and can introduce variability because the interpretations of professionals may differ from each other. The lack of comprehensive guidelines for annotations of specific medical issues such as white matter hyperintensities (WMH) exacerbates this variability. In a federated learning environment, this challenge is further amplified because annotations from multiple annotators at different sites are likely to introduce higher interpretation variability.

[0004] Furthermore, for AI models, especially those based on deep learning, regular updates to adapt to new changes and incorporate feedback from users are crucial. A common practice is to collect feedback annotations from radiologists or pathologists during their daily use of the application. However, given their primary responsibilities and time constraints, these professionals usually provide rough annotations, such as bounding box annotations or hand-drawn annotations, rather than detailed annotations. Unfortunately, these rough annotations on the images are not precise enough to be used directly to retrain the model. In most cases, a team of skilled technicians must reprocess the annotated image data provided by medical practitioners as feedback in order to obtain more accurately annotated images that can be used as training data for the model. Such an approach requires training of skilled technicians and is very time-consuming. In addition, it is prone to errors and changes.

[0005] Therefore, an automated method is needed to bridge this gap, thereby effectively transforming the rough annotations or image-level annotations from medical practitioners into detailed voxel-level annotations that can be used to train models. The present invention aims to address this need, thereby introducing an innovative automated method for refining medical data with annotations. Summary of the invention

[0006] The present invention proposes an innovative method for refining and enhancing medical image annotations, in particular, an innovative method for moving from the coarse annotations or image-level annotations typically provided by medical professionals to more precise voxel-level annotations that are crucial for training robust deep learning models. It should be noted that the proposed solution for changing (improving the precision of the annotations) can also be used as a standalone tool to refine the standard designation of regions of interest (ROIs) on annotated images so that the annotations better fit the target region, such as a pathology. This also facilitates appropriate analysis of the annotated medical images.

[0007] The process begins by focusing on an area or region of interest through image cropping, using as reference hand-drawn annotations or bounding boxes around pathologies provided by radiologists or pathologists. This initial cropping step serves a dual purpose: it emphasizes the target pathology and significantly reduces the initial computational load by limiting the amount of data to be processed in subsequent steps.

[0008] After this, a histogram analysis of the cropped region is performed. This analysis allows different image objects to be distinguished based on their pixel intensity distribution. Interval thresholds are determined by analyzing these histogram patterns and combining them with existing knowledge about the image, such as the specific organ in question, image modality, or imaging sequence. These values ​​play an important role in segmenting images, distinguishing pathologies or regions of interest (ROIs) from other surrounding structures.

[0009] The segmented images resulting from the application of these thresholds then become the input data for an expectation maximization algorithm, which is a tool used to generate an initial refined segmentation. For greater flexibility, one may also choose to use a convolutional neural network (CNN) model for this step.

[0010] Another novel aspect of the present invention is that it incorporates a structural similarity index (SSIM) calculation. Using tools from Python such as the "scikit-image" package, the SSIM between the newly generated cropped volume or segmentation and similar structures in the training set is calculated. The SSIM metric, which ranges between -1 and 1, provides a quantitative measure of similarity, with a value of 1 indicating a perfect match.

[0011] Using this similarity metric, the segmented structures from the training set with the highest SSIM are used to fine-tune and refine the segmentation produced in the previous step. This may involve processes such as edge smoothing to ensure that the final annotation is as accurate as possible.

[0012] In practical applications, the use of the invention has resulted in the successful conversion of coarse annotations of structures such as kidneys from CT images or white matter hyperintensities from FLAIR images and MPRAGE images into highly accurate voxel-level annotations. The significance of the invention is far-reaching: it provides a cost-effective, time-efficient solution for generating high-quality annotations, greatly reducing the variability introduced by different annotators, reducing errors and reducing the required computer resources (required storage and computing power). BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims.

[0014] Figure 1 is a diagram showing a conventional method for performing update training of a model.

[0015] Figure 2 It is a diagram showing the general concept of the method in the embodiment of the present invention.

[0016] Figure 3is a diagram of a method for obtaining training data from coarsely annotated data.

[0017] Figure 4 Several steps (a) to (f) of a method for automatic annotation are shown, which aims to obtain voxel-level annotations according to an embodiment of the invention starting from a coarse annotation of the kidney from a CT image.

[0018] Figure 5 Several steps (a) to (f) of a method for automatic annotation are shown, which aims to obtain voxel-level annotations according to an embodiment of the invention starting from a coarse annotation of WMHs from FLAIR images.

[0019] Figure 6 Several steps (a) to (f) of a method for automatic annotation are shown, which aims to obtain voxel-level annotations according to an embodiment of the invention starting from a coarse annotation of WMHs from MPRAGE images.

[0020] Figure 7 Shown are the starting point (a) and the result (b) of a method for automatic annotation, which aims to obtain voxel-level annotations starting from a rough ellipse annotation of a WMH according to an embodiment of the invention. DETAILED DESCRIPTION

[0021] The terms used in this specification are selected to include currently widely used general terms. In some cases, the terms may be terms arbitrarily established by the applicant. In such cases, the meaning of the terms will be defined in the relevant parts of the specific embodiments. Therefore, the terms used in the specification should not be simply defined by the names of the terms, but should be defined based on the meaning of the terms and the overall description of the present disclosure.

[0022] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.It should be noted that throughout the drawings, the same reference numerals are used to designate the same or similar elements.

[0023] It should also be noted that the present application relates to medical images, however, its teachings can be used in any type of image that includes coarse image-level annotations that need to be refined to obtain pixel-level or voxel-level annotations. The description mainly relates to three types of medical imaging techniques: CT, FLAIR, and MPRAGE. However, the teachings of the present invention are not limited to use with these types of medical images.

[0024] Computed tomography, commonly referred to as CT or CAT scans, is a medical imaging technique that produces detailed pictures of the body's internal structures by taking a series of X-ray images from different angles. This method allows the creation of cross-sectional images in greater detail than conventional X-rays, which can be used to examine various internal organs, bones, soft tissues, and blood vessels.

[0025] Fluid Attenuated Inversion Recovery, commonly referred to as FLAIR, is a specific sequence used in magnetic resonance imaging (MRI). The main purpose of FLAIR is to suppress the signal from fluid, especially the cerebrospinal fluid found in the brain. By doing so, it makes abnormalities like multiple sclerosis plaques or some strokes that are surrounded by this fluid more obvious and easier to identify.

[0026] Magnetization Prepared Rapid Acquisition Gradient Echo, abbreviated as MPRAGE, is another sequence used in MRI that is specifically designed to produce detailed, T1-weighted images of the brain. This imaging technique is particularly useful for capturing fine details of brain structures. Typically, MPRAGE images are utilized in studies that require detailed structural brain imaging, such as studies on neurodegenerative diseases.

[0027] The present invention revolves around a comprehensive approach to refine image annotation in the field of medical imaging. The main goal is to move from the typically coarse and general image-level annotations to more precise and detailed accurate voxel-level annotations using classical image processing techniques and the fusion of available training data.

[0028] Figure 1 A conventional method according to the conventional technology is shown.

[0029] In step 101, a radiologist or medical practitioner roughly annotates a medical image. For example, such annotation can be done using a mouse on a computer or using a pen-like object on a tablet computer or using any other type of rough selection performed from a user interface of any type of computer. The image data can be a two-dimensional image or a three-dimensional image. In the two-dimensional case, the medical practitioner can make a rough selection by drawing an ellipse, a circle, a square or any two-dimensional geometric figure that defines the perimeter of a region of interest (ROI) in the image. This ROI can be used for the description of a pathology or the identification of any specific element in the image. The image data can also be a three-dimensional image. In that case, the selection can be made by the following operation: in the same way as a computer extrude a 3D volume from a 2D input, or by selecting any other type of volume in a 3D image with the help of known tools for human-computer interaction. The annotated image data obtained is then stored.

[0030] In step 102, annotators may be trained to refine annotations in the image data. Alternatively or additionally, computer technicians may be trained to perform such refinement of annotations.

[0031] In step 103, a trained annotator or technician will perform the modification of the annotations.

[0032] In step 104, annotation correction is performed by the radiographer.

[0033] In step 105, the annotations are reviewed by a clinical expert.

[0034] In step 106, the annotation consistency is checked using a predefined threshold. In case the annotation consistency is below the threshold, step 103 is restarted. If this check is successful, the refined training data is stored in step 107. In step 108, this data can be used to retrain the model.

[0035] This method is essentially based on human expertise, which comes from medical practitioners who make rough annotations, from trained annotators who perform modifications and corrections, and from clinical experts who perform the review of annotations. In other words, this method is very time-consuming, error-prone, and has a lot of variations due to the number and type of participants. In addition, it requires storing a significant amount of data corresponding to the work in progress data at each step of the process, performing several calculation steps, and performing several transmissions of these work in progress data. That is, also from a purely computer technology perspective, this method has several disadvantages, which are related to the inefficient use of computer resources.

[0036] Figure 2 The method according to the present invention is shown. Step 201 is similar to Figure 1 Step 101 is performed in the method of the conventional technology shown.

[0037] However, in step 202, an annotation refinement tool implemented on a computer is used on the coarsely annotated image data as input to obtain - after computation - an accurate voxel-level annotated image as output.

[0038] The step 203 of storing and the step 204 of retraining the model are similar to the steps 107 and 108 discussed above for the conventional method.

[0039] Figure 3 A diagram showing a method for obtaining training data from coarsely annotated data according to an embodiment of the present invention.

[0040] In a first step 301, i.e., in an image cropping step, the roughly annotated image data is processed so that the image-level annotations provided by the radiologist or medical practitioner (e.g., as freehand annotations, see steps 101 and 201 discussed above) result in cropping. This step simplifies the subsequent process because the region of interest (a two-dimensional surface or a three-dimensional volume) annotated by the radiologist or pathologist is cropped to focus on the pathology or element of interest in the image. By focusing on this cropped region, regions of no interest are discarded, resulting in a more efficient and accurate annotation refinement process, and also reducing the amount of data stored and processed in the subsequent steps of the method.

[0041] The second step 302 after cropping is a multimodal histogram analysis of the cropped area.

[0042] Multimodal histogram analysis is a method for understanding the distribution of pixel intensities (or voxels in the case of 3D images) in an image. When the distribution of these pixel intensities is represented in graphical form, the resulting graphical depiction is called a histogram. In particular, a histogram that exhibits multiple significant peaks or "modes" indicates that there are several major groups of pixel intensities in the image.

[0043] In the field of medical imaging, these patterns can represent different tissue types or specific anatomical structures or pathologies. For example, a specific peak may be associated with the intensity value characteristics of bone tissue, while another different peak may indicate adipose tissue or a tumor. Such analysis helps to determine the range of pixel intensities associated with specific anatomical structures or pathological structures, thereby facilitating the differentiation or "segmentation" of these structures within the image. In addition, when images derived from different imaging modalities such as magnetic resonance imaging (MRI) and computed tomography (CT) are evaluated simultaneously, each modality contributes its unique set of pixel intensity values. A comprehensive analysis of the aggregated histogram can enhance the understanding of the anatomical and pathological structures under scrutiny.

[0044] In an embodiment of the present invention, this multimodal histogram analysis delves into the distribution of pixel values ​​or voxel values ​​within a cropped surface or volume.By examining patterns in the histogram, different image objects, such as organs, tumors, and background areas, can be identified.

[0045] The third step 303 is a step of determining a threshold value of an interval.

[0046] This step is intricately linked to the multimodal histogram analysis step performed on the region of interest within the cropped image. Its main purpose is to achieve a clear and accurate delineation of the structures or entities present in that particular region. After isolating the desired region by cropping the image around the annotations provided by the radiologist or pathologist, the histogram of this cropped region is thoroughly examined. This examination attempts to identify different patterns in the histogram that correspond to different structures, such as organs or abnormalities like tumors, compared to the background region. To perform this step, the process incorporates available prior knowledge about the image in question. This can include: specific attributes of the organ or condition being viewed, the type of imaging modality used (such as CT or MRI), and potential unique image sequences (if relevant). In this step, a database with image information about organs, modalities, and sequences can be used to feed the data.

[0047] The method then either obtains input from the user to set an appropriate threshold around the detected pattern or automatically determines this appropriate threshold. These values ​​effectively act as dividing lines within the histogram. They distinguish between pixel intensities corresponding to different structural entities within the image. As an illustrative example, if the image shows a tumor with higher intensity values ​​than the surrounding tissue, the selected threshold will be optimally located between these two intensity groups, enabling accurate segmentation of the tumor.

[0048] This segmented image created using the determined threshold is then used as the starting point for a fourth step 304 using expectation maximization (EM). It is used to improve the accuracy of the preliminary image segmentation derived from the threshold. This refinement improves the results of the preliminary segmentation, particularly suitable for the inherent challenges posed by medical images, such as variability in tissue contrast and potential artifacts. Initially, the segmented image from the thresholding is used as a base guess or prior for the EM algorithm. This preliminary segmentation provides a starting point for the optimization process of EM. The EM algorithm operates through two main iterative steps: an expectation step and a maximization step.

[0049] In the expectation step, for each pixel in the image, the algorithm calculates the probability that the pixel belongs to a particular segment or structure. This is based on the model parameters, effectively assigning soft or probabilistic labels to the pixels, suggesting their likely classification.

[0050] Next comes the maximization step, which uses the probabilities determined in the expectation step to adjust and refine the model parameters. The goal of this stage is to slightly adjust these parameters to provide the most appropriate explanation for the observed data, pushing the segmentation towards higher accuracy.

[0051] This cycle of estimating probabilities and then maximizing the fit based on these probabilities is repeated. As it iterates, the segmentation of the image becomes increasingly refined. This iterative process ends when the change in the model parameters between iterations is minimal, indicating that the optimal segmentation solution has been achieved.

[0052] In cases where more advanced methods are needed, a Convolutional Neural Network (CNN) model can be trained to generate the initial segmentation using available prior data.

[0053] To ensure the quality and accuracy of the generated segmentation, the method includes a fifth step 305 of performing similarity matching. This is achieved by using a "structural similarity index (SSIM) calculation". This is performed using the initial training data of the model or by utilizing known data such as the "scikit-image" package from Python, and then calculating the SSIM between the newly generated cropped volume or initial segmentation and similar structures found in the training set. The SSIM provides a range from -1 to 1, where a value of 1 indicates a perfect match.

[0054] Finally, to maximize the quality of the annotation, a "fine-tuning" step 307 is performed. Here, the structure from the training dataset with the highest SSIM value is used, indicating the closest match. This structure is used to refine and optimize the segmentation mask generated in the previous step, thereby ensuring the highest accuracy annotation.

[0055] In practical applications, this approach has shown impressive results. For example, coarse annotations of kidneys from CT images were seamlessly converted to voxel-level annotations. Similarly, white matter hyperintensities (WMH) annotations from both FLAIR images and MPRAGE images were meticulously refined to voxel level using the described steps.

[0056] Figure 4 is an illustration of such a practical application of the method for automatic annotation starting from a rough annotation of the kidney from a CT image.

[0057] Figure 4 Sub-figure (a) of shows a CT image 401 of a human body 402 focused on the pelvic region. A rough annotation 403 has been performed around the kidneys 404. This roughly annotated image is used to Figure 3 Inputs to the methods shown.

[0058] Figure 4 Sub-figure (b) of shows a cropped image 405 (and hence an image of the cropped data) of the region of interest defined by the rough annotation.

[0059] Figure 4Sub-figure (c) of shows the result of a histogram analysis with the distribution of pixel intensities. Three modes M1, M2 and M3 can be identified as three peaks in the histogram. For this image, the object of interest is bright, which means that the mode with the brightest intensity is used. For example, a threshold 406 can be set in this mode by user selection.

[0060] Figure 4 Sub-figure (d) of shows pixels selected for the segmented image 407 .

[0061] Figure 4 Sub-figure (e) of FIG. 4 shows a mask 408 obtained by the method of the present invention, and Figure 4 Sub-figure (f) of shows mask 409 according to the ground truth.

[0062] Figure 5 is another illustration of such a practical application of the method for automatic annotation starting from a rough annotation of WMHs from FLAIR images.

[0063] Figure 5 Sub-figure (a) of shows a FLAIR image 501 of a human body 502 with the skull and brain regions focused. A rough annotation 504 has been performed around white matter hyperintensities (WMH) 503. This image is used for Figure 3 Inputs to the methods shown.

[0064] Figure 5 Sub-figure (b) of shows a cropped image 505 (and hence an image of the cropped data) of the region of interest defined by the rough annotation.

[0065] Figure 5 Sub-figure (c) of shows the result of a histogram analysis with the distribution of pixel intensities. Two modes M1 and M2 can be identified as two peaks in the histogram. For this image, the object of interest is bright, which means that the mode with the brightest intensity is used. For example, a threshold 506 can be set in this mode by user selection.

[0066] Figure 5 Sub-figure (d) of shows selected pixels for the segmented image 507 .

[0067] Figure 5 Sub-figure (e) of shows a mask 508 obtained by the method of the present invention, and Figure 5 Sub-figure (f) of shows mask 509 according to the true value.

[0068] Figure 6 is another illustration of such a practical application of the method for automatic annotation starting from a rough annotation of WMH from MPRAGE images.

[0069] Figure 6 Sub-figure (a) of shows an MPRAGE image 601 of a human body 602 with the skull and brain regions focused. A rough annotation 604 has been performed around white matter hyperintensities (WMH) 603. This image is used for Figure 3 Inputs to the methods shown.

[0070] Figure 6 Sub-figure (b) of shows a cropped image 605 (and hence an image of the cropped data) of the region of interest defined by the rough annotation.

[0071] Figure 6 Sub-figure (c) of shows the result of a histogram analysis with the distribution of pixel intensities. Two modes M1 and M2 can be identified as two peaks in the histogram. For this image, the object of interest is dark, which means that the mode with the lowest brightness intensity is used. For example, a threshold 606 can be set in this mode by user selection.

[0072] Figure 6 Sub-figure (d) of shows selected pixels for the segmented image 607 .

[0073] Figure 6 Sub-figure (e) of shows a mask 608 obtained by the method of the present invention, and Figure 6 Sub-figure (f) of shows mask 609 according to the true value.

[0074] Figure 7 The beginning and the end of a roughly annotated FLAIR image are shown. Figure 7 The sub-figure (a) shows that the result is Figure 7 It is shown in the sub-figure (b) of Figure 7 In sub-figure (b) of , voxel-level annotations 702 of the WMH are obtained. In this context, it should be noted that this also shows how it is possible to use the solution as a standalone tool as described above.

[0075] The refined image data with voxel-level annotations thus obtained have the necessary features to be used in the retraining of the model.

[0076] In an embodiment of the present invention, the model is retrained using the newly acquired refined image data and the model is used for diagnosis.

[0077] In summary, the present invention addresses the need for accurate and consistent annotations. Traditional methods of manual annotation, which although considered the gold standard, are time consuming and can introduce significant interpretation variability. This is particularly evident in the field of federated learning, which is primarily used for medical image annotation.

[0078] The present invention provides an innovative method for refining coarse image-level annotations to generate precise voxel-level annotations. Through an innovative combination of traditional image processing techniques and modern deep learning methods, the present invention successfully transforms general image-level instructions into detailed voxel-level annotations. This breakthrough not only ensures improved accuracy, but also significantly reduces the time and cost associated with the annotation process.

[0079] Furthermore, the adoption of the present invention ensures a significant reduction in interpretation variability, ultimately leading to consistent and reliable model performance. As the cornerstone of the effectiveness of deep learning methods, the quality of annotations is significantly improved, translating into superior model performance results.

[0080] It should be noted that the method outlined in the present invention also produces a significant improvement in the efficiency of computer resource utilization. The reduced computational overhead produced by the present invention ensures that computer resources, which are often limited in large-scale medical imaging analysis, are used more effectively and efficiently. In fact, the present invention limits the storage of working images in progress and limits the storage and computation of segmented images.

Claims

1. A method for refining annotations in an image, comprising the following steps: obtaining an initially annotated image (401, 501, 601) as an image including the annotation (403, 504, 604); Cropping (S301) the initially annotated image to obtain a cropped image (405, 505, 605), the cropped image (405, 505, 605) retaining only the portion of the initially annotated image indicated by the annotation; analyzing ( S302 ) pixel intensity distribution within the cropped image; Segmenting (S303) the cropped image based on pixel intensity analysis to obtain a segmented image (407, 507, 607); thinning (S304) the segmented image to obtain a thinned segmented image; as well as Similarity matching is performed (S305) on the thinned image to obtain a delineation mask (408, 508, 608).

2. The method according to claim 1, wherein: The annotations (403, 508, 608) in the initially annotated image are image-level annotations.

3. The method according to claim 1 or 2, wherein: The initially annotated image is a medical image.

4. A method according to any one of the preceding claims, wherein: The cropping step (S301) includes retaining only the portion of the image within the perimeter defined by the annotation to focus on the region of interest defined by the annotation, and removing the portion of the image outside the perimeter defined by the annotation.

5. A method according to any one of the preceding claims, wherein: The analyzing step (S302) includes performing a histogram analysis on the pixel intensities within the cropped image.

6. The method according to claim 5, wherein: Performing a histogram analysis of pixel intensities within the cropped image also includes detecting a pattern and determining a threshold.

7. The method according to claim 6, wherein: The segmentation step (S303) further includes segmenting the cropped image based on the determined threshold.

8. A method according to any one of the preceding claims, wherein: The step of thinning (S304) the segmented image comprises using an iterative expectation-maximization EM algorithm.

9. The method according to claim 8, wherein: The iterative EM algorithm uses the segmented image as an initial guess or prior.

10. The method according to claim 9, wherein: The EM algorithm using the iterations includes: Perform the desired steps to calculate the probability of each pixel belonging to a particular segment or structure, and A maximization step is performed to adjust model parameters based on the probabilities derived from the expectation step.

11. A method according to any one of the preceding claims, wherein: Performing similarity matching on the refined image includes using a structural similarity index SSIM calculation and performing a fine-tuning step in which the delineation mask is optimized using the structure from the training dataset with the highest SSIM value.

12. A method according to any one of the preceding claims, wherein: The delineation mask (408, 508, 608) is a voxel-level annotation.

13. A data processing system comprising a processor adapted to be configured to perform the steps of the method according to any one of claims 1 to 12.

14. A computer program product comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method according to any one of claims 1 to 12.

15. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method according to claim 1.

Images