Method for generating image annotation tool

Abstract

The invention describes a computer-implemented method of a tool (T) for generating annotations for an image (M), preferably a medical image (M), the method comprising the steps of: providing access to a base model (13); providing a graphical user interface (140) adapted to enable a user (P) to specify a result of a desired image annotation task; training the client-side machine learning model (12) using the local database (11) of annotated images to adapt the base model (13) to perform the desired image annotation task based on the specified results; and augmenting the graphical user interface (140) with a graphical element (142T) representing the tool (T) to perform the desired image annotation task. The invention also describes an image processing system (1) adapted to perform the steps of such a method.

Description

Background Art

[0001] Medical imaging plays an important role in the diagnosis and treatment of cancer. Images are obtained using modalities such as radiography or magnetic resonance imaging, which are analyzed to detect abnormalities, such as lesions or nodules. Typically, a practitioner such as a radiologist can view the images in an image reader application, such as an application that complies with the DICOM (Digital Imaging and Communications in Medicine) standard. Such an image reader application may have a graphical user interface (GUI) that allows the user to annotate any relevant features in the image. For example, an image reader application may allow the practitioner to add distance lines to the image to indicate the size of an abnormality in the image, the distance to another anatomical feature visible in the image, etc., and add labels. These annotations are stored with the image in the patient's medical record. Because subsequent health care decisions may be based on quantitative measurements of any abnormalities visible in the patient's images, it is important to measure accurately and consistently.

[0002] However, image reader applications deployed in today's medical imaging platforms typically provide only a limited set of tools with which users can annotate images. This means that obtaining the desired quantification may be too difficult or time consuming, for example, the image reader's toolset may not provide any means for easily measuring the distance between the center of an abnormality and further anatomical features. When it is too difficult to label an image with such information, that information - even though it may be highly relevant - may not be written into the medical record.

[0003] If a user wishes to utilize a particular tool, it is necessary to specify what the tool should do and give the tool specification to the tool developer. The tool developer then programs the tool and updates the image reader application software to include the tool. The tool can then be provided to the user as part of an update to the image reader application. However, the user must wait for the tool for some time, and the associated costs can be high. This effort is repeated each time the user requires a particular tool.

[0004] It is therefore an object of the present invention to facilitate the annotation of medical images.

[0005] This object is achieved by the claimed computer-implemented method of generating an image annotation tool and the claimed image processing system. Summary of the invention

[0006] According to the present invention, a computer-implemented method for generating an image annotation tool includes the following steps: providing access to a base model that has been previously trained to perform self-supervised or supervised tasks on images, such as denoising, compression, filling in gaps ("inpainting"), etc.; providing a graphical user interface suitable for enabling a user to specify the results of a desired image annotation task; training a client-side machine learning model using a local database of previously annotated images to adapt the base model to perform the desired image annotation task based on the specified results; and augmenting the graphical user interface with graphical elements representing a tool for performing the desired image annotation task.

[0007] One advantage of the computer-implemented method of the present invention is that it enables a user (eg, a radiologist) to define a custom tool based on previously made annotations already stored in a local database.

[0008] The method of the present invention provides advantages over the prior art, in which it is difficult or even impossible for radiologists to define custom image annotation tasks. In some systems, in order to incorporate a specific annotation task into the image analysis workflow, it may be necessary to commission an external developer to program a tool in the image processing software, obtain regulatory approval for the tool, and then install a software update locally to enable the new tool to be invoked. However, the cost of providing custom tools in this way may be prohibitive.

[0009] Instead, the computer-implemented method of the present invention allows the user to specify what the desired tool should do, and then adapts the underlying model to perform the desired task. Because any such custom tool is essentially defined and validated by qualified practitioners, regulatory approval of the custom tool is not necessary.

[0010] According to the invention, an image processing system adapted to perform the steps of the method of the invention comprises means for accessing a base model; a user interface adapted to enable a user to specify the result of a desired image annotation task; a client-side machine learning model configured to be trained using a local database of annotated images to adapt the base model to perform the image annotation task based on the specified result, i.e., the base model is adapted to behave as a "custom tool" that performs the task. The user interface of the image processing system of the invention is also adapted to display an image and enable a user to invoke a custom tool to perform a new image annotation task on the displayed image.

[0011] Because the image processing system of the present invention is capable of creating customized tools to perform image annotation tasks, the image processing system may be referred to herein as a "custom tool creator."

[0012] In the context of the present invention, the images are preferably diagnostic medical images and the image processing system is preferably distributed as an application comprising a DICOM standard image reader.

[0013] The object of the invention is also achieved by a computer program product comprising instructions which, when executed by a processor of the image processing system of the invention, cause the processor to perform the steps of the method of the invention. The invention also describes a computer readable data carrier having such a computer program product stored thereon.

[0014] The dependent claims give particularly advantageous embodiments and features of the invention, as revealed in the following description. Features of different claim categories may be combined as appropriate to give further embodiments not described here.

[0015] In the context of the present invention, a machine learning model or machine learning algorithm (MLA) may be implemented as a module of a computer program that may run on any suitable platform. The expressions "machine learning model", "machine learning algorithm", "neural network" and "artificial intelligence tool" should be understood as synonyms and may be used interchangeably herein.

[0016] Medical images may be obtained using any of several imaging modalities, such as computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), single photon emission computed tomography (SPECT), X-ray imaging, ultrasound imaging, optical coherence tomography (OCT), photoacoustic imaging, etc. Depending on the imaging modality, the medical images may be in the form of 3D volumes or 2D slices.

[0017] A "finding" is understood to be an annotation superimposed on a medical image and used to mark an abnormality detected in that image. A finding may be a straight distance line placed between two points, a closed line drawn around a contour, etc. A finding may include a label, such as a number specifying the length of the line. The method of the present invention makes use of such a "finding database", i.e. a database comprising any medical image previously annotated by a practitioner to show relevant findings.

[0018] The base model or "basic model" used in the context of the present invention should be understood as a large machine learning model that has been trained using a large number of unlabeled medical images to perform one or more self-supervised tasks, such as denoising, image compression, gap filling, etc., and can be adapted or fine-tuned to a variety of tasks. The base model can be trained using any suitable transfer learning or representation learning algorithm. For example, a medical image database of at least several hundred training data sets (each data set includes medical images), and more preferably, several thousand training data sets can be used to train the base model.

[0019] The base model discussed here may include all kinds of medical images across the entire anatomical range. Likewise, the base model may include images taken using a variety of different modalities, such as CT scans, MRI angiograms, X-rays, etc. However, in particularly preferred embodiments of the present invention, the base model includes images taken using a single type of modality, because useful information can be more easily obtained from comparisons of images originating from the same type of imaging device.

[0020] The system allows the user to start the custom tool development process. In a particularly preferred embodiment of the present invention, when the user starts this process, the image reader application shows one or more candidate results or "prompts" in the user interface, allowing the user to select any result that matches the desired result of the custom tool. The candidate results are preferably selected from images in the discovery database. Each candidate result can be regarded as a suggestion on how the custom tool can edit the image to include the discovery. For example, a PACS (picture archiving and communication system) for storing diagnostic images may include an annotation tool that allows the user to overlay annotations (e.g., distance lines, dimensions) on the image, and these annotations are stored in the PACS together with the image. Candidate results or prompts can be selected from these annotated images. By selecting a prompt, the user specifies the result of the desired image annotation task. Based on the user's selection, the client-side machine learning algorithm ("client-side MLA") is trained to adapt or "fine-tune" the base model to perform the desired task. The candidate results or "candidate findings" will be the basic facts of the client-side MLA. In fact, the client-side MLA modifies or adapts the neural network parameters of the base model to fine-tune it to perform the custom task, such as by adjusting one or more layers of the base model network, by reconfiguring the entire base model network, etc. The custom model can now perform the specific task for which it has been adjusted, referred to herein as a "custom tool" or "custom model". Any suitable machine learning classifier can be deployed as a client-side MLA. For example, using a 3D image volume as input, a 3D machine learning classifier such as a residual network with multiple layers can take an image subvolume as input and classify the subvolume as a "finding subvolume" or a "normal patch subvolume". This approach ensures that findings such as distance lines can be distinguished from normal patches in the last layer of the neural network. In order to increase the number of subvolume samples from the finding database, any point on the finding (not just the endpoint of the distance line) can be used as the center of the finding subvolume. However, this approach may not be realistic enough. Therefore, in a particularly preferred embodiment of the present invention, the client-side MLA is preferably implemented as a linear classifier, such as a binary multilayer perceptron that recognizes distance lines in images of the finding database. In order to successfully train the perceptron model, the training set must be linearly separable, that is, the training set must include positive examples (annotated images in the finding database) and negative examples. Negative samples can be obtained by randomly sampling common medical images from general PACS and computing descriptors on these random samples.

[0021] Alternatively, the client-side MLA may be a neural network such as a convolutional neural network (CNN). The supervised training process of the client-side MLA preferably includes a back-propagation technique to find the best set of parameters that will enable it to infer ground truth from the medical image (in this case, the location of the distance lines in the medical image).

[0022] Custom tools or customized models can be easily derived from the base model, which are adapted based on the prompt(s) selected by the user. In the single-shot approach, the training of the client-side MLA is based on a single prompt.

[0023] The client-side MLA is "small" because, for example, it is configured to avoid overfitting. For example, the client-side MLA can be a single-layer linear classifier detector. The terms "client-side MLA", "small MLA" or "custom tool builder MLA" can be used interchangeably below.

[0024] The accuracy of a machine learning algorithm depends largely on the amount of data set used to train it. For example, a database of findings in a clinic or practice may be the result of annotations by users over a period of months or years, and a PACS database may include many more unannotated images. Using a base model as described above enables training of a customized machine learning tool with an advantageously smaller number of examples.

[0025] In a preferred embodiment of the present invention, when developing a custom tool, one or more appropriate filters are applied.

[0026] For example, if the custom tool is to identify and mark specific distances or contours in cardiac CT images, appropriate filters ensure that only this image type and / or an appropriate segmentation algorithm (e.g., a dedicated segmentation tool available for cardiac CT images) is used by the client-side MLA and the base model during the custom tool development phase. In a further example, if the custom tool is to show the maximum size of a kidney stone in an abdominal X-ray, appropriate filters ensure that only this image type is used by the client-side MLA and the base model during the custom tool development phase.

[0027] In a preferred embodiment of the invention, semantic descriptors of findings may be stored in a findings database. For example, for a finding of this type, a semantic descriptor such as "maximum edge-to-edge width" may be stored along with an image showing the finding. Other examples of semantic descriptors may be "endpoint", "midpoint of distance line", etc. In a further preferred embodiment of the invention, a thumbnail of the image area including the finding may be stored in the findings database along with the image. Any such thumbnail may be retrieved and used as a prompt whenever the user initiates the custom tool definition process. Storing semantic descriptors and / or thumbnails along with each image in the findings database may significantly reduce computation time during the custom tool definition process.

[0028] The definition of the customized tool can be considered complete when it is verified by the user, for example using a cross-validation technique. When the user is satisfied with the performance of the newly trained tool on a set of test images, the user can confirm that the tool can be added to the user interface. The image processing system then adds a corresponding element to the graphical user interface, such as a new icon or a new button in the tool pane, so that the user can call the customized tool to perform the desired task on the image displayed in the image reader application.

[0029] In a further preferred embodiment of the present invention, the base model is developed in a federated learning or collaborative learning manner so that several users (e.g., customers of a medical imaging software platform) can participate in building a powerful base model without the need for users to share data. In federated learning, only the neural network parameters of the "fine-tuned" base model (developed by a small MLA) are shared between participating sites, while any patient-related data remains in place and is not shared. In this way, a base model at one site or location can learn from adaptations developed at any number of other sites without the need for users to share any potentially sensitive patient data. Federated learning with a self-supervised objective will not be affected by the quality of labels in the image annotations in the findings database, because optimization objectives (e.g., denoising, image compression, etc.) using semi-supervised or unsupervised learning do not require labels. Federated learning also advantageously distributes computational overload between sites, so that advantageously large base models can be developed in a collaborative manner.

[0030] As an alternative or in addition to training the base model using supervised or semi-supervised methods, fully unsupervised methods such as autoencoder image inpainting, a self-supervised task for learning robust feature representations, can be deployed. Here, image inpainting is an alternative representation learning method that involves reconstructing the “blacked out” or cut-off image regions through MLA. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Other objects and features of the present invention will become apparent from the following detailed description considered in conjunction with the accompanying drawings. It should be understood, however, that the drawings are designed only for the purpose of illustration and not as a definition of the limits of the present invention.

[0032] Figure 1 is a schematic representation of a method of the present invention for defining a custom image annotation tool;

[0033] Figure 2 is a schematic representation of a system of the present invention for defining a custom image annotation tool;

[0034] Figure 3 A stage in the method of the present invention is shown;

[0035] Figure 4 shows the image reader application after invoking the custom tool;

[0036] Figure 5 It shows how the method of the present invention can be deployed in a federated learning approach;

[0037] Figure 6 A prior art approach to developing custom image annotation tools is illustrated;

[0038] In the drawings, like numbers refer to like objects throughout. Objects in the drawings are not necessarily drawn to scale. DETAILED DESCRIPTION

[0039] Figure 1 is a simplified schematic diagram showing the main elements 11, 12, 13 of a custom tool creator 1, indicating the stages in the method of the invention for creating a custom annotation tool; and Figure 2 is a simplified schematic diagram of an image reader application 14 for user interaction with the custom tool creator 1. It should be understood that the present invention is based on interaction with a user through a suitable front end, such as an image reader application 14, which complies with the DICOM (Digital Imaging and Communications in Medicine) standard and has a graphical user interface 140 or GUI to allow a user to at least retrieve, view, annotate and store medical images. The GUI 140 can be displayed on a monitor of an image reading station, such as in a radiology department of a clinic, on a handheld device such as a tablet computer, etc. The image reader application 14 is used by practitioners such as radiologists, oncologists, surgeons, etc. to view or "read" medical images and also to annotate medical images with findings. To this end, the user may be able to use the tools of the GUI 140 to place lines and labels on the imaged body areas, such as measurements, dimensions, etc. In the image processing system 1 of the present invention, the user is also given the option of defining custom tools to be used during image reading and annotation, which will be explained below.

[0040] In an exemplary scenario, a user wishes to generate a customized tool that will automatically annotate medical images in a particular manner, such as a tool that adds a particular type of annotation to a cardiac CT scan. To do this, the user selects a "generate tool" function from the GUI (e.g., by selecting a corresponding menu item). In a first stage 21, the image reader application 14 then presents the user with a list of "prompts" 141A, 141B, 141C, 141D, such as thumbnails of previously annotated images retrieved from a local findings database 11 and presented in a drop-down menu 141. Each thumbnail or prompt 141A, 141B, 141C, 141D can show a candidate finding FA, FB, FC, FD, such as a measured distance or size. The user can select one or more of these thumbnails (as indicated by a check box). The GUI 140 can present any number of prompts. If the user selects only one of the tip thumbnails, the system 1 may interpret this to mean that the other thumbnails do not show suitable candidate findings for the tool the user wants, and may update the tip menu 141 to show thumbnails with other more suitable candidate findings from the findings database 11 (more similar to the findings in the selected tip).

[0041] In a subsequent step 22, based on the selected cues, the small machine learning model 12 is trained to adapt to the base model 13, so that the customized version 13C of the model 13 will annotate medical images in the same way. Figure 3 As shown, the small MLA 12 may be a binary multilayer perceptron that is arranged to be trained using randomly sampled images from the PACS image database 15 ("negative" examples) and images in the local findings database 11 ("positive" examples).

[0042] The base model 13 is understood to have been previously trained in a self-supervised or supervised manner on a large image dataset, for example using any suitable representation learning algorithm. The small machine learning model 12 is trained locally (i.e. on the client side) to fine-tune the base model 13 to perform the desired task. The result of the adaptation or fine-tuning step of the MLA 12 is a tailor-made or customized model 13C, i.e. a modified version of the base model 13 that can perform the desired annotation task.

[0043] As mentioned above, when training MLA, suitable filters can be applied. Figure 3 As shown, filters 16 can ensure that only images related to a certain anatomical region are used during the tool development phase. Continuing with this example, when a custom tool is intended to add a specific type of annotation to a cardiac CT scan, the filter will ensure that the client-side MLA only uses cardiac CT scans of the findings database.

[0044] In a final stage 23, after completing the training stage 22, the image reader application 14 allows the user to specify a name or icon for the custom tool and add the name or icon to a menu 142 or tool pane 142 of the GUI 140 so that the new tool 14N can be called up during any subsequent image reading, e.g., by clicking on its icon or name. The descriptors of the findings in the selected prompt image(s) can be saved as additional tool information or can be used to suggest an appropriate tool name. For example, a custom tool 13C trained to measure the width of nodules in lung x-rays can be named "Lung Nodule Maximum Width".

[0045] The result of the custom tool synthesis is a tool 13C that will automatically perform the desired action when a user viewing a medical image M in an image reader application 14 selects the corresponding custom tool icon T. Selection of the tool icon T causes the custom model 13C to search for a candidate location in the current image M. If a candidate location is found, the custom model 13C will add the corresponding finding to the image M. Using the above example, the custom model will superimpose a line of the widest dimension of any nodule visible in the image. For example, if Figure 4 As shown, when the user opens a lung scan M in the image reader application 14 and selects the "maximum width of lung nodules" tool icon T from the menu 142, the custom tool 13C will perform segmentation of the image M and annotate the image M with the corresponding findings F. Here, the medical image M has been automatically annotated by the custom model 13C to include the longest side-to-side dimension of each of the two nodules identified in the image. Of course, the mini-MLA 12 can also be trained to adapt the base model 13 to perform classification, so that in this case, the image M can be augmented by the custom model 13C with the descriptors of the lesion class of each identified nodule.

[0046] Any custom tool defined in this way can be shared between users of the same platform, even when based at different medical imaging facilities, as long as all users have access to the same base model 13. As described above, the patient-related data in any image database used to develop a custom tool for a user remains "in situ" and is not shared between distributed facilities. Figure 5 It is shown how the method of the present invention can be deployed in a federated learning approach. Here, various medical facilities 50 may be able to access a common base model 13. At each facility 50, practitioners can use their image reader system 14 to develop custom tools. With each custom tool development process, the base model 13 becomes stronger. This is done without the medical facilities 50 having to share data, because any images used to train the client-side MLA are retrieved from the local database 11, 15, and this data is not required by the base model 13.

[0047] Figure 6 A prior art method for developing a custom image annotation tool is illustrated. In the first stage 61, a clinician specifies the functionality of a desired image annotation tool, such as a tool that will automatically overlay a lung scan with the maximum size of any nodules present in the image. A tool specification 610 is given to the tool developer. In stage 62, the tool developer modifies the image reader application to perform the desired functionality. In stage 63, the tool is provided to the user as part of an image reader application update 630. Obtaining a custom tool in this manner can take weeks and can be costly. Both expense and delay are repeated each time a practitioner needs a tool to perform a specific annotation task.

[0048] Although the present invention has been disclosed in the form of preferred embodiments and variations thereof, it will be appreciated that many additional modifications and variations may be made thereto without departing from the scope of the invention. As described above, using a small client-side model to adapt a base model provides a direct way to obtain customized tools without significant cost, as an alternative to the more expensive method of directly adapting the base model.

[0049] Instead of using client-side MLA, an alternative approach can be used to adapt the base model to perform the desired type of annotation. For example, the user can compute a descriptor vector for one or more suitable images in a PACS. The descriptors of the selected examples are populated to check the similarity of candidate locations in the target image. At runtime, all possible locations are calculated and similar locations are sought in images of the image database. The similarity is calculated by the base model based on a predefined threshold. When one or more detected similar locations are found, the base model has effectively learned a new tool.

[0050] For the sake of clarity, it should be understood that the use of "a" or "an" throughout the application does not exclude a plurality, and "comprising" does not exclude other steps or elements. References to a "unit" or a "module" do not exclude the use of more than one unit or module.

[0051] Reference numerals list Custom Tool Creator 1 Discovery Database11 Small MLA 12 Base Model 13 Custom Model 13C Image Reader 14 GUI 140 Tips Menu 141 Tips 141A, 141B, 141C, 141D Tools Menu 142 PACS Database 15 Filter 16 Method stage 21, 22, 23 Medical Modality 50 Prior art method stages 61, 62, 63 Tool Specification 610 Software Update 630 Discovery F Candidate results: FA, FB, FC, FD Custom Tool T Image M Practitioner P.

Claims

1. A computer-implemented method for generating a tool (T) for annotation of an image (M), preferably a medical image (M), comprising the following steps: - provide access to the base model (13); - providing a graphical user interface (140) adapted to enable a user (P) to specify a desired result of an image annotation task; - training a client-side machine learning model (12) using a local database of annotated images (11) to adapt a base model (13) to perform a desired image annotation task based on the specified results; as well as - augmenting the graphical user interface (140) with graphical elements (142T) representing tools (T) for performing the desired image annotation task.

2. A computer-implemented method according to the preceding claim, wherein the step of enabling a user (P) to specify a result of a desired image annotation task comprises presenting a plurality of candidate results (F1, F2, F3, F4) to the user (P) for selection.

3. A computer-implemented method according to any one of the preceding claims, wherein the candidate results (F1, F2, F3, F4) are annotations in an image selected from a database (11) of previously annotated images.

4. A computer-implemented method according to the preceding claim, wherein the candidate results (F1, F2, F3, F4) are presented in thumbnails (141A, 141B, 141C, 141D) of the corresponding images.

5. A computer-implemented method according to any one of the preceding claims, wherein: A graphical element (142T) is added to a tool pane (142) of a graphical user interface (140).

6. A computer-implemented method according to any of the preceding claims, wherein the client-side machine learning model (12) adapts the base model (13) to perform the customized task by adjusting the neural network parameters of the base model (13).

7. An image processing system (1) adapted to perform the steps of the method according to any one of claims 1 to 6, comprising: - means for accessing the base model (13); - a user interface (140) adapted to enable a user (P) to specify a desired result of an image annotation task; - a client-side machine learning model (12) configured to be trained using a local database (11) of annotated images to adapt a base model (13) to perform the image annotation task based on the specified results; and wherein - The user interface (140) is adapted to enable the user (P) to invoke a tool (142T) to perform the image annotation task on the displayed image (M).

8. An image processing system according to the preceding claim, wherein the base model (13) comprises a machine learning model trained on a plurality of images to perform any one of image denoising, image compression, image inpainting.

9. The image processing system according to claim 7 or 8, wherein: The base model (13) is trained on at least one thousand medical images (M).

10. An image processing system according to any one of claims 7 to 9, wherein the client-side machine learning model (12) is configured to avoid overfitting.

11. An image processing system according to any one of claims 7 to 10, wherein the client-side machine learning model (12) is any one of a linear classifier, a binary classifier, a neural network, and a similarity lookup table.

12. The image processing system according to any one of claims 7 to 11, wherein: An additional database (15) of normal images (M) is used to train the client-side machine learning model (12).

13. An image processing system according to any one of claims 7 to 12, wherein the data set of the annotated image database (11) comprises an image (M) and at least one annotation, wherein the annotation comprises at least a line (F) superimposed on the image (M).

14. A computer program product comprising instructions which, when executed by a processor of an image processing system according to any one of claims 7 to 13, cause the processor to perform the steps of the method according to any one of claims 1 to 6.

15. A computer-readable data carrier having stored thereon a computer program product according to claim 14.

Images