Medical image processing

Abstract

The invention relates to medical image processing. The invention relates to processing of a medical image while providing the medical image with at least one annotation data element. Regions of interest that are currently relevant to a user are identified. Upon identification of such a region of interest, at least one annotation data element is provided with an appearance that causes the region of interest to be occluded as little as possible, which identification may be done dynamically or continuously.

Description

Technical Field

[0001] The present invention relates to a method and apparatus for processing medical images. In particular, the present invention relates to medical image processing for outputting a medical image together with one or more annotation data elements. Background Art

[0002] In the context of medical examinations, imaging methods have become very important. Such imaging methods may include, for example, ultrasound images, radiological images, in particular images from computed tomography or magnetic resonance tomography.

[0003] After capturing one or more images of the examination area of ​​the patient, the images are analyzed and recorded by a medical expert (e.g., a radiologist). For this purpose, the medical expert typically adds additional information, such as in the form of annotations, notes, marks, labels, symbols, and / or any other kind of other information to the captured images. In addition, additional information may also be provided in association with the captured medical images, such as patient information (e.g., name, date of birth), information about the time and / or place when the image was recorded, information about the device and the (one or more) settings applied for capturing the image, and / or the reason for the relevant medical examination (e.g., a medical problem). If the additional information is displayed with the image, some details in the image may be obscured by the representation of the additional information. On the other hand, if the additional information is completely hidden or faded out, the user may no longer be able to access potentially important information.

[0004] Therefore, there is a need for a method of processing medical data in the context of a medical image in a manner that enables the output of the medical image together with the additional information in a manner that enables no loss of information. In particular, a user should be able to grasp many details of relevant locations in the image, and at the same time provide the user with as much additional information as possible. In particular, in view of the needs and requirements of the present application, there is a need to provide the medical image and the related additional information in a manner that allows the output of the medical image and the related additional information to be dynamically adjusted. Summary of the invention

[0005] The invention therefore provides a method and an apparatus for processing medical images and a computer program having the features of the independent claims. Further advantageous embodiments are the subject matter of the dependent claims.

[0006] According to a first aspect, a method for processing a medical image is provided. The method comprises a step of receiving a medical image. The image may be received in any suitable manner and format. For example, the medical image may be received directly from a medical imaging device. Alternatively, the medical image may be received from a local or remote storage device. The method further comprises a step of determining at least one annotation data element. The annotation data element is to be displayed together with the received medical image. In particular, the annotation data element is related to the medical image or a portion of the medical image. In addition, the method comprises a step of determining a region of interest in the received medical image. The region of interest may be a location or region of particular interest to a user. The determination of the region of interest may be performed manually based on an indication of a user, or automatically, for example, by analyzing the user's activities. The method further comprises a step of calculating the appearance of at least one annotation data element. The appearance may be calculated in response to the determined region of interest. As will be described in more detail below, the appearance may be related to a visual effect for providing the location of each annotation data element and / or, for example, automatically applied to the annotation data element. Finally, the method comprises a step of outputting the calculated appearance of at least one annotation data element together with the medical image. For this purpose, an output device may be used. The medical image and the calculated appearance of the at least one annotation data element may be output in any suitable manner, for example by being displayed on a display or computer screen to be used in the interventional procedure.

[0007] According to a second aspect, a device for processing a medical image is provided. The device may be configured to perform the method according to the first aspect.

[0008] The device may include a receiving interface, an annotation unit, a processing unit, and an output interface. The receiving interface is configured to receive a medical image for processing the medical image. The annotation unit is configured to determine at least one annotation data element. The annotation data element may be displayed together with the received medical image. In addition, the annotation data element may be related to the medical image or a part of the medical image. The processing unit is configured to determine a region of interest in the received medical image. The processing unit is also configured to calculate the appearance of the at least one annotation data element in response to the determined region of interest. The output interface is configured to output the calculated appearance of the at least one annotation data element together with the medical image.

[0009] According to a third aspect, a computer program is provided. The computer program may be loadable into a memory unit of a computing unit, the computer program comprising program code portions causing the computing unit to perform the method for processing a medical image according to the first aspect when the computer program is executed in the computing unit.

[0010] A medical image within the meaning of the present invention may be an image that can be used for medical purposes. For example, a medical image may be used for diagnostic purposes of an organism, in particular a human or an animal. For example, a medical image may provide a two-dimensional representation of a body or a part of a body. For example, a medical image may be obtained from an ultrasound imaging modality, an X-ray imaging modality, or any other type of imaging system that can be used to obtain images for medical purposes. The medical image may be obtained directly from the modality in which the medical image is acquired, or may be obtained from a storage device for storing data of the medical image. The medical image may represent: the entire body of an organism, a part of an organism such as an arm, a leg, an organ, a structure within an organism such as an implant or a stent, etc. In particular, the medical image may be a two-dimensional representation, which may be derived, for example, from a two-dimensional data set or a three-dimensional data set. For example, such a three-dimensional data set may be obtained by means of a computed tomography, a positron emission tomography device, or a magnetic resonance tomography. Data for medical images that may be used in the present invention may be provided in any suitable data structure. For example, a medical image may be received according to the DICOM standard. However, any other suitable format such as TIFF, JPG, etc. may also be possible.

[0011] To receive the medical image, a receiving interface may be used. The receiving interface may be communicatively coupled to a data source for providing the medical image. For example, the data source may be an imaging device such as an ultrasound modality or an X-ray modality. Alternatively, the medical image may be received from a storage device, such as a local storage such as a hard disk, a network storage, a cloud service, and / or any other suitable storage for storing and providing image data.

[0012] In particular, a medical image may be received to perform processing on the medical image. Such further processing (operation) may include, for example, annotation, analysis, reporting, archiving and / or comparing the image with other medical images, such as similar images acquired at different time points.

[0013] An annotation data element in the context of the present invention may be a data element that may be provided and / or displayed together with a medical image. The annotation data element may provide additional information related to the medical image. In particular, the annotation data element may include additional information.

[0014] For example, annotation data elements may provide: information about the patient to whom the medical image relates, information about the location and / or tissue at which the medical image was acquired, information about the configuration of the device, in particular the device used to acquire the medical image, timestamp information, or any other technical information.

[0015] An annotation data element may be related to the indication of a specific location or region within a medical image. For example, an annotation data element may include an arrow, a cross, or any other kind of mark for indicating a specific location in an image. Furthermore, an annotation data element may show a specific structure such as a line, a curve, in order to indicate the course or development of a structure (of "something") within the image.

[0016] The annotation data element may also include an indication of an area, such as a geometric object, such as a circle, rectangle or a hand-drawn marked area, which may be used to indicate a specific area within the medical image. Additionally, the annotation data element may be, for example, a text box or any other element for providing (additional) text in order to add instructions or comments to a specific location within the medical image. However, the annotation data element is not limited to the above examples. Additionally, the annotation data element may be any kind of supplementary element that may be added to the original medical image in order to provide additional information.

[0017] The method comprises determining at least one annotation data element. This may be performed by generating the annotation data element (through user interaction, online or in real time), or by receiving or reading in an already generated annotation data element.

[0018] Therefore, the annotation data element can be obtained, for example, from an external data source such as a storage device. For example, the annotation data element can be provided from a local storage device, a network storage, or a cloud service. In order to receive the annotation data element, the annotation module can be communicatively coupled to an external data source for providing the data of the annotation data element. For example, the data of the annotation data element can be provided and / or generated by an additional field in a DICOM file and / or in a DICOM header that provides the relevant medical image. Additionally or alternatively, data related to the annotation data element can also be obtained from another data source. In this case, the data of the annotation data element can be related to the medical image, for example, by a unique identifier indicating the relationship between the annotation data element and the medical image. The annotation data element can be provided as an overlay element on the medical image.

[0019] When using a medical image superimposed with annotation data elements, the user may pay special attention to a specific area within the medical image. In the present invention, such a specific area within the medical image is named as a region of interest. In particular, the region of interest may be a specific area or part of the image that is considered to be particularly relevant. For example, such a region of interest may be an area that the user wants to analyze or record. Therefore, the user may want to identify as many details as possible in the region of interest. For this purpose, the region of interest should include as few annotation data elements as possible so as to minimize the visual occlusion caused by the annotation data elements. The region of interest may be a specific area within the medical image that the user is currently paying attention to. Such a region of interest can be determined by monitoring the user's input and operation. For example, the region of interest in the medical image can be determined algorithmically, for example, based on a pointer controlled by the user. For example, such a pointer can be a mouse pointer of a computer mouse, a touch signal of a touch screen, or an indication obtained by means of eye tracking of the user. However, any other indication that can provide information about the user's operation or interaction may also be possible. Alternatively, the user can manually specify an area in the medical image as a desired region of interest. For this purpose, the user can draw the region of interest freehand, or use geometric elements such as circles, rectangles, polygons, etc. to indicate a specific area that should be considered as a region of interest. Alternatively or additionally, the region of interest can also be automatically identified by analyzing the content of the image. For example, a segmentation algorithm can be performed to identify specific elements such as organs, lesions, tumors, blood vessels, bones, implants, etc. within the medical image. Since such segmentation is only related to identifying specific areas within the medical image in order to identify the region of interest, the segmentation for this purpose can be performed at a lower resolution, in particular a resolution significantly lower than that usually used for diagnostic purposes. Based on this analysis, a specific region of interest can be automatically selected. In the case where multiple options for the region of interest can be identified, a menu can be provided to the user, and the user can select one or more options of the identified region, which one or more options will be considered as the region of interest in the further process.

[0020] In order to minimize the occlusion effect of annotation data elements in the medical image, the appearance of the annotation data elements can be adjusted. In this context, the expression experience may include features such as the position of the annotation data elements and / or the visual effects applied to the individual annotation data elements. In this context, the position of the annotation data elements can be regarded as the position of the annotation data elements within the entire area that can be used to output the medical image and the annotation data elements. In the case where the medical image does not cover the entire area that can be used for output, the area outside the medical image can be used to output the annotation data elements. For example, such an area outside the medical image can be above the medical image, below the medical image, or on the left or right side of the medical image. In addition, the annotation data elements can also be arranged within the medical image but outside the region of interest. Alternatively, it is also possible that the annotation data elements can be arranged at a position such that (only) a portion of the annotation data elements covers the region of interest.

[0021] In the case where a specific position is indicated in the original data of the specified annotation data element, such originally specified position can be regarded as the initial position. In this case, if the annotation data element does not cover the region of interest, the initial position can be used as the position for arranging the corresponding annotation data element. However, if the annotation data element with the initial position partially or completely covers the region of interest, the position of the annotation data element can be changed. In this way, a modified position, i.e., a target position, can be determined. In particular, the target position can be a position such that the annotation data element does not cover or at least only partially covers the region of interest.

[0022] The appearance of the annotation data element may also include any kind of features related to the visual effect. For example, the appearance of the annotation data element may also be related to the size of the annotation data element, which is used to provide the annotation data element with the medical image. The size of the annotation data element may be the extension of the entire annotation data element in the horizontal and / or vertical dimensions. The size of the annotation data element may be changed by scaling the extension of the annotation data element. In the case where the annotation data element includes text, the font size may also be reduced in order to minimize the extension of the annotation data element. In addition, the spacing between lines, words or characters may also be used in order to modify the size of the entire annotation data element. In addition, the font style may also be changed, abbreviations may be used, text wrapping may be adjusted or any other method may be performed in order to reduce the size of the annotation data element. The appearance of the annotation data element may also be related to the style of the visual representation. For example, the solid representation of the annotation data element may be changed to a representation that uses only lines, particularly lines for indicating boundaries, contours or specific structures, uses dashed or dotted lines instead of solid lines, applies shadows or shading, and / or modifies the transparency level of the annotation data element. The above possibilities for modifying the appearance, in particular the position and / or the visual effect of the annotation data element can be combined. In addition, these are only examples and do not limit the present invention. In addition, any other kind of modification to the appearance of the annotation data element may also be possible.

[0023] In order to output the resulting representation, in particular the medical image combined with one or more annotation data elements, to the user, the annotation data elements can be output together with the relevant medical image. For this purpose, the individual parts, i.e., the medical image and the annotation data elements according to the applied appearance, can be output. The output may include outputting the medical image and one or more annotation data elements according to the calculated appearance on a suitable output device. For this purpose, the medical image and the annotation data elements according to the calculated appearance can be provided to a suitable output device such as a display (e.g., a monitor or a computer display) or any other suitable output device via an output interface. For example, the output interface can be a data interface coupled to the output device in communication. For example, the output interface can be a rendering device, which renders the medical image and the annotation data elements according to the calculated appearance so as to present the result of the rendering process and output the result of the rendering process to the output device. For example, the output interface can generate a graphic signal, which is provided to a screen or monitor via a graphic interface such as HDMI or a display port.

[0024] In order to receive user interactions, such as user instructions for specifying a current position, determining a region of interest, and / or inputting additional (e.g., text) commands, a suitable input device can be communicatively coupled to an input interface for receiving user interactions. For example, the input interface can receive a signal from a computer mouse in order to detect the current position intended by the user. Alternatively, the user can specify a specific position by means of touch on a touch screen. The position of the user's attention can also be determined, for example, by a device for tracking the user's eyes. Therefore, such a device can provide an indication for specifying the position at which the user is currently looking. In addition, it is possible that the user can enter any kind of text by means of a keyboard or a speech recognition system. In this way, the user can enter text that can be used, for example, for reporting or archiving medical images. In this case, the entered text can be monitored for semantic analysis. Based on the semantic analysis, further operations can be triggered, such as determining a region of interest or changing a previously determined region of interest to another region of interest. Additionally or alternatively, the user can enter any kind of command in order to initiate appropriate processing, such as selecting a specific region of interest, triggering automatic segmentation of medical images, changing expected, assumed operations, etc.

[0025] In a possible embodiment, at least one annotation data element is first displayed together with the medical image at an initial position. The step of calculating the appearance of the annotation data element may include adjusting the initial position to a target position. In this way, an initial position, i.e., an initially intended or automatically preset position, may be assigned to the annotation data element and may be used where appropriate. Furthermore, if the initial position conflicts with the region of interest, the initial position may be changed to another suitable position, i.e., the target position. In this way, visual obstructions due to the annotation data element may be avoided or at least reduced.

[0026] In a possible embodiment, after the step of determining at least one annotation data element and before determining the region of interest, the method may further comprise the step of displaying the medical image for the purpose of determining the region of interest. After displaying the medical image, a manual or automatic method for identifying the region of interest in the displayed medical image may be performed. In this way, the region of interest may be easily adjusted depending on the actually displayed medical image.

[0027] In a possible embodiment, the determination of the region of interest may be based on the interaction of the user with the displayed medical image. In particular, the determination of the region of interest may be based on monitoring of the position of the pointer, eye tracking of the user and / or semantic analysis of the displayed medical image. Additionally or alternatively, determining the region of interest may include executing a segmentation algorithm to identify a region of the body, a predetermined part of the body, or an organ or structure within the displayed medical image. In this way, automatic determination of a suitable region of interest in a displayed medical image may be achieved.

[0028] In a possible embodiment, calculating the appearance may include calculating the position of at least one annotation data element relative to the medical image output. In particular, the appearance may be calculated based on the spatial distance between the initial position of the at least one annotation data element and the determined region of interest. Additionally or alternatively, the appearance may be calculated such that at least one annotation data element does not overlap with the determined region of interest, or only slightly overlaps with the determined region of interest. In this regard, "slightly" may be defined as a percentage of overlap. For example, it may be preconfigured that the percentage of overlap is allowed to be in the range of 10% to 50%, particularly 20% to 40%, and more particularly 30% of the region of interest. In this way, a suitable position may be selected for the annotation data element so that the region of interest in the medical image is affected as little as possible. The percentage of overlap may be configured before the step of calculating the appearance or even during the step of calculating the appearance.

[0029] In a possible embodiment, the step of calculating the appearance may include adjusting the size of at least one annotation data element. In particular, the appearance may be calculated by adjusting the font size of the text, replacing expressions with abbreviations, adjusting text wrapping, adjusting the transparency level of at least one annotation data element, displaying only the outline or contour of at least one annotation data element, replacing at least one annotation data element with a thumbnail, and / or applying shading, shadows and / or dashed graphic structures. Thus, the impairment of the additional data element may be reduced by applying suitable visual effects, so that details in the region of interest of the medical image may be identified in the best possible way.

[0030] In a possible embodiment, calculating the appearance of at least one annotation data element may include determining an output area for displaying the at least one annotation data element. In this case, the output area may be an area outside the medical image, and / or an area outside an area in the medical image representing a body or a part of a body, and / or an area outside a determined region of interest. By identifying such a suitable output area for displaying the annotation data element, a simple concept of providing the annotation data element in a suitable area that does not affect the region of interest may be achieved.

[0031] In a possible embodiment, outputting the calculated appearance of at least one annotation data element together with the medical image may be based on a predetermined attribute of the at least one annotation data element and / or a predetermined attribute of the medical image. The attribute may be any kind of feature related to the annotation data element. For example, the attribute may be related to the visual appearance of the annotation data element (e.g. the size of the annotation data element required to output the annotation data element). Furthermore, the attribute may be related to a specific kind of information of the annotation data element (e.g. a label specifying a specific element within the medical image). However, any other kind of attribute may also be possible. In this way, the appearance of the annotation data element may be adjusted in different ways depending on the importance of the individual attributes of the annotation data element or any other specific feature.

[0032] In a possible embodiment, the step of calculating the appearance may include determining a ranking of the plurality of annotation data elements. In this case, the calculation of the appearance of the annotation data elements may be performed based on the determined ranking. Thus, annotation data elements with a higher ranking (e.g., priority) or a higher level of relevance may be provided with an appearance that is better recognized, while the appearance of annotation data elements with a lower ranking may be adjusted so that these annotation data elements may have less influence on the region of interest.

[0033] In a possible embodiment, the medical image may be received from an imaging device, such as an ultrasound imaging modality. Additionally or alternatively, the medical image may be received from an X-ray modality. In particular, the medical image may be received from a computed tomography scan, from a positron emission tomography device and / or from a magnetic resonance tomography scan. However, any other kind of device for capturing images relevant for medical purposes may also be possible.

[0034] In possible embodiments, at least one annotation data element may include a graphical element, a text element and / or a numerical element. Additionally or alternatively, at least one annotation data element may include metadata related to the patient, data of the configuration of the medical imaging modality used to capture the medical image and / or a specified date and / or data related to the purpose of capturing the medical image. Apart from this, any other kind of information that may be related to the medical image may also be possible. In this way, the annotation data elements may provide important information for manually or automatically (with the aid of a software application) analyzing and understanding the medical image.

[0035] In a possible implementation, the at least one annotation data element may be determined based on metadata and / or supplementary data provided in association with the medical image. For example, the annotation data element may be determined from information provided in an additional data field of a data set including image data of the medical image. For example, a header of the medical image may include additional information that may be used to derive the at least one annotation data element.

[0036] In a possible embodiment, the appearance of the annotation data element is adjusted in response to the zoom level of the medical image. In this way, the appearance of at least one annotation data element can be dynamically adjusted according to the current zoom level used to display the medical image. For example, in the case where the zoom level of the medical image increases, the area related to the region of interest may also increase, and therefore, the appearance of the annotation data element may be adjusted so that the increased area of ​​the region of interest may not be affected by the annotation data element. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In the following detailed description of the accompanying drawings, non-limiting exemplary embodiments having features and further advantages thereof will be discussed with reference to the accompanying drawings. In the accompanying drawings:

[0038] Figure 1 : A schematic diagram comprising a representation of a medical image and additional annotation data elements;

[0039] Figure 2 : A schematic block diagram showing a device for processing a medical image according to an embodiment;

[0040] Figure 3 : A flowchart of a method for processing a medical image according to an embodiment;

[0041] Figures 4 to 9 : A schematic diagram of possible modifications to the appearance of an annotation data element. DETAILED DESCRIPTION

[0042] Figure 1 A schematic diagram of a representation 1 of a medical image 10 and a plurality of annotation data elements 21 to 25 is shown.

[0043] The medical image 10 may be a two-dimensional image captured for medical purposes. The medical image 10 may be an image of a body or at least a part of a body of a human or animal. The medical image 10 may be received, for example, by an ultrasound image modality or an X-ray modality, a computed tomography scan, a positron emission tomography device, or a magnetic resonance tomography scan. However, any other device for acquiring a medical image may also be possible. The data of the medical image 10 may be captured in a two-dimensional space or a three-dimensional space, and based on the captured data, a two-dimensional image may be provided as the medical image 10.

[0044] As in Figure 1As can also be seen in the representation 1, the representation 1 may include one or more annotation data elements 21 to 25. The annotation data elements 21 to 25 may be any kind of additional element (in addition to the image itself) in the representation 1 for providing additional information and / or indicating a specific location or area. The annotation data elements 21 to 25 may include graphic, text and / or digital elements. In addition, the annotation data elements 21 to 25 may include: the configuration of the medical imaging modality that has been used to acquire or capture the medical image, data specifying a date, or further data related to the purpose of capturing the medical image 10 (e.g., a medical question). For example, the annotation data elements 21 to 25 may include a text element 21 for providing additional text. In this way, additional information may be provided, such as information about the patient, the intended purpose of capturing the medical image, the location and / or configuration of the device used to record the medical image, or any other kind of information. In addition, the annotation data elements 21 to 25 may include a box 22, such as a text box including additional information, in particular annotation text. The annotation data elements 21 to 25 may also include a graphic element 23, which may, for example, be used to indicate a specific area within the medical image. The graphic element 23 may be a graphic element manually generated by a user and / or a graphic element automatically generated, for example, based on processing of the medical image 10. For example, segmentation may be applied to the medical image 10 in order to identify a specific region, such as an organ, a lesion, a tumor, or a structure such as an implant or a stent. The annotation data elements 21 to 25 may also include one or more marks 24, which may be used to indicate a specific location within the medical image 10. In addition, any other kind of annotation data element, such as a graphic element 25, may be used, which may, for example, be used to show the orientation of the two-dimensional medical image 10 in three-dimensional space. The above list of annotation data elements 21 to 25 only provides some exemplary possibilities for the annotation data elements 21 to 25. In addition, any other suitable kind of annotation data elements 21 to 25 may also be possible.

[0045] As from Figure 1 As can be appreciated from the illustrations provided in , in the prior art, it is possible that details of the medical image 10 may be at least partially hidden due to a large number of annotation data elements 21 to 25. In particular, if a user wants to work on a certain area within the medical image 10, the user may assign several annotation data elements 21 to 25 to the area, but at the same time, the user wants to identify as many details of the area as possible. Therefore, for such a task, it is not appropriate to completely hide the annotation data elements 21 to 25. In view of this, according to the present disclosure, a dynamic modification of the appearance of the annotation data elements 21 to 25 is provided.

[0046] Figure 2A schematic block diagram of an apparatus 100 for processing a medical image 10 is shown. The apparatus 100 may include a receiving interface 110, an annotation unit 120, a processing unit 130 and an output interface 140.

[0047] The receiving interface 110 may receive the data of the medical image 10. The data of the medical image 10 may be received, for example, from an associated storage device 210, or directly from a scanner (image acquisition modality). For this purpose, the receiving interface 110 may be communicatively coupled to the storage device 210. The data of the medical image 10 may be received in any suitable format, such as the corresponding DICIM standard. However, any other image format may also be possible.

[0048] The annotation module 120 may receive data of at least one annotation data element 21 to 25. For example, the data of at least one annotation data element 21 to 25 may be received from an associated further storage device 220. For this purpose, the annotation module 120 may be communicatively coupled to the further storage device 220. Based on the received data, the annotation module 120 may also determine at least one annotation data element 21 to 25. For example, the annotation module 120 may determine which kind of annotation data element the received data relates to, and which position in the medical image 10 the annotation data element relates to. For example, the annotation module 120 may determine whether the received data relates to text, a graphical element, an indication of a specific position or region, etc. For example, the data of each annotation data element 21 to 25 may be determined based on metadata provided in association with the data of the medical image 10. For example, the data of the annotation data elements 21 to 25 may be obtained from a data field in a DICOM header or file. However, the data of the annotation data elements 21 to 25 may also be provided in any other suitable manner, for example by means of a separate additional data file.

[0049] The processing unit 130 may perform processing on the received medical image 10 and the related annotation data elements 21 to 25. In particular, the processing unit 130 may adjust the appearance of the annotation data elements 21 to 25 when outputting the annotation data elements 21 to 25 together with the related medical image 10. In this regard, the term "appearance" may relate to the position where the annotation data elements 21 to 25 are output and the visual effect applied to the respective annotation data elements 21 to 25 when outputting the annotation data elements 21 to 25.

[0050] For this purpose, the processing unit 130 determines a region of interest in the received medical image 10. Such a region of interest may be related to a specific area within the medical image 10. For example, the region of interest may be the location of an area that the user is currently focusing on. For example, if the user is currently focusing on a specific organ or a specific part of an organ such as a lesion or a tumor, the processing unit 130 may automatically identify the area as a region of interest.

[0051] With regard to automatically identifying such regions of interest, the processing unit 130 may apply any suitable method. For example, monitoring of a pointer (e.g., movement of a computer mouse), eye tracking, semantic analysis of text entered by a user, etc. may be applied in order to identify such regions of interest. In addition, the medical image 10 may also be automatically analyzed, such as by semantic analysis and / or segmentation, in order to identify specific regions that may be considered regions of interest. Since the results of such segmentation are only used to identify specific regions of interest, the segmentation may be based on a medical image with reduced resolution. In this way, segmentation may be performed faster and with less computational requirements. However, any other suitable method for determining regions of interest may also be possible.

[0052] After determining the region of interest, the processing unit 130 may also calculate a suitable appearance for each annotation data element 21 to 25 that should be provided in association with the relevant medical image 10. In particular, the appearance of the annotation data elements 21 to 25 may be adjusted so that the user can recognize that the region of interest is as unobstructed as possible. For example, the position of the annotation data elements 21 to 25 may be selected so that the annotation data elements 21 to 25 are output outside the determined region of interest.

[0053] Additionally or alternatively, the representation of the annotation data elements 21 to 25 may also be adjusted in response to the determined region of interest. For example, the transparency level of the annotation data elements 21 to 25 may be automatically adjusted or set, the size of the annotation data elements 21 to 25, the text size of the text included in the annotation data elements 21 to 25 may be reduced, and / or any other suitable modification may be applied. Figures 4 to 9 Further possibilities for adjusting the appearance of the annotation data elements 21 to 25 are described.

[0054] After the processing unit 130 calculates the appropriate appearance of each of the annotation data elements 21 to 25 that should be provided together with the relevant medical image 10, the calculated appearance of the annotation data elements 21 to 25 is output together with the relevant medical image 10 through the output interface 140. For this purpose, the medical image 10 and the annotation data elements with the calculated appearance may be sent to the output device 141 via the output interface 140. The sent medical image 10 and the relevant annotation data elements 21 to 25 may be output through the output device 140. For example, the output device 140 may display the medical image 10 and the relevant annotation data elements 21 to 25 on a computer screen.

[0055] Furthermore, the device 100 may include an input interface 150 for receiving user interactions. For example, the user interactions may be received by means of a computer mouse, a device for eye tracking of a user, a touch screen, a keyboard, a system for speech recognition, or any other suitable means for receiving user input. In case such user interactions are received, the processing unit 130 may take the user interactions into account in order to determine the region of interest. In this way, the appearance of the region of interest and the individual annotation data elements 21 to 25 thereon may be dynamically adjusted in response to the received user interactions.

[0056] Figure 3 A block diagram illustrating a method for processing a medical image according to an embodiment is shown. The method may include any steps that may be suitable for execution by the apparatus 100 as described above. Likewise, the apparatus 100 described above may include components that may be suitable for executing the method as described below.

[0057] In step S1 , a medical image and, optionally, data for processing the medical image 10 may be received.

[0058] In step S2, at least one annotation data element 21 to 25 may be determined. For example, the at least one annotation data element 21 to 25 may be determined based on supplementary data related to the medical image 10. For example, the supplementary data may be obtained from a data field of a DICOM file (e.g., a DIMCOM header) providing data of the medical image 10. However, any other suitable way of providing data of the annotation data elements 21 to 25 may also be possible. For example, the data may be provided by an additional data file related to the medical image 10.

[0059] In step S3, a region of interest in the received medical image may be determined. The determination may be performed as described above in conjunction with the device 100. For example, the region of interest may be determined based on automatic image analysis of the medical image and / or user interaction, such as monitoring pointer movement, eye tracking, semantic analysis of user input, segmentation of the medical image, etc.

[0060] In step S4, the appearance of the annotation data elements 21 to 25 may be calculated. In particular, the appearance of the annotation data elements 21 to 25 may be determined in response to the determined region of interest. For example, the appearance of the annotation data elements 21 to 25 may be adjusted so that the annotation data elements 21 to 25 occlude the region of interest as little as possible. Further details on adjusting the appearance will be described in more detail below.

[0061] After the appearance of the annotation data elements 21 to 25 is calculated, in step S5, the calculated appearance of the annotation data elements 21 to 25 can be output to the output device 141 together with the medical image 10. For this purpose, the medical image 10 and the calculated appearance of the annotation data elements 21 to 25 can be sent to the output device 141 via the output interface 140.

[0062] The appearance of the annotation data elements 21 to 25 may relate to the position of the respective annotation data elements 21 to 25 in the representation 1 output together with the medical image 10 and / or the appearance may relate to the visual effects applied to the annotation data elements 21 to 25 when outputting the annotation data elements 21 to 25.

[0063] In order to calculate suitable positions of the annotation data elements 21 to 25 , a (target) position may be determined at which the respective annotation data element 21 to 25 does not occlude the region of interest or occludes the region of interest as little as possible.

[0064] In a possible embodiment, the annotation data elements 21 to 25 may first be displayed at an initial position. The initial position may be a position where the annotation data elements 21 to 25 are displayed without taking the region of interest into account. When the region of interest is determined, it is desirable that the annotation data elements 21 to 25 should obscure the region of interest as little as possible. For this purpose, the initial position of the annotation data elements 21 to 25 may be changed. In particular, the position of the annotation data elements 21 to 25 may be changed to a target position. The target position may be a position away from the region of interest. In this way, the annotation data elements 21 to 25 may not cover or only partially cover the region of interest to a minimum extent.

[0065] The positions of the output annotation data elements 21 to 25 may also be adjusted based on additional rules, which may be stored in a rule base. For example, the positions of the annotation data elements 21 to 25 may be adjusted based on the spatial distance between the initial positions of the annotation data elements 21 to 25 and the region of interest. In the case where the initial positions of the annotation data elements 21 to 25 are far from the region of interest, the initial positions of the annotation data elements 21 to 25 may remain unchanged, or may only be slightly changed. The closer the annotation data elements 21 to 25 are to the region of interest, the more the positions of the corresponding annotation data elements 21 to 25 may be changed, in particular moved away from the region of interest. In particular, it is desirable that the annotation data elements 21 to 25 do not overlap or only overlap slightly with the region of interest.

[0066] In addition to determining a suitable position when calculating the appearance of the annotation data elements 21 to 25, suitable visual effects may also be applied to the annotation data elements 21 to 25. For example, the appearance of the annotation data elements 21 to 25 may be adjusted by changing the overall size of the individual annotation data elements 21 to 25. For this purpose, scaling of the annotation data elements 21 to 25 may be applied. In the case where the annotation data elements 21 to 25 may include text, the text size may also be adjusted, the font used for the text may be changed, at least some expressions may be replaced with abbreviations, text wrapping may be adjusted, or any other kind of suitable modification may be applied. Furthermore, the original annotation data elements 21 to 25 may also be replaced with suitable icons or thumbnails. In this way, the size of the area required to provide the individual annotation data elements 21 to 25 together with the medical image in the representation 1 may be reduced.

[0067] Additionally or alternatively, further visual effects may also be applied that may reduce the obstruction of the visibility of the region of interest in the medical image 10. For example, the transparency level of the annotation data elements 21 to 25 may be set or adjusted. Furthermore, only the outlines or contours of the annotation data elements 21 to 25 may be displayed in order to improve the visibility of the details in the region of interest. Shading or shadowing of the annotation data elements 21 to 25 may also be applied, or dashed graphic structures may be used instead of solid lines. However, any other suitable way of applying a visual effect, in particular a visual effect that may improve the visibility of the region of interest may also be possible.

[0068] In the case where a plurality of annotation data elements 21 to 25 should be provided with the medical image, the order of the individual annotation data elements 21 to 25 can be determined. The order can be used to prioritize some elements over other elements. For example, annotation data elements 21 to 25 with a higher priority can be kept within the region of interest, or can be positioned close to the region of interest, while annotation data elements 21 to 25 with a lower priority can be positioned farther from the region of interest. Different visual effects can also be applied to the annotation data elements 21 to 25 according to the order or priority. For example, annotation data elements 21 to 25 with a higher priority can be represented with a larger size, lower transparency, more details, etc., while annotation data elements 21 to 25 with a lower priority can be displayed with a smaller size, a higher transparency level, less details, or by icons or thumbnails, etc.

[0069] The ordering may have been determined in advance and provided in association with the individual annotation data elements 21 to 25. Furthermore, the ordering may also be determined by analyzing the annotation data elements 21 to 25 that should be output. For this purpose, any kind of suitable analysis may be applied, such as a semantic analysis. In this way, the ordering or priority may be determined automatically. Alternatively, a fixed, predetermined ordering may be applied.

[0070] Furthermore, the medical image 10 and / or the annotation data elements 21 to 25 that should be provided together with the medical image 10 may also be analyzed in order to determine the appearance of the respective annotation data elements 21 to 25. For example, the annotation data elements 21 to 25 may be analyzed in order to identify one or more predetermined attributes. In this case, the annotation data elements 21 to 25 related to such identified attributes may be provided at or close to the initial position. Additionally or alternatively, such annotation data elements 21 to 25 may be displayed with visual effects that enable better (e.g., faster) processing or good recognition, such as a larger size, lower transparency, solid lines, etc. On the other hand, other annotation data elements 21 to 25 may be displayed at a position farther from the region of interest and / or by applying a visual effect that reduces the visual occlusion caused by such annotation data elements 21 to 25.

[0071] In order to easily find suitable positions of the annotation data elements 21 to 25, a suitable output area may be determined for displaying such annotation data elements 21 to 25. For example, if the medical image 10 in the representation 1 does not cover the complete area available in the representation 1, the output area for providing at least some of the annotation data elements 21 to 25 may be selected as an output area outside the medical image 10.

[0072] Additionally or alternatively, an automatic image analysis of the medical image 10 may also be performed. Based on the analysis, regions in the medical image 10 that are related to the body, a part of the body, a specific organ or a structure that is considered to be particularly relevant to the image processing or medical problem may be identified. Therefore, in this case, the output region may be selected as a region outside the relevant region. For example, a medical image includes a region showing a part of the body (e.g., a certain bone) and one or more other less relevant regions of the part of the body, for example, surrounded by air or tissue. In this case, the surrounding region (air or tissue or less relevant structure) may be selected as a suitable output region. In this case, at least some of the annotation data elements 21 to 25 may be located at positions within the output region.

[0073] When generating a representation 1 having a medical image 10 and one or more annotation data elements 21 to 25, a calculation of the appearance of the annotation data elements 21 to 25 may be performed once. Furthermore, the representation may be dynamically adjusted during operation, and in particular, the appearance of the annotation data elements 21 to 25 may be adjusted. For example, the appearance of the annotation data elements 21 to 25 may be adjusted in response to user interaction. For example, user interaction may be monitored in order to dynamically adjust a region of interest. In response thereto, the appearance of the annotation data elements 21 to 25 may be adjusted when the region of interest changes.

[0074] For example, the region of interest may be determined by monitoring a pointer (eg, a mouse icon of a computer mouse), receiving touches on a touch screen, and performing eye tracking of a user (to identify where the user is looking).

[0075] The appearance of the annotation data elements 21 to 25 may also be adjusted when the zoom level of the medical image 10 changes.

[0076] In addition, user inputs, such as texts, in particular texts related to the archiving of medical images, can also be analyzed. For example, such texts can be input via a keyboard or a speech recognition system. In this case, the text can be semantically analyzed, for example to identify keywords. Based on this analysis of the text, a specific area in the medical image 10 that the user is currently considering can be identified. Based on this, the region of interest can be adjusted accordingly. For example, if the user enters "the image shows a tumor cell in the liver...", such a tumor cell in the medical image 10 can be automatically identified and the region of interest can be determined as the tumor cell. If the user also enters "the right border of the liver...", the region of interest can be changed to the right border of the liver shown in the medical image 10.

[0077] When a specific position in the medical image 10 that the user is currently focusing on is identified, a region of interest can be determined based on the position. For example, the region of interest can be determined as a region having a predetermined distance from the position that the user is currently focusing on. Alternatively, the image data of the medical image 10 can be analyzed to identify a specific structure related to the position that the user is currently focusing on. For example, segmentation of the medical image 10 can be performed, and the region of interest can be selected as an element at or near the position that the user is currently focusing on. For example, if the user focuses on a specific blood vessel, for example, because a pointer controlled by the user moves over the blood vessel, the corresponding blood vessel can be automatically selected as the region of interest. When the user changes his focus, the region of interest can change automatically, and therefore, the appearance of the annotation data elements 21 to 25 can also change in response to the change of the region of interest.

[0078] Therefore, in a preferred embodiment, the region of interest is determined dynamically, in particular repeatedly.

[0079] In addition, it is also possible that the user can manually identify a specific region of interest. For example, the user can select a specific location. This can be done, for example, by touching or clicking a mouse. The user can also manually indicate the desired region of interest by any other suitable interaction, for example, by drawing a circle, rectangle, polygon or free hand drawing. Alternatively, a specific region of interest can be determined based on a semantic analysis of the text entered by the user. After such a process of identifying the region of interest, the user's attention can be monitored, for example, by monitoring the mouse pointer or tracking eye movement. Based on this monitoring of the user's attention, the appearance of the annotation data elements 21 to 25 can change in response to the distance between the user's attention and the region of interest.

[0080] The change of appearance, in particular the change of the visual effect, can be performed, for example, by switching between a predetermined number of visual effects. For example, the transparency can be changed between two or more different transparency levels, the size can be changed between a larger size and a smaller size, the graphical representation can be changed between the original representation and the thumbnail representation, etc. As an alternative to switching between only two or a smaller number of different effects, it is also possible that the appearance of the annotation data elements 21 to 25 can be changed dynamically in response to the position currently focused on by the pointer or the user and the (pre-selected) region of interest. For example, the appearance, in particular the visual effect, of the annotation data elements 21 to 25 can be changed gradually or continuously based on the distance between the region of interest and the position currently focused on by the user.

[0081] Some examples for adjusting the appearance of the annotation data elements 21 to 25 are described below. The described appearance modifications can be applied individually or in combination. For example, different modifications for adjusting the appearance of the annotation data elements 21 to 25 can be applied to different annotation data elements 21 to 25. In addition, multiple examples of the described examples for adjusting the appearance can also be combined and applied to the annotation data elements 21 to 25. In addition, it should be noted that the described methods for adjusting the appearance may only be some examples, which do not limit the scope of the present invention. In addition, any other kind of suitable method for adjusting the appearance of the annotation data elements 21 to 25 may also be possible.

[0082] In the following, reference numeral 400 relates to a representation comprising annotation data elements 401 to 403 with an initial appearance (i.e., initial position and / or initially desired visual effect without taking the region of interest into account). Furthermore, reference numeral 500 relates to a representation with a further modification of the appearance applied to the annotation data elements, such that the annotation data elements are provided with a modified / adjusted appearance as annotation data elements 501 to 503.

[0083] For greater clarity, only the annotation data elements 401 to 403 and the annotation data elements 501 to 503 are shown in the following example, wherein the related medical images have been omitted.

[0084] Figure 4 On the left side an exemplary representation 400 of three annotation data elements 401 to 403 is shown. In this example, the first annotation data element 401 relates to text, the second annotation data element 402 relates to a line and the third annotation data element 403 relates to a graphical element, eg indicating an object.

[0085] As in Figure 4 As can be seen on the right side of the diagram, the positions of the annotation data elements 501 to 503 can be changed. In this way, the annotation data elements 501 to 503 can be located at positions that enable the identification of a region of interest in a medical image (not shown) without or at least with less interference due to obstruction caused by the annotation data elements.

[0086] Figure 5 Another example of a representation 400 having three annotation data elements 401 to 403 is shown, wherein the left annotation data elements 401 to 403 are provided as text elements with complete text. Figure 5 As shown on the right side of , the size of each annotation data element can be reduced by shortening the text or applying modified text wrapping. For example, the text in the first annotation data element 501 is shortened by providing only the beginning of the text followed by three dots. Figure 5In the second annotation data element 502, the original complete text is shortened by using abbreviations. In addition, in the third annotation data element 503, the text wrapping is changed so that the width of the corresponding text box is reduced, thereby increasing the height of the text box.

[0087] Figure 6 Another example for adjusting the appearance of the annotation data elements 401 to 403 in the representation 400 is shown. In this case, the distance between the individual elements of the annotation data elements 401 to 403 is reduced. For example, in the annotation data element 501, the line spacing between the individual lines of the text is reduced. Similarly, the spacing between the individual letters or words of the text can also be reduced, different font styles can be used, or other similar operations can be performed. In the second annotation data element 502, the spacing between the individual elements (circles) is reduced in the horizontal and vertical dimensions. In the third annotation data element 503 of this example, only the spacing in the vertical direction is reduced, and no modification is applied in the horizontal direction. Obviously, similar operations can be performed by changing the spacing in the horizontal direction only.

[0088] Figure 7 A further exemplary possibility for adjusting the appearance of the annotation data elements 401 to 403 is shown. Figure 7 As can be seen on the right, the appearance of the first annotation data element 501 is changed by showing only the outer border of a single element. Depending on the individual data elements, further details of the annotation data element 501 may also be shown by some internal lines within the individual annotation data elements. The appearance of the second annotation data element 502 is changed by showing the corresponding annotation data element 402 only in dotted lines. Obviously, any other way for modified line styles (e.g., dotted lines, etc.) may also be possible instead of dotted lines. The appearance of the third annotation data element 503 is changed by showing the corresponding annotation data element 403 in a shaded illustration. For this purpose, any kind of shade may be possible. Similar to shade, any similar modification of the appearance may also be applied to shadows. Although not explicitly shown in this figure, the transparency level of the annotation data element may also be changed.

[0089] Figure 8 A further exemplary possibility for adjusting the appearance of the annotation data elements 401 to 403 is shown. As can be seen on the right side of the figure, the size of the individual annotation data elements 501 to 503 can be changed, in particular reduced. For example, the text size of the text in the annotation data element 501 can be reduced. In a similar manner, the length of the line in the annotation data element 502 is shortened and the size of the third annotation data element 503 is reduced. Instead of simply reducing the size of the annotation data elements 501 to 503, it is also possible to replace the original annotation data element with a minimized representation, such as an icon or thumbnail.

[0090] at last, Fig. 9 A further possibility for adjusting the appearance of the annotation data elements 401 to 403 is shown. In this example, the annotation data elements are moved from an initial position to a completely different area of ​​the representation 500. In particular, the annotation data elements 501 to 503 may be arranged at an output area outside the medical image 10.

[0091] In summary, the present invention relates to the processing of medical images when they are provided together with at least one annotation data element. It is proposed to identify a specific region of interest that is currently relevant to the user. After such a region of interest has been identified (this identification can be done dynamically or continuously), at least one annotation data element is provided with an appearance that makes the region of interest as little obscured as possible.

[0092] To the extent not explicitly described, the various embodiments described with reference to the drawings or their various aspects and / or features may be combined or interchanged with one another without limiting or expanding the scope of the invention described, provided that such combination or interchange is useful and within the spirit of the invention. The advantages described with respect to a particular embodiment of the invention or with respect to a particular drawing are also advantages of other embodiments of the invention where applicable.

Claims

1. A method for processing a medical image (10), comprising the following steps: - receiving (S1) a medical image (10); - determining (S2) at least one annotation data element (21 to 25) to be displayed together with the received medical image (10) and relating to the medical image (10) or a part of the medical image (10); - determining (S3) a region of interest in the received medical image (10); - calculating (S4) the appearance of said at least one annotation data element (21 to 25) in response to the determined region of interest; - outputting (S5) the calculated appearance of the at least one annotation data element (21 to 25) together with the medical image (10) via an output device (141).

2. The method according to claim 1, wherein: The at least one annotation data element (21 to 25) is first displayed together with the medical image (10) at an initial position, and wherein calculating (S4) the appearance of the annotation data element (21 to 25) comprises adjusting the initial position to a target position.

3. The method according to claim 1 or 2, wherein: After the step of determining (S2) the at least one annotation data element (21 to 25) and before determining (S3) the region of interest, the method further comprises the following method steps: - displaying the medical image (10) for the purpose of determining the region of interest.

4. The method according to claim 3, wherein: Determining the region of interest is based on user interaction with the displayed medical image (10), in particular based on monitoring a pointer position, eye tracking of the user and / or semantic analysis of the displayed medical image (10), and / or wherein, Determining the region of interest comprises executing a segmentation algorithm in order to identify a region of a body, a predetermined part of the body or an organ or structure within the displayed medical image.

5. A method according to any one of the preceding claims, wherein: Calculating (S4) the appearance includes: calculating the position of outputting the at least one annotation data element (21 to 25) relative to the medical image (10) based on the spatial distance between the initial position of the at least one annotation data element (21 to 25) and the determined region of interest, and / or making the at least one annotation data element (21 to 25) not overlap with the determined region of interest, or only slightly overlap with the determined region of interest.

6. A method according to any one of the preceding claims, wherein: Calculating (S4) the appearance comprises: adjusting the size of the at least one annotation data element (21 to 25), in particular adjusting the font size of the text, replacing expressions with abbreviations, adjusting text wrapping, adjusting the transparency level of the at least one annotation data element (21 to 25), displaying only the shape or outline of the at least one annotation data element (21 to 25), replacing the at least one annotation data element (21 to 25) with a thumbnail and / or applying shading, shadows and / or dashed graphic structures.

7. A method according to any one of the preceding claims, wherein: Calculating (S4) the appearance of the at least one annotation data element (21 to 25) comprises: determining an output area for displaying the at least one annotation data element (21 to 25), The output area is: an area outside the medical image (10), and / or an area outside an area in the medical image (10) representing a body or a part of a body, and / or an area outside a determined region of interest.

8. A method according to any one of the preceding claims, wherein: Outputting (S5) the calculated appearance of the at least one annotation data element (21 to 25) together with the medical image (10) is based on a predetermined attribute of the at least one annotation data element (21 to 25) and / or the medical image (10).

9. A method according to any one of the preceding claims, wherein: Calculating (S4) the appearance comprises determining an ordering of a plurality of annotation data elements (21 to 25), and wherein the calculation of the appearance of the annotation data elements (21 to 25) is performed based on the determined ordering.

10. The method according to any one of the preceding claims, wherein: The medical image (10) is received from an ultrasound imaging modality and / or from an X-ray modality, in particular from a computed tomography scan, from a positron emission tomography device and / or from a magnetic resonance tomography scan.

11. A method according to any one of the preceding claims, wherein: The at least one annotation data element (21 to 25) comprises a graphical element and / or a text element and / or a numerical element, metadata relating to the patient, a configuration of the medical imaging modality used to capture the medical image (10) and / or data of a specified date and / or data relating to the purpose of capturing the medical image (10).

12. A method according to any one of the preceding claims, wherein: The at least one annotation data element (21 to 25) is determined based on metadata and / or supplementary data provided in association with the medical image (10).

13. A method according to any one of the preceding claims, wherein: The appearance of annotation data elements (21 to 25) is adjusted in response to a zoom level of the medical image (10).

14. A device (100) for processing a medical image (10), the device (100) being configured to perform a method according to any one of the preceding method claims, the device (100) comprising: - a receiving interface (110) configured to receive a medical image (10) for processing the medical image (10); - an annotation unit (120) for determining at least one annotation data element (21 to 25) to be displayed together with the received medical image (10) and relating to the medical image (10) or a part of the medical image (10); - a processing unit (130) for determining a region of interest in the received medical image (10); - wherein the processing unit (130) is further configured to: calculate the appearance of the at least one annotation data element (21 to 25) in response to the determined region of interest; - an output interface (140) for outputting the calculated appearance of the at least one annotation data element (21 to 25) together with the medical image (10).

15. A computer program loadable into a memory unit of a computing unit, the computer program comprising a program code portion which, when executed in the computing unit, causes the computing unit to perform a method for processing medical images according to the method claimed in any one of the preceding method claims.

Images