Method, system, and computer program element for controlling an interface displaying medical images

Abstract

The present invention relates to a method, system, and computer program element for controlling an interface that displays medical images to a user on a display, comprising the steps of receiving medical images of a patient, the user displaying the medical images on a display to perform image interpretation, identifying a region of interest in the medical image using an eye tracker configured to provide tracked eye tracking data from the user, analyzing the region of interest using a first artificial intelligence (AI) module, determining anatomical structures from the medical image by correlating the identified region of interest with data from the first AI module, assessing the user's condition using a second artificial intelligence (AI) module indicative of the user's ability to perform image interpretation, and controlling the interface and the displayed medical images depending on the eye tracking data.

Description

[Technical Field]

[0001] The present invention relates to the field of controlling devices for user medical image reading, and more particularly to a method for controlling an interface for displaying medical images, a system for controlling an interface for displaying medical images, and a computer program element. [Background technology]

[0002] Image reading, such as reading chest X-rays (CXRs) in medical units, for example emergency departments, is performed in a systematic manner, thereby avoiding any time consuming, thus avoiding improvisation and misreading of life-threatening medical images, and improving cost efficiency. The image reading device is highly systematic, for example indicating which parts of the medical image need to be read.

[0003] Image interpretation is performed manually, which means a user, usually a radiologist, clicks with a computer mouse and manually navigates through the image and its review. This mode of interaction is time-consuming; for example, a typical CXR image interpretation lasts an average of 91 seconds. Additionally, radiologists are subject to numerous distractions while performing image interpretation, increasing their workload and potentially causing failure during image interpretation and analysis. Summary of the Invention [Problem to be solved by the invention]

[0004] Therefore, there is a need to optimize and improve the medical image reading process, as well as to optimize the control of the interface used to read medical images. In particular, there is a need to be able to control the interface used to display medical images on a display while the user is reading the image. The analysis of image reading should be improved, and new ways of interacting with the interface, i.e. the radiologist's workstation, can be implemented, which are less cumbersome and therefore faster.

[0005] It is an object of the present invention to provide an improved method, system and computer program element for controlling an interface for displaying medical images. [Means for solving the problem]

[0006] The object of the present invention is solved by the subject matter of the independent claims, further embodiments are incorporated in the dependent claims.

[0007] It should be noted that any feature, function, and / or element described below with reference to a method applies equally to a system, and vice versa. Thus, any feature, function, step, and / or element described below with reference to one aspect of the disclosure applies equally to any other aspect of the disclosure.

[0008] Furthermore, it should be noted that some embodiments may be described with reference to chest X-ray images. Nevertheless, the embodiments are not limited to this application. The embodiments described below with reference to one particular medical image apply equally to any other medical image, such as an X-ray image, a mammography image, an MRI image, a CT image, etc., whose interface for displaying the image may be controlled by the present invention; this list is merely exemplary and not limiting.

[0009] According to a first aspect of the present invention, a method for controlling an interface that displays medical images to a user on a display is described. The method includes receiving a medical image of a patient, displaying the medical image on the user's display for performing image interpretation, identifying a region of interest of a user in the medical image using an eye tracker configured to provide eye tracking data from the user, and analyzing the region of interest using a first artificial intelligence (AI) module. An anatomical structure is then determined in a further step from the medical image by correlating the identified region of interest with data from the first AI module, assessing the user's condition using a second AI module indicative of the user's ability to perform image interpretation, and controlling the interface and the displayed medical image in response to the eye tracking data.

[0010] In the context of the present invention, the term "image reading" is understood to describe the analysis of a medical image by a user, preferably a physician. Image reading is performed by the user by viewing the interface display according to image reading guidelines, which represent medical standards for analyzing medical images of patients. These guidelines may vary depending on the country and the applicable standard.

[0011] In the context of the present invention, the term "region of interest (ROI)" should be understood to describe a sample within a medical image of particular interest for medical diagnosis used by a user. For example, the region of interest may be the boundary of a tumor, a bronchus, a lung, a hearth, a bronchial tube, and this list is not limited. The type and size of the ROI may depend on the medical image to be analyzed, the anatomical structures that may be found in the medical image, and the part of the patient's body that undergoes medical image analysis.

[0012] In the context of the present invention, the term "user state" is understood to describe whether the user is in a physical and / or cognitive state to perform medical image reading. In particular, the user state may indicate whether the user is tired or not, indicating that the user's reading comprehension may be insufficient. Therefore, the user's fatigue state may be monitored and / or analyzed.

[0013] In other words, a method is described for controlling an interface configured to display one or more medical images to a user, allowing the user to perform medical image interpretation of the displayed one or more medical images. The interface may be controlled by a user, who may be, for example, a radiologist. In particular, the display of image information, image portions, medical image display, and ROI display may be controlled based on eye tracking data. The monitored / tracked eye tracking data may be used by a processor and stored in any suitable memory (local, online, cloud-based, etc.). The processor may be configured to process the eye tracking data, for example, to provide this data to the respective AI modules used in the method and system. Furthermore, the processor may be configured to perform the method steps using the respective electronic and computer elements. In particular, the method may be a computer-implemented method in which the processor may be configured to perform at least some or all of the method steps described in the embodiments. An advantage of the present invention is that the user can control the interface without using any other device, meaning that the user can control the interface without using a computer mouse or computer keys. Typically, interaction with a workstation, e.g., the interface, is performed via the usual modality of a mouse and keyboard. These conventional modalities may be omitted here, and in particular may be replaced with the described method and system, and the mouse and keyboard may be replaced with an eye tracker. One or more artificial intelligence modules may use eye tracking to interpret the user's view or the user's gaze direction to provide eye-guided navigation. Furthermore, the user's condition may be evaluated, in particular whether the user is able to perform satisfactory medical image reading. This means that artificial intelligence (AI) modules may be used to help the user know their own limitations during the image reading process.The method may assess the user's state and indicate the user's ability to perform the image reading, meaning that the image reading can be assessed at the start and / or during the image reading. Thus, the user's state can be assessed when starting a new image reading or when continuing a stopped / paused image reading. The eye tracker can determine the position within the medical image where the user is currently looking at at any time. By correlating this position with knowledge obtained from the first AI module, it is possible to infer the type of anatomical structure currently being investigated.

[0014] The control of the eye-guided navigation depends on the respective eye tracking data, which can be implemented in the method and system in a coded manner. For example, the detected eye tracking data is the frequency with which the user blinks, and depending on the blinks, different actions can be performed. Possible actions are, for example, whether an image or part of an image should be displayed, enlarged, and / or removed from the display. The eye tracking data may be monitored by an eye tracker, which will be described in more detail below.

[0015] This method of interaction and controlling the interface can be less cumbersome and therefore faster. Tracking the user's eyes makes it possible to control and monitor compliance with the systematic review plan. Furthermore, when using the described method, the time taken to read each medical image can be reduced, which can result in a significant cumulative reduction in resources allocated to medical image reading and, further, in a significant reduction in utilization costs.

[0016] According to an exemplary embodiment of the present invention, the method may further include simultaneously displaying the current medical image and the region of interest on a display, where the medical image and the region of interest may be displayed next to each other on the display. For example, when examining a chest X-ray image, there are at least two images that may be displayed side-by-side, with the entire medical image displayed on the left and the ROI displayed on the right. The images may also be positioned one above the other. Furthermore, the region of interest may be displayed on the medical image itself, and the ROI may be displayed on the medical image in a designated manner. For example, the ROI may be displayed / designated by a marker and / or indicator within the medical image.

[0017] According to an exemplary embodiment of the present invention, the displaying step may further include simultaneously displaying the patient's current medical image and / or displaying the patient's past medical images on the display, and the first AI module may analyze and indicate regions of interest in the past medical images that are similar to regions of interest in the current medical image. According to this embodiment, the past medical images and the current medical image may be displayed side by side or one above the other. In addition, ROIs in the past and / or current medical images may be simultaneously displayed on the display of the interface. The ROIs in the past and / or current medical images may be shown on the respective images themselves or may be further displayed on additional images on the display. The user may be able to control which images and which ROIs are displayed and how they are displayed, and the control may be described in detail in further embodiments of the present invention.

[0018] According to an exemplary embodiment of the present invention, the method may further include displaying a region of interest in the medical image displayed on the display by a first AI module in response to the received eye tracking data, generating an augmented view of the region of interest indicated by the first AI module, and removing the augmented view of the region of interest indicated by the first AI module. In other words, by using the eye tracker, a user may be able to control whether an ROI should be generated and whether the ROI should be displayed, as well as how long the ROI should be displayed on the display of the interface. An enhanced view of the ROI may be displayed together with the current medical image on the display, and the medical image and the ROI may be displayed next to, alongside, and / or on top of each other. An enhanced view may be understood to include (mean) a zoomed-in or zoomed-out view of the ROI or a rotated view of the ROI, meaning that the ROI is displayed, e.g., a zoomed-in and rotated image to enable better medical image interpretation.

[0019] According to an exemplary embodiment of the present invention, when the eye tracker determines that the user is gazing at the region of interest for a period of time, the first AI module can be used to show the region of interest having a boundary, the boundary can be rectangular, generating the enhanced view can include improving the quality of this view, and the eye tracker removes the enhanced view of the region of interest when the user determines that they want to remove their view from the region of interest.

[0020] Improving the quality of the enhanced view can include contrast adaptation, particularly contrast enhancement using a pre-calculated lookup table, where a respective pre-calculated lookup table is provided for each respective organ being examined. Contrast adaptation can also include fly's contrast adaptation using either min-max normalization, histogram normalization, or contrast-limited adaptive histogram equalization. Furthermore, the quality improvement can be a region-specific image filter, edge enhancement, and / or a high-pass sharpening filter, and / or a Canny or Sobel edge-finding filter. While improving the quality of the enhanced view can apply at least one of the above-mentioned features, multiple features can also be applied to improve quality. Furthermore, the improved quality of the enhanced view of the ROI can include improved, increased resolution of the medical image in the region of the ROI or increased resolution of the displayed ROI.

[0021] According to an exemplary embodiment of the present invention, analyzing the region of interest using a first artificial intelligence (AI) module may include at least one of object detection AI, semantic segmentation AI, or instance segmentation AI. Therefore, the analysis of the ROI may be performed using all of the AI ​​modules, one after the other, simultaneously, or only on one AI module, or even by two AI modules. The object detection AI may use a deep convolutional neural network configured for object detection, which may be configured to find objects in the image and further configured to indicate the objects, for example, using rectangles. The semantic segmentation AI may use a deep convolutional neural network configured for segmentation, which may mean assigning a corresponding unique class label to each pixel in the image. For example, both lungs are output (segmented) in the same color. Instance segmentation may combine both object detection and segmentation, specifically identifying all terms or instances of a class visible in the image and providing them with distinct masks or bounding boxes. For example, instance segmentation may separate the left and right lungs, which may be output in different colors. Furthermore, object detection AI can use the Faster RCNN or YOLO algorithm. Semantic segmentation AI can use DeepLab, UNet, or HRNet, for example. Instance segmentation AI can use Mask RCNN, MaskLab, TensorMask, YOLACT, SOLO or SOLOv2, or CenterMask, for example. Using object detection AI, it is possible to determine the anatomical structure the user is currently looking at. Furthermore, with information about which anatomical structure the user is focusing on, the system can appropriately highlight the ROI and provide appropriate guidance.

[0022] According to an exemplary embodiment of the present invention, the method may further include using a first AI module to correlate a user's image reading of the medical image with the reading guidelines and indicate whether the image reading was performed in accordance with the reading guidelines. For example, there may be several reasons why a user may want to pay more attention to a particular portion of an image. For example, for a chest X-ray image, a specific goal may be to find and analyze a specific abnormality / disease that may be located in a specific portion of the image. However, an incomplete reading of a medical image should / is flagged, for example, using a visual marker displayed on the interface display. Alternatively, or additionally, an incompletely read medical image may be marked by the processor or one of the AI ​​modules, indicating that the reading was not performed in accordance with the reading guidelines. Furthermore, an incompletely read medical image may be added to a hanging queue for further reading and checking with the guidelines at a later time. Thus, a user may be allowed to perform an incomplete reading and therefore an incomplete review of the image, or only the incomplete review may be indicated. For example, in a life-threatening situation, it may be important for the user to be able to perform only a partial reading and then complete a full formal reading at a later time.

[0023] According to an exemplary embodiment of the present invention, controlling the interface may include at least one of controlling the size of a medical image, controlling a portion of a medical image, controlling the size of a portion of a medical image, scrolling through multiple medical images, controlling displayed medical image data, controlling highlighting of a medical image and / or a portion of a medical image, and controlling patient data displayable on the display with reference to the medical image. The control may be performed for at least one of the above-mentioned features and / or for multiple of these features. In particular, the controlling may include a single step of one of the above-mentioned features or may include several, multiple, and / or all of them. Accordingly, detected eye tracking data may be associated with each control feature. For example, a single blink may confirm the display of an image or ROI. Another amount of blinking may be used to control the size of a displayed ROI or the size of an extended view. Eye or head movement may be used to control scrolling through multiple images. The examples are not intended to be limiting, and other eye tracking data may be used for other control features.

[0024] According to an exemplary embodiment of the present invention, eye tracking may be used to monitor at least one of the following eye tracking data: a user's eye movement, a user's eye gaze direction, an average eye fixation time, a pupil area, the time the user looks at a point on a display, eyelid movement, eye color, and a user's head movement. For example, the eye movement may be any eye movement, which may be up-and-down eye movement and / or left-and-right eye movement. The monitored eye tracking data may be only one of the above-mentioned eye tracking data, or may be multiple or all of these characteristics. One, multiple, or all of the eye tracking data may be tracked simultaneously. Eye tracking may be used to monitor the movement and behavior of each of the user's monocular eyes. The user's eye observation direction may be the gaze determined from both eyes. The average eye fixation time may be the average of any time the user gazes at a specific point in a medical image, and the duration of the average fixation time may be predefined by a respective reading standard. Pupil area and changes in this area may be measured for each monocular eye. Furthermore, the eyelid movement may be an eyelid opening and closing, and / or one or more blinks. The tracked eye tracking data may be used by a second AI module to determine the user's fatigue. For example, the average fixation time and pupil area may be used as features for fatigue detection using a fuzzy K-means clustering method. These features may be extracted from pupil segmentation using computer vision methods such as thresholding, the modified elliptical Hough transform method, or by deep learning methods, for example, using the DeepLab architecture.

[0025] According to an exemplary embodiment of the present invention, the first AI module may be trained using data from medical training images containing similar information to the displayed medical image, and the medical training images used to train the first AI module may be annotated by a medical expert who annotates the anatomical structures in each of the medical training images. The trained dataset may be an annotated dataset, where the annotation is a pixel-by-pixel decision as to whether an object is present at this location. Alternatively, the first AI module may be a pre-trained model and may be trained on non-medical data.

[0026] According to an exemplary embodiment of the present invention, the second AI module may be trained using a dataset of iris and pupil images acquired by an eye tracker, with the pupil annotated pixel by pixel.

[0027] According to an exemplary embodiment of the present invention, the eye tracker, the first AI module, and the second AI module operate simultaneously during the display of the medical image to the user on the display, and thus the processor can be configured to perform all steps simultaneously, which can make the image reading process faster.

[0028] According to an exemplary embodiment of the present invention, the second AI module can evaluate at least one of eye movement, eyelid movement, eye color, or pupil dilation to evaluate the user's ability to perform image reading. One or more of the above-mentioned features may be evaluated by the AI ​​module, or all of these features may be evaluated. Furthermore, the evaluation of multiple features may be performed simultaneously or sequentially. In particular, the second AI module can evaluate the user's fatigue state to determine the user's ability to perform sufficient reading of medical images. In particular, fatigue can be evaluated by eye tracking. For example, eye movement and pupil dilation can be suitable markers for fatigue prediction. This has the advantage of notifying the user that their reading may be insufficient. For example, eye color monitoring can monitor for irritated eyes or red eyes, which, when detected, can be used to indicate that the user is unable to properly view the displayed image.

[0029] According to a further exemplary embodiment of the present invention, the medical image to be displayed and controlled on the interface may be a chest X-ray image, an MRI image, a mammography image, an ultrasound image, a CT image, or a PET image.

[0030] According to a second aspect of the present invention, a system for controlling an interface that displays medical images to a user is described. The system includes a display having an interface for interfacing with a user. The system further includes an eye tracker configured to provide tracked eye tracking data from the user. The system further includes a processing unit configured to communicate with the display, the processing unit being further configured to receive medical images of a patient, display the medical images on the display for the user to perform image interpretation, identify a region of interest of the user in the medical image using the eye tracker, analyze the region of interest using a first artificial intelligence (AI) module, determine anatomical structures from the medical image by correlating the identified region of interest with data from the first AI module, assess the user's condition using a second artificial intelligence (AI) module indicative of the user's ability to perform image interpretation, and control the interface and the displayed medical image depending on the eye tracking data.

[0031] In other words, the system includes a user interface that can be controlled by a user using an eye tracker. Additionally, the system can further include at least two AI modules, at least one of which analyzes specific portions of the medical image currently being viewed by the user to enhance the presentation of the structure currently under investigation and verify the user's adherence to systematic review guidelines. At least one of the modules can assess user fatigue and, therefore, the user's ability to perform image interpretation. The system can, for example, use an eye tracker to control conventional viewer software, which can be implemented by the at least two AI modules.

[0032] According to an exemplary embodiment of the present invention, the system may further include a camera configured to communicate with the processor and configured to capture at least one of the following: a user's eye movements, a user's eye gaze direction, an average eye fixation time, a pupil area, a time the user looks at a point on the display, eyelid movements, or a user's head movements. In particular, the camera may be a respective eye tracker or may be used as part of the eye tracker. The camera / eye tracker may be located near the display. The eye tracker allows the user to know at any time the position on the medical image on which the user is currently focused, which may be achieved by using at least one of the above-mentioned eye tracking features that the camera may be configured to monitor. The camera may be configured to capture at least one of the above-mentioned features, or may capture all of them, or may capture only multiple of these features. In particular, the features captured by the camera may be captured simultaneously.

[0033] According to a third aspect of the present invention, a computer program element for controlling an interface for displaying medical images to a user is described. The computer program element, when executed by a processor of the system, is adapted to cause the system to receive medical images of a patient and display the medical images on a user's display for performing image interpretation, identify a region of interest of the user in the medical image using an eye tracker configured to provide tracked eye tracking data from the user, analyze the region of interest using a first artificial intelligence (AI) module, determine anatomical structures from the medical image by correlating the identified region of interest with data from the first AI module, assess the user's condition using a second artificial intelligence (AI) module indicative of the user's ability to perform image interpretation, and control the interface and the displayed medical image in dependence on the eye tracking data.

[0034] A computer program element may be part of a computer program, but may also be an entire program in its own right, for example a computer program element may be used to update an existing computer program to achieve the present invention.

[0035] The program elements may be stored on a computer readable medium, which may be considered as a storage medium such as a USB stick, a CD, a DVD, a data storage device, a hard disk or any other medium on which such program elements may be stored.

[0036] According to various embodiments of the present disclosure, the methods described herein may be implemented using a hardware computer system executing a software program. Furthermore, in exemplary, non-limiting embodiments, implementations may include distributed processing, component / object distributed processing, and parallel processing. Virtual computer system processing may implement one or more of the methods or functions described herein, and the processors described herein may be used to support virtual processing environments.

[0037] It should be noted that embodiments of the present invention are described with reference to different subject matters. In particular, some embodiments are described with reference to apparatus / system type claims, while other embodiments are described with reference to method type claims. However, those skilled in the art will know from the above and following description that, unless otherwise notified, any combination of features belonging to one type of subject matter, as well as any combination of features relating to different subject matters, such as between features of a particular apparatus type claim and features of a method type claim, is considered to be disclosed in the present application.

[0038] The above and further aspects of the present invention will be apparent from and explained with reference to the following examples of embodiments. The present invention will be explained in more detail below with reference to the following examples, but the present invention is not limited to the embodiments. [Brief explanation of the drawings]

[0039] [Figure 1] 1 shows a flow diagram of a method according to one embodiment of the present invention. [Figure 2] 1 illustrates a medical image displayed on an interface according to one embodiment of the present invention. [Figure 3] 10 illustrates additional medical images displayed on an interface according to one embodiment of the present invention. [Figure 4] 10 illustrates additional medical images displayed on an interface according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0040] The figures in the drawings are schematic and it should be noted that in different figures, similar or identical elements are provided with the same reference signs. FIG. 1 shows a flow diagram including method steps according to one embodiment of the present invention. A method for controlling an interface that displays medical images to a user on a display is described using the following steps: In step 101, a medical image of a patient is received, and in step 102, the medical image is displayed on a display for a user so that the user can perform image interpretation. In step 103, a region of interest of the user within the medical image is identified using an eye tracker configured to provide eye tracking data tracked from the user. Once the region of interest is identified, in step 104, the region of interest is analyzed using a first artificial intelligence (AI) module. In step 105, an anatomical structure from the medical image is determined by correlating the identified region of interest with data from the first AI module. In step 106, the user's condition is evaluated using a second artificial intelligence (AI) module indicating the user's ability to perform image interpretation. In step 107, the interface and the displayed medical image are controlled depending on the eye tracking data. The steps of the method may be performed one after the other, as the order of the steps may vary and the order presented herein is not limiting. Furthermore, some or at least a plurality of method steps may be performed simultaneously. For example, the evaluation of the user's condition may be performed simultaneously during the entire image acquisition by the user. Thus, the evaluation of the user's condition may be performed simultaneously with the identification of the ROI, the analysis of the ROI, and the determination of the anatomical structure. Furthermore, the method may further include additional substeps, for example, after the evaluation of the user's condition, a display may be displayed, which may be displayed in another method step on the display for the user. It is also possible to restart the method after controlling the interface if the image acquisition is not complete and the user wishes to continue the image acquisition when the image acquisition is flagged as incomplete. Thus, a step may be restarted after step 107, for example, by displaying an image in step 102 and then redisplaying the same image when continuing the image acquisition.Meanwhile, it may be possible that a new image is received from the same patient and a further image reading process should be performed on this image. The step of displaying the medical image may include the following further substeps: displaying a region of interest using a first AI module in the medical image; and generating a highlight of the region of interest indicated by the first AI module, wherein the display of the ROI, the display of the highlight, and the display of the medical image itself are controlled depending on the eye tracking data.

[0041] In particular, the step of controlling the interface may further comprise the steps of controlling a size of the medical image, controlling a portion of the medical image, controlling a size of a portion of the medical image, scrolling through a plurality of medical images, controlling displayed medical image data, controlling highlighting of the medical image and / or portions of the medical image, and controlling patient data that can be displayed on the display with reference to the medical image, each of the sub-steps may be controlled by the user depending on the eye tracking data.

[0042] FIG. 2 illustrates a system for controlling an interface 100 that displays a medical image 201 to a user, according to one embodiment of the present invention. The system includes a display 204 that includes the interface 100 for interfacing with a user. The system further includes an eye tracker 205 configured to provide tracked eye tracking data from the user. Furthermore, the system includes a processing unit configured to communicate with the display 204. The processing unit is configured to receive the medical image 201 of a patient and display the medical image 201 on the display 204 for the user to perform image interpretation. Furthermore, the processing unit is configured to identify a region of interest of the user within the medical image 201 using the eye tracker 205. The region of interest in FIG. 2 is depicted as a rectangle on the left side of the display 204. The region of interest is analyzed using a first artificial intelligence (AI) module. An anatomical structure 206 is determined from the medical image 201 by correlating the identified region of interest with data from the first AI module. The user's condition is evaluated by a processor using a second artificial intelligence (AI) module, indicating the user's ability to perform image interpretation. The interface 100 and the displayed medical image 201 are controlled depending on the tracked eye tracking data. The current medical image 201 and a region of interest (rectangle) are simultaneously displayed on the display 204. In FIG. 2, the medical image 201 and the highlight 202 of the region of interest are displayed next to each other on the display 204. The current medical image (201) and the ROI are shown on the left, and the highlight 202 is shown on the left side of the display 204. When the user looks at a particular structure in the medical image 201, the interface highlights the boundary 203 of the ROI with a rectangular line. The boundary 203 is known from the first AI module, which uses object detection and / or semantic segmentation to determine the anatomical structure 206 the user is gazing at. For example, the highlighted view 202 is a zoomed-in version of the ROI with adjusted contrast to optimize and facilitate image interpretation by the user.The AI ​​module's control over the detection of the ROI can be confirmed by the user's blink. For example, the identified ROI is marked with a rectangle 203 in the medical image 201, and the user confirms this identification with a single blink for a simple blink code for "yes / correct." If the ROI is not a corrected ROI, the user can use a different blink code, such as two blinks for "no," or the eye tracker can analyze head movement, such as shaking the user's head from left to right, or vice versa, to interpret it as "no." When the user is satisfied with their investigation and / or has finished reading the extended view 202, the user can return their gaze to the overview 201, i.e., the entire medical image 201 on the left side of the display 204 and the extended view 202 and / or the highlighted ROI 203 are removed from the display 204. In FIG. 2, the displayed medical image 201 is an image of a patient's chest, and the identified ROI, indicated by the boundary 203, is the airway. The enhanced view 202 (the quality of the enhanced view 202 can be improved using the features described in the embodiments in the above section) having improved image quality is of the airways, particularly the bronchioles.

[0043] The eye tracker 205 is placed in close proximity to the user's screen, here on top of the display 204. The eye tracker is a camera 205 configured to communicate with the processor and configured to capture at least one or more of the user's eye movements, the direction of gaze of the user's eyes, the average fixation time of the eyes, the pupil area, the period during which the user looks at a point on the display, and the user's head movements. Thus, the eye tracker 205 always knows where on the medical image 201 the user is currently looking.

[0044] FIG. 3 illustrates another display of medical images 3 according to an embodiment of the present invention. In FIG. 3 , historical medical images are displayed, with multiple historical images 311-314 displayed in a row at the bottom of the display 204. The ROI of the historical medical image 310 is displayed on the left side of the display, and the current ROI of the current medical image is displayed on the right side of the display 204. Thus, the displaying step includes simultaneously displaying the patient's current medical image 201 and / or displaying the patient's past medical images on the display 204, and the first AI module analyzes and indicates regions of interest 303 in the past medical images that are similar to the region of interest 203 in the current medical image 201. The indication of the regions of interest, e.g., boundaries 303, 203, can include color coding to avoid confusion regarding the nature of the image, i.e., current or historical. For example, the boundary of the region of interest 303 in the historical image is colored a different color than the boundary 203 in the current medical image 201.

[0045] The interpretation of eye tracking data allows for several possibilities. For example, three blinks can be defined as a signal that brings up a patient history screen. The interpretation of eye tracking data can also be user-specified, meaning the user can change the settings according to their preferences. In FIG. 3, three blinks present the patient's history images 311-314, which are displayed in a row below, with one specific history image highlighted to the left. The selected history medical image is confirmed by the user with an eye blink or a head nod. For example, scrolling through the history images 311-314 in a row can be controlled by moving the head to the left for the picture on the left side of the row, or by moving the head to the right for the picture on the right side of the row. The eye tracker 205 tracks these head movements, and the eye tracking data is analyzed by a processor, which controls the interface for what should be displayed. Furthermore, the respective image data 315 can be displayed on the medical image and / or on the ROI 303; for example, this information can be displayed in the upper left corner of the ROI 303. The information displayed may include the date the image was taken (in the case of historical medical images) or any other relevant patient data such as age, illness, etc.

[0046] FIG. 4 illustrates additional medical images displayed on an interface according to an embodiment of the present invention. The present invention can also be applied to the 3D medical images shown in FIG. 4. During the generation of CT, MRI, or PET images, a patient's body part is subdivided into a set of slices. A single image may be such a slice, and to analyze the whole body part, each image slice must be read by the user. As shown in FIG. 4, each slice may be displayed on the interface 100 as a single medical image 201. The user can control the interface 100 using head movements to scroll through each single slide, for example, by moving their head left to select the slide 420 before the current image 201 and / or by moving their head right to select the slide 421 after the current image 201. The interface 100 can also be controlled by similar eye movements to the left or right to scroll through different medical image slides.

[0047] While the present invention has been illustrated and described in detail in the drawings and the foregoing description, such illustration and description are to be considered exemplary or illustrative and not restrictive, and the present invention is not limited to the disclosed embodiments. Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. It should be noted that the term "comprising" does not exclude other elements or steps, and that "a" or "an" does not exclude a plurality. Also, elements described in association with different embodiments may be combined. It should also be noted that reference signs in the claims should not be construed as limiting the scope of the claims. [Explanation of symbols]

[0048] 100 interfaces 201 Medical Imaging 202 Highlighting 203, 303 boundary 204 Display 205 Eye Tracker 206, 306 anatomical structure 310 Historical Images 311-314 Past images 315 Image Data 420 image slices 421 Image Slices S101-S107 Steps

Claims

1. 1. A method for controlling an interface for displaying medical images to a user on a display, comprising: receiving a medical image of a patient; displaying the medical image on a display for the user to perform image interpretation; identifying a region of interest of a user in the medical image using an eye tracker configured to provide tracked eye tracking data from the user; analyzing the region of interest using a first artificial intelligence module; determining anatomical structures from the medical image by correlating the identified region of interest with data from the first artificial intelligence module; assessing the user's condition using a second artificial intelligence module indicative of the user's ability to perform the image interpretation; controlling the interface and the displayed medical image in dependence on the eye tracking data; A method comprising:

2. displaying the current medical image and the region of interest simultaneously on the display. and the medical image and the region of interest are displayed next to each other on the display; The method of claim 1.

3. the displaying step further comprises simultaneously displaying current medical images of the patient and / or past medical images of the patient on the display; The method of claim 1 or 2, wherein the first artificial intelligence module analyzes and indicates regions of interest in previous medical images that are similar to regions of interest in the current medical image.

4. dependent on the received eye tracking data; displaying the region of interest in the medical image displayed on the display by the first artificial intelligence module; generating, by the first artificial intelligence module, an enhanced view of the indicated region of interest; The method according to claim 1 , further comprising the step of removing, by said first artificial intelligence module, an enhanced view of said indicated region of interest.

5. the eye tracker determining that the user is fixating the region of interest for a period of time, and using the first artificial intelligence module to indicate the region of interest having a boundary, the boundary being rectangular; generating the enhanced view comprises improving a quality of the enhanced view of the medical image; The method of claim 4 , wherein the eye tracker removes the enhanced view of the region of interest when the eye tracker determines that the user removes their view from the region of interest.

6. analyzing the region of interest using the first artificial intelligence module comprises at least one of an object detection artificial intelligence, a segmentation artificial intelligence, or an instance segmentation artificial intelligence; The object detection artificial intelligence uses a deep convolutional neural network configured for object detection; The segmentation artificial intelligence uses a deep convolutional neural network configured for segmentation.

6. The method according to any one of claims 1 to 5.

7. using the first artificial intelligence module to correlate the user's image interpretation of the medical image with interpretation guidelines; indicating on the medical image whether the image reading has been performed in accordance with the reading guidelines; 7. The method according to claim 1, comprising:

8. 8. The method of claim 1, wherein the step of controlling the interface comprises at least one of the steps of: controlling a size of the medical image; controlling a portion of the medical image; controlling a size of a portion of the medical image; controlling scrolling through a plurality of medical images; controlling displayed medical image data; controlling highlighting of the medical image and / or portions of the medical image; and controlling patient data that can be displayed on the display with reference to the medical image.

9. 9. The method of claim 1, wherein the eye tracking is used to monitor eye tracking data that is at least one of the following: eye movements of the user, gaze direction of the user's eyes, average fixation time of the eyes, pupil area, duration during which the user looks at a point on the display, eyelid movement, eye color, or head movement of the user.

10. the first artificial intelligence module is trained using data from medical training images having information similar to the medical image to be displayed, the medical training images used to train the first artificial intelligence module being annotated by a medical expert who annotates anatomical structures in each of the medical training images; and / or The first artificial intelligence module is a pre-trained artificial intelligence module trained on non-medical data.

10. The method according to any one of claims 1 to 9.

11. 11. The method of claim 1, wherein the eye tracker, the first artificial intelligence module, and the second artificial intelligence module operate simultaneously while displaying the medical image to the user on the display.

12. 12. The method of claim 1, wherein the second artificial intelligence module evaluates at least one of eye movement, eyelid movement, eye color, or pupil dilation to evaluate the user's ability to perform the image reading.

13. 1. A system for controlling an interface that displays medical images to a user, comprising: a display having an interface for interfacing with the user; an eye tracker configured to provide tracked eye tracking data from the user; a processing unit configured to communicate with a display, the processing unit further comprising: receiving a medical image of a patient; displaying the medical image on a display for the user to perform image interpretation; using the eye tracker to identify a region of interest of a user in the medical image; analyzing the region of interest using a first artificial intelligence module; determining anatomical structures from the medical image by correlating the identified region of interest with data from the first artificial intelligence module; assessing the user's condition using a second artificial intelligence module indicative of the user's ability to perform the image reading; controlling the interface and the displayed medical image in dependence on the eye tracking data; a processing unit configured to cause the A system having:

14. a camera configured to communicate with the processor and configured to capture at least one of the user's eye movements, eyelid movements, gaze direction of the user's eyes, average fixation time of the eyes, pupil area, duration of time the user looks at a point on the display, and head movement of the user; The system of claim 12 further comprising:

15. 1. A computer program element for controlling an interface for displaying medical images to a user, the computer program element, when executed by a processor of a system, providing the system with: receiving a medical image of a patient; displaying the medical image on a display for the user to perform image interpretation; identifying a region of interest of a user in the medical image using an eye tracker configured to provide tracked eye tracking data from the user; analyzing the region of interest using a first artificial intelligence module; determining anatomical structures from the medical image by correlating the identified region of interest with data from the first artificial intelligence module; assessing the user's condition using a second artificial intelligence module indicative of the user's ability to perform the image reading; controlling the interface and the displayed medical image in dependence on the eye tracking data; A computer program element configured to cause the execution of

Images