Method, system and computer program unit for controlling an interface for displaying medical image

Abstract

The present invention relates to a method, system and computer program element for controlling an interface for displaying a medical image to a user on a display, comprising and configured to perform the steps of: receiving a medical image of a patient; displaying the medical image on a display for a user to perform image reading; identifying a region of interest in the medical image using an eye tracker configured to provide eye tracking data tracked from the user; analyzing the region of interest using a first artificial intelligence (AI) module; determining an anatomical structure from the medical image by correlating the identified region of interest with data from the first AI module; evaluating, using a second artificial intelligence (AI) module, a state of the user indicating an ability of the user to perform image reading; the interface and the displayed medical image are controlled in accordance with the eye tracking data.

Description

Technical Field

[0001] The present invention relates to the field of control devices for the reading of medical images for a user, and more particularly to a method for controlling an interface for displaying medical images, a system for controlling an interface for displaying medical images, and a computer program unit. Background Art

[0002] Image reading, such as the reading of chest X-rays (CXR) in a medical unit (e.g., in an emergency department), is performed in a systematic manner so that any time-consuming process can be avoided, and thus life-threatening improvisation and misreading of medical images can be avoided, and cost efficiency can also be improved. The arrangement for performing image reading is extremely coded, for example, which part of the medical image must be read.

[0003] Image reading is performed manually, which means that a user, typically a radiologist, clicks using a computer mouse and manually navigates through the images and studies using the mouse. This interaction mode is time-consuming. For example, the reading of a typical CXR image lasts on average 91 seconds. In addition, there are many distractions for the radiologist when performing image reading, and the workload increases, which may lead to failures during image reading and image analysis. Summary of the Invention

[0004] Therefore, there is a need to optimize and enhance the process of reading medical images, and there is a need to optimize the control of the interface used for medical image reading. In particular, it is necessary to be able to control the interface used for displaying medical images on a display during image reading by a user. The analysis of image reading should be improved, and new ways of interacting with the interface, i.e., the radiologist's workstation, can be implemented, which are less cumbersome and thus faster.

[0005] The object of the present invention is to provide an improved method, system, and computer program unit for controlling an interface for displaying medical images.

[0006] The object of the present invention is solved by the subject matter of the independent claims, wherein further embodiments are incorporated in the dependent claims.

[0007] It should be noted that any feature, function, and / or element described below with reference to the method equally applies to the system, and vice versa. Therefore, any feature, function, step, and / or element described below with reference to one aspect of the present disclosure equally applies to any other aspect of the present disclosure.

[0008] In addition, it should be noted that some of the embodiments may be described with reference to chest X-ray images. However, the embodiments are not limited to this application. The following description of embodiments with reference to a particular medical image description is equally applicable to any other medical image, such as X-ray images, mammography images, MRI images, CT images can also be analyzed, and the interface for displaying the image can be controlled by the present invention, where this list is merely exemplary and not limiting.

[0009] According to a first aspect of the present invention, a method for controlling an interface for displaying a medical image to a user on a display is described. The method includes the following steps: receiving a medical image of a patient; displaying the medical image at the display for the user to perform image reading; in addition, the method includes the following steps: using an eye tracker to identify a region of interest in the medical image by the user, the eye tracker being configured to provide eye tracking data tracked from the user; using a first artificial intelligence (AI) module to analyze the region of interest. Then, in another step, the anatomical structure is determined based on the medical image by the following operations: correlating the identified region of interest with the data from the first AI module, using a second artificial intelligence (AI) module to evaluate the state of the user, which indicates the user's ability to perform image reading; and controlling the interface and the displayed medical image based on the eye tracking data.

[0010] In the context of the present invention, the term "image reading" should be understood as describing the analysis of a medical image by a user, where the user is preferably a medical staff member, a doctor. The user performs image reading by viewing the display of the interface according to image reading guidelines, where these image reading guidelines represent medical standards for analyzing a patient's medical image. These guidelines may vary according to the country and the applied standards.

[0011] In the context of the present invention, the term "region of interest (ROI)" should be understood as describing a sample within a medical image of particular interest for medical diagnosis by a user. For example, the region of interest may be a tumor, a bronchus, a lung, a heart, the boundary of a bronchus, where this list is not restrictive. The type and size of the ROI may depend on the medical image to be analyzed, the anatomical structures that can be found in the medical image, and the part of the patient's body undergoing medical image analysis.

[0012] In the context of the present invention, the term "state of the user" should be understood as describing whether the user is in a physical and / or cognitive condition for performing the reading of a medical image. In particular, the state of the user should indicate a possible deficiency in the user's reading, regardless of whether the user is tired. Thus, the fatigue state of the user can be monitored and / or analyzed.

[0013] In other words, a method for controlling an interface is described, the interface being configured to display one or more medical images to a user, where the user can perform a medical image reading on the one or more displayed medical images. The interface can be controlled by a user, who can be, for example, a radiologist. In particular, the display of the following can be controlled based on eye-tracking data: image information, parts of the image, the display of the medical image, and the display of the ROI. The monitored / tracked eye-tracking data is used by a processor and can be stored in any suitable memory (local or online, cloud-based, etc.). The processor is configured to process the eye-tracking data, for example, to provide the data to the corresponding AI modules used in the method and system. In addition, the processor can be configured to perform the steps of the method using the corresponding electronic and computer components. In particular, the method can be a computer-implemented method, where the processor can be configured to perform at least some or all of the method steps described in the embodiments. The advantage of the described method is that the user can control the interface without using any other device, which means that the user can control the interface without using a computer mouse or computer keys. Typically, the interaction with a workstation (such as the interface) occurs through the usual modalities, which are the mouse and keyboard. Now these usual modalities can be omitted, and in particular, the described method and system can be used instead, where the mouse and keys can be replaced by an eye tracker. One or more artificial intelligence modules can use eye tracking to interpret the user's view or the direction of the user's gaze in order to provide eye-guided navigation. In addition, the state of the user can be evaluated, in particular, whether the user is capable of performing a sufficient medical image reading. This means that artificial intelligence (AI) modules can be used to help the user become aware of his own limitations during the image reading process. The method can evaluate the state of the user and indicate the user's ability to perform the image reading, which means that the image reading can be evaluated at the start and / or during the image reading. Thus, the state of the user can be evaluated when starting a new image reading or continuing an image reading that has been stopped / paused. The eye tracker can be able to determine at any time the position in the medical image that the user is currently looking at. By correlating this position with the knowledge obtained from the first AI module, the type of anatomical structure currently being studied can be inferred.

[0014] The control of the eye-guided navigation depends on the corresponding eye-tracking data, where each eye-tracking data can be implemented in an encoded manner in the method and system. For example, the detected eye-tracking data is the frequency of the user's blinks, and different actions can be performed according to the blinks. Possible actions are, for example, whether an image or a part of the image should be displayed, magnified, and / or removed from the display. The eye-tracking data that can be monitored by the eye tracker will be further elaborated in detail below.

[0015] This method of interaction and method of control interface can be less cumbersome and thus faster. Tracking of the user's eyes allows for control and monitoring of compliance with a systematic review protocol. In addition, when using the method, the time spent reading each medical image can be reduced, which can lead to a significant cumulative reduction in the resources allocated to medical image reading, which can also lead to a substantial reduction in utilization costs.

[0016] According to an exemplary embodiment of the present invention, the method may further include the step of simultaneously displaying a current medical image and a region of interest on a display, wherein the medical image and the region of interest may be displayed side by side with each other on the display. For example, when examining a chest X-ray image, there are at least two images that can be displayed in such a way that the images are arranged side by side, wherein the entire medical image is displayed on the left and the ROI is displayed on the right. It is also possible that the images may be arranged above each other. In addition, the region of interest may be displayed within the medical image itself, wherein the ROI is displayed in an indicative manner within the medical image. For example, markers and / or indicators within the medical image are used to display / indicate the ROI.

[0017] According to an exemplary embodiment of the present invention, the display step further includes: simultaneously displaying a current medical image of a patient and / or displaying a historical medical image of the patient on a display, wherein a first AI module analyzes and indicates a region of interest in the historical medical image that is similar to the region of interest of the current medical image. According to this embodiment, the historical medical image and the current medical image may be displayed side by side or may also be displayed above each other. Additionally, the ROIs of the historical and / or current medical images may be simultaneously displayed on the display of the interface. The ROIs of the historical and / or current medical images may be indicated within the respective images themselves and may also be displayed in an additional image on the display. The user may be able to control which image and which ROI should be displayed and in what manner they should be displayed, wherein this control may be elaborated in further embodiments of the present invention.

[0018] According to an exemplary embodiment of the present invention, the method may further include the following steps: displaying a region of interest in a medical image displayed on a display according to received eye tracking data by a first AI module; generating an enhanced view of the indicated region of interest by the first AI module; and removing the enhanced view of the indicated region of interest by the first AI module. In other words, the user may be able to control the interface by using an eye tracker as to whether an ROI should be generated and whether the ROI should be displayed and for how long the ROI should be displayed on the display of the interface. The enhanced view of the ROI may be displayed at the display together with the current medical image, where the medical image and the ROI may be displayed side by side, juxtaposed, and / or above each other. The enhanced view may be understood to include (mean) a magnified or reduced view of the ROI or a rotated view of the ROI, which means that the ROI is displayed, for example, magnified and rotated images to allow better medical image reading.

[0019] According to an exemplary embodiment of the present invention, wherein when the eye tracker can determine that the user has gazed at a region of interest for a period of time, the first AI module is used to indicate the region of interest by using a boundary, wherein the boundary may be rectangular, wherein generating the enhanced view may include improving the quality of the view, and wherein when the eye tracker determines that the user removes his view from the region of interest, the enhanced view of the region of interest is removed.

[0020] Improving the quality of the enhanced view may include: contrast adaptation, in particular contrast enhancement using a pre-computed look-up table, wherein for each respective organ to be examined, a corresponding pre-computed look-up table is provided. Contrast adaptation may also use fly-by contrast adaptation, which uses min-max normalization, histogram normalization, or contrast-limited adaptive histogram equalization. In addition, the quality improvement may be a region-specific image filter, edge enhancement, and / or high-pass sharpening filter and / or an edge finding filter such as Canny or Sobel. The improvement of the quality of the enhanced view may apply at least one of the above features, and on the other hand, multiple of these features may be applied to improve the quality. In addition, the improved quality of the enhanced view of the ROI may include: improved and magnified resolution of the medical image in the region of the ROI, or magnified resolution of the displayed ROI.

[0021] According to an exemplary embodiment of the present invention, the step of using a first artificial intelligence (AI) module to analyze a region of interest may include at least one of the following: object detection AI, semantic segmentation AI, or instance segmentation AI. Thus, it may also be possible that the analysis of the ROI is performed using all AI modules (one after another or simultaneously), or only by an AI module or by two AI modules. The object detection AI may use a deep convolutional neural network configured for object detection, where object detection may be configured to find one or more objects in an image and may also be configured to indicate the object using, for example, a rectangle. The semantic segmentation AI may use a deep convolutional neural network configured for segmentation, which may mean assigning a corresponding and unique class label to each pixel in the image. For example, the two lungs will be output in the same color (segmented). Instance segmentation may combine both object detection and segmentation. In particular, it may identify each item or instance of a visible class in the image and give it a different mask or bounding box. For example, instance segmentation may separate the lungs on the right and left sides, and the output colors may be different. In addition, the object detection AI may use faster RCNN or YOLO algorithms. The semantic segmentation AI may use, for example, DeepLab, UNet, or HRNet. The instance segmentation AI may use, for example, Mask RCNN, MaskLab, TensorMask, YOLACT, SOLO, or SOLOv2, or CenterMask. By using the object detection AI, the anatomical structure that the user is currently looking at can be determined. In addition, using the information about which anatomical structure the user is focusing on, the system may be able to appropriately enhance the ROI and provide appropriate guidance.

[0022] According to an exemplary embodiment of the present invention, the method may further include the following steps: using a first AI module to correlate an image reading of a medical image by a user with a reading guideline, and indicating at the medical image whether the image reading is performed in accordance with the reading guideline. For example, a user may have several reasons to pay more attention to a specific part of an image. For example, for a chest X-ray image, a specific objective may be to find and analyze a specific abnormality / disease that may be located in a specific part of the image. However, an incomplete reading of the medical image can / should be alerted and marked, for example, by using a visual marker displayed on a display of the interface. Alternatively or additionally, a medical image that is not fully read can be marked, which can be indicated by a processor or by one of the AI modules in the AI module that the reading is not performed according to the reading guideline. In addition, a medical image with an incomplete reading can be added to a pending queue so that a further reading and checking against the guideline can be performed later. Thus, a user can be authorized to perform an incomplete reading and thus an incomplete review of the image, and can only indicate the incomplete review. For example, in a life-threatening situation, it is important that a user should be able to perform only a partial reading and be able to complete a full formal reading later.

[0023] According to an exemplary embodiment of the present invention, the step of controlling the interface may include at least one of the following: controlling the size of the medical image, controlling a part of the medical image, controlling the size of the part of the medical image, controlling scrolling through multiple medical images, controlling the medical image data to be displayed, controlling highlighting of the medical image and / or the part of the medical image, controlling patient data that can be displayed at the display with reference to the medical image. Control can be performed on at least one of the above features and / or on multiple of these features. In particular, the step of control may include a single step of the above features, or may include some and / or multiple and / or all of them. Thus, the detected eye-tracking data can be associated with the corresponding control features. For example, blinking once can confirm the display of an image or ROI. Another amount of blinking can be used to control the size of the ROI to be displayed or the size of an enhanced view. Movements of the eyes or head can be used to control scrolling through multiple images. The examples are not intended to be restrictive, and other eye-tracking data can be used for other control features.

[0024] According to an exemplary embodiment of the present invention, eye tracking can be used to monitor eye tracking data, which is at least one of the following: the eye movement of the user, the viewing direction of the user's eyes, the average fixation time of the eyes, the pupil area, the time period during which the user has looked at a point on the display, eyelid movement, eye color, and the head movement of the user. For example, eye movement can be any movement of the eyes, which can be the up and down movement of the eyes and / or the left and right movement of the eyes. The monitored eye tracking data can be only one of the above eye tracking data, or can be multiple of these features or all features. It can be possible to track one or more or all of the eye tracking data simultaneously. Eye tracking can be used to monitor the movement and behavior of each single eye of the user. The viewing direction of the user's eyes can be the gaze determined from both eyes. The average fixation time of the eyes can be the average value of any time when the user gazes at a specific point in the medical image, where the duration of the average fixation time can be predefined by the corresponding reading standard. The pupil area and the change in this area can be measured for each single eye. In addition, eyelid movement can be the opening or closing of the eyelids, and / or one or more blinks. The tracked eye tracking data can be used by a second AI module to determine the fatigue of the user. For example, the average fixation time and the pupil area can be used as features for fatigue detection using the fuzzy K-means clustering method. These features can be extracted from pupil segmentation using computer vision methods (such as thresholding, the corrected elliptical Hough transform method) or by deep learning methods (such as using the DeepLab architecture).

[0025] According to an exemplary embodiment of the present invention, data from medical training images can be used to train a first AI module, the medical training images including information similar to the medical images to be displayed, wherein the medical training images used to train the first AI module can be annotated by medical experts, and the medical experts annotate the anatomical structures in each of the medical training images. The trained data set can be an annotated data set, where the annotation is a per-pixel decision on whether an object exists at that location. On the other hand, the first AI module can be a pre-trained model that can be trained on non-medical data.

[0026] According to an exemplary embodiment of the present invention, a data set of iris images and pupil images obtained by an eye tracker can be used to train a second AI module, where the pupil is annotated on a per-pixel basis.

[0027] According to an exemplary embodiment of the present invention, the eye tracker, the first AI module, and the second AI module work simultaneously during the display of medical images to the user on the display. Therefore, the processor can be configured to execute all steps simultaneously, so that the image reading process can be carried out faster.

[0028] According to an exemplary embodiment of the present invention, the second AI module may evaluate at least one of eye movement, eyelid movement, eye color, or pupil dilation to evaluate the ability of the user to perform the image reading. The AI module may be capable of evaluating one, more than one, or all of the above features. In addition, the evaluation of more than one feature may be performed simultaneously or one after another. In particular, the second AI module may evaluate the fatigue state as the state of the user to determine the ability of the user to perform a sufficient reading of a medical image. In particular, fatigue may be evaluated by eye tracking. For example, eye movement and pupil dilation may be suitable markers for fatigue prediction. This has the advantage of notifying the user that his or her reading may be insufficient. For example, the monitoring of eye color may monitor irritated eyes or red eyes, which may be used to indicate that the user cannot properly look at the displayed image when such irritated eyes or red eyes are detected.

[0029] According to another exemplary embodiment of the present invention, the medical image to be displayed and whose display should be controlled on the interface may be a chest X-ray image, an MRI image, a mammography image, an ultrasound image, a CT image, or a PET image.

[0030] According to a second aspect of the present invention, a system for controlling an interface for displaying a medical image to a user is described. The system includes a display that includes an interface for interfacing with the user. The system further includes an eye tracker configured to provide eye tracking data tracked from the user. In addition, the system includes a processing unit configured to communicate with the display, wherein the processing unit is further configured to: receive a medical image of a patient and display the medical image at the display for the user to perform image reading; use the eye tracker to identify an area of interest of the user in the medical image; use a first artificial intelligence (AI) module to analyze the area of interest; determine an anatomical structure based on the medical image by correlating the identified area of interest with data from the first AI module; use a second artificial intelligence (AI) module to evaluate the state of the user, which indicates the ability of the user to perform image reading; and control the interface and the displayed medical image based on the eye tracking data.

[0031] In other words, the system includes a user interface that can be controlled by the user using an eye tracker. Additionally, the system can further include at least two AI modules, where at least one module analyzes a specific part of a medical image that the user is currently looking at in order to enhance the presentation of the structure currently under study and to verify the user's compliance with the systematic review guidelines. At least one of the modules can evaluate the user's fatigue and thus his or her ability to perform image reading. The system uses the eye tracker to control, for example, a conventional viewer software, which can be implemented by at least two AI modules.

[0032] According to an exemplary embodiment of the invention, the system can further include a camera that is configured to communicate with the processor and is configured to capture at least one of the following: the eye movement of the user, the viewing direction of the user's eyes, the average fixation time of the eyes, the pupil area, the period of time the user has looked at a point on the display, eyelid movement, or the movement of the user's head. In particular, the camera can be a corresponding eye tracker or can be used as part of an eye tracker. The camera / eye tracker can be placed near the display. The eye tracker allows knowing at any time the position on the medical image that the user is currently focusing on, which can be achieved by using at least one of the above-mentioned eye tracking features that the camera can be configured to monitor. The camera can be configured to capture at least one of the above features, or can capture all of them, or only a plurality of these features. In particular, the features captured by the camera can be captured simultaneously.

[0033] According to a third aspect of the invention, a computer program unit for controlling an interface for displaying a medical image to a user is described. The computer program unit, when run by a processor of the system, is adapted to cause the system to: receive a medical image of a patient and display the medical image at a display for the user to perform image reading, use an eye tracker configured to provide eye tracking data tracked from the user to identify an area of interest of the user in the medical image; use a first artificial intelligence (AI) module to analyze the area of interest; determine an anatomical structure based on the medical image by correlating the identified area of interest with data from the first AI module; use a second artificial intelligence (AI) module to evaluate the state of the user, which indicates the user's ability to perform image reading; control the interface and the displayed medical image based on the eye tracking data.

[0034] The computer program unit can be part of a computer program, but it can also be the entire program itself. For example, the computer program unit can be used to update an existing computer program to implement the invention.

[0035] The program unit can be stored on a computer-readable medium. The computer-readable medium can be regarded as a storage medium, such as a USB stick, a CD, a DVD, a data storage device, a hard disk, or any other medium on which the program unit as described above can be stored.

[0036] According to various embodiments of the present disclosure, the methods described herein can be implemented using a hardware computer system running a software program. Additionally, in exemplary non-limiting embodiments, the implementation can include distributed processing, component / object distributed processing, and parallel processing. Virtual computer system processing can implement one or more of the methods or functions described herein, and the processors described herein can be used to support a virtual processing environment.

[0037] It must be noted that embodiments of the present invention have been described with reference to different subject matters. In particular, some embodiments have been described with reference to apparatus / system type claims, while other embodiments have been described with reference to method type claims. However, those skilled in the art will appreciate from the above and following descriptions that, unless otherwise indicated, any combination of features related to different subject matters, especially any combination between the features of apparatus type claims and the features of method type claims, in addition to any combination of features belonging to one type of subject matter, is also considered to be disclosed by this application. Brief Description of the Drawings

[0038] The aspects defined above and other aspects of the present invention are apparent from the examples of the embodiments to be described hereinafter and are explained with reference to the examples of the embodiments. The present invention will be described in more detail hereinafter with reference to the examples of the embodiments, but the present invention is not limited thereto.

[0039] Figure 1 A flowchart of a method according to an embodiment of the present invention is illustrated.

[0040] Figure 2 A medical image displayed on an interface according to an embodiment of the present invention is illustrated.

[0041] Figure 3 Another medical image displayed on an interface according to an embodiment of the present invention is illustrated.

[0042] Figure 4 Another medical image displayed on an interface according to an embodiment of the present invention is illustrated.

[0043] List of Reference Numerals:

[0044] 100 Interface

[0045] 201 Medical Image

[0046] 202 Enhanced View

[0047] 203 and 303 boundaries

[0048] 204 Display

[0049] 205 Eye tracker

[0050] 206 and 306 Anatomical structures

[0051] 310 Historical images

[0052] 311 - 314 Historical images

[0053] 315 Image data

[0054] 420 Image slices

[0055] 421 Image slices

[0056] Steps S101 - S107 Detailed implementation manners

[0057] The illustrations in the accompanying drawings are schematic. It should be noted that in different accompanying drawings, similar or identical elements are provided with the same reference numerals.

[0058] Figure 1The figure illustrates a flowchart including method steps according to an embodiment of the present invention. The method for controlling an interface for displaying medical images to a user is described using the following steps. In step 101, a medical image of a patient is received, and in step 102, the medical image is displayed to the user at a display such that the user can perform image reading. In step 103, an eye tracker is used to identify a region of interest in the medical image, the eye tracker being configured to provide eye tracking data tracked from the user. When the region of interest is identified, then in step 104, a first artificial intelligence (AI) module is used to analyze the region of interest. In 105, an anatomical structure from the medical image is determined by correlating the identified region of interest with data from the first AI module. In step 106, a second artificial intelligence (AI) module is used to evaluate the state of the user, which indicates the user's ability to perform image reading. In step 107, the interface and the displayed medical image are controlled based on the eye tracking data. The steps of the method may be performed one after another, where the order presented herein is not restrictive as the order of the steps may be different. Additionally, some or at least a plurality of the method steps may be performed simultaneously. For example, the evaluation of the state of the user may be performed simultaneously during the entire image reading of the user. Thus, the evaluation of the state of the user may be performed simultaneously with the identification of the ROI, the analysis of the ROI, and the determination of the anatomical structure. Additionally, the method may further include additional sub-steps, such as after the evaluation, the state of the user may be indicated and an indication may be displayed to the user on the display in another method step. When the image reading is marked as incomplete because the image reading is not finished and the user wants to continue the image reading, it may also be possible to restart the method after controlling the interface. Thus, the steps may start again after step 107, for example, by displaying the image in step 102 when continuing the image reading, and again the same image may be displayed in step 102. On the other hand, it may be possible to receive a new image from the same patient, and a further image reading process should be performed for the image. The step of displaying the medical image may include the following additional sub-steps: displaying the region of interest in the medical image using the first AI module, and generating an enhanced view of the indicated region of interest by the first AI module, wherein the display of the ROI, the display of the enhanced view, and the display of the medical image itself are controlled based on the eye tracking data.

[0059] In particular, the step of controlling the interface may further include the following additional sub-steps: controlling the size of the medical image, controlling a portion of the medical image, controlling the size of the portion of the medical image, controlling the scrolling through multiple medical images, controlling the medical image data to be displayed, controlling the highlighting of the medical image and / or a portion of the medical image, and controlling the patient data that can be displayed at the display with reference to the medical image. Each of the sub-steps may be controlled by the user based on the eye tracking data.

[0060] Figure 2 FIG. 4 shows a system for controlling an interface 100 for displaying a medical image 201 to a user according to an embodiment of the present invention. The system includes a display 204, and the display 204 includes the interface 100 for connection with the user interface. The system further includes an eye tracker 205 configured to provide eye tracking data tracked from the user. In addition, the system includes a processing unit configured to communicate with the display 204. The processing unit is configured to receive the medical image 201 of the patient and display the medical image 201 at the display 204 for the user to perform image reading. In addition, the processing unit is configured to use the eye tracker 205 to identify the region of interest of the user in the medical image 201. The region of interest is indicated by a rectangle on the left side of the display 204. Figure 2 The region of interest in FIG. 6 is analyzed using a first artificial intelligence (AI) module. The anatomical structure 206 is determined from the medical image 201 by correlating the identified region of interest with the data from the first AI module. The state of the user, which indicates the user's ability to perform image reading, is evaluated by the processor using a second artificial intelligence (AI) module. The interface 100 and the displayed medical image 201 are controlled according to the tracked eye tracking data. The current medical image 201 and the region of interest (rectangle) are simultaneously displayed on the display 204. In Figure 2In [the figure], the medical image 201 and the enhanced view 202 of the region of interest are displayed side by side with each other on the display 204. On the left side, the current medical image 201 with the indicated ROI is shown, and on the left side of the display 204, the enhanced view 202 is shown. If the user gazes at a specific structure of the medical image 201, the interface highlights the boundary 203 as the ROI using a rectangular line. The boundary 203 is known from a first AI module that uses object detection and / or semantic segmentation to determine the anatomical structure 206 that the user is gazing at. For example, the enhanced view 202 is an enlarged version of the ROI with adjusted contrast for optimizing and simplifying the user's image reading. The control of the detection of the ROI by the AI module can be confirmed by the user through a blink. For example, the identified ROI is marked with a rectangle 203 in the medical image 201, and for a simple blink code of "yes / correct", the user confirms the identification by a single blink. If the ROI is not the corrected ROI, the user can use another blink code, such as blinking twice for "no", or the eye tracker can analyze head movement so that, for example, the user's head shaking from left to right, or vice versa, can be understood as "no". If the user is satisfied with his study and / or has finished reading the enhanced view 202, the user can return his or her gaze to the overview 201, that is, the entire medical image 201 on the left side of the display 204, and remove the enhanced view 202 and / or the highlighted ROI 203 from the display 204. In Figure 2 In [the figure], the displayed medical image 201 is an image of a patient's chest, and the identified ROI indicated by the boundary 203 is the airway. The enhanced view 202 with improved image quality (the quality of the enhanced view 202 can be improved using the features described in the embodiments in the above section) is the airway, particularly the bronchioles.

[0061] The eye tracker 205 is placed closely adjacent to the user's screen, which is here at the top of the display 204. The eye tracker is a camera 205 that is configured to communicate with a processor and is configured to capture at least one or more of the following: the eye movement of the user, the viewing direction of the user's eyes, the average fixation time of the eyes, the pupil area, the time period during which the user has looked at a point on the display, and the head movement of the user. Thus, the eye tracker 205 always knows the position on the medical image 201 that the user is currently looking at.

[0062] Figure 3 The display of another medical image 3 according to an embodiment of the present invention is shown. In Figure 3In it, historical medical images are displayed, where multiple historical images 311 - 314 are displayed in a row at the bottom of the display 204. On the left side of the display, the ROI of the historical medical image 310 is displayed, and on the right side of the display 204, the current ROI of the current medical image is displayed. Thus, the display step includes: simultaneously displaying the current medical image 201 of the patient and / or displaying the historical medical images of the patient on the display 204, where the first AI module analyzes and indicates the region of interest 303 in the historical medical image that is similar to the region of interest 203 of the current medical image 201. The indication of the region of interest (such as boundaries 303, 203) may include color coding so as to avoid confusion regarding the nature of the image: current or historical. For example, the boundary of the region of interest 303 of the historical image is colored with a different color from the boundary 203 of the current medical image 201.

[0063] The interpretation of the eye - tracking data allows for several possibilities. For example, blinking three times can be defined as a signal for opening the screen of the patient's history. The interpretation of the eye - tracking data can also be specified by the user, which means the user can change the settings according to his or her preferences. In Figure 3 it, blinking three times brings up the patient's historical images 311 - 314, which are displayed on the lower row, and one specific historical image is highlighted on the left. The user confirms the selected historical medical image by blinking or nodding. For example, the scrolling through the historical images 311 - 314 in the row can be controlled by moving the head to the left for the picture on the left side of the row or moving the head to the right for the picture on the right side of the row. The eye - tracker 205 is tracking these head movements, the eye - tracking data is analyzed by the processor, and the interface is controlled for what is to be displayed.

[0064] In addition, corresponding image data 315 can be displayed on the medical image and / or on the ROI 303. For example, this information can be displayed in the upper - left corner of the ROI 303. The information displayed can include the date when the image was taken (for historical medical images), or any other relevant patient data, such as age, disease, etc.

[0065] Figure 4 Another medical image displayed on the interface according to an embodiment of the present invention is shown. The present invention can also be applied to Figure 4 the 3D medical image illustrated. During the generation of CT, MRI, or PET images, a body part of the patient is subdivided into a set of slices. A single image can be such a slice, and in order to analyze the entire body part, each image slice must be read by the user. As Figure 4As illustrated, each slice can be displayed as a single medical image 201 on the interface 100. The user can control the interface 100 by using head movement to scroll through each individual slice, for example, by moving the head to the left to select slice 420 before the current image 201 and / or by moving the head to the right to select slice 421 after the current image 201. It may also be possible for the interface 100 to be controlled by eye movement similar to left or right to scroll through different medical image slices.

[0066] Although the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive; the invention is not limited to the disclosed embodiments. Other variations of the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. It should be noted that the term "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. Also, elements associated with different embodiments can be combined. It should also be noted that the reference signs in the claims should not be construed as limiting the scope of the claims.

Claims

1. A method for controlling an interface that displays medical images to a user on a display, the method comprising the steps of: Receiving medical images of a patient, Displaying the medical images at the display for the user to perform image reading, Using an eye tracker to identify a region of interest in the medical images by the user, the eye tracker being configured to provide eye tracking data tracked from the user, Using a first artificial intelligence (AI) module to analyze the region of interest, Determining an anatomical structure based on the medical images by correlating the identified region of interest with data from the first AI module, Using a second artificial intelligence (AI) module to evaluate the state of the user, the state indicating the user's ability to perform the image reading, Controlling the interface and the displayed medical images based on the eye tracking data.

2. The method according to claim 1, further comprising the steps of: Simultaneously displaying the current medical image and the region of interest on the display, wherein the medical image and the region of interest are displayed side by side with each other on the display.

3. The method according to claim 1 or 2, Among them, The displaying step further comprises: simultaneously displaying the current medical image of the patient and / or displaying historical medical images of the patient on the display, wherein the first AI module analyzes and indicates regions of interest in the historical medical images similar to the region of interest of the current medical image.

4. The method according to any one of claims 1 to 3, further comprising the steps of: Performing the following operations according to the received eye tracking data: Displaying the region of interest in the medical image displayed on the display by the first AI module, Generating an enhanced view of the indicated region of interest by the first AI module, Removing the enhanced view of the indicated region of interest by the first AI module.

5. The method according to claim 4, Among them, When the eye tracker determines that the user has gazed at the region of interest for a period of time, using the first AI module to indicate the region of interest with a boundary, wherein the boundary is rectangular, wherein generating the enhanced view includes: improving the quality of the enhanced view of the medical image, wherein when the eye tracker determines that the user removes his view from the region of interest, removing the enhanced view of the region of interest.

6. The method according to any one of claims 1 to 5, Among them, The step of using the first artificial intelligence (AI) module to analyze the region of interest includes at least one of the following: object detection AI, segmentation AI, or instance segmentation AI, wherein the object detection AI uses a deep convolutional neural network configured for object detection, wherein the segmentation AI uses a deep convolutional neural network configured for segmentation.

7. The method according to any one of claims 1 to 6, further comprising the steps of: Use the first AI module to correlate the image reading of the medical image by the user with the reading guide, indicate at the medical image whether the image reading is performed in accordance with the reading guide.

8. The method according to any one of claims 1 to 7, Among them, The step of controlling the interface includes at least one of the following: controlling the size of the medical image, controlling a portion of the medical image, controlling the size of the portion of the medical image, controlling scrolling through multiple medical images, controlling the medical image data to be displayed, controlling highlighting of the medical image and / or the portion of the medical image, and controlling patient data that can be displayed at the display with reference to the medical image.

9. The method according to any one of claims 1 to 8, Among them, Eye tracking is used to monitor eye tracking data, which is at least one of the following: eye movement of the user, viewing direction of the user's eyes, average fixation time of the eyes, pupil area, period of time during which the user has looked at a point on the display, eyelid movement, eye color, or head movement of the user.

10. The method according to any one of claims 1 to 9, Among them, The first AI module is trained using data from medical training images, which include information similar to the medical image to be displayed, wherein the medical training images used to train the first AI module are annotated by medical experts who annotate anatomical structures in each of the medical training images, and / or wherein the first AI module is a pre-trained AI module trained on non-medical data.

11. The method according to any one of claims 1 to 10, Among them, When the medical image is displayed to the user on the display, the eye tracker, the first AI module, and the second AI module work simultaneously.

12. The method according to any one of claims 1 to 11, Among them, The second AI module evaluates at least one of eye movement, eyelid movement, eye color, or pupil dilation for evaluating the user's ability to perform the image reading.

13. A system for controlling an interface for displaying a medical image to a user, the system comprising: a display including an interface for interfacing with the user, an eye tracker configured to provide eye tracking data tracked from the user, a processing unit configured to communicate with the display, wherein the processing unit is further configured to: receive a medical image of a patient, and display the medical image at the display for the user to perform image reading, use the eye tracker to identify an area of interest of the user in the medical image, use a first artificial intelligence (AI) module to analyze the area of interest, determine an anatomical structure based on the medical image by correlating the identified area of interest with data from the first AI module Use a second artificial intelligence (AI) module to evaluate the state of the user, the state indicating the user's ability to perform the image reading. Control the interface and the displayed medical image based on the eye tracking data.

14. The system according to claim 12, further comprising: A camera configured to communicate with the processor and configured to capture at least one of the following: eye movement of the user, eyelid movement, viewing direction of the user's eyes, average fixation time of the eyes, pupil area, period of time the user has looked at a point on the display, and head movement of the user.

15. A computer program unit for controlling an interface for displaying a medical image to a user. Among them, The computer program unit, when run by a processor of the system, is adapted to cause the system to: Receive a medical image of a patient, and Display the medical image at the display for the user to perform image reading. Use an eye tracker to identify an area of interest of the user in the medical image, the eye tracker being configured to provide eye tracking data tracked from the user. Use a first artificial intelligence (AI) module to analyze the area of interest. Determine an anatomical structure based on the medical image by correlating the identified area of interest with data from the first AI module. Use a second artificial intelligence (AI) module to evaluate the state of the user, the state indicating the user's ability to perform the image reading. Control the interface and the displayed medical image based on the eye tracking data.

Images