Apparatus for an optical imaging system, optical imaging system, method and computer program

Abstract

Examples relate to an apparatus for an optical imaging system, the apparatus comprising one or more processors and one or more storage devices. The apparatus is configured to obtain sensor data of an optical imaging sensor of the optical imaging system. The sensor data is indicative of an image of a sample. Further, the apparatus is configured to obtain gaze direction data of a sensor. The gaze direction data is indicative of a gaze direction of a user of the optical imaging system. The apparatus is further configured to generate region data indicative of a region of interest on which the user gazes on the image of the sample. The region data is generated based on the gaze direction data. Further, the apparatus is configured to generate output data indicative of an adjusted image of the sample to be displayed on a display device. The output data is generated based on the sensor data and the region data. The adjusted image of the sample comprises an emphasized region indicative of the region of interest such that the emphasized region is emphasized relative to the region of interest in the image of the sample. The apparatus is further configured to transmit the output data for display on the display device.

Description

Technical Field

[0001] Examples relate to an apparatus for an optical imaging system (such as a surgical optical imaging system), an optical imaging system, a method, and a computer program. Background Technology

[0002] Communication between users of optical imaging systems (e.g., surgeons during surgery) can be crucial for saving time and / or for teaching purposes. Effective communication often requires pointing to a specific point within the surgical cavity. For example, teaching relies heavily on pointing during explanation. This is difficult for surgeons to do, and without proper pointing, students / residents may find it hard or impossible to follow. However, pointing to a specific point within the surgical cavity is time-consuming because the surgeon must use the tip of the instrument in their hand, meaning they need to stop and interrupt their workflow. Furthermore, due to workflow disruptions, surgeons may sometimes avoid communicating to express ideas or solicit opinions. Additionally, pointing by touching tissue with instruments within the surgical cavity is not entirely without risk. Therefore, there may be a desire for improved concepts of communication between different users. Summary of the Invention

[0003] This expectation is addressed through the subject matter of the independent claims.

[0004] The concept presented in this disclosure is based on the insight that an adjusted image of a sample to be displayed on a display device can be generated based on gaze direction data indicating the gaze direction of a user in an optical imaging system. Based on the gaze direction data, regions of interest (ROIs) that will be emphasized in the adjusted image of the sample can be determined. Therefore, an adjusted image of sample 110 can be generated including emphasized regions indicating ROIs, said emphasized regions being highlighted relative to the ROIs in the image of the sample. In this way, the ROI that the user is viewing can be highlighted in the adjusted image of the sample. Thus, for example, the recognition of the ROI by another person can be improved.

[0005] An example provides an apparatus for an optical imaging system, the apparatus including one or more processors and one or more storage devices. The apparatus is configured to acquire sensor data from an optical imaging sensor of the optical imaging system. The sensor data indicates an image of a sample. Furthermore, the apparatus is configured to acquire gaze direction data from the sensor. The gaze direction data indicates the gaze direction of a user of the optical imaging system. The apparatus is also configured to generate region data indicating a region of interest that the user is gazing at in the image of the sample. The region data is generated based on the gaze direction data. Additionally, the apparatus is configured to generate output data indicating an adjusted image of the sample to be displayed on a display device. The output data is generated based on the sensor data and the region data. The adjusted image of the sample includes an emphasized region indicating the region of interest, such that the emphasized region is emphasized relative to the region of interest in the image of the sample. The apparatus is also configured to transmit the output data for display on the display device. Acquiring the gaze direction data allows for the generation of the region of interest of interest that the user is interested in. For example, during surgery, a user can view a portion of an anatomical feature, such as a vein, artery, or bone of interest, for example, to proceed with the next procedure. The region of interest can be used to adjust the image of the sample to generate an adjusted image of the sample. This adjusted image allows another user to easily identify the region of interest that the user is gazing at. In this way, communication between different users can be facilitated. Therefore, using the sensor data and the gaze direction data allows for the display of an image of the sample that indicates a view of the sample and a visualization of the user's gaze on the microscope.

[0006] In the example, the device can be configured to generate the output data including the emphasized region by adjusting the brightness, sharpness, color settings, size, and / or orientation of the emphasized region or its surrounding area. Therefore, the region of interest, or its surrounding area, can be adjusted. In this way, the gaze direction of another user can be triggered, causing the other user to instinctively gaze at the user's region of interest.

[0007] In the example, the gaze direction can indicate the time the user has that gaze direction. The device can be configured to determine the region of interest based on the time the user has that gaze direction. Using the time the user has that gaze direction can allow for improved reliability in determining the region of interest. For example, when the user searches for anatomical features in an image of the sample with their eyes, adjustments to the image of the sample should not be triggered. In this way, false positives that indicate regions of interest that the user is not actually interested in can be reduced or even avoided.

[0008] In the example, the device can be configured to determine the region of interest (ROI) by comparing the time the user has been gazing in the specified direction with a threshold time. When the time the user has been gazing in the specified direction exceeds the threshold time, the device can be configured to identify the region the user is gazing at as a ROI. In this way, the reliability of determining the ROI can be improved. Furthermore, the threshold time can allow the user to control the adjustment of the image of the sample based on the time the user gazes at the ROI. This can provide the user with improved control over the selection of the ROI to be highlighted.

[0009] In the example, the device can be configured to determine a second region of interest based on the time the user has the gaze direction, and to generate the output data based on the second region of interest. In this way, multiple regions of interest can be visualized in the adjusted image of the sample. For example, multiple anatomical features of the sample at different regions can be highlighted by the multiple regions of interest. This can improve the usability of the adjusted image of the sample, especially when the user is interested in multiple regions simultaneously.

[0010] In the example, the device can be configured to determine when the user is no longer interested in the region of interest based on the time the user has the gaze direction, and adjust an adjusted image of the sample, including the emphasized region indicating the region of interest, such that the emphasized region is no longer emphasized relative to the region of interest in the image of the sample. In this way, regions that the user was interested in and therefore emphasized can be adjusted to no longer be emphasized based on the time the user has the gaze direction. Therefore, outdated regions of interest that might distract another user can be avoided.

[0011] In the example, the device can be configured to adjust the sharpness of the region of interest by generating focus data that indicates the adjusted focus of the microscope of the optical imaging system. Furthermore, the device can be configured to transmit focus data for adjusting the focus of the microscope so that the region of interest is focused in the adjusted image of the sample. In this way, the perception of the region of interest can be increased by increasing the sharpness of the region of interest. Therefore, the user can instinctively gaze at the region of interest.

[0012] In the example, the device can be configured to adjust the orientation of the emphasized region by shifting its position so that the emphasized region is located in the center of the adjusted image of the sample. In this way, the perception of the region of interest can be increased by shifting the region of interest to the center (e.g., the center of the display device on which the adjusted image of the sample is displayed). Therefore, the user can instinctively gaze at the region of interest.

[0013] In the example, the device can be configured to determine an accuracy score indicating the accuracy of the generated gaze direction data, and to generate an adjusted image of the sample based on the accuracy score. For example, the emphasized region can be more strongly emphasized to achieve increased accuracy. For instance, the emphasis on the emphasized region can be strengthened when the user gazes at the region of interest for a specific period of time. In this way, the reliability of the emphasized region can be improved.

[0014] In the example, the device can be configured to obtain display data instructing a display device for displaying an adjusted image of the sample, and to generate the adjusted image of the sample based on the display data. In this way, the region of interest can be determined based on sensor data, gaze direction data, and display data, allowing the region of interest to be inferred even when the user's actual viewpoint is uncertain.

[0015] In this example, the device can be configured to obtain user data indicating a user to display an adjusted image of the sample to it, and to generate an adjusted image of the sample based on the user data. In this way, a user-specific adjusted image of sample 110 can be generated and displayed to a specific user. Therefore, information overload on the user can be reduced.

[0016] In the example, the device can be configured to acquire second gaze direction data from a second sensor. The second gaze direction data indicates the gaze direction of a second user of the optical imaging system. Furthermore, the device can be configured to generate second region data based on the second gaze direction data, the second region data indicating a second region of interest gazed upon by the second user. The device can also be configured to generate output data based on the second region data, representing an adjusted image of the sample including a second emphasis region indicating the second region of interest, such that the second emphasis region is emphasized relative to the second region of interest in the image of the sample. In this way, multiple emphasis regions for multiple users can be displayed by showing the adjusted image of the sample. Therefore, communication between multiple users can be facilitated.

[0017] In the example, the device can be configured to generate output data including an indicator element that indicates the sample. The indicator element indicates which user of the optical imaging system the emphasized region belongs to. In this way, it can facilitate the assignment of the emphasized region to a specific user of the optical imaging system.

[0018] In the example, the device can be configured to generate the output data by generating marker elements that indicate the location of the region of interest, such that the emphasized region in the adjusted image of the sample is emphasized relative to the region of interest in the image of the sample by the location of the marker elements. In this way, markers (such as geometric objects at least partially surrounding the region of interest) can be used to indicate the region of interest. Therefore, highlighting of the emphasized region can be facilitated. For example, the image of the sample can be adjusted by simply adding the markers.

[0019] The example provides an optical imaging system that includes the device as described above.

[0020] An example provides a method for an optical imaging system, the method comprising obtaining sensor data of an optical imaging sensor of the optical imaging system, the sensor data indicating an image of a sample. Furthermore, the method includes: obtaining gaze direction data of the sensor, the gaze direction data indicating the gaze direction of a user of the optical imaging system; and generating region data based on the gaze direction data, the region data indicating a region of interest gazed upon by the user on an image of the sample. The method further includes: generating output data based on the sensor data and the region data, the output data indicating an adjusted image of the sample to be displayed on a display device, the adjusted image of the sample including an emphasized region indicating the region of interest, such that the emphasized region is emphasized relative to the region of interest in the image of the sample; and transmitting the output data for display on the display device. The method can be performed by the device described above.

[0021] The various examples disclosed herein relate to corresponding computer programs having program code for performing the methods described above when the computer program is executed on a processor. Attached Figure Description

[0022] The following description will use only examples of devices and / or methods, with reference to the accompanying drawings, in which...

[0023] Figure 1a and 1b A schematic diagram of an example of a device for an optical imaging system and a corresponding optical imaging system including the device is shown;

[0024] Figure 2 An example of an adjusted image of the sample is shown;

[0025] Figure 3 A flowchart illustrating an example method for an optical imaging system is shown; and

[0026] Figure 4 A schematic diagram of a system including a microscope and a computer system is shown. Detailed Implementation

[0027] Various examples will now be described more fully with reference to the accompanying drawings, some of which illustrate certain aspects. For clarity, the thickness of lines, layers, and / or regions may be exaggerated in the drawings.

[0028] Figure 1a and 1bA schematic diagram is shown of a device 130 for an optical imaging system 100 and an example of a corresponding optical imaging system 100 including device 130. The task of device 130 is to control the microscope 120 and various aspects of the optical imaging system 100 (which may be a surgical optical imaging system), and / or to process various types of sensor data from the optical imaging system 100. Therefore, device 130 can be implemented as a computer system that interfaces with various components of the optical imaging system 100 (e.g., sensor 122). Device 130 can be part of the optical imaging system 100. Alternatively, device 130 can be communicatively coupled to the optical imaging system 100. For example, device 130 can be a ready-to-use module that can be connected to the optical imaging system 100.

[0029] like Figure 1a As shown, device 130 includes one or more processors 134 and one or more storage devices 136. Optionally, device 130 also includes one or more interfaces 132. The one or more processors 134 are coupled to one or more storage devices 136 and optionally one or more interfaces 132. Typically, the functionality of device 130 may be provided by one or more processors 134 (e.g., for generating area data or for generating output data) in combination with one or more interfaces 132 (for exchanging information, such as exchanging information with sensor 122 or display device to transmit output data) and / or one or more storage devices 136 (for storing and / or retrieving information).

[0030] Device 130 is configured to acquire sensor data from the optical imaging sensor 122 of the optical imaging system 100. The sensor data indicates an image of sample 110. For example, the sensor data can be acquired by receiving data from a sensor (e.g., the optical imaging sensor 122 of microscope 120) or a frame buffer (e.g., part of the optical imaging system 100). Optionally or alternatively, the sensor data can be acquired by measurement by device 130. For example, the optical imaging sensor 122 may be part of device 130. Therefore, device 130 can control the optical imaging sensor 122 to measure the sensor data. The sensor data can be raw data from the sensor, meaning device 130 can perform post-processing on the sensor data. Alternatively, the sensor data can be post-processed data from the sensor, meaning further post-processing by device 130 may not be required.

[0031] Furthermore, the device is configured to acquire gaze direction data from sensor 124. The gaze direction data indicates the gaze direction of the user of the optical imaging system 100. For example, the gaze direction data can be obtained from a sensor (e.g., a display device (e.g., a sensor)). Figure 1bThe gaze direction data is obtained by receiving an eye-tracking sensor 124 and / or a camera (e.g., part of an optical imaging system 100) in the display device 180 shown. Optionally or alternatively, gaze direction data can be obtained by measurement by device 130. For example, eye-tracking sensor 124 and device 130 can be used in the display device (e.g., Figure 1b The device 130 is part of the display device 180 shown. Therefore, the device 130 can control the eye-tracking sensor 124 to measure gaze direction data. The gaze direction data can be raw data from the sensor, meaning the device 130 can post-process the gaze direction data. Alternatively, the gaze direction data can be post-processed data from the sensor, meaning further post-processing by the device 130 may not be required. The sensor 124 (e.g., an eye-tracking sensor) is part of the display device 180 (such as... Figure 1b The arrangement on the monitor 180 shown may be advantageous for head-raising surgery. Alternatively, Figure 1b The display device 180 shown can be a stereoscopic display device, also known as a 3D monitor. Therefore, a user can wear polarized glasses to view the stereoscopic display device 180. In this case, the sensor 124 can be part of the polarized glasses or can be arranged on the polarized glasses.

[0032] Obtaining gaze direction data allows for the identification of regions of interest (ROIs). Therefore, device 130 is also configured to generate region data indicating the ROIs that a user is gazing at on an image of a sample. This region data is generated based on the gaze direction data. For example, a ROI can be a region of interest to the user. A ROI can refer to a specific defined area or portion within an image (e.g., an image of the sample) or dataset (e.g., sensor data) selected for more detailed examination or analysis and / or the next operational procedure. For example, a ROI can be used to focus a particular region and / or feature of particular interest within an image. ROIs can help focus the efforts and / or attention of the users(s) of optical imaging system 100 on relevant areas, improve accuracy by eliminating unnecessary information, and / or facilitate communication, for example, for diagnostic purposes. Therefore, ROIs can facilitate communication among users of optical imaging system 100.

[0033] Therefore, device 130 is configured to generate output data indicating an adjusted image of sample 110 to be displayed on display device 180. The output data is generated based on the sensor data and the region data. The adjusted image of sample 110 includes an emphasis region indicating a region of interest, such that the emphasis region is highlighted relative to the region of interest in the image of sample 110. That is, device 130 can adjust the image of sample 110 so that the region of interest is highlighted in the adjusted image of sample 110. For example, the emphasis region can improve the perceptibility of the region of interest. In this way, a user of optical imaging system 100 can easily perceive the region of interest of another user of optical imaging system 100. Therefore, the adjusted image of sample 110 can be used to facilitate communication between multiple users of the optical imaging system.

[0034] An adjusted image of sample 110 can be generated by adjusting the image of sample 110 based on region data. For example, the region of interest can be transformed into an emphasis region by brightening and / or sharpening the region of interest relative to other parts of the image of sample 110. That is, the adjusted image of sample 110 can be an image of sample 110 that has actually been adjusted. Therefore, device 130 can post-process the image of sample 110 to generate output data indicating the adjusted image of sample 110. Optionally or alternatively, the adjusted image of sample 110 can be generated by combining information from region data and sensor data. For example, the adjusted image of sample 110 can be generated by generating marker elements as described below. The marker elements can be superimposed on the image of sample 110. The marker elements can emphasize the region of interest, for example, by emphasizing the region of interest around it. That is, the adjusted image of sample 110 can include an image of sample 110 superimposed with additional information (e.g., marker elements). Therefore, the image of sample 110 can generate output data indicating the adjusted image of sample 110 without post-processing by the device. Alternatively, the image of sample 110 can be combined with additional information to generate an adjusted image of sample 110, such as a composite image including the image of sample 110 and the marker elements.

[0035] Furthermore, device 130 is configured to transmit output data for display on display device 180. For example, output data may be transmitted to display device 180. Optionally or alternatively, output data may be transmitted to storage device, such as a frame buffer. Thus, display device 180 can retrieve output data from the frame buffer. Therefore, device 130 can trigger and / or control the display of an adjusted image of sample 110.

[0036] The inventors have discovered that using sensors (e.g., eye-tracking sensors or cameras) to detect the user's gaze direction in an optical imaging system 100 allows for visualization of the user's gaze direction. That is, the user's gaze direction can be used to determine the region of interest the user is gazing at. The determined region of interest can be displayed using an adjusted image of sample 110. For example, the user's gaze direction or region of interest can be emphasized in the image of sample 110, which can allow for easier, more efficient, faster, and / or less risky communication between multiple users (e.g., a patient) regarding sample 110.

[0037] Device 130 can enable easy, rapid, precise, and / or intuitive communication between different users of an optical imaging system. For example, when different users are performing surgery, device 130 can improve communication without disrupting the surgical workflow. For example, for teaching and training purposes, students can observe the surgeon's gaze direction and correlate it with the lecturer's comments. Optionally, professors can assess and evaluate students' gaze directions to provide better guidance.

[0038] The proposed concept can be built around two main components: an optical imaging system 100 (which includes optical components and may include a display device 180 for viewing sample 110) and a device 130 (which can be used to control the optical imaging system 100, process sensor data from microscope 120 (e.g., sensor 122) and / or generate output data).

[0039] Typically, microscopes such as microscope 120 are optical instruments suitable for examining objects too small to be examined (by the human eye alone). For example, microscope 120 can provide samples (such as...) Figure 1a The optical magnification of sample 110 shown is illustrated. In modern microscopes, optical magnification is typically provided for cameras or imaging sensors, such as the optical imaging sensor 122 of microscope 120.

[0040] Various types of optical imaging systems exist. If the optical imaging system 100 is used in the medical or biological field, the sample 110 can be a sample of organic tissue, for example, arranged in a petri dish or present in a part of a patient's body. In some examples of this disclosure, such as... Figure 1b As shown, the optical imaging system 100 can be a surgical optical imaging system, such as an optical imaging system that will be used during or during surgical procedures (such as tumor surgery). However, the proposed concept can also be applied to other types of microscopy, such as laboratory microscopy or microscopy for materials examination purposes.

[0041] Figure 1bA schematic diagram of an example of a surgical optical imaging system 100 including a microscope 120 and an instrument 130 is shown. Generally, a (surgical) optical imaging system is a system that includes a microscope 120 and additional components that operate in conjunction with the microscope 120. In other words, a (surgical) optical imaging system is a system that includes a microscope 120 and one or more additional components, such as an instrument 130 (which may be a computer system adapted to control the microscope 120 and, for example, generate output data), an illumination system (which is used to illuminate the sample imaged by the microscope 120), additional sensors, a display, etc.

[0042] Figure 1b The surgical optical imaging system 100 shown includes a number of optional components, such as a base unit 105 with a support (which may include device 130), eyepiece displays 140, 145 arranged at a microscope 120, a display device 180, and a (robotic or artificial) arm 160 that holds the microscope 120 in place and is coupled to the base unit 105 and the microscope 120. Typically, these optional and non-optional components can be coupled to device 130, which can be configured to control and / or interact with the respective components.

[0043] In the example, device 130 can be configured to generate output data including the emphasized region by adjusting the brightness, sharpness, color settings, size, and / or orientation of the emphasized region or its surrounding area.

[0044] The adjusted image of sample 110 makes it easier and / or more instinctively possible to identify the region of interest (ROI) of interest to the user. For example, the ROI can be indicated by emphasizing the ROI and / or its surroundings. For example, the boundaries of the emphasized region can be blurred, such as the periphery of the ROI being blurred and / or darkened / brightened. For example, the adjusted image of sample 110 can include a smooth transition from the emphasized region to the periphery. For example, portions of the image of sample 110 that are not part of the ROI can be darkened, so that the perceptibility of the ROI can be increased by the reduced perceptibility of other portions of the image of sample 110. In this way, the gaze direction of another user of the optical imaging system 100 can be drawn. That is, another user can instinctively gaze at the emphasized region of the adjusted image of sample 110.

[0045] For example, an adjusted image of sample 110 can be generated by adjusting the brightness (such as lightness, contrast, illuminance), sharpness, color settings (such as hue, level, tint), size (such as width, height), and / or orientation of the emphasized area or its surroundings.

[0046] The region of emphasis can be substantially the same as the region of interest (ROI). That is, the ROI can be adjusted so that it is emphasized compared to the image of sample 110. Alternatively, the region of emphasis can be different from the ROI. For example, the region of emphasis can be larger than the ROI, to allow for a smooth transition between the ROI's periphery and the ROI. That is, the region of emphasis can be made more prominent by adjusting the periphery of the ROI. In this way, users can more easily and / or instinctively perceive the region of emphasis that may contain the ROI.

[0047] In the example, the gaze direction can indicate the time the user has been gazing in that direction. Device 130 can be configured to determine the region of interest based on the time the user has been gazing in that direction. The time the user has been gazing in that direction can allow for improved reliability in determining the region of interest. Furthermore, device 130 can generate region data only if the gaze direction data indicates that the user has been gazing at the same location for a specific period of time. Therefore, the computational workload of device 130 can be reduced.

[0048] For example, region data can be generated by device 130 only if the user has been gazing at the same location for more than a threshold time. In this example, device 130 can be configured to determine regions of interest (ROIs) by comparing the time the user has been gazing in a direction with the threshold time. When the user has been gazing in a direction for more than the threshold time, device 130 can be configured to identify the region the user is gazing at as a ROI. Using a threshold time can allow for improved determination of ROIs. For example, a threshold time can be used to reduce false positives in ROI determinations. In this way, the reliability of determining the ROI can be improved. Furthermore, a threshold time can allow for improved user interaction. For example, a user who knows the threshold time can use it to control the visualization of the desired ROI. For example, a user can intentionally look at a region of an image of a sample for more than the threshold time to trigger highlighting that region as a ROI. Alternatively, another trigger from the user (such as blinking and / or vocalization) can be used to trigger the generation of region data. In this case, device 130 can obtain trigger data indicating the user's intention to identify the region the user is gazing at as a ROI, allowing device 130 to generate an adjusted image that includes the highlighted region.

[0049] In the example, device 130 can be configured to determine a second region of interest based on the time the user has a gaze direction, and generate output data based on the second region of interest. Determining the second region of interest allows for the generation of output data of an adjusted image of sample 110 including multiple emphasis regions (or only one emphasis region indicating multiple regions of interest). In this way, multiple regions of interest can be visualized to another user of the optical imaging system 100. For example, the chief surgeon may need feedback on two different regions of an image of sample 110. Therefore, device 130 can generate output data of an adjusted image of the sample including two emphasis regions, one for each region of interest. In this way, an assistant can easily identify the chief surgeon's region of interest.

[0050] In the example, device 130 can be configured to determine when a user is no longer interested in a region of interest based on the duration of the user's gaze, and adjust the adjusted image of sample 110, including the emphasized regions indicating the regions of interest, such that the emphasized regions are no longer emphasized relative to the regions of interest in the image of sample 110. For example, when the user has not gazed at a region of interest for a specific time, device 130 can stop generating output data indicating the adjusted image of sample 110. In this case, device 130 can transmit the image of sample 110, instead of the output data, for display on display device 180. That is, the output data can include the image of sample 110. Therefore, only the image of sample 110 can be displayed to the user. In this way, outdated information can be avoided from being displayed to another user. Alternatively, when device 130 generates output data indicating multiple regions of interest and the user has not gazed at the first region of interest for a specific time, device 130 can stop emphasizing the first region of interest and can generate output data indicating other regions of interest(s).

[0051] In the example, device 130 can be configured to adjust the sharpness of the emphasized region by generating focus data indicating the adjusted focus of microscope 120 of optical imaging system 100. Furthermore, device 130 can be configured to transmit focus data for adjusting the focus of microscope 120 such that the region of interest is focused in the adjusted image of sample 110. Device 130 can transmit focus data for adjusting the focus of microscope 120. For example, device 130 can transmit focus data to an actuator of the microscope to control the adjustment of the focus of microscope 120. Optionally or alternatively, device 130 can transmit focus data to a storage device (e.g., storage device 136) to trigger the adjustment of the focus of microscope 120, for example, the actuator of microscope 120 can retrieve focus data from the storage device. In this case, the generation of output data can be based on the adjustment of the settings of optical imaging system 100. Therefore, by adjusting the settings of optical imaging system 100, such as the focus of microscope 120, an adjusted image of sample 110 including the emphasized region (the region focused by microscope 120) can be generated. In other words, the output data can indicate the image of sample 110 captured using optical imaging sensor 122 without further post-processing of the image of sample 110. An adjusted image can be generated by adjusting the settings of optical imaging system 100 based on region data. Therefore, output data indicating an adjusted image of sample 110 can be generated by device 130 by post-processing the image of sample 110, overlaying additional information onto the image of sample 110, and / or adjusting the settings of optical imaging system 100 (e.g., microscope 120) to adjust the captured image of sample 110.

[0052] In the example, device 100 can be configured to adjust the orientation of the emphasis region by shifting its position so that the emphasis region is centered in the adjusted image of sample 110. For example, the adjusted image of sample 110 for display on display device 180 may be a cropped portion of the image of sample 110. The center of the cropped portion may be the emphasis region. In this case, further post-processing or adjustment of the adjusted image of sample 110 may not be necessary. That is, for example, the emphasis region can be emphasized by adjusting the image of sample 110 so that the emphasis region is displayed in the center of display device 180. In this way, another user can instinctively gaze at the area of ​​interest.

[0053] In the example, device 130 can be configured to determine an accuracy score that indicates the accuracy of the generated gaze direction data, and to generate an adjusted image of sample 110 based on the accuracy score. For example, a higher accuracy score indicates higher accuracy. The accuracy score can be used to indicate the reliability of the identified region of interest. Therefore, when a higher accuracy score is present, the emphasized region can be emphasized more strongly. For example, the brightness or sharpness of the region of interest can increase with an increasing accuracy score. For example, the size or perimeter of the marker element surrounding the region of interest can increase with a decreasing accuracy score.

[0054] Optionally or alternatively, the adjusted image of sample 110 may be generated by device 130 only when a certain accuracy score is achieved or exceeded. In this way, unreliable output data can be avoided. Optionally or alternatively, device 130 may determine the metric used to emphasize the highlighted area based on the accuracy score. For example, for a lower accuracy score, the highlighted area may be emphasized simply by adjusting brightness, and for a higher accuracy score, it may be emphasized by adjusting both brightness and sharpness. For example, the position of the highlighted area may be shifted only when a certain accuracy score is achieved, such that the highlighted area is centered in the adjusted image. Optionally or alternatively, accuracy may be used to control the size of individual gaze marker elements.

[0055] In the example, device 130 can be configured to receive display data from display device 180 instructing for displaying an adjusted image of sample 110, and generate the adjusted image of sample 110 based on the display data. In this way, the region of interest (ROI) can be determined based on sensor data, gaze direction data, and display data, allowing the ROI to be inferred even when the user's actual viewpoint is uncertain. For example, an assistant may wear a head-mounted display on which an image of sample 110 can be displayed. Therefore, device 130 can determine the ROI based on the gaze direction, the image of sample 110 displayed on the head-mounted display, and information about the display device. For example, device 130 may receive display data from the display device and / or retrieve display data from a storage device (such as storage device 136). Based on the geometric relationship between the user's eyes and the display device, device 130 can determine the ROI.

[0056] In the example, device 130 can be configured to acquire user data indicative of multiple users of the optical imaging system and generate output data, such as an adjusted image of sample 110, based on the user data. For example, multiple users may use the optical imaging system 100, for instance, during surgery. The optical imaging system 100 may include at least one display device 180 or be communicatively coupled to said at least one display device. For example, the optical imaging system 100 may include multiple display devices 180 (…). Figure 1b (Not shown in the diagram) or can be communicatively coupled to the plurality of display devices. In this case, multiple users of the optical imaging system 100 can use multiple different display devices 180, such as those described below with respect to multiple head-mounted display devices. The description of the head-mounted display device is exemplary and also applicable to other display devices, such as... Figure 1b The (3D) monitor 180 or a combination thereof is shown. For example, a first user may use a head-mounted display device, and a second user may use the (3D) display device 180.

[0057] In this example, device 130 can be configured to obtain user data instructing a user to display an adjusted image of sample 110 to it, and to generate an adjusted image of sample 110 based on the user data. For example, device 130 may receive user data from an input device. Optionally or alternatively, device 130 may retrieve user data from a storage device (such as storage device 136). The user data may allow the generation of an adjusted image of sample 110 for a specific user, such as a specific user display device. Therefore, device 130 may generate output data to transmit the output data to a specific display device. In this way, device 130 may generate different images of sample 110 for display on different display devices. For example, the attending surgeon and assistant may each wear a head-mounted display device. Device 130 may determine the region of interest of the attending surgeon and may generate output data based on the region of interest. The output data may be intended for use by the assistant. That is, device 130 may transmit the output data for display on the assistant's head-mounted display device. The attending surgeon will still see an image of sample 110, but not an adjusted image of sample 110 (because the attending surgeon knows where he is gazing). This means that output data can be transmitted for display on a display device 180 for a different user. That is, device 130 can transmit output data for display on an assistant's head-mounted display and an image of sample 110 for display on a head-mounted display for the chief surgeon. In this way, a specific image of sample 110 can be displayed to a specific user of the optical imaging system 100. If display device 180 is a head-mounted display (… Figure 1b (Not shown in the image), then the sensor (e.g., eye-tracking sensor 124) can be part of the head-mounted display or can be disposed on the head-mounted display. The description of the surgeon and assistant is exemplary and also applies to other users of the optical imaging system 100. This means that the surgeon can be understood as the first user and the assistant can be understood as the second user. For example, the user can be the chief surgeon, assistant (such as an operating room nurse or surgical technician), an anesthesiologist, a medical intern, or a remote participant.

[0058] In principle, user data can indicate the number of users of the optical imaging system 100. The number of users can be indicated by the number of display devices used in conjunction with the optical imaging system 100. For example, a (3D) monitor 180 and two head-mounted displays can be communicatively coupled to the optical imaging system 100. This means that at least two users can use the optical imaging system 100. Alternatively, three users can also use the optical imaging system 100, two via head-mounted displays and one via monitor 180. Optionally or alternatively, user data can provide the number of users of the optical imaging system. For example, the optical imaging system may include a camera (such as sensor 124) and can capture images of the environment of the optical imaging system 100. Device 130 can be configured to generate user data by identifying users based on this image, for example, through facial recognition.

[0059] In this example, user data can indicate the user to whom or to whom the adjusted image of sample 110 is displayed. This means that user data can indicate a target user for viewing the adjusted image of sample 110. The target user can be associated with a display device (e.g., display device 180). For example, the target user can view the (3D) monitor 180 within their line of sight, or can wear a head-mounted display device. This means that output data can be transmitted to the display device associated with the target user. In this way, it can be ensured that the adjusted image of the sample is displayed on the correct display device. For example, the target user may be different from a user with a gaze direction indicated by gaze direction data, or all users of the optical imaging system 100 can be the target user (i.e., each display device can display the same adjusted image of sample 110).

[0060] For example, user data could instruct multiple users associated with multiple display devices, all of whom should see the same information. In this case, the adjusted image could be transmitted to multiple display devices for display to multiple users. Alternatively, the user data could specify a limited number of users (e.g., using a monitor) to whom the adjusted image of sample 110 is displayed or will be displayed, or a single user associated with only one display device (e.g., using a monitor or head-mounted display). This means that the user to whom output data should be displayed can be indicated in the user data.

[0061] For example, the optical imaging system 110 may include two display devices. A first display device may be associated with a first user (e.g., a chief surgeon), and a second display device may be associated with a second user (e.g., an assistant). An adjusted image of the sample 110 may be generated based on the first user's gaze direction. This means that the sensor 124 can track the first user's gaze direction. Since the first user knows where they are looking, adjusting the image of the sample 110 for the first user may confuse them. Therefore, the adjusted image of the sample 110, which can be generated based on the first user's gaze direction, can only be transmitted to the second display device. Thus, the adjusted image of the sample 110 can only be displayed to the second user. In this way, as described above, a specific image of the sample 110 can be displayed to a specific user of the optical imaging system 100.

[0062] In the example, the output data can be transmitted for display on a display device 180 for a different user. That is, the other user can be a second user, and the display device 180 (e.g., a (3D) monitor 180 or a head-mounted display) can be associated with the other user. Therefore, the gaze direction of the first user can be communicated to the other user in an improved manner. In this way, cooperation among multiple users of the optical imaging system 100 can be improved. In particular, cooperation can be improved when using different display devices.

[0063] In the example, user data may indicate second gaze direction data from a second sensor, which in turn indicates the gaze direction of a second user of the optical imaging system 100. The second user may be different from the first user. Furthermore, device 130 may be configured to generate an adjusted image of sample 110 by generating second region data indicating a second region of interest (ROI) based on the second gaze direction data. Device 130 may also be configured to generate output data based on the second region data, indicating a second emphasized region of interest (ROI) in sample 110, such that the second emphasized region is emphasized relative to the ROI in the image of sample 110. This means that the output data may include information about the emphasized region (the user's ROI) and the second emphasized region (the second user's ROI).

[0064] In the example, the adjusted image of sample 110 may include an emphasis region and a second emphasis region. This means that the second emphasis region can also be different from the emphasis region. In this case, the emphasis region and the second emphasis region can be emphasized differently (see also...). Figure 2 If the adjusted image of sample 110 is displayed on only one display device, or if multiple display devices display the same adjusted image of sample 110, this can allow for differentiation of the gaze direction of different users.

[0065] In the example, device 130 can be configured to generate output data by generating an adjusted image of sample 110 to be displayed on display device 180, and generating another adjusted image of sample 110 to be displayed on another display device 180, the other adjusted image of sample 110 including a second emphasis area. Furthermore, device 130 can be configured to transmit output data by transmitting an adjusted image of sample 110 for display on a display device and transmitting another adjusted image of sample 110 for display on another display device. The other display device can be different from the first display device. For example, the first display device can be used by a second user, while the other display device can be used by a first user. This means that several users with independent display devices (e.g., head-mounted displays, monitors, stereoscopic displays (3D) or combinations thereof) can use the optical imaging system 100. A specific image of sample 110 can be displayed to each user, which includes the other user's gaze direction or gaze point. This means that the display device associated with the first user (i.e., the other display device) can receive the other adjusted image of sample 110. The display device associated with the second user (i.e., display device 180) can receive the adjusted image of sample 110. In this way, the gaze direction or gaze point of other users(s) can be displayed on different display devices specific to different users. For example, the first user can be a chief surgeon, and the second user can be an assistant surgeon with their own monitor 180 within their respective lines of sight. Therefore, a customized image of sample 110 can be displayed to each user. In this way, collaboration among multiple users of the optical imaging system 100 can be improved.

[0066] In the example, device 130 can be configured to acquire second gaze direction data from a second sensor. The second gaze direction data indicates the gaze direction of a second user of the optical imaging system. The second gaze direction data can be acquired in the same manner as the gaze direction data (which may be referred to as the first gaze direction data when acquired). The second gaze direction data can be acquired simultaneously with the first gaze direction data. For example, the first and second gaze direction data can be received from a gaze signal from sensor 124.

[0067] Alternatively, the second gaze direction data can be obtained independently of the first gaze direction data. In the example, the gaze direction data can be obtained by receiving from a first display device or a user equipment configured to view the display device, and the second gaze direction data can be obtained by receiving from a second display device or a second user equipment configured to view the display device or another display device. The user equipment can be polarized glasses for viewing a 3D monitor. This means that sensor 124 can be part of the polarized glasses. The display device or user equipment can be different from the second display device or the second user equipment. The other display device can be different from or the same as this display device. If the other display device is the same as this display device, multiple users can use polarized glasses to view the same display device. In this case, the display device can display an image of sample 110 including an emphasis area and a second emphasis area (see...). Figure 2 ).

[0068] Furthermore, device 130 can be configured to generate second region data based on second gaze direction data, which indicates a second region of interest (ROI) being gazed at by a second user. Device 130 can also be configured to generate output data based on the second region data of an adjusted image of sample 110 including a second emphasis region indicating the second ROI, such that the second emphasis region is emphasized relative to the second ROI in the image of sample 110. In this way, multiple emphasis regions for multiple users can be displayed by showing the adjusted image of sample 110. Therefore, communication between multiple users can be facilitated.

[0069] In the example, device 130 can be configured to generate output data indicating an adjusted image of sample 110 containing an indicator element. The indicator element indicates which user of the optical imaging system 100 the emphasized area belongs to. In this way, it can facilitate the assignment of the emphasized area to a specific user of the optical imaging system 100. For example, the indicator element can be a portrait, username, color-coded marker element, or marker element with a specific geometry.

[0070] In the example, device 130 can be configured to generate output data by generating marker elements that indicate the location of the region of interest, such that the emphasized region in the adjusted image of sample 110 is emphasized relative to the region of interest in the image of sample 110 by the location of the marker elements. In this way, markers (such as geometric objects surrounding the region of interest) can be used to indicate the region of interest. In this case, additional information can be overlaid on the image of sample 110 to generate the adjusted image of sample 110.

[0071] like Figure 1aAs shown, one or more optional interfaces 132 are coupled to corresponding one or more processors 134 at device 130. In the example, the one or more processors 134 may be implemented using one or more processing units, one or more processing devices, or any means for processing, such as a processor, computer, or programmable hardware component that can operate with appropriately adapted software. Similarly, the functionality of the described one or more processors 134 may also be implemented in software, which is then executed on one or more programmable hardware components. Such hardware components may include general-purpose processors, digital signal processors (DSPs), microcontrollers, etc. The one or more processors 134 are capable of controlling one or more interfaces 132 such that any data transfers occurring through the one or more interfaces 132 and / or any interactions that may be involved in the one or more interfaces 132 can be controlled by the one or more processors 134.

[0072] In an implementation, device 130 may include a memory, such as one or more storage devices 136, and at least one or more processors 134 operatively coupled to the memory and configured to perform the methods described below.

[0073] In the example, one or more interfaces 132 may correspond to any means for acquiring, receiving, transmitting, or providing analog or digital signals or information, such as any connector, contact, pin, register, input port, output port, conductor, channel, etc., that allows the provision or acquisition of signals or information. One or more interfaces 132 may be wireless or wired, and may be configured to communicate with other internal or external components, such as transmitting or receiving signals or information.

[0074] Device 130 may be a computer, processor, control unit, (Field-Programmable Array) ((F)PLA), (Field-Programmable Gate Array) ((F)PGA), graphics processing unit (GPU), application-specific integrated circuit (ASIC), integrated circuit (IC), or system-on-a-chip (SoC) system. Device 130 may be part of optical imaging system 100. Alternatively, device 130 may be located outside optical imaging system 100 and may be connected to, as referenced... Figure 2 The device communicates.

[0075] Further details and aspects will be mentioned in conjunction with the examples described below. The examples shown in Figure 1 may include one or more aspects mentioned in conjunction with the proposed concepts or as described below (e.g., Figure 2 –4) One or more optional or additional features corresponding to one or more examples described above.

[0076] Figure 2An example of an adjusted image 202 of sample 210 is shown. The adjusted image can be generated by a device as described with reference to FIG1. Figure 2 As can be seen, the adjusted image 202 of sample 210 may include two emphasis regions 208 and 218. The emphasis regions 208 and 218 may be emphasized by marker elements 220 and 230. The marker elements may be superimposed on the image of sample 210 to generate the adjusted image 202 of sample 210.

[0077] Each marker element 220, 230 can be assigned to a user of the optical imaging system. For example, a first marker element 220 can be assigned to the (chief) surgeon, and a second marker element 230 can be assigned to an assistant. The first marker element 220 may have a different color and / or a different shape than the second marker element 230. Figure 2 (Not shown in the image). Optionally, the indicator elements 220' and 230' can indicate which user the highlighted area belongs to. For example, as shown in the image. Figure 2 As can be seen, the indicator elements 220' and 230' can be text containing the username.

[0078] Using this device, the gaze direction of at least one user can be visualized. For example... Figure 2 As shown, the device can also visualize the gaze direction of multiple users. In this way, it becomes clear which user is gazing at which region of interest based on each marker element 220, 230, especially when more than three users are using the optical imaging system. For example, marker elements 220, 230 may include different colors, different symbols, and / or different indicative elements, such as displayed names. In the example, marker elements 220, 230 may be designed so as not to obstruct observation of the sample. For example, marker elements 220, 230 may be hollow symbols and / or transparent labels.

[0079] Further details and aspects will be mentioned in conjunction with the examples described above and / or below. Figure 2 The examples shown may include one or more aspects mentioned in conjunction with the proposed concept, or the preceding (e.g., Figure 1) and / or the following (e.g., Figure 3 –4) One or more optional or additional features corresponding to one or more examples described above.

[0080] Figure 3A flowchart illustrating an example of a method for an optical imaging system is shown. Method 300 can be performed by an apparatus as described with reference to FIG1. ​​Method 300 includes obtaining sensor data 310 of an optical imaging sensor of an optical imaging system, the sensor data indicating an image of a sample. Furthermore, method 300 includes: obtaining 320 gaze direction data of a sensor, the gaze direction data indicating the gaze direction of a user of the optical imaging system; and generating 330 region data based on the gaze direction data, the region data indicating a region of interest that the user is gazing at in the image of the sample. Method 300 further includes: generating 340 output data based on the sensor data and the region data, the output data indicating an adjusted image of the sample to be displayed on a display device, the adjusted image of the sample including an emphasized region indicating a region of interest, such that the emphasized region is emphasized relative to the region of interest in the image of the sample; and transmitting 350 the output data for display on a display device. The method can be performed by an apparatus as described above.

[0081] Further details and aspects will be mentioned in conjunction with the examples described above and / or below. Figure 3 The examples shown may include one or more aspects mentioned in conjunction with the proposed concept, or the preceding (e.g., Figures 1–2) and / or the following (e.g., Figure 4 One or more optional or additional features corresponding to one or more examples described above.

[0082] Some embodiments relate to a microscope that includes the device as described in conjunction with Figure 1. Alternatively, the microscope may be communicatively connected to the device as described in conjunction with Figure 1. Figure 4 This shows the configuration to execute, for example, the reference. Figure 3 A schematic diagram of system 400 (e.g., an optical imaging system) described herein. System 400 includes a microscope 410 and a computer system 420. The microscope may include devices as described above, for example, with reference to FIG1. ​​Microscope 410 is configured to capture images and is connected to computer system 420. Computer system 420 is configured to perform at least a portion of the methods described herein. Computer system 420 may be configured to execute machine learning algorithms. Computer system 420 and microscope 410 may be separate entities, but may also be integrated together in a common housing. Computer system 420 may be part of the central processing system of microscope 410, and / or computer system 420 may be part of sub-components of microscope 410, such as sensors, actuators, cameras, or illumination units of microscope 410.

[0083] Computer system 420 may be a local computer device (e.g., a personal computer, laptop computer, tablet computer, or mobile phone) having one or more processors and one or more storage devices, or it may be a distributed computer system (e.g., a cloud computing system having one or more processors and one or more storage devices distributed in various locations (e.g., distributed at local clients and / or one or more remote server farms and / or data centers)). Computer system 420 may include any circuitry or combination of circuitry. In one embodiment, computer system 420 may include one or more processors of any type. As used herein, a processor may mean any type of computing circuitry, such as, but not limited to, a microprocessor, microcontroller, complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, graphics processor, digital signal processor (DSP), multi-core processor, field-programmable gate array (FPGA), or any other type of processor or processing circuitry. Other types of circuitry that may be included in computer system 420 may be custom circuitry, application-specific integrated circuits (ASICs), etc., such as one or more circuits (e.g., communication circuits) used in wireless devices such as mobile phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. Computer system 420 may include one or more storage devices, which may include one or more memory elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard disk drives, and / or one or more drives for disposing of removable media (such as optical discs (CDs), flash memory cards, digital video discs (DVDs), etc.). Computer system 420 may also include a display device, one or more speakers, and a keyboard and / or controller, which may include a mouse, trackball, touchscreen, voice recognition device, or any other device that allows a system user to input information into and receive information from computer system 420.

[0084] In conjunction with the examples mentioned above, further details and aspects will be discussed. Figure 4 The examples shown may include one or more optional or additional features corresponding to one or more aspects mentioned in conjunction with the proposed concept or one or more examples described above (e.g., Figures 1–3).

[0085] Some or all of the method steps may be performed by (or using) hardware devices (such as processors, microprocessors, programmable computers, or electronic circuits). In some embodiments, one or more of the most important method steps may be performed by such devices.

[0086] Depending on certain specific implementation requirements, embodiments of the present invention can be implemented in hardware or software. This implementation can be executed using a non-transitory storage medium (such as a digital storage medium, e.g., floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or FLASH memory) that stores electronically readable control signals thereon, which cooperate with (or are capable of cooperating with) a programmable computer system to cause the corresponding method to be executed. Therefore, the digital storage medium can be computer-readable.

[0087] Some embodiments of the invention include a data carrier having electronically readable control signals that are capable of cooperating with a programmable computer system to enable the execution of one of the methods described herein.

[0088] Generally, embodiments of the present invention can be implemented as a computer program product having program code that, when run on a computer, is operable to perform one of the methods. The program code may, for example, be stored on a machine-readable medium.

[0089] Other implementations include a computer program stored on a machine-readable medium for performing one of the methods described herein.

[0090] Therefore, in other words, an embodiment of the present invention is a computer program having program code for performing one of the methods described herein when the computer program is run on a computer.

[0091] Therefore, a further embodiment of the invention is a storage medium (or data carrier, or computer-readable medium) including a computer program stored thereon for performing one of the methods described herein when executed by a processor. Data carriers, digital storage media, or recording media are generally tangible and / or non-transitory. A further embodiment of the invention is an apparatus as described herein, including a processor and a storage medium.

[0092] Therefore, a further embodiment of the invention is a data stream or signal sequence representing a computer program for performing one of the methods described herein. The data stream or signal sequence may, for example, be configured to be transmitted via a data communication connection (e.g., via the Internet).

[0093] Further embodiments include a processing element (e.g., a computer or programmable logic device) configured or adapted to perform one of the methods described herein.

[0094] A further embodiment includes a computer on which a computer program is installed for performing one of the methods described herein.

[0095] A further embodiment of the invention includes an apparatus or system configured to transmit, for example, electronically or optically to a receiver, a computer program for performing one of the methods described herein. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may include, for example, a file server for transmitting the computer program to the receiver.

[0096] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, the field-programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. Generally, these methods are preferably performed by any hardware device.

[0097] If certain aspects of an apparatus or system have been described, those aspects should also be understood as a description of the corresponding method, and vice versa. For example, a block, device, or functional aspect of an apparatus or system may correspond to a feature of the corresponding method, such as a method step. Therefore, aspects of a method description should also be understood as a description of the corresponding block, element, characteristic, or functional feature of the corresponding apparatus or system.

[0098] The following text provides some examples of the proposed concepts:

[0099] Example (e.g., Example 1) relates to an apparatus for an optical imaging system, including one or more processors and one or more storage devices, wherein the apparatus is configured to: obtain sensor data of an optical imaging sensor of the optical imaging system, the sensor data indicating an image of a sample; obtain gaze direction data of the sensor, the gaze direction data indicating a user's gaze direction of the optical imaging system; generate region data based on the gaze direction data, the region data indicating a region of interest gazed at by the user on the image of the sample; generate output data based on the sensor data and the region data, the output data indicating an adjusted image of the sample to be displayed on a display device, the adjusted image of the sample including an emphasized region indicating the region of interest, such that the emphasized region is emphasized relative to the region of interest in the image of the sample; and transmit the output data for display on the display device.

[0100] Another example (e.g., Example 2) relates to the previous example (e.g., Example 1) or any other example, and further includes: the device is configured to generate the output data including the emphasized region by adjusting at least one of the brightness, sharpness, color settings, size, and orientation of the emphasized region or the periphery of the emphasized region.

[0101] Another example (e.g., Example 3) relates to the preceding example (e.g., one of Example 1 or 2) or any other example, and further includes the gaze direction indicating the time the user has the gaze direction, and wherein the device is configured to determine the region of interest based on the time the user has the gaze direction.

[0102] Another example (e.g., Example 4) relates to the previous example (e.g., Example 3) or any other example, and further includes the device being configured to: determine the region of interest by comparing the time the user has the gaze direction with a threshold time; and identify the region the user is gazing at as a region of interest when the time the user has the gaze direction exceeds the threshold time.

[0103] Another example (e.g., Example 5) relates to the preceding example (e.g., one of Example 3 or 4) or any other example, and further includes the device being configured to: determine a second region of interest based on the time the user has the gaze direction; and generate the output data based on the second region of interest.

[0104] Another example (e.g., Example 6) relates to the preceding example (e.g., one of Examples 3, 4, or 5) or any other example, wherein the device is configured to: determine when the user is no longer interested in the region of interest based on the time the user has the gaze direction; and adjust an adjusted image of the sample including the emphasized region indicating the region of interest, such that the emphasized region is no longer emphasized relative to the region of interest in the image of the sample.

[0105] Another example (e.g., Example 7) relates to the preceding example (e.g., one of Examples 2, 3, 4, 5, or 6) or any other example, wherein the device is configured to: adjust the sharpness of the region of emphasis by generating focus data indicating the adjusted focus of the microscope of the optical imaging system; and transmit focus data for adjusting the focus of the microscope such that the region of interest is focused in the adjusted image of the sample.

[0106] Another example (e.g., Example 8) relates to the previous example (e.g., one of Examples 2, 3, 4, 5, 6 or 7), wherein the device is configured to adjust the orientation of the emphasis region by shifting the position of the emphasis region so that the emphasis region is located in the middle of the adjusted image of the sample.

[0107] Another example (e.g., Example 9) relates to the preceding example (e.g., one of Examples 1 to 8) or any other example, and further includes the device being configured to: determine an accuracy score indicating the accuracy of the generated gaze direction data; and generate an adjusted image of the sample based on the accuracy score.

[0108] Another example (e.g., Example 10) relates to the preceding example (e.g., one of Examples 1 to 9) or any other example, and further includes the device being configured to: obtain user data indicating a plurality of users of the optical imaging system; and generate an adjusted image of the sample based on the user data.

[0109] Another example (e.g., Example 11) relates to the preceding example (e.g., one of Examples 1 to 10) or any other example, and further includes the device being configured to: obtain user data indicating a user to display an adjusted image of the sample thereto; and generate an adjusted image of the sample based on the user data.

[0110] Another example (e.g., Example 12) relates to the preceding example (e.g., one of Examples 1 to 11) or any other example, and further includes the device being configured to: obtain second gaze direction data from a second sensor, the second gaze direction data indicating the gaze direction of a second user of the optical imaging system; generate second region data based on the second gaze direction data indicating a second region of interest gazed upon by the second user; and generate, based on the second region data, the output data indicating an adjusted image of the sample including a second emphasis region indicating the second region of interest, such that the second emphasis region is emphasized relative to the second region of interest in the image of the sample.

[0111] Another example (e.g., Example 13) relates to the preceding example (e.g., one of Examples 1 to 12) or any other example, and further includes the device being configured to generate the output data by generating marker elements that indicate the location of the region of interest, such that the emphasized region in the adjusted image of the sample is emphasized relative to the region of interest in the image of the sample by the location of the marker elements.

[0112] Examples (e.g., Example 14) relate to an optical imaging system including a device according to any of the foregoing examples.

[0113] Example (e.g., Example 15) relates to a method for an optical imaging system, comprising: obtaining sensor data of an optical imaging sensor of the optical imaging system, the sensor data indicating an image of a sample; obtaining gaze direction data of the sensor, the gaze direction data indicating a gaze direction of a user of the optical imaging system; generating region data based on the gaze direction data, the region data indicating a region of interest gazed at by the user on the image of the sample; generating output data based on the sensor data and the region data indicating an adjusted image of the sample to be displayed on a display device, the adjusted image of the sample including an emphasized region indicating the region of interest, such that the emphasized region is emphasized relative to the region of interest in the image of the sample; and transmitting the output data for display on the display device.

[0114] The appended claims are incorporated herein by reference in the detailed description, wherein each claim may be taken independently as a separate example. It should also be noted that while in the claims a dependent claim refers to a specific combination with one or more other claims, other examples may also include combinations of dependent claims with the subject matter of any other dependent or independent claim. Such combinations are explicitly stated herein unless it is stated in individual cases that a particular combination is not intended to be used. Furthermore, for any other independent claim, the features of the claim should also be included, even if the claim is not directly defined as dependent on the other independent claim.

[0115] Aspects and features related to a specific example in the previous examples can also be combined with one or more other examples to replace the same or similar features of the other examples, or to introduce features additionally into the other examples. List of reference numerals 100 Optical Imaging System 105 Base 110 samples 120 microscope 122 Sensors 130 equipment 132 interface 134 processor 136 storage devices 140 and 145 eyepiece displays 160 arms 180 display device 202 Adjusted Image 210 samples Areas of Interest 218 and 228 220, 230 marking elements 220', 230' indicator elements 300 Methods for optical imaging systems 310 Obtain sensor data 320 Obtain gaze direction data 330 Generate regional data 340 Generate output data 450 Transmit Output Data 400 system 410 Microscope 420 Computer System

Claims

1. An apparatus (130) for an optical imaging system (100), comprising one or more processors (134) and one or more storage devices (136), wherein the apparatus (130) is configured to: Sensor data of the optical imaging sensor (122) of the optical imaging system (100) is obtained, the sensor data indicating an image of the sample (110); Gazing direction data of the sensor is obtained, the gaze direction data indicating the gaze direction of the user of the optical imaging system (100); Region data is generated based on the gaze direction data, the region data indicating the region of interest that the user is gazing at on the image of the sample (110); Output data is generated based on the sensor data and the region data. The output data indicates an adjusted image of the sample (110) to be displayed on the display device (180). The adjusted image of the sample (110) includes an emphasis region indicating the region of interest, such that the emphasis region is emphasized relative to the region of interest in the image of the sample (110). as well as The output data is transmitted for display on the display device (180).

2. The device (130) according to claim 1, wherein the device (130) is configured to The output data including the emphasized region is generated by adjusting at least one of the brightness, sharpness, color settings, size, and orientation of the emphasized region or its surrounding area.

3. The device (130) according to any one of the preceding claims, wherein The gaze direction indicates the time during which the user has the gaze direction, and The device (130) is configured to determine the region of interest based on the time the user has the gaze direction.

4. The device (130) according to claim 3, wherein the device (130) is configured to: The region of interest is determined by comparing the time the user has the gaze direction with a threshold time; and When the user gazes in the specified direction for a period exceeding the threshold time, the area the user is gazing at is identified as a region of interest.

5. The device (130) according to claim 3 or 4, wherein the device (130) is configured to: A second region of interest is determined based on the time the user has the stated gaze direction; and The output data is generated based on the second region of interest.

6. The device (130) according to any one of claims 3-5, wherein the device (130) is configured to: Determine when the user is no longer interested in the region of interest based on the time the user has the gaze direction; and The image of the sample (110) including the emphasized region indicating the region of interest is adjusted such that the emphasized region is no longer emphasized relative to the region of interest in the image of the sample (110).

7. The device (130) according to any one of claims 2-6, wherein the device (130) is configured to The sharpness of the emphasized area is adjusted by generating focus data of the microscope, which indicates the optical imaging system (100), to indicate the focused area; and Focus data is transmitted to adjust the focus of the microscope so that the region of interest is focused in the adjusted image of the sample (110).

8. The device (130) according to any one of claims 2-7, wherein the device (130) is configured to The orientation of the emphasis region is adjusted by shifting its position so that it is located in the center of the adjusted image of the sample (110).

9. The device (130) according to any one of the preceding claims, wherein the device (130) is configured to: Determine the accuracy score for the accuracy of the generated gaze direction data; and An adjusted image of the sample (110) is generated based on the accuracy score.

10. The device (130) according to any one of the preceding claims, wherein the device (130) is configured to: Obtain user data indicating multiple users of the optical imaging system; and The output data is generated based on the user data.

11. The device (130) according to claim 10, wherein The user data indicates the user to whom or to whom the adjusted image of the sample (110) is displayed.

12. The device (130) according to claim 11, wherein The output data is transmitted for display on the display device (180) for a different user than the user.

13. The device (130) according to claim 10, 11 or 12, wherein The user data indicates the second gaze direction data of the second sensor, and the second gaze direction data indicates the gaze direction of the second user of the optical imaging system (100); and The device (130) is configured to generate an adjusted image of the sample by: A second region data is generated based on the second gaze direction data, the second region data indicating the second region of interest of the second user's gaze; and The output data is generated based on the second region data to produce an adjusted image of the sample (110) including a second emphasis region indicating the second region of interest, such that the second emphasis region is emphasized relative to the second region of interest in the image of the sample (110).

14. The device (130) according to claim 13, wherein the device (130) is configured to The output data is generated by generating an adjusted image of the sample (110) to be displayed on the display device (180) and another adjusted image of the sample (110) to be displayed on another display device (180), the other adjusted image of the sample including the second emphasized area; and The output data is transmitted by transmitting an adjusted image of the sample (110) for display on the display device and another adjusted image of the sample (110) for display on the other display device.

15. The device (130) according to claim 13, wherein The adjusted image of the sample (110) includes the emphasized region and the second emphasized region.

16. The device (130) according to claim 13, 14 or 15, wherein The gaze direction data is obtained by receiving it from a first display device or a user device configured to view the display device; and The second gaze direction data is obtained by receiving it from a second display device or a second user equipment configured to view the display device or another display device.

17. The device (130) according to any one of the preceding claims, wherein the device (130) is configured to The output data is generated by generating marker elements that indicate the location of the region of interest, such that the emphasized region in the adjusted image of the sample (110) is emphasized relative to the region of interest in the image of the sample (110) by the location of the marker elements.

18. An optical imaging system (100), comprising: The device (130) according to any one of the preceding claims.

19. A method (300) for an optical imaging system, comprising: Obtain sensor data from the optical imaging sensor of the optical imaging system (310), the sensor data indicating an image of the sample; Obtain gaze direction data from (320) the sensor, the gaze direction data indicating the gaze direction of the user of the optical imaging system; (330) region data is generated based on the gaze direction data, the region data indicating the region of interest that the user is gazing at on the image of the sample; (340) Output data is generated based on the sensor data and the region data, the output data indicating an adjusted image of the sample to be displayed on a display device, the adjusted image of the sample including an emphasized region indicating the region of interest, such that the emphasized region is emphasized relative to the region of interest in the image of the sample; as well as Transmit (350) the output data for display on the display device.

20. A computer program having program code for executing the method of claim 19 when the computer program is executed on a processor.

Images