Apparatus for providing a stream of images and method therefor

Abstract

A first aspect of this disclosure relates to an apparatus for providing an image stream, configured to: - obtain a plurality of stereo images of a sample from a stereo imaging system, wherein the stereo images include a left image and a right image as stereo information; and - obtain user position information; - for each image, determine a single-view image based on the left image, the right image and the user position information; and - provide the single-view image for display.

Description

Technical Field

[0001] This disclosure relates to an apparatus for providing an image stream and a method of operating such an apparatus. Background Technology

[0002] Stereo cameras, also known as stereo imaging systems, measure depth by capturing two offset viewpoints. Due to their ability to provide three-dimensional visualization of anatomical structures, this technology has demonstrated significant application value in medical and diagnostic imaging. In surgery, stereo cameras can guide surgeons by providing a 3D view of the surgical area, enhancing depth perception and accuracy. They assist in minimally invasive surgery by improving spatial understanding of tissue and instrument placement, reducing the risk of accidental injury. Furthermore, stereo vision can generate accurate 3D reconstructions for diagnostic purposes, aiding in the early detection of abnormalities. For example, stereo endoscopy can explore internal pathways and detect subtle abnormalities in tissue morphology. However, 3D display systems can be both expensive and bulky, requiring improvement. Summary of the Invention

[0003] The purpose of this disclosure is to improve the display of 3D information.

[0004] This objective is achieved through the disclosed embodiments, which are specifically defined by the subject matter of the independent claims. The dependent claims provide information for other embodiments. Various aspects and embodiments of these aspects, which provide additional features and advantages, are also disclosed below in the abstract and description.

[0005] The first aspect of this disclosure relates to a device for providing an image stream, configured to:

[0006] - Obtain multiple stereo images of the sample from a stereo imaging system, wherein the stereo images include a left image and a right image as stereo information; and

[0007] - Obtain user location information;

[0008] - For each image, determine the single-view image based on the left image, the right image, and the user's location information;

[0009] - Provides a single-view image for display.

[0010] The device according to the first aspect can be a control device (e.g., a specially programmed computer) configured to control an imaging device and / or may be part of an imaging device. The imaging device can be configured to observe a sample from two perspectives in a medical and / or diagnostic setting. A dual-view medical and / or diagnostic imaging device can be a microscope, endoscope, etc., wherein the dual-view device can include a system with at least two cameras and / or be associated with two independent cameras. The two cameras are configured to observe the sample from two different perspectives / viewpoints. An imaging device providing a single-view image can include a 2D display or be connected to a 2D display. Alternatively or additionally, the device according to the first aspect can provide a single-view image to a separate 2D display.

[0011] A sample can be any object and / or specimen that can be observed with imaging equipment. In surgical or diagnostic settings, a sample can be a patient's organ and / or part of an organ. This can be a biopsy, particularly a biopsy involving in situ dissection and analysis. A sample can be or include the patient's bone and / or blood. A sample can also include sutures, incisions, or any other artificial objects placed inside or on the body during surgery.

[0012] In the sense of this disclosure, obtaining an image may include receiving and / or extracting image information.

[0013] Stereoscopic images can be provided by a stereo camera in an imaging system that can be associated with an operator (e.g., a surgeon operating a microscope). The images can display visual information (e.g., ordinary or visible light) and / or non-visual information (e.g., infrared information). The first viewpoint can be based on the viewpoint or mounting point of the first stereo camera. A stereo imaging system (e.g., a stereo camera) can be a device that uses two lenses spaced apart to simultaneously capture two separate images of the same scene (simple stereo imaging). Stereo cameras can also include more complex designs beyond two fixed lenses. Such cameras can include multiple lenses or sensor arrays, adjustable inter-axis distances, automatic alignment systems, synchronized shutters, and / or integrated computational processing.

[0014] User location information may be information about the viewing angle (also referred to herein as "gaze angle" or "observation angle") of a user viewing a sample on a 2D display providing a single-view image. User location information may also include user distance, i.e., the distance between the user and the display and / or the distance of a predefined point relative to the display (where the display corresponds to the observation point of a stereo camera). User distance may be a gaze distance, for example, the distance between the user and a ranging device (e.g., a camera) and / or the distance of another focal point relative to the ranging device. Additionally or alternatively, user distance may be the shortest distance between the user and the display and / or the shortest length of a predefined point relative to the display.

[0015] Specifically, the single-view image can be based on different viewpoint information included in the left and right images of the stereoscopic image. In the sense of this disclosure, providing a single-view image may include sending and / or storing information for retrieval by others, for example, storing the information in shared memory. In particular, multiple consecutive images can be determined and provided as an image stream (i.e., a video stream) for display. The single-view image can be determined by adjusting the left and / or right images of the stereoscopic camera. Furthermore, the single-view image is based on the depth information of the stereoscopic image, i.e., the information included in the left and right images of the stereoscopic image.

[0016] The first aspect of the embodiment provides viewpoint-related image information, thereby providing a depth experience for the user on a 2D display.

[0017] The first aspect of the embodiment relates to a device for providing an image stream, comprising either or both of the following:

[0018] - Stereo camera;

[0019] - A monitor used to display single-view images.

[0020] Specifically, the device can be a software device and configured to run on the computer of the stereo camera and / or the computer of the display. Alternatively or additionally, the device may also include or be a stereo microscope, and the above-described functions can be run on the computer of the stereo microscope.

[0021] The first aspect of the embodiment relates to a device for providing an image stream.

[0022] The user location information includes one or more of the following:

[0023] - The position of the user's head;

[0024] - The position of the user's single or both eyes;

[0025] -The position of the user's torso;

[0026] - The position of the user's arm;

[0027] - The location of the marker on the user.

[0028] Suitable methods for obtaining user position may include measuring or estimating the user's head position, eye position, torso position, arm position, and / or markers on the user. In each case, a direction vector can be calculated and compared to a screen surface or a known reference.

[0029] Head-based methods rely on cameras or depth sensors to locate facial feature points and can fit a 3D model and / or triangulate the feature locations to generate a head pose vector that is compared to the screen normal. Eye-position-based methods measure user position information and can utilize infrared illumination and / or high-resolution cameras to detect corneal or pupillary reflections. Trunk-position-based and / or arm-based position tracking methods involve pose estimation of body keypoints to generate a coarse vector. The head or eye position can be derived from this vector based on a body model. Tag-based tracking can include one or more reference tags placed on the user (e.g., on a helmet or near the eyes). Based on the derived information, 3D position and orientation can be determined to approximate the gaze direction relative to the display. User position information can be observed by obtaining information from passive or active markers such as QR codes or infrared tags. The system detects the position and orientation of these markers through image processing, enabling precise tracking of the user's position.

[0030] The first aspect of the embodiment relates to a device for providing an image stream.

[0031] Includes devices for observing the user's location, and in particular, includes one or more of the following:

[0032] -camera;

[0033] - Eye tracker;

[0034] - Head tracker;

[0035] - Body tracker;

[0036] - Receivers of signals from active markers;

[0037] - A device for detecting passive markers.

[0038] One or more methods can be used to observe the user's location.

[0039] A camera can be used to determine the direction of gaze by recognizing the user's eyes from the camera image. The camera can be mounted such that it observes an image parallel to the display screen, for example, by attaching the camera to the screen, or by allowing it to observe "through" the center of the display screen. Therefore, by determining the angle relative to the user's single eye, the direction of gaze can be effectively identified. Alternatively, methods can be employed that track other parts of the body, such as the forehead and / or nose.

[0040] An eye tracker is a device or system that measures the position and / or movement of a user's eyes. Alternatively or additionally, an eye tracker can also directly provide the gaze direction. The gaze direction can be determined relative to a display surface. Based on the gaze direction, a single-view image is determined, allowing the user to experience a gaze-dependent view. Eye trackers may include dedicated cameras and infrared illumination for accurate and real-time tracking. Similar to cameras, head trackers can also be attached to a display or mounted in a manner that allows for efficient calculation of the gaze angle (in one or more dimensions) relative to the display, thereby minimizing computational latency.

[0041] A head tracker is a device or system that (specifically, uses sensors or cameras) measures the orientation and position of a person's head. Based on head tracker information, the gaze direction can be determined relative to a display surface. This can be based on a model of the user's head. Similarly, a model of the user's body can be used to infer the gaze direction relative to the display from information from a body tracker.

[0042] An active marker worn by the user emits a signal that allows external sensors or cameras to track its position, particularly its precise 3D orientation. This marker can be aligned with the user's line of sight (e.g., placed near the user's eyes). The device can then compare the marker's orientation vector to a reference axis of the display to determine the user's gaze angle relative to the screen. Alternatively, one or more active markers can be used that provide an absolute position within a specific room, and the gaze angle relative to the display (which may also have an absolute position within the same room) can be calculated, for example, by converting the information to the display's coordinate system.

[0043] Passive markers, such as tags or other recognizable patterns, can be placed near a user's eyes or on a head-mounted device. One or more cameras can detect and track this marker by analyzing its unique visual pattern. The system can then estimate the marker's 3D orientation relative to the camera coordinate system. By comparing this orientation vector to a reference axis of the display (e.g., the screen normal), the user's gaze angle can be determined.

[0044] The first aspect of the embodiment relates to a device for providing an image stream, configured to:

[0045] - Perform user gaze angle detection based on user location information, especially based on the determination of whether the user has one or both eyes.

[0046] The user's gaze angle can be a one-dimensional gaze angle or a multi-dimensional gaze angle.

[0047] The first aspect of the embodiment relates to a device for providing an image stream.

[0048] The user's location information, especially the gaze angle, is obtained under the following conditions:

[0049] -At constant intervals;

[0050] - If the user's location changes.

[0051] User location information (e.g., gaze angle and / or distance to the display) can be observed at a constant sampling interval (e.g., 1 ms). Alternatively, user location information can be observed only when the user's position changes and the change is greater than a predefined threshold. In this case, it is necessary to observe changes in location information, which can be accomplished using sensors that are less complex and / or less expensive than position sensors.

[0052] The first aspect of the embodiment relates to a device for providing an image stream, configured to:

[0053] -Map the user's position to a virtual location between the left and right images of the stereoscopic image;

[0054] - Single-view images are determined based on weighted virtual locations.

[0055] For example, the user's position includes the user's gaze angle and the shortest distance from the user to the display. Based on this information (and the fact that the display corresponds to the viewpoint of the stereo camera providing the left and right stereo images), the user's position can be mapped to a straight line defined by the line / distance between the two lenses of the stereo camera and / or by the centers of the left and right images of the stereo camera. If this line is limited to the line between the lenses / images, then in stereo vision, this line is also called the "baseline." In other words, the baseline of a stereo camera is the distance between the optical centers of its two lenses.

[0056] The shortest distance between the user and the camera system can be on the stereo camera's baseline. The mapping of the user's position to the baseline is then complete. Alternatively, the shortest distance can also be to the left of the left lens / image or to the right of the right lens / image, meaning the shortest distance between the user and the display (corresponding to the camera) does not end on the stereo camera's baseline. A user position with a shortest user distance that does not end on the stereo camera's baseline can be forced to be mapped to one of two starting points of the stereo camera's baseline: if the user is to the left of the left lens, their position is mapped to the center of the left lens (the leftmost point of the baseline). If the user is to the right of the right lens, their position is mapped to the center of the right lens (the rightmost point of the baseline). This is merely one example of how the mapping to the virtual position between the left and right images of the stereo image is determined.

[0057] The first aspect of the embodiment relates to a device for providing an image stream, configured to:

[0058] - Determine the disparity map based on the left and right images of the stereo image;

[0059] - Determine single-view images based on disparity maps.

[0060] A disparity map can be a two-dimensional representation, for example, indicating the positional difference between the left and right views for each pixel in a stereo image. The intensity / value of each pixel corresponds to the disparity, allowing for the recovery of relative depth and supporting a three-dimensional understanding of a scene from both the two-dimensional left and right images and the right image. Based on the disparity (map), pixels at any viewpoint on the baseline (i.e., the straight line between the left and right images) can be determined. A disparity map can be determined by mapping the left image from the stereo camera to the right image, and / or another disparity map can be determined by mapping the right image to the left image. Disparity maps can be created for stereo images. Since stereo cameras are vertically aligned, a line-by-line correspondence problem can be solved, i.e., finding the corresponding sample pixels in the left camera image within the right camera image.

[0061] Based on the determined disparity map, a virtual single-view image can be generated for points on the baseline to which the user's position is mapped. When the user's position is mapped to the center of the baseline (i.e., a 0° gaze angle is detected), the determination of the single-view image based on the disparity map can be performed as follows: each pixel of the original left or right image can be shifted to the virtual single-view image based on a weighted disparity (the disparity value at the same pixel coordinates in the disparity map determined from the left and right images). For example, the pixel at position (1,1) in both the left and right images has a disparity value of 12, and can then be mapped to pixel (1,7) in the single-view image based on half of the disparity (=6). This can be performed on each pixel in the original image. This will provide a single-view image in the middle of the baseline, corresponding to a 0° gaze angle.

[0062] The first aspect of the embodiment relates to a device for providing an image stream, configured to:

[0063] - Determine a first single-view image based on a first stereo image and a first disparity map based on the first stereo image; and

[0064] -The second single-view image is determined based on the following;

[0065] --A second stereoscopic image obtained after the first stereoscopic image;

[0066] --A second disparity map based on the second stereo image; and

[0067] --First stereoscopic image and / or first parallax map.

[0068] By using previously determined information from previous single-view images, the processing resources required for subsequently determining another single-view image can be reduced. For example, the processing time for determining the mapping to a baseline and / or for determining another single-view image based on a disparity map can be shortened. This enables real-time determination of single-view image streams for users of 2D displays.

[0069] The first aspect of the embodiment relates to a device for providing an image stream.

[0070] The single-view image is determined based on one or more of the following:

[0071] -AI;

[0072] -Flight time information;

[0073] - Structured light information.

[0074] For example, algorithms using deep learning models trained on large stereo datasets can accurately match features between left and right images. These matches can generate detailed disparity maps that indicate depth, and these detailed disparity maps can be processed using user location information to generate single-view images as described above.

[0075] A time-of-flight sensor emits light pulses and measures their return time to derive precise distance information for each pixel. This depth data (disparity map) can be compared with disparity data based on stereo images. Figure 1 It can be used, or replaced, as a disparity map based on stereo images in order to map sample images to user location information.

[0076] A structured light projector projects a known pattern onto a sample, and a stereo camera captures the deformed pattern. By analyzing these deformations, depth measurements can be obtained. This depth data (disparity map) can be compared with disparity data based on stereo images. Figure 1 It can be used, or replaced, as a disparity map based on stereo images in order to map sample images to user location information.

[0077] The first aspect of the embodiment relates to a device for providing an image stream, configured to:

[0078] - Obtain updated information for a single-view image; and

[0079] - Determine and / or provide single-view images based on the obtained updated information.

[0080] Update information can be correlated with user information. For example, a user can input a frequency, which defines the frequency at which a new single-view image is determined. Alternatively or concurrently, update information can be correlated with changes in the user's position (e.g., positional shift). If the change in the user's position exceeds a predefined threshold, a new single-view image can be determined. This change needs to be correlated not only with the user's position (e.g., gaze angle or distance) but also with the derived user position, such as gaze velocity and / or gaze acceleration. Therefore, the user can effectively determine the rate of view updates.

[0081] The first aspect of the embodiment relates to a device for providing an image stream, configured to:

[0082] - Indicates blind spots in a single-view image.

[0083] Determining a single-view image based on the left and / or right images may result in each pixel of the single-view image not being correctly defined. This is because the left image may include information not present in the right image, and vice versa.

[0084] To avoid user confusion, these pixels can be indicated, for example, using predefined colors. Alternatively, they can be filled with existing information from either the left or right image of the stereoscopic image. In one embodiment, undefined single-view image information is replaced with image information from the left image of the first color and the right image of the second color. In this way, the user can see which camera or which side the replacement information comes from.

[0085] The first aspect of the embodiment relates to a device for providing an image stream, configured to:

[0086] Provide single-view images in single-view image streams (especially video streams).

[0087] It can provide image streams, for example, in any known video format (such as MP4, AVI, MOV, or WMV).

[0088] The first aspect of the embodiment relates to a device for providing an image stream, configured to:

[0089] - Switch between multiple users.

[0090] For example, switching can be achieved through user identification and / or manual user assignment. Based on the user to whom the device has been switched, user location information is determined, and a single-view image is calculated. This facilitates collaboration among different people in front of the 2D display of the imaging device (e.g., in front of a microscope).

[0091] A second aspect of this disclosure relates to a method for providing an image stream, comprising the following steps:

[0092] - Obtain multiple stereo images from a stereo imaging system, wherein the stereo images include a left image and a right image as stereo information; and

[0093] - Obtain user location information;

[0094] - For each image, determine the single-view image based on the left image, the right image, and the user's location information;

[0095] - Provides a single-view image for display.

[0096] The method according to the second aspect may include steps of any embodiments disclosed in connection with the first aspect of this disclosure and / or features and / or functions of one or more embodiments disclosed in connection with the following drawings.

[0097] Another aspect of this disclosure relates to a computing device including a processor configured to perform a method according to any of the foregoing aspects / embodiments.

[0098] Another aspect of this disclosure relates to a computer program product comprising instructions that, when executed by a computer system, cause the computer system to perform a method according to any of the foregoing aspects / embodiments.

[0099] Another aspect of this disclosure relates to a computer-readable medium comprising instructions that, when executed by a computer system, cause the computer system to perform a method according to any of the foregoing aspects / embodiments. Attached Figure Description

[0100] Further advantages and features arise from the following embodiments, some of which are illustrated in the accompanying drawings. The drawings are not always to scale. In particular, for clarity, the dimensions of various features may be enlarged or reduced. Therefore, the drawings are at least partially schematic.

[0101] Figure 1 Operation of the device according to an embodiment of this disclosure is illustrated.

[0102] Figure 2 Operation of the device according to an embodiment of this disclosure is illustrated.

[0103] Figure 3 Different operations of the device according to embodiments of this disclosure are illustrated.

[0104] Figure 4 A microscope system according to or used in an embodiment of this disclosure is shown.

[0105] Although some aspects have been described in the context of the apparatus (or system) of this disclosure, the description of these aspects also represents a description of the corresponding method, wherein the blocks or devices correspond to method steps or features of method steps.

[0106] Similarly, the aspects described in the context of method steps also represent descriptions of corresponding blocks, items, or features of a corresponding device or system, in particular, which may be distributed in different locations and configured to exchange information between different locations via corresponding communication devices.

[0107] Generally, the disclosure of the described methods also applies to corresponding devices (or apparatuses) for performing the methods, or to corresponding systems comprising one or more devices, and vice versa. For example, if specific method steps are described, the corresponding device may include features for performing the described method steps, even if such features are not explicitly described or represented in the figures. On the other hand, for example, if a specific device is described based on functional units, the corresponding method may include one or more steps for performing the described function, even if such steps are not explicitly described or represented in the figures. Similarly, corresponding device features or features for performing specific method steps may be provided for the system. Unless otherwise explicitly stated, features of the various exemplary aspects and embodiments described above or below may be combined.

[0108] As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items and may be abbreviated as “ / ”. Unless otherwise expressly stated, expressions such as “e.g.,” “as,” or “in particular” indicate that they may represent optional or alternative features that may be combined with all other (mandatory, optional, or alternative) features of aspects or embodiments of this disclosure.

[0109] In the following description, reference will be made to the accompanying drawings, which form part of this disclosure, illustrating specific aspects of this disclosure. Like reference numerals refer to the same or at least functionally or structurally similar features. Detailed Implementation

[0110] Figure 1 Operation 100 of a device according to an embodiment of this disclosure is illustrated, showing some basic features. A fluoroscopic medical and / or diagnostic imaging device (such as a microscope) includes a stereo camera comprising a left lens 120 and a right lens 122. The distance between the lenses is the baseline 124 of the stereo camera. Images of a sample 102 can be captured through these two lenses. The sample is represented by the letters "ABC". As can be seen from the cones of view of lenses 120 and 122, the fields of view of each camera partially overlap.

[0111] Based on the images from lenses 120 and 122, a virtual camera 130 is simulated. The virtual camera 130 can have an arbitrarily defined viewpoint on the baseline 124 of the stereo camera, and thus can obtain different viewing angles 134 of sample 102. The signal processing procedure is shown in the lower half of the figure. The left image 110 provided by the left lens 120 and the right image 112 provided by the right lens 122 are transmitted to a processor 132. This processor can be a computer equipped with appropriate software or a hardware-based processing device, such as an FPGA. Based on the processor, a single-view image 136 is determined and then provided to a two-dimensional display (e.g., a display of a fluoroscopic medical and / or diagnostic imaging device) for display. As shown, image 136 differs from images 110 and 112, as it includes a portion of both. As described above, the generation of image 136 is based on the information provided by images 110 and 112 from both lenses 120 and 122.

[0112] Figure 2 This illustrates a potential manner in which device operation 200, according to an embodiment of the present disclosure, is based, particularly on the viewpoint information 134 upon which the single-view image 136 is generated. A user 202 views a sample on a 2D screen 204. A camera 206, positioned at the top of the screen, captures the user's gaze. The camera's gaze information is transmitted to the device's processor 210 to determine the user's gaze angle 208, and optionally, the distance from the user to the screen. Based on this information, the processor 210 calculates the virtual viewpoint 134 (e.g., ...). Figure 1 (As shown in one example). This calculation process can be performed continuously or triggered by events, such as when the user's position changes beyond a predefined threshold. The virtual viewpoint 134 is then transmitted to the processor 132 to calculate the virtual single-view image 136. Processors 210 and 132 can also be implemented in a single processor.

[0113] Based on the aforementioned operation loop, consecutive left and right images 210 can be processed to determine the final image stream 236 transmitted to the 2D display 204. The image stream 236 provides the user 202 with a sample view dynamically adapted to her / his gaze 208.

[0114] In an embodiment not shown, the user's gaze is measured horizontally and vertically. Furthermore, a first stereo camera observes the sample horizontally, and a second camera observes the sample vertically. A first virtual image is then determined based on the horizontal portion of the user's gaze and the stereoscopic image from the horizontally oriented stereo camera (as described above). The first virtual image is then further adjusted based on the vertical portion of the user's gaze and the left and right images from the vertically aligned camera. The determination of the vertical adjustment is, in principle, performed in the same manner as the horizontal adjustment (e.g., as described above). Based on this two-step process, a two-dimensional user image (stream) that takes into account the user's vertical and horizontal head movements can be provided.

[0115] Figure 3 The operation 300 of the device according to an embodiment of this disclosure is illustrated, particularly the mapping of the measured gaze angle 208 to a point on the baseline of the stereo camera. As described in the preceding figures, sample 102 is captured by two lenses 120, 122 of a stereo camera with baselines of 124, 324. The gaze angle is measured from the display plane 310. In practice, this is represented by camera 206, which captures the user's viewing angle toward the 2D display 204.

[0116] The mapping from viewing angle 208 to baselines 124 and 324 can now be performed as follows: Viewing angles are mapped to the baselines such that positive viewing angle 312 is mapped to the left image / view (i.e., the leftmost part of the baseline), and negative viewing angle 312 is mapped to the right image / view (i.e., the rightmost part of the baseline). The zero-angle view 316 is mapped to the center between the two views (i.e., the center of the baseline). Intermediate viewing angles are mapped proportionally to baseline 124; for example, user 202's viewing angle 208 is mapped to point 330. This does not require the user's distance from the 2D display. The amounts of angles 312 and 314 can be selected to optimize 3D perception.

[0117] In a more complex embodiment, the distance from the user to the 2D display can be used to modulate angles 312, 314 during operation, as user perception may change with viewing distance, thus providing a better user experience.

[0118] Some embodiments relate to microscopes, which include combinations Figures 1 to 3 One or more of the systems described herein. Alternatively, the microscope may be a combination of Figures 1 to 3 One or more of the systems described herein are part of or connected to that system.

[0119] Figure 4 A schematic diagram of a system 400 configured to perform the methods described herein is shown. System 400 includes a microscope 410 having a 2D screen (not shown) and a computer system 420. Microscope 410 is configured to capture images and is connected to computer system 420. Computer system 420 is configured to perform at least a portion of the methods described herein. Computer system 420 may be configured to execute machine learning algorithms. Computer system 420 and microscope 410 may be separate entities, but may also be integrated into a common housing. Computer system 420 may be part of the central processing system of microscope 410, and / or computer system 420 may be part of a sub-component of microscope 410 (e.g., a sensor, actuator, camera, or illumination unit, etc.).

[0120] Computer system 420 may be a local computer device (e.g., a personal computer, laptop, tablet, or mobile phone) having one or more processors and one or more storage devices, or it may be a distributed computer system (e.g., a cloud computing system whose one or more processors and one or more storage devices are distributed across various locations, such as local clients and / or one or more remote server farms and / or data centers). Computer system 420 may include any circuitry or combination of circuitry. In one embodiment, computer system 420 may include one or more processors, which may be of any type. As used herein, a processor may refer to any type of computing circuitry, such as, but not limited to, a microprocessor, microcontroller, Complex Instruction Set Computing (CISC) microprocessor, Reduced Instruction Set Computing (RISC) microprocessor, Very Long Instruction Word (VLIW) microprocessor, graphics processor, digital signal processor (DSP), multi-core processor, field-programmable gate array (FPGA) for example, a microscope or microscope components (e.g., a camera), or any other type of processor or processing circuitry. Other types of circuitry that may be included in computer system 420 may be custom circuits, application-specific integrated circuits (ASICs), etc., such as one or more circuits (e.g., communication circuits) for wireless devices such as mobile phones, tablets, laptops, two-way radios, and similar electronic systems. Computer system 420 may include one or more storage devices, which may include one or more storage elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard disk drives, and / or one or more drives for processing removable media, such as compact disks (CDs), flash memory cards, digital video disks (DVDs), etc. Computer system 420 may also include a display device, one or more speakers, and a keyboard and / or controller, which may include a mouse, trackball, touchscreen, voice recognition device, or any other device that allows a system user to input and receive information from computer system 420.

[0121] Some or all of the method steps may be performed by (or using) hardware devices, such as processors, microprocessors, programmable computers, or electronic circuits. In some embodiments, one or more of the most important method steps may be performed by such devices.

[0122] Depending on certain implementation requirements, embodiments of the present invention can be implemented in hardware or software. This implementation can be performed using a non-transient storage medium (e.g., a digital storage medium such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or flash memory) storing electronically readable control signals that cooperate (or are capable of cooperating with) a programmable computer system to perform the corresponding methods. Therefore, the digital storage medium can be computer-readable.

[0123] Some embodiments of the invention include a data carrier having electronically readable control signals, which is capable of cooperating with a programmable computer system to perform one of the methods described herein.

[0124] Typically, embodiments of the present invention can be implemented as a computer program product having program code that, when run on a computer, can be operated to perform one of the methods. The program code may, for example, be stored on a machine-readable medium.

[0125] Other embodiments include a computer program for performing one of the methods described herein, the computer program being stored on a machine-readable medium.

[0126] In other words, therefore, an embodiment of the present invention is a computer program having program code that, when run on a computer, performs one of the methods described herein.

[0127] Therefore, another embodiment of the invention is a storage medium (or data carrier or computer-readable medium) comprising a computer program stored thereon, which, when executed by a processor, performs one of the methods described herein. Data carriers, digital storage media, or recording media are typically tangible and / or non-transient. Another embodiment of the invention is an apparatus as described herein, comprising a processor and a storage medium.

[0128] Therefore, another embodiment of the invention represents a data stream or signal sequence for performing one of the methods described herein. For example, the data stream or signal sequence may be configured to be transmitted via a data communication connection (e.g., via the Internet).

[0129] Another embodiment includes a processing means, such as a computer or a programmable logic device, configured or adapted to perform one of the methods described herein.

[0130] Another embodiment includes a computer on which a computer program for performing one of the methods described herein is installed.

[0131] Another embodiment of the invention includes an apparatus or system configured to transmit (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may, for example, include a file server for transmitting the computer program to the receiver.

[0132] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, the field-programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware device.

[0133] The term “and / or” as used herein includes any and all combinations of one or more of the associated listed items and may be abbreviated as “ / ”.

[0134] Although some aspects have been described in the context of the apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or feature of a method step. Similarly, aspects described in the context of a method step also represent a description of a corresponding block, item, or feature of the corresponding apparatus.

[0135] Reference tag list

[0136] Operation of 100 Imaging Processing Equipment

[0137] 102 samples

[0138] 110 left image

[0139] 112 Right Image

[0140] 120 left lens

[0141] 122 Right Lens

[0142] 124 baseline

[0143] 130 Virtual Camera

[0144] 132 processor

[0145] 134 Virtual Perspective

[0146] 136 single-view images

[0147] Operation of 200 Imaging Processing Equipment

[0148] 202 users

[0149] 204 screen

[0150] 206 camera

[0151] 208 gaze angle

[0152] 210 continuous original images

[0153] 236 image streams

[0154] Operation of 300 Imaging Processing Equipment

[0155] 310 monitor flat panel

[0156] Viewing angle of the right image 312

[0157] 316 Zero Observation Angle

[0158] 324 baseline

[0159] Mapping from 330° gaze angle to baseline

[0160] 400 system

[0161] 410 microscope

[0162] 420 Computer System

Claims

1. A device for providing an image stream, particularly a stereo microscope. Configured as: - Obtain multiple stereo images of sample (102) from a stereo imaging system, wherein, The stereoscopic image includes a left image (110) and a right image (112) that serve as stereoscopic information; and - Obtain user location information; - For each image, a single-view image is determined based on the left image, the right image and the user's location information (134). - Provides a single-view image for display.

2. The device according to the preceding claims, Includes one or both of the following: - Stereo camera; - A display for showing single-view images (134).

3. The device according to any one of the preceding claims, in, User location information includes one or more of the following: - The position of the user's head; - The position of the user's single or both eyes; -The position of the user's torso; - The position of the user's arm; - The location of the marker on the user.

4. The device according to any one of the preceding claims, This includes devices for observing a user's location, particularly one or more of the following: -camera; - Eye tracker; - Head tracker; - Body tracker; - Receivers of signals from active markers; - A device for detecting passive markers.

5. The device according to any one of the preceding claims, Configured as: - Based on user location information, especially based on the determination of whether the user has one or both eyes, perform user gaze angle detection.

6. The device according to any one of the preceding claims, in, User location information, especially gaze angle, is obtained under the following circumstances: -At constant intervals; -If the user's location changes.

7. The device according to any one of the preceding claims, Configured as: -Map the user's position to a virtual position (136) between the left image (110) and the image (112) of the stereoscopic image. - The single-view image (134) is determined based on the weighting of the virtual location (136).

8. The device according to any one of the preceding claims, Configured as: - Determine the disparity map based on the left image (110) and right image (112) of the stereo image; - Determine single-view images based on disparity maps (134).

9. The device according to any one of the preceding claims, Configured as: - Determine a first single-view image based on a first stereo image and a first disparity map of the first stereo image; and -The second single-view image is determined based on the following: --The second stereo image acquired after the first stereo image; --A second disparity map based on the second stereo image; and --First stereoscopic image and / or first parallax map.

10. The device according to any one of the preceding claims, in, The single-view image is determined based on one or more of the following (134): -AI; -Flight time information; - Structured light information.

11. The device according to any one of the preceding claims, Configured as: - Obtain updated information for a single-view image; and - Determine and / or provide single-view images based on the obtained updated information.

12. The device according to any one of the preceding claims, Configured as: - Indicates blind spots in a single-view image (134).

13. The device according to any one of the preceding claims, Configured as: Provide single-view images in single-view image streams, especially in video streams.

14. The device according to any one of the preceding claims, Configured as: - Switch between multiple users.

15. A method for providing an image stream, particularly for stereomicroscopy. Includes the following steps: - Obtain multiple stereo images from a stereo imaging system, wherein the stereo images include a left image and a right image as stereo information; and - Obtain user location information; - For each image, determine the single-view image based on the left image, the right image, and the user's location information; - Provides a single-view image for display.

Images