Data processing device and computer-implemented method for motion-dependent image processing in a medical observation device and medical observation device
The data processing device improves low-light imaging in medical devices by identifying still image segments and combining pixel values across multiple frames, addressing noise and blur issues in low-light conditions.
Patent Information
- Application Number
- JP2025507463
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2022-08-10
- Filing Date
- 2023-08-10
- Publication Date
- 2025-08-15
AI Technical Summary
Low-light imaging in medical observation devices, such as microscopes and endoscopes, suffers from excessive noise and motion blur when capturing images at video frame rates, particularly in fluorescence and surgical microscopy.
A data processing device that derives an input set of images from a time sequence, applies a motion detection scheme to identify still image segments, and determines output image segments by combining intensity values from corresponding pixels across multiple input images, adjusting camera settings dynamically to improve image quality.
Enhances image quality by reducing noise and motion blur, allowing for better image acquisition with less sensitive and potentially cheaper image acquisition devices, while maintaining real-time adjustments.
Smart Images
Figure 2025526723000001_ABST
Abstract
Description
[Technical Field]
[0001] The present invention relates to a data processing device for motion-dependent image processing in a medical observation device such as a microscope or endoscope, a medical observation device such as a microscope or endoscope, a computer-implemented method for motion-dependent image processing of medical upset-related input images and a corresponding computer-readable medium or computer program product. [Background technology]
[0002] In low-light imaging applications, such as fluorescence and surgical microscopy, the sensitivity of the image acquisition device, e.g., a camera, can be considered a bottleneck in the imaging process. In low-light imaging scenarios, the fluorescence intensity may be too weak to provide a clear picture of the imaged scene, especially when images are captured at video frame rates. As the camera gain increases, image noise degrades image quality.
[0003] When the object being observed is stationary, i.e., not moving, one possible approach to improving image quality is to apply an increased integration time at the expense of a reduced frame rate. This approach can provide a significant improvement in image quality for stationary objects, but suffers from motion blur when the object is moving.
[0004] Thus, in the art, low light imaging at video frame rates suffers from images that are excessively noisy or exhibit motion blur. Summary of the Invention [Problem to be solved by the invention]
[0005] Accordingly, one object of the present invention is to improve the efficiency of the imaging process, for example by providing substantially better image quality with a given image acquisition device, or by providing the same image quality with a less sensitive and / or cheaper and / or smaller image acquisition device. [Means for solving the problem]
[0006] This object is solved by the present invention for a data processing device mentioned in the beginning in that the data processing device is configured to: derive an input set of input images from a plurality of input images, the plurality of input images being part of one or more time sequences of input images, all of the input images of the input set being from the same time sequence of input images, each input image comprising input pixels, each input pixel comprising an intensity value; determine still input image segments within the input images of the input set by applying a motion detection scheme to the input set, the still input image segments being representative of still image content in the input set; determine still output image segments within an output image, the still output image segments having the same position in the output image as the still input image segment has in one of the input images of the input set; and determine intensity values of output pixels of the output image within the still output image segments from combinations of intensity values combining corresponding input pixels from different input images of the subset of input images, the corresponding pixels being within the still input image segments.
[0007] This has the advantage that the imaging process can be improved as the camera settings can be dynamically adjusted in real time.
[0008] The medical observation device mentioned in the introduction achieves the above-mentioned object by comprising a data processing device according to the invention and an input camera for recording one of one or more time sequences. The medical observation device also has the advantage of an improved imaging process. The medical observation device may be, for example, a microscope or an endoscope.
[0009] The computer-implemented method for motion-dependent image processing of medical observation input images as mentioned at the beginning solves the above problem by including the method steps of: deriving an input set of input images from a plurality of input images, the plurality of input images being part of one or more time sequences of input images, all of the input images of the input set being from the same time sequence of input images, each input image comprising input pixels, each input pixel comprising an intensity value; determining still input image segments within the input images of the input set by applying a motion detection scheme to the input set, the still input image segments representing still image content within the input set; determining still output image segments within an output image, the still output image segments having the same position within the output image as the still input image segment has in one of the input images of the input set; and determining intensity values of output pixels of the output image within the still output image segments from combinations of intensity values of corresponding input pixels from different input images of a subset of input images, the corresponding pixels being within the still input image segments.
[0010] The computer-implemented method of the present invention also has the advantage of improved image processing.
[0011] The computer readable medium or computer program product of the present invention solves the above problem by containing instructions that, when executed by a computer, cause the computer to carry out the computer-implemented method according to the present invention.
[0012] Further improvements to the data processing device of the present invention, the medical observation device of the present invention, the computer-implemented method and the computer-readable medium or computer program product of the present invention are described below. Each of the described embodiments is advantageous in itself, and additional features may be combined with each other in any way or may be omitted.
[0013] The improvements and advantages thereof described with respect to the data processing device or medical observation device also relate to the computer-implemented method or computer-readable medium or computer program product. Similarly, the improvements and advantages mentioned with respect to the method for the computer-readable medium or computer program product also apply to the data processing device or medical observation device according to the invention. Features described with respect to the data processing device also relate to the medical observation device.
[0014] A static input image segment may correspond to an entire input image, in other words, the entire input image of an input set of input images from a plurality of input images is static.
[0015] Thus, the static output image segment in the output image also corresponds to the entire output image. In this embodiment, all pixels of the output image are therefore treated equally, i.e., by determining the intensity values of all output pixels in the static output image from a combination of intensity values that combine with each other corresponding input pixels from different input images of the subset of input images, and corresponding pixels in the static input image.
[0016] The intensity value of an output pixel of the output image may then be determined from a combination of intensity values of corresponding input pixels from different input images of the subset of input images, where the corresponding pixels are in the input image corresponding to the static input image segment.
[0017] A still output image segment may have the same position in the output image as the still input image segment had in the most recent of the input images in the input set.
[0018] A stationary segment may be defined in particular by pixels whose intensity values vary less than a predetermined variation threshold over a predetermined number of input images.
[0019] A still input image segment may be determined for each input image in the input set. In particular embodiments, each still input image segment may correspond to the entire corresponding input image in the input set.
[0020] A still input image segment may include contiguous and / or non-contiguous clusters of pixels. In other words, a still input image segment may include one cluster of adjacent pixels or two or more such clusters of adjacent pixels. Thus, still input image segments may be separated from one another by non-static input image segments.
[0021] The static input image segments may, in particular, have the same position, shape, orientation and / or size in each of the input images of the input set.
[0022] The static input image segments can represent image patterns. The same image pattern may be processed in each of the input images. Such image patterns may be determined by pattern recognition methods such as k-means clustering, machine learning, object recognition, or other known processes.
[0023] In one embodiment, the image pattern may be at different locations in at least some of the input images. This may be the case, for example, when the sample is scanned slowly and the image pattern may be moved at the scanning speed. Relative movement may also be visible within the field of view. Such image patterns represent stationary input image segments.
[0024] To prevent any erroneous determination of still image content, each still image content may represent image content whose motion is below a predetermined motion threshold, so that vibrations that may occur during the image formation process will not erroneously determine the still input image segment.
[0025] The motion detection scheme mentioned above can be any known motion detection or motion segmentation scheme, such as moving object detection, frame differencing, time differencing, or similar schemes.
[0026] Corresponding pixels are pixels at the same location in the static input image segment. Advantageously, the output pixels of the output image are obtained pixel by pixel. In another embodiment, clusters of output pixels of the output image are obtained from clusters of pixels of the input image.
[0027] In some embodiments, there may be two or more time sequences of input images. Each of the two or more time sequences may be generated by a different camera. In other embodiments, some of the time sequences may be generated by a first camera and other time sequences may be generated by a second camera. A camera represents any possible image acquisition device.
[0028] Accordingly, a corresponding embodiment of the medical observation device of the present invention further comprises a second camera for recording a second time sequence of the one or more time sequences, the second camera being capable of generating a second input image.
[0029] In one advantageous embodiment of the medical observation device, the input camera may be a white light camera and / or the second camera may be a fluorescence camera configured to record the emission spectrum of at least one fluorophore.
[0030] This has the advantage that information can be extracted from different spectral regions that reveal different information, which can then be efficiently combined with each other.
[0031] The data processing device may provide a threshold parameter that indicates the size of the subset. This threshold parameter may be provided as a default value or may be changed by a user of the data processing device. The threshold parameter may be received by the data processing device or, in another embodiment, stored in a storage unit. This has the advantage that a single pixel or a cluster of two, three, four, or generally a small number of pixels that exhibit some noise in the input image data is determined to be non-stationary. Any number of pixels may be covered by the threshold parameter. This can suppress the effects of pixel noise and improve image quality.
[0032] In another advantageous embodiment, the data processing device is further configured to determine intensity values of output pixels of an output image outside the static output image segment from one or more corresponding input pixels of a subset of input images outside the static input image segment, wherein the number of input pixels used to determine the intensity values of the output pixels outside the static output image segment is less than the number of input pixels used to determine the intensity values of the output pixels within the static output image segment. A movable object may be detected outside the static input image segment. In a sequence of multiple input images of the input set, this moved object is located at different positions in each input image. Different input pixels of each input image are used to determine the intensity values of the output pixels outside the static output image segment. Different input pixels of adjacent input images can describe the trajectory of a moving element represented by the output pixel in the output image. Therefore, multiple input pixels can determine the intensity value of one output pixel. Therefore, the number of input pixels is greater than the number of output pixels.
[0033] In a further advantageous embodiment of the data processing device, the data processing device is further configured to determine intensity values of output pixels outside the static output image segment as the intensity values of corresponding input pixels of the latest input image of the subset of input images. This has the advantage that no image blurring can occur. Thus, intensity values of non-static image content are not calculated from multiple input images, but only from the latest input image.
[0034] The data processing device may be further improved in that the number of input pixels used to determine the intensity value of an output pixel in a static output image segment is in the range of 3 to 9. Thus, the number of input pixels used may be 3, 4, 5, 6, 7, 8, or 9. In other embodiments, other numbers of input pixels are contemplated, for example, 1 or 2, or 10 or more. The number of input pixels can be changed depending on the intensity values of the input pixels. By changing the number of input pixels for determining the intensity value of an output pixel, the intensity value of the static output image segment can be increased or decreased.
[0035] The data processing device may be further configured to: derive a second set of second input images from the plurality of input images, the second set of second input images all from the same time sequence, the time sequence of the second set of input images being different from the time sequence of the input set, each second input image comprising a second input pixel, each second input pixel comprising an intensity value; determine a second still output image segment in each of the second input images of the second set, the second still output image segment being in the second output image and having the same position in the second output image as the still input image segment had in one of the input images of the input set; and determine an intensity value of an output pixel in the second still output image segment of the second output image from a combination of intensity values combining corresponding input pixels from different input images of a subset of the second set, the corresponding pixel being in the still input image segment.
[0036] Also, for a second set of second input images, the second still output image segment may correspond to the entire second input image, i.e., the entire image content of the second input image is determined to be still.
[0037] The second output image may be calculated from a subset of the second set of second input images, and in particular the size of the subset may be controlled by a user-controllable parameter, which may be input by a user and stored in a memory unit.
[0038] The input images of the input set and the second input images of the second set may be aligned images, and at least some of the images of the input set may have been recorded simultaneously with or as the second input images of the second set.
[0039] A further advantageous embodiment of the data processing device is further configured to align an input image of the input set with a second input image of the second set. When different input images are aligned, they can be distinguished from one another.
[0040] Furthermore, the input image and the second input image may be recorded simultaneously and / or synchronously. The input image and the second input image may be recorded at the same time or within a time span that is preferably less than the flicker fusion threshold or less than 50 ms.
[0041] The data processing device may be further improved by being further configured to determine a size of the still input image segment and, when the determined size of the still input image segment exceeds a combination threshold, determine from the combination an intensity value of an output pixel of the output image in the still output image segment.
[0042] This has the advantage that image content smaller than the combination threshold, i.e., image content extending over fewer than a certain number of pixels, is not processed by the data processing device, which may result in faster image processing. Thus, the size of the still input image segment may include a count of pixels.
[0043] In another advantageous embodiment, the data processing device is further configured to determine a standard deviation of output pixels in the static output image segment from corresponding pixels of different input images in the static input image segment, which has the advantage that it is possible to reject intensity values from the combined result of combining corresponding pixels when the intensity values are outside a threshold based on the determined standard deviation.
[0044] The data processing device may be further configured to calculate an output image, a static input image segment, or a static output image segment, and provide the output image to a display device. Alternatively or additionally, the output image may be provided to a recording device, which may in particular be a digital recording device that allows further processing of the recorded image, for example storing it in a memory.
[0045] The present invention will be described below with reference to the accompanying drawings. The illustrated embodiments are exemplary and not limiting of the present invention. Features shown in different embodiments may be combined with each other in any way or may be omitted. Furthermore, features described with reference to a data processing device may be applied to a medical observation device as well as a computer-implemented method. Accordingly, features described with reference to a method may be transferred to the corresponding apparatus. [Brief explanation of the drawings]
[0046] [Figure 1] FIG. 1 is a schematic diagram of a medical observation device. [Figure 2] FIG. 1 is a flow diagram of a computer-implemented method. [Figure 3] 1 is a flowchart of a method for evaluating an input image sequence. [Figure 4a] 10 is a flowchart for determining a still image area. [Figure 4b] 10 is a flowchart for determining a still image area. [Figure 5] 1 is a flowchart of a method for determining a still image region. [Figure 6] 10 is another schematic flow chart for determining a still image region. DETAILED DESCRIPTION OF THE INVENTION
[0047] An example of one embodiment of the present invention will be described with reference to FIG.
[0048] 1 schematically illustrates a medical observation device 100. The medical observation device 100 may be a fluorescence-based optical imaging device. The medical observation device 100 may also be realized as a microscope or an endoscope. Although the medical observation device in FIG. 1 is only schematically illustrated as a microscope, the following description also applies to an endoscope.
[0049] The medical observation device 100 can be used for surgical or laboratory assistance. A user can use the medical observation device 100 to observe an object or specimen 110 in an object scene 120.
[0050] The medical observation device 100 may include a camera 114. The camera 114 may record images of an object scene. The camera 114 may be used as a video camera that records and provides an image sequence 132. The data processing device 100 may retrieve the image sequence from the camera as the input image sequence 132. The data processing device 100 may be integrated with the medical observation device 100. Alternatively, the data processing device 100 may be coupled to the medical observation device 100.
[0051] The medical observation device 100 may optionally comprise a secondary camera 116 in addition to the (primary) camera 114. The secondary camera may simultaneously observe the object scene 120 to generate another image sequence 152. The secondary camera 116 may therefore be embodied as a video camera that records the object scene 120 and provides the image sequence 152 to the data processing device 100. Any or all of the cameras 114 and 150 are configured to generate an image sequence of chronologically ordered images representing the object scene 120 observed by the medical observation device 100. Thus, according to an embodiment in which there is only one camera in the medical observation device 100, the data processing device 100 retrieves an input image sequence 132 of chronologically ordered primary input images.
[0052] According to another embodiment, when two cameras, such as the primary camera 114 and the secondary camera 116, simultaneously observe the object scene 120, the data processing device 100 is configured to retrieve a primary input image sequence 132 of chronologically ordered primary input images representing the object scene 120 and a secondary input image sequence 152 of chronologically ordered secondary input images representing the object scene 120. As described above, the camera 114 may be embodied as a fluorescence camera. Alternatively, it may be embodied as a white-light camera. According to an embodiment comprising two cameras, the primary camera may be embodied as a white-light camera, and the secondary camera may be embodied as a fluorescence camera, the difference between the fluorescence camera and the white-light camera being primarily that the fluorescence camera is configured to observe the object scene 120 through a passband filter (not shown in FIG. 1 ) integrated into the camera or part of the medical observation device 100. The passband filter may be selective to the emission light spectrum of fluorophores present in or added to the specimen 110 within the object scene 120.
[0053] Because a passband filter can block any light components outside the passband spectrum, images captured by a fluorescence camera are affected by low light levels. This drawback is even more pronounced when the camera operates with a very short exposure time. This is the case for any camera operating in video mode, which generates an image sequence of chronologically ordered images. Whether the camera is a white-light camera, a fluorescence camera, or any camera in the medical observation device 100, the frame rate and exposure time settings can be dynamically adjusted by the data processing device 100. For example, the data processing device 100 can increase the frame rate of the camera 114 and / or the secondary camera 116. Increasing the frame rate can be useful for eliminating motion blur in a single image of an image sequence, which can occur when the specimen 110 moves faster than the exposure time of the observation camera.
[0054] Furthermore, the primary camera 114 and the secondary camera 116 of the medical observation device 100 may be positioned, adjusted, or calibrated to produce images that can be overlaid. In other words, an image of a primary input image sequence from a first camera can be overlaid with a secondary image of a secondary input image sequence. In this case, the primary input image of the primary camera 114 and the secondary input image of the secondary camera 116 may have the same aspect ratio, the same pixel dimensions, the same image size and ratio, and / or a predetermined, known transformation relationship that allows input pixels of an image from the first camera to be mapped to other input pixels of an image from the second camera. In other words, if the primary input image of the first camera includes a primary input pixel, the known mapping relationship allows the primary input pixel to be mapped to a secondary input pixel of the secondary input image of the secondary camera.
[0055] The medical observation device may further include an output device 112. The output device 112 can receive an output image 170 (106) from the data processing device 100. The output device 112 can be a display, smart glasses, a projector, a holographic display, etc.
[0056] The output image 170 (106) comprises output pixels, each of which comprises at least one intensity value. The data processing device 100 is configured to generate and / or output the output image 170 (106).
[0057] According to one embodiment, the data processing device 100 can retrieve an input image sequence 132 from the camera 114. The input image sequence can include a chronologically ordered sequence of input images representing an object scene 120 observed by the medical observation device 100. The data processing device 100 can then process the retrieved input image sequence 132 to output an output image 170 (106). In particular, the data processing device 100 can evaluate the image sequence 132 to determine motion within the object scene 120. If the image sequence 132 does not indicate the presence of motion within the observed object scene 120, the data processing device 100 can digitally enhance the most recent image of the image sequence 132 and provide the enhanced output image 170 (106) to the output device 112. For example, the data processing device 100 can stack the most recent image of the retrieved image sequence 132 with a previous image in the image sequence. For example, multiple images of the input image sequence 132 can be averaged to calculate the output image 170 (106). Alternatively, the output image 106 may be computed as a median filtered image from the input image sequence 132. The output image 170 (106) may be computed by signal clipping from the extracted input image sequence 132.
[0058] According to another embodiment, when two cameras, i.e., the primary camera 114 and the secondary camera 116, are simultaneously observing the object scene 120 instead of only one camera, the data processing device 100 can analyze the primary input image sequence 132 to determine whether motion is present in the simultaneously observed object scene 120. If the data processing device determines that motion is not present in the observed object scene 120, the data processing device 100 can alternatively calculate the output image 170 (106) from the secondary input image sequence in the same way as it is calculated when only one camera is used. This embodiment is particularly useful when light is very abundant on one imaging channel, such as a white light camera, and the amount of light received by this camera is very low on another imaging channel, such as a fluorescence camera.
[0059] However, the data processing device is not limited to generating only one output image 170 (106) when operating the two cameras. Thus, the data processing device 100 can also generate and / or output secondary output images that are calculated from the primary input image sequence by stacking or combining the images in the sequence.
[0060] In response to determining that there is no actual motion in the observed object scene, the data processing device 100 may stack or combine the most recent 3, 4, 5, 6, 7, 8, 9, 10, 11, 12 (or any other number) images of the input image sequence.
[0061] If the data processing device 100 determines that motion is present in the object scene 120, the data processing device 100 may adjust the frame rate setting of the camera 114 to account for motion blur artifacts. If the medical observation device 100 is operated with two cameras simultaneously observing the object scene 120, the data processing device 100 may adjust the frame rate setting of both cameras. In either case, the data processing device may increase the frame rate setting of the primary camera 114 and / or the secondary camera 116.
[0062] Furthermore, in response to determining that motion is present in the observed object scene 120, the data processing device can determine a still image region or a still image segment in the input image sequence. Hereinafter, the terms still image region and still image segment are used synonymously. A still image region may be an (entire) image or an image mask, and indicates input pixels in the input images of the input image sequence that do not substantially change in the input image sequence. In other words, these input pixels indicate or represent a still portion of the recorded object scene 120. This still portion may be, for example, the background behind a moving object.
[0063] The data processing device 100 can determine still image regions by evaluating or analyzing the input image sequence to determine motion within the input image sequence. For example, if the result of evaluating the input image sequence is a mask of input pixels that changed throughout the input image sequence, the complement of these input pixels represents the still image regions. Alternatively, the data processing device may directly generate the still image regions as a result of evaluating the input image sequence. In either case, the still image regions indicate or represent the locations of input pixels that are not subject to variation within the images of the input image sequence being evaluated or analyzed.
[0064] This image may be, for example, the most recent image in the input image sequence. The data processing device 100 can generate and / or output the output image 170 (106) by assembling or composing different regions of the output image depending on the location of the output pixel within the output image. If the location of the output pixel is in a static image region, the intensity value of the output pixel is calculated from a combination of intensity values from a subset of the predetermined input image sequence. If only one camera is operating, the predetermined input image sequence is input image sequence 132.
[0065] If two cameras are used, alternatively or additionally, the predetermined input image sequence may be a secondary input image sequence taken from a secondary camera simultaneously observing the object scene 120. In combining, input pixels of different input images of a subset that match the output pixel with respect to their position in the still image region are combined. This combining may be calculated as sigma matching, averaging, or median filtering. For completeness, it is noted that if two cameras are used, the data processing device calculates the output pixel combination by combining secondary input pixels of different secondary input images of a subset that match the output pixel with respect to their position in the still image region.
[0066] In summary, the intensity values of output pixels of the output image, which are within a static image region, are calculated from corresponding input pixels of different input images from a subset of the input image sequence. The density of the subset can be between 2 and 12. The data processing device 100 can generate a subset of the input image sequence by filtering or omitting input images from the input image sequence that have a low signal-to-noise ratio. This can further improve the image characteristics in the output image 170 (106). The data processing device 100 can calculate dynamic regions of the output image 170 (106), i.e., regions of output pixels outside the static image region, by employing the intensity values of each input pixel in the most recent input image that matches each output pixel in the output image. This allows a static background to be output with a high signal-to-noise ratio in the static image region of the output image, while at the same time, the dynamic image region of the output image, i.e., the region complementary to the static image region, is not affected by motion blur artifacts. This improves the overall imaging characteristics of the medical observation device 100.
[0067] If the medical observation device 100 operates with two cameras, the data processing device 100 may alternatively or additionally similarly generate and / or output the output image 170 (106) from a combination of the secondary input image sequence. This embodiment is particularly useful when a fluorescence camera observes the object scene 120 simultaneously with the white-light camera 114. In this case, the white-light camera 114 provides the primary input image sequence to the data processing device 100 for assessing whether motion is present in the observed object scene 120. If motion is present, the data processing device 100 uses this information to provide an enhanced output image 170 (106) from the secondary input image sequence of the fluorescence camera 116. In particular, the data processing device 100 can calculate intensity values for output pixels that lie within static image regions determined from the primary input image sequence of the white-light camera 114.
[0068] FIG. 2 shows a flow diagram of a computer-implemented method for image processing of medical observation input images representing an object scene observed by a medical observation device. A data processing device 100 may be configured to perform the computer-implemented method 200. In step 210, an input image sequence of time-ordered input images representing the object scene observed by the medical observation device 100 may be retrieved. Furthermore, in step 215, a secondary input image sequence of time-ordered secondary input images representing the object scene observed by a second camera of the medical observation device may be received in addition to the input image sequence, which may also be referred to as the primary input image sequence. In this alternative embodiment, the input images of the primary input image sequence are also referred to as primary input images. The input images include input pixels, each of which includes an intensity value. Similarly, the input pixels of the primary input images will hereinafter be referred to as primary input pixels, and the input pixels of the secondary input images will correspondingly be referred to as secondary input pixels. In either case, each input pixel includes an intensity value. In step 220, the input image sequence or the primary input image sequence may be evaluated to determine motion within the observed object scene. If the result of the evaluation of the image sequence is negative, i.e., if no motion is determined in the evaluated image sequence 132, the output image 170 may be generated by combining the images of the image sequence 132. For example, the output image 170 may be the result of a stacking operation based on the image sequence or the primary input image sequence. For example, in either case, a subset of the input image sequence may be used to generate the output image. If the signal-to-noise ratio in the subset is below a predetermined threshold, one or more input images or primary input images may be filtered from the input image sequence.
[0069] In step 230, an output image may be generated by stacking the input image sequence or the primary input image sequence, respectively.
[0070] In step 240, still image regions may be determined. In particular, still image regions may be determined from the results of evaluating the input image sequence or the primary input image sequence.
[0071] Optionally, in step 250, the frame rate setting of the medical observation device 100 can be dynamically adjusted. For example, the frame rate setting can be increased. In this case, the amount of time that elapses between successive images in the input image sequence or primary input image sequence is shortened. This allows for reducing motion blur artifacts. The frame rate setting can be increased for the primary camera 114 or for both cameras simultaneously, i.e., the secondary camera 116.
[0072] In step 260, intensity values for output pixels generated for the output image may be calculated. The intensity values may be calculated from the input image sequence or a subset of the input image sequence, the primary input image sequence or a subset thereof, and / or the secondary input image sequence or a subset thereof. Intensity values for output pixels in the determined still image region may be calculated, where the intensity values are calculated from a combination of intensity values of a subset of a predetermined input image sequence. The predetermined input image sequence may be indicated by an operational setting of the medical observation device 100.
[0073] If the medical observation device is operated with only one camera, the predetermined input image sequence may not be indicated by the operating settings and is identical to the input image sequence. If the medical observation device 100 is operated with two cameras, the predetermined input image sequence may be a secondary input image sequence, and thus the intensity values of the output pixels in the determined still image region are calculated from a combination of intensity values of a subset of the secondary input image sequence.
[0074] In either case, in the combination of intensity values, input pixels of different input images of the subsets that match the output pixel with respect to location within the still image region are combined. For example, primary input pixels of different primary input images of the subset of a given input image sequence that match the output pixel with respect to location within the still image region are combined. According to another example, in the combination, secondary input pixels of different secondary input images of the subset of a secondary input image sequence that match the output pixel with respect to location and still image region are combined.
[0075] The locations of the output pixels are associated with the still image regions by one or more indices or addresses associated with the output image. Additionally, output pixels of the output image that are outside the still image regions may be calculated from the input image sequence or a secondary input image sequence.
[0076] The intensity value of an input pixel or a secondary input pixel may be taken as the intensity value of the output pixel, and the position of the input pixel corresponds to the position of the output pixel within the still image region. Alternatively, the intensity value of the output pixel may be calculated from a combination of input pixels that correspond to the output pixel within the sub-region. For example, the intensity value of the output pixel is the average or median of the neighboring input pixels of the input pixel that corresponds to the output pixel in terms of position, and the number of neighboring input pixels within the same input image is defined by a configurable parameter. For example, the configurable parameter for the size of neighboring pixels to be considered to calculate the output pixel may be dynamically adjusted in response to adjustments to the frame rate setting. This may compensate if the frame rate setting used introduces too much noise into the recorded image.
[0077] In step 270, output image 170 may be output. For example, the output image may be output to output device 112. Output device 112 may receive output image 170 from data processing device 100. Similarly, two separate output images may be provided to output device 112, one for each camera simultaneously observing object scene 120.
[0078] 3 shows a schematic flow diagram of a method for evaluating a sequence of input images to determine motion in an object scene. In the following, the method is described with respect to input pixels. However, the same method applies without limitation to primary or secondary input pixels.
[0079] Because an input image, such as a primary input image or a secondary input image, is a digital image that has been or can be processed by a computer, the location of any input pixel within the input image can be represented by a pair of numbers or indices. Such a pair of numbers is also known to those skilled in the art as pixel coordinates. The pair of numbers represents the location of the input pixel in a two-dimensional area.
[0080] The method 300 for estimating motion in an object scene can be performed in parallel or sequential manner, where step 310 involves setting a first input pixel from which to begin the calculation.
[0081] In step 320, the variance or standard deviation of a given or set input pixel is calculated. Since variance is defined as the square of the standard deviation, preferably, only the absolute value of the standard deviation is calculated. To calculate the above-mentioned statistical measure, the intensity values of the input pixel in the input image sequence are used, with each input pixel located at the same position in different input images of the input image sequence. Since the position of the given or set input pixel is converted to a position in the object scene, the input image sequence including the input image provides a time series of intensity values at the given or set position of the input pixel. In step 330, the calculated measure, such as the standard deviation, variance, or absolute value of the standard deviation, may be compared with a predetermined threshold. The statistical measure may be compared to determine whether it exceeds the predetermined threshold. If the calculated statistical measure is low, it may indicate the absence of motion at the given or set input pixel.
[0082] In step 340, any input pixel in a given input image, such as the most recent input image in the sequence of evaluated input images, may be marked or labeled. For example, the input pixel may be marked as a moving pixel. Alternatively, instead of marking a moving pixel, any pixel that does not exceed a predetermined threshold may be marked. In this case, they may be marked as a stationary pixel.
[0083] In step 350, the above steps 310-340 may be repeated for all input pixels in the case of sequential processing.
[0084] In step 360, the total number of marked input pixels may be determined.
[0085] In step 370, the sum may be compared to another predetermined threshold t2 that indicates motion in the object scene.
[0086] If the sum exceeds another predetermined threshold t2, motion within the object scene can be determined, in other words, variations in intensity values in subsequent input images indicate motion in the object scene.
[0087] Optionally, in step 380, still image regions can be determined from the result of marking the input pixels. For example, if the input pixels are marked as moving pixels, a complementary image mask can be determined for the marked input pixels. This mask indicates the locations of input pixels where no motion is present in the object scene, and therefore represents still image regions. Alternatively, if still pixels are marked in step 340, the result can be directly taken as the still image region.
[0088] 4a and 4b show a flow diagram for determining still image regions. In step 410a, FIG. 4a shows that a statistical measure for each input pixel can be calculated for each pixel. For example, the statistical measure can be the absolute value of the variance and deviation or the standard deviation. The statistical measure M can be calculated along the same input pixel in the input image sequence.
[0089] In step 420a, the calculated statistical measure M for each input pixel may be compared to a given threshold t3. If it is determined that the statistical measure M exceeds the predetermined threshold t3, the input pixel may be marked as a moving pixel or a dynamic object region in step 425a. If it is determined that the statistical measure M is less than the predetermined threshold t3, the input pixel may be marked as a stationary pixel or a stationary object region in step 426a.
[0090] In step 430a, a complementary mask for the marked pixels may be determined, which represents input pixels that do not represent moving pixels, in other words, static input pixels, and may be used as the static image region.
[0091] In Fig. 4b, the statistical measure M is calculated in step 410b in the same way as it was calculated in step 410a of Fig. 4a. Similarly, in step 420b, the calculated statistical measure M is compared with a predetermined threshold t3. However, in contrast to the method of Fig. 4a, in step 426b, only input pixels for which the calculated statistical measure M is below the predetermined threshold t3 are marked. This allows for the direct generation of a mask representing the static image regions in step 430b.
[0092] Figure 5 shows a schematic flow chart of a method for determining still image regions. An exemplary input image sequence 510 is shown at the top of Figure 5. The input image sequence of chronologically ordered input images 502-506 may be processed in step 520 by a motion detection scheme, a motion estimation scheme, a segmentation analysis scheme, etc. Input images 503 and 504 are not shown.
[0093] Step 520 may be part of methods 300, 400a, and 400b described above. As a result of step 520, a still image region may be obtained. For example, it may be obtained from the result of a motion detection scheme, a motion estimation scheme, or a segmentation analysis scheme. For example, any or all of the above schemes may generate a mask that segments an input image of an input image sequence into at least two parts, one part or segment representing a still image region and another part representing a dynamic image region.
[0094] The dynamic image region can represent moving pixels. At the bottom of Figure 5, an exemplary mask 530 is shown that includes a dynamic image region 540 (shaded) and a static image region 550. In the given example, the static image region 550 is discontinuous in that the central input pixel is labeled by a moving pixel in the dynamic image area 540.
[0095] 6 shows another schematic flow chart for determining still image regions starting from another image sequence, in which chronologically ordered input images 602-606 represent an object scene as observed by the camera of the medical observation device 100.
[0096] In the given example, a structure 601 in the center of an image or object scene moves from one direction to another through the center of the image. In step 620, static image regions may be determined from analyzing the input image sequence 610. In particular, as a result of the analysis or evaluation of the input image sequence 610, an image mask 630 may be generated, which also includes pixels, and depending on whether the pixel indicates motion within the object scene, the intensity value of the pixel is set to a value that indicates motion or that does not indicate motion.
[0097] In the illustrated image mask 630, the static image region 650 is contiguous and completely surrounds the dynamic image region 640 in the center.
Claims
1. A data processing device (100) for motion-dependent image processing in a medical observation device such as a microscope or endoscope, comprising: - said data processing device (100) is configured to derive an input set (102a) of input images from a plurality of input images (102), said plurality of input images being part of one or more time sequences of input images; all of the input images in the input set (102a) are from the same time sequence of input images; Each input image includes input pixels, each input pixel including an intensity value; the data processing device (100) is configured to determine still input image segments (104) within the input images of the input set (102a) by applying a motion detection scheme to the input set (102a), the still input image segments (104) representing still image content within the input set (102a); - the data processing device (100) is configured to determine a still output image segment (108) in an output image (106), the still output image segment (108) having the same position in the output image (106) as the still input image segment (104) has in one of the input images of the input set (102a); - the data processing device (100) is configured to determine intensity values of output pixels of the output image (106) in the static output image segment (108) from a combination of intensity values of corresponding input pixels from different input images of a subset of input images, the corresponding pixels being in the static input image segment (104); A data processing device (100).
2. The data processing device (100) - further configured to determine intensity values of output pixels (110) of said output image (106) outside said static output image segment (108) from one or more corresponding input pixels of said subset of said input image outside said static input image segment; the number of input pixels used to determine the intensity values of the output pixels (110) outside the still output image segment (108) is less than the number of input pixels used to determine the intensity values of output pixels (110) within the still output image segment (108); The data processing device (100) of claim 1.
3. The data processing device (100) - further configured to determine the intensity values of output pixels (110) outside the static output image segment (108) as the intensity values of corresponding input pixels of a most recent input image of the subset of input images, The data processing device (100) of claim 2.
4. the number of input pixels used to determine the intensity value of an output pixel in the still output image segment (108) is in the range of 3 to 9; A data processing device (100) according to claim 2 or 3.
5. The data processing device (100) - configured to derive a second set of second input images (102b) from said plurality of input images; the second set (102b) of second input images are all from the same time sequence, the time sequence of the second set (102b) of input images being different from the time sequence of the input set (102a); each second input image includes second input pixels, each second input pixel including an intensity value; the data processing device (100) is configured to determine a second still output image segment in each of the second input images of the second set (102b), the second still output image segment being in a second output image, the still input image segment (104) having the same position in the second output image as it has in one of the input images of the input set (102a); the data processing device (100) is configured to determine an intensity value of an output pixel in the second still output image segment of the second output image from the combination of intensity values combining corresponding input pixels from different input images of a subset of the second set (102b), the corresponding pixel being in the still input image segment (104); A data processing device (100) according to any one of claims 1 to 4.
6. The data processing device (100) - further configured to align said input images of said input set with said second input images of said second set; A data processing device (100) according to any one of claims 1 to 5.
7. the input image and the second input image are recorded simultaneously and / or synchronously; The data processing device (100) of claim 6.
8. The data processing device (100) - determining the size of said still input image segment; - determining the intensity values of the output pixels of the output image (106) in the still output image segment (108) from the combination when the determined size of the still input image segment exceeds a combination threshold; further configured as follows: A data processing device (100) according to any one of claims 1 to 7.
9. The data processing device (100) - further configured to determine the standard deviation of the output pixels in the still output image segment (108) from the corresponding pixels of the different input images in the still input image segment (104); A data processing device (100) according to any one of claims 1 to 8.
10. The data processing device (100) - calculating the output image (106), the static input image segment (104) or the static output image segment (108); - providing said output image (106) to a display device (112); It is configured as follows: A data processing device (100) according to any one of claims 1 to 9.
11. A medical observation device such as a microscope or an endoscope, the medical observation device comprising: - a data processing device (100) according to any one of claims 1 to 10, an input camera (114) for recording one of said one or more time sequences; A medical observation device comprising:
12. The medical observation device includes: - further comprising a second camera (116) for recording a second time sequence of said one or more time sequences, The medical observation device according to claim 11.
13. said input camera (114) is a white light camera, and / or - said second camera (116) is a fluorescence camera configured to record the emission spectrum of at least one fluorophore; The medical observation device according to claim 12.
14. 1. A computer-implemented method for motion dependent image processing of medical observation input images, the computer-implemented method comprising: - deriving an input set (102a) of input images from a plurality of input images, said plurality of input images being part of one or more time sequences of input images; all of the input images in the input set (102a) are from the same time sequence of input images; each input image includes input pixels, each input pixel including an intensity value; - determining still input image segments (104) within the input images of the input set (102a) by applying a motion detection scheme to the input set (102a), the still input image segments (104) representing still image content within the input set (102a); - determining a still output image segment (108) in an output image (106), said still output image segment (108) having the same position in said output image as said still input image segment (104) has in one of said input images of said input set (102a); - determining an intensity value of an output pixel in the static output image segment (108) of the output image from a combination of intensity values of corresponding input pixels from different input images of a subset of input images, said corresponding pixels being in the static input image segment (104); 10. A computer-implemented method comprising:
15. A computer-readable medium or computer program product containing instructions, The instructions, when executed by a computer, cause the computer to perform the computer-implemented method of claim 14. A computer-readable medium or computer program product.