Data processing device and computer-implemented method for motion-dependent image processing in medical observation device, and medical observation device

Abstract

The invention relates to a data processing device (100) for motion-dependent image processing in a medical viewing device, such as a microscope or endoscope. In the field, weak light imaging at a video frame rate either has the disadvantage of too noisy images or has the disadvantage of motion blur. Image quality is improved by a data processing apparatus (100) configured to retrieve an input set (102a) of input images from a plurality of input images (102), the plurality of input images being part of one or more time series of input images, where the input images in the input set (102a) are all from the same time series of input images, and wherein each input image comprises input pixels, each input pixel comprising an intensity value; determining a static input image segment (104) in the input image in the input set (102b) by applying a motion detection scheme to the input set (102a), the static input image segment (104) representing static image content in the input set (102a); determining a static output image segment (108) in the output image (106), a position of the static output image segment (108) in the output image (106) being the same as a position of the static input image segment (104) in one of the input images in the input set (102a); and determining intensity values for output pixels of an output image (106) in the static output image segment (108) from a combination of intensity values combining corresponding input pixels of different input images from the subset of input images with each other, where the corresponding pixels are located within the static input image segment (104).

Description

Technical Field

[0001] The present invention relates to a data processing device for motion-correlated image processing in a medical observation device (such as a microscope or an endoscope), to a medical observation device (such as a microscope or an endoscope), to a computer-implemented method for motion-correlated image processing of medical interference relationship input images, and to corresponding computer-readable media or computer program products. Background Art

[0002] In low-light imaging applications such as fluorescence and surgical microscopy, the sensitivity of the image acquisition device (e.g., camera) may be considered a bottleneck in the imaging process. In low-light imaging scenarios, the fluorescence intensity may be too weak to provide a clear image of the scene being imaged, especially when the image is captured at video frame rates. If the camera gain is increased, image noise can degrade the image quality.

[0003] If the observed object is static, i.e. not moving, one possible way to improve the image quality is to apply an increased integration time at the expense of a reduced frame rate. Even though this approach can significantly improve the image quality for static objects, it will also suffer from motion blur when the object is moving.

[0004] Therefore, in the art, low-light imaging at video frame rates either has the disadvantage of an image being too noisy or having motion blur. Summary of the invention

[0005] It is therefore an object of the present invention to improve the efficiency of the imaging process, for example by providing significantly better image quality using a given image acquisition device, or by providing the same image quality using a less sensitive and / or cheaper and / or smaller image acquisition device.

[0006] The data processing device mentioned at the beginning of the present invention achieves this purpose, wherein the data processing device is configured to retrieve an input set of input images from multiple input images, the multiple input images are part of one or more input image time series, wherein the input images in the input set are all from the same input image time series, and wherein each input image includes input pixels, each input pixel includes an intensity value, the data processing device is configured to determine a static input image segment in the input images in the input set by applying a motion detection scheme to the input set, the static input image segment represents the static image content in the input set, the data processing device is configured to determine a static output image segment in the output image, the position of the static output image segment in the output image is the same as the position of the static input image segment in one of the input images in the input set, and the data processing device is configured to determine the intensity value of the output pixel of the output image in the static output image segment based on a combination of intensity values ​​of corresponding input pixels of different input images from a subset of the input images combined with each other, wherein the corresponding pixels are located within the static input image segment.

[0007] This has the advantage that imaging processing can be improved, as camera settings can be adjusted dynamically in real time.

[0008] The medical observation device mentioned at the beginning achieves the above-mentioned object by comprising a data processing device according to the invention and an input camera for recording a time series in one or more time series. The medical observation device also has the advantage of improved imaging processing. The medical observation device can be, for example, a microscope or an endoscope.

[0009] The computer-implemented method for motion-related image processing of medical observation input images mentioned at the beginning solves the above-mentioned problem by including the following method steps: retrieving an input set of input images from multiple input images, the multiple input images are part of one or more input image time series, wherein the input images in the input set are all from the same input image time series, and wherein each input image includes input pixels, each input pixel includes an intensity value; determining a static input image segment in the input images in the input set by applying a motion detection scheme to the input set, the static input image segment representing the static image content in the input set; determining a static output image segment in the output image, the position of the static output image segment in the output image being the same as the position of the static input image segment in one of the input images in the input set; and determining the intensity value of the output pixel in the static output image segment of the output image based on a combination of intensity values ​​of corresponding input pixels of different input images from a subset of the input images combined with each other, wherein the corresponding pixel is located within the static input image segment.

[0010] The computer implemented method of the invention also has the advantage of improved image processing.

[0011] The computer-readable medium or computer program product of the present invention solves the above-mentioned problems by comprising instructions which, when the program is executed by a computer, cause the computer to perform the computer-implemented method according to the present invention.

[0012] In the following, further improvements of the data processing device of the present invention, the medical observation device of the present invention, the computer-implemented method of the present invention and the computer-readable medium or computer program product of the present invention will be described. Each of the described embodiments is advantageous in itself, and the additional features can be arbitrarily combined with each other or omitted.

[0013] The improvements and advantages described with respect to the data processing device or the medical observation device also relate to the computer-implemented method or the computer-readable medium or the computer program product. Likewise, the improvements and advantages mentioned with respect to the method with respect to the computer-readable medium or the computer program product also apply to the data processing device or the medical observation device according to the invention. The features described with respect to the data processing device also relate to the medical observation device.

[0014] The static input image segment may correspond to the entire input image. In other words, the entire input image of the input set of input images from the plurality of input images is static.

[0015] Accordingly, the static output image segment in the output image also corresponds to the entire output image. In this embodiment, each pixel of the output image is therefore processed identically, i.e. by determining the intensity value of each output pixel in the static output image based on a combination of intensity values ​​of corresponding input pixels from different input images of the subset of input images combined with each other, wherein the corresponding pixels are located within the static input image.

[0016] In this case, an intensity value of an output pixel of the output image may be determined from a combination of intensity values ​​of corresponding input pixels from different input images of the subset of input images combined with each other, wherein the corresponding pixels are located within the input image corresponding to the static input image fragment.

[0017] The position of the static output image segment in the output image may be the same as the position of the static input image segment in the most recent input image in the input set.

[0018] Static segments may in particular be defined by pixels whose intensity values ​​vary over a predetermined number of input images below a predetermined variation threshold.

[0019] A static input image segment may be determined for each input image in the input set. In certain embodiments, each static input image segment may correspond to the entire corresponding input image in the input set.

[0020] The static input image segments may include continuous and / or discontinuous pixel clusters. In other words, the static input image segments may include one adjacent pixel cluster or more than one such adjacent pixel clusters. Thus, the static input image segments may be separated from each other by non-static input image segments.

[0021] A static input image segment may specifically have the same position, shape, orientation and / or size in each input image in the input set.

[0022] Static input image segments can be represented as image patterns. The same image pattern can be processed in each input image. Such image patterns can be determined by pattern recognition schemes, such as k-means clustering, machine learning, object recognition, or other known processes.

[0023] In one embodiment, the image pattern may be located at different positions in at least some of the input images. This may be the case, for example, if the sample is scanned slowly, where the image pattern may move at the scanning speed. Relative motion may also be seen in the field of view. Such an image pattern represents a static input image segment.

[0024] In order to prevent any erroneous determination of static image content, each static image content may represent an image content whose motion is below a predetermined motion threshold. Thus, vibrations that may occur in the imaging process do not falsify the determination of static input image segments.

[0025] The motion detection scheme mentioned above may be any known motion detection scheme or motion segmentation scheme, such as moving object detection, frame difference, time difference or the like.

[0026] Corresponding pixels are pixels at the same position within the static input image segment. It is advantageous if the output pixels of the output image are obtained pixel by pixel. In other embodiments, clusters of output pixels of the output image are obtained from clusters of pixels of the input image.

[0027] In some embodiments, there may be more than one input image time series. Each of the two or more time series may be generated by a different camera. In other embodiments, some time series may be generated by a first camera and other time series may be generated by a second camera. The camera represents any possible image acquisition device.

[0028] Therefore, a corresponding embodiment of the medical observation device of the present invention further comprises a second camera for recording a second time series of the one or more time series. The second camera may generate a second input image.

[0029] In an advantageous embodiment of the medical viewing device, the input camera may be a white light camera and / or the second camera may be a fluorescence camera configured to record the emission spectrum of at least one fluorophore.

[0030] This has the advantage that information can be retrieved from different spectral regions which reveal different information. This information can be efficiently combined with one another.

[0031] The data processing device may provide a threshold parameter representing the size of the subset. The threshold parameter may be provided as a default value or may be changed by a user of the data processing device. The threshold parameter may be received by the data processing device and, in another embodiment, may be stored in a storage unit. This has the advantage that a single pixel or a pixel cluster consisting of two, three, four or more generally a small number of pixels showing noise in the input image data is determined to be non-static. The threshold parameter may cover any number of pixels. This allows the effects of pixel noise to be suppressed to improve image quality.

[0032] In another advantageous embodiment, the data processing device is further configured to determine the intensity value of the output pixel of the output image outside the static output image segment based on one or more corresponding input pixels of the subset of the input image outside the static input image segment, wherein the number of input pixels used to determine the intensity value of the output pixel outside the static output image segment is less than the number of input pixels used to determine the intensity value of the output pixel in the static output image segment. Outside the static input image segment, a movable object can be detected. In a sequence of multiple input images in the input set, the moving object will be located at different positions in each input image. Different input pixels of each input image are used to determine the intensity value of the output pixel outside the static output image segment. Different input pixels of adjacent input images can describe the trajectory of the moving element represented by the output pixel in the output image. Therefore, a large number of input pixels can determine the intensity value of one output pixel. Therefore, the number of input pixels is higher than the number of output pixels.

[0033] In another advantageous embodiment of the data processing device, the data processing device is further configured to determine the intensity value of the output pixel outside the static output image fragment as the intensity value of the corresponding input pixel of the latest input image in the subset of input images. This has the advantage that no image blurring occurs. Thus, the intensity value of the non-static image content is not calculated from a large number of input images, but only from the latest input image.

[0034] The data processing device may be further improved, wherein the number of input pixels used to determine the intensity value of the output pixel in the static output image segment is in the range of 3 to 9. Thus, the number of input pixels used may be three, four, five, six, seven, eight or nine. In other embodiments, other numbers of input pixels may be envisaged, such as one or two, or 10 or more. The number of input pixels may be varied depending on the intensity value of the input pixels. By varying the number of input pixels used to determine the intensity value of the output pixel, the intensity value of the static output image segment may be increased or decreased.

[0035] The data processing apparatus may also be configured to retrieve a second set of second input images from the plurality of input images, wherein the second set of second input images are all from the same temporal sequence, the temporal sequence of the second set of input images being different from the temporal sequence of the input set, and wherein each second input image comprises a second input pixel, each second input pixel comprising an intensity value. Such data processing apparatus may also be configured to determine a second static output image segment in each second input image in the second set, the second static output image segment being located in the second output image and having a position in the second output image that is the same as a position of the static input image segment in one of the input images in the input set, and to determine an intensity value of an output pixel in the second static output image segment of the second output image based on a combination of intensity values ​​combining corresponding input pixels of different input images from a subset of the second set, wherein the corresponding pixel is located within the static input image segment.

[0036] Likewise in the case of the second set of second input images, the second static output image segment may correspond to the entire second input image, ie the entire image content of the second input image is determined to be static.

[0037] The second output image may be calculated based on a subset of the second set of second input images, and in particular, the subset size may be controlled by a user controllable parameter. Such user controllable parameters may be input by a user and stored in a storage unit.

[0038] The input images in the input set and the second input images in the second set may be registered images. At least some of the images in the input set may be recorded simultaneously with or as the second input images in the second set.

[0039] Another advantageous embodiment of the data processing device is further configured to register an input image in the input set with a second input image in the second set.If different input images are registered, they can be distinguished from each other.

[0040] Furthermore, the input image and the second input image may be recorded simultaneously and / or synchronously.The input image and the second input image may be recorded at the same time or within a period that is preferably less than a flicker fusion threshold or less than 50 ms.

[0041] The data processing device is further improved by further configuring the data processing device to: determine the size of the static input image segment, and when the determined size of the static input image segment exceeds the combination threshold, determine the intensity value of the output pixel of the output image in the static output image segment based on the combination, so as to further improve the data processing device.

[0042] This has the advantage that image content that is smaller than the combination threshold (i.e. image content that extends over less than a certain number of pixels) is not processed by the data processing means. This may result in faster image processing. Thus, the size of the static input image segment may comprise a count of pixels.

[0043] In another advantageous embodiment, the data processing means is further configured to determine a standard deviation of an output pixel in the static output image segment from a corresponding pixel of a different input image located within the static input image segment. This has the advantage that an intensity value can be rejected from the result of the combination combining the corresponding pixels when the intensity value is outside a threshold value based on the determined standard deviation.

[0044] The data processing device may also be configured to calculate an output image, a static input image segment or a static output image segment and to provide the output image to a display device. Alternatively or additionally, the output image may be provided to a recording device. The recording device may in particular be a digital recording device, which allows the recorded image to be further processed, for example saved in a memory.

[0045] In the following, the present invention will be described with reference to the accompanying drawings. The embodiments shown are exemplary and do not limit the present invention. The features shown in the different embodiments may be combined with each other or omitted at will. In addition, the features described with reference to the data processing device may be applied to the medical observation device and the computer-implemented method. Therefore, the features described with reference to the method may be transferred to the corresponding device. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a schematic diagram of a medical observation device;

[0047] Figure 2 is a flow chart of a computer implemented method;

[0048] Figure 3 is a flow chart of a method for evaluating an input image sequence;

[0049] Figure 4a and 4bis a flow chart for determining a static image region;

[0050] Figure 5 is a flow chart of a method for determining a static image region; and

[0051] Figure 6 is another schematic flow chart for determining a static image region. DETAILED DESCRIPTION

[0052] First, refer to Figure 1 Examples of embodiments of the present invention are described.

[0053] Figure 1 The medical observation device 100 is schematically shown. The medical observation device 100 may be an optical imaging device based on fluorescence. In addition, the medical observation device 100 may be embodied as a microscope or an endoscope. Figure 1 The medical observation device is only schematically shown as a microscope, but the following description also applies to an endoscope.

[0054] The medical observation device 100 may be used to assist surgery or may be used in a laboratory. A user may observe an object or sample 110 in an object scene 120 by using the medical observation device 100 .

[0055] The medical viewing device 100 may include a camera 130. The camera 130 may record images of a subject scene. The camera 130 may also be used as a video camera that records and provides an image sequence 132. The data processing device 140 may retrieve the image sequence from the camera as an input image sequence 132. The data processing device 140 may be integrated into the medical viewing device 100. Alternatively, the data processing device 140 may be coupled to the medical viewing device 100.

[0056] In addition to the (main) camera 130, the medical viewing device 100 may optionally include a secondary camera 150. The secondary camera may simultaneously observe the object scene 120 to generate another image sequence 152. Thus, the secondary camera 150 may also be embodied as a video camera, which records the object scene 120 and provides the image sequence 152 to the data processing device 140. Any or all of the cameras 130 and 150 are configured to generate an image sequence representing time-sequentially arranged images of the object scene 120 observed by the medical viewing device 100. Thus, according to an embodiment in which only one camera is present in the medical viewing device 100, the data processing device 140 retrieves the input image sequence 132 of time-sequentially arranged primary input images.

[0057] According to another embodiment, when two cameras (e.g., a primary camera 130 and a secondary camera 150) simultaneously observe the object scene 120, the data processing device 140 is configured to retrieve a primary input image sequence 132 of primary input images representing the object scene 120 in chronological order and a secondary input image sequence 152 of secondary input images representing the object scene 120 in chronological order. As described above, the camera 130 may be embodied as a fluorescence camera. Alternatively, it may be embodied as a white light camera. According to an embodiment including two cameras, the primary camera may be embodied as a white light camera, and the secondary camera may be embodied as a fluorescence camera, the difference between a fluorescence camera and a white light camera mainly being that a fluorescence camera is configured to pass a passband filter ( Figure 1 The passband filter is also integrated in the camera or is part of the medical viewing device 100 to observe the object scene 120. The passband filter can be selective for the emission spectrum of the fluorophore of the sample 110 present in or added to the object scene 120.

[0058] Since the passband filter can block any light components outside the passband spectrum, images captured by a fluorescence camera are affected by low light levels. This disadvantage is magnified when the camera is operated with very short exposure times. This is the case for any camera operating in video mode, in which the camera produces an image sequence of images arranged in time order. Whether the camera is a white light camera or a fluorescence camera, or any camera of the medical observation device 100, the data processing device 140 can dynamically adjust the frame rate of the exposure time setting. For example, the data processing device 140 can increase the frame rate of the camera 130 and / or the secondary camera 150. When the sample 110 moves faster than the exposure time of the observation camera, increasing the frame rate may help to eliminate motion blur that may occur in individual images in the image sequence.

[0059] In addition, the main camera 130 and the secondary camera 150 of the medical observation device 100 can be arranged, adjusted or calibrated to produce images that can be superimposed. In other words, the images in the main input image sequence from the first camera can be superimposed with the secondary images in the secondary input image sequence. In this case, the main input image of the main camera 130 and the secondary input image of the secondary camera 150 can have the same aspect ratio, the same pixel size, the same image size and ratio, and / or have a predetermined known transformation relationship that allows the input pixels of the image from the first camera to be mapped to other input pixels of the image of the second camera. In other words, when the main input image of the first camera includes a main input pixel, the known relationship for mapping allows the main input pixel to be mapped to the secondary input pixel of the secondary input image of the secondary camera.

[0060] The medical observation device may further include an output device 160. The output device 160 may receive an output image 170 from the data processing device 140. The output device 160 may be a display, smart glasses, a projector, a holographic display, or the like.

[0061] The output image 170 comprises output pixels, and each pixel comprises at least one intensity value. The data processing device 140 is configured to generate and / or output the output image 170 .

[0062] According to an embodiment, the data processing device 140 may retrieve an input image sequence 132 from the camera 130. The input image sequence may include input images that are arranged in time order and represent the object scene 120 observed by the medical viewing device 100. The data processing device 140 may then process the retrieved input image sequence 132 to output an output image 170. In particular, the data processing device 140 may evaluate the image sequence 132 to determine motion in the object scene 120. When the image sequence 132 does not indicate any motion present in the observed object scene 120, the data processing device 140 may digitally enhance the latest image of the image sequence 132 to provide an improved output image 170 to the output device 160. For example, the data processing device 140 may stack the latest image of the retrieved image sequence 132 with the previous image in the image sequence. For example, multiple images in the input image sequence 132 may be averaged to calculate the output image 170. Alternatively, the output image 170 may be calculated as a median filtered image from the input image sequence 132. The output image 170 may also be calculated by signal cropping from the retrieved input image sequence 132 .

[0063] According to another embodiment, when not only one camera but two cameras, namely the first camera 130 and the second camera 150, are simultaneously observing the object scene 120, the data processing device 140 may analyze the primary input image sequence 132 to determine whether there is motion in the simultaneously observed object scene 120. In the event that the data processing device determines that there is no motion in the observed object scene 120, the data processing device 140 may alternatively calculate the output image 170 from the secondary input image sequence in the same manner as when only one camera is used. This embodiment is particularly useful when the light on one imaging channel (e.g., a white light camera) is very sufficient, while on another imaging channel (e.g., a fluorescence camera), the amount of light received by the camera is very low.

[0064] However, the data processing device when operating the two cameras is not limited to generating only one output image 170. Therefore, the data processing device 140 may also generate and / or output a secondary output image calculated from the primary input image sequence by stacking or combining the images in the sequence.

[0065] In response to determining that there is indeed no motion in the observed object scene, the data processing device 140 may stack or combine the most recent three, four, five, six, seven, eight, nine, ten, eleven, twelve (or any other number) images in the input image sequence.

[0066] When the data processing device 140 determines that there is motion in the object scene 120, the data processing device 140 can adjust the frame rate setting of the camera 130 to use motion blur artifacts. When the medical viewing device 100 operates with two cameras simultaneously viewing the object scene 120, the data processing device 140 can adjust the frame rate settings of the two cameras. In any case, the data processing device can increase the frame rate setting of the primary camera 130 and / or the secondary camera 150.

[0067] In addition, in response to determining that there is motion in the observed object scene 120, the data processing device can determine a static image area or a static image segment in the input image sequence. In the following, the terms static image area and static image segment are used synonymously. The static image area can be a (whole) image or a mask of an image, and indicates input pixels in the input image in the input image sequence that do not substantially change in the input image sequence. In other words, these input pixels indicate or represent a static part of the recorded object scene 120. For example, the static part can be the background behind a moving object.

[0068] The data processing device 140 may determine the static image region by evaluating or analyzing the input image sequence to determine motion in the input image sequence. For example, when the result of evaluating the input image sequence is a mask of input pixels that have changed throughout the input image sequence, the complementary set of these input pixels represents the static image region. Alternatively, the data processing device may directly generate the static image region as a result of evaluating the input image sequence. In any case, the static image region indicates or represents the location of input pixels that have not been subject to fluctuations in the images of the evaluated or analyzed input image sequence.

[0069] For example, the image may be the latest image in the sequence of input images. The data processing device 140 may generate and / or output the output image 170 by assembling or composing different regions of the output image according to the position of the output pixel in the output image. When the position of the output pixel is located in the static image region, the intensity value of the output pixel is calculated according to the combination of intensity values ​​from a subset of the predetermined input image sequence. When only one camera is in operation, the predetermined input image sequence is the input image sequence 130.

[0070] When two cameras are used, alternatively or additionally, the predetermined input image sequence may be a sequence of secondary input images retrieved from a secondary camera simultaneously observing the object scene 120. In the combination, input pixels of different input images in the subset that match the position of the output pixel with respect to the static image area are combined. The combination may be calculated by sigma stacking, averaging or median filtering. For the sake of completeness, it is worth noting that when two cameras are used, the data processing device calculates the combination of output pixels by combining the secondary input pixels of different secondary input images in the subset that match the position of the output pixel with respect to the static image area.

[0071] In summary, the intensity values ​​of output pixels of the output image (where the output pixels are located within the static image region) are calculated based on corresponding input pixels of different input images from a subset of the input image sequence. The cardinality of the subset may be 2 to 12. The data processing device 140 may generate a subset of the input image sequence by filtering or omitting input images from the input image sequence having a low signal-to-noise ratio. This may further increase the image characteristics in the output image 170. The data processing device 140 may calculate the dynamic region of the output image 170, i.e., the region of output pixels located outside the static image region, by adopting the corresponding intensity values ​​of the input pixels in the latest input image that match the corresponding output pixels in the output image. Therefore, in the static image region of the output image, a stationary background may be output with a high signal-to-noise ratio, while the dynamic image region of the output image (i.e., the region complementary to the static image region) does not suffer from motion blur artifacts. Therefore, the overall imaging characteristics of the medical observation device 100 are improved.

[0072] When the medical viewing device 100 is operated using two cameras, the data processing device 140 may alternatively or additionally generate and / or output the output image 170 from the combination of the secondary input image sequences in a similar manner. This embodiment is particularly useful when the fluorescence camera and the white light camera 130 simultaneously observe the object scene 120. In this case, the white light camera 130 provides the data processing device 140 with the primary input image sequence to evaluate whether there is motion in the observed object scene 120. If there is motion, the data processing device 140 will use this information to provide an improved output image 170 from the secondary input image sequence of the fluorescence camera 150. In particular, the data processing device 140 can calculate the intensity values ​​of the output pixels located in the static image area determined from the primary input image sequence of the white light camera 130.

[0073] Figure 2A flow chart of a computer-implemented method for image processing of a medical observation input image representing an object scene observed by a medical observation device is shown. The data processing device 140 may be configured to implement the computer-implemented method 200. In step 210, an input image sequence of input images representing a chronologically arranged object scene observed by the medical observation device 100 may be retrieved. In addition, in step 215, in addition to the input image sequence (which may also be referred to as the main input image sequence), a secondary input image sequence of secondary input images representing a chronologically arranged object scene observed by a second camera of the medical observation device may also be received. In this alternative embodiment, the input images in the main input image sequence are also referred to as the main input images. The input image includes input pixels, and each input pixel includes an intensity value. Similarly, the input pixels of the main input image are referred to as the main input pixels hereinafter, and accordingly, the output pixels of the secondary input images are referred to as the secondary input pixels accordingly. In any case, each input pixel includes an intensity value. In step 220, the input image sequence or the main input image sequence may be evaluated separately to determine the motion in the observed object scene. If the result of the evaluation of the image sequence is negative, i.e. when no motion is determined in the evaluated image sequence 132, an output image 170 may be generated by combining the images in the image sequence 132. For example, the output image 170 may be the result of a stacking operation based on the image sequence or, respectively, based on the main input image sequence. For example, in any case, a subset of the input image sequence may be used to generate the output image. In the subset, one or more input images (respectively, the main input image) may be filtered out from the input image sequence when the signal-to-noise ratio is below a predetermined threshold.

[0074] In step 230, an output image may be generated by stacking a sequence of input images or stacking a sequence of master input images respectively.

[0075] In step 240, the static image region may be determined. In particular, the static image region may be determined from the result of evaluating the input image sequence or respectively the master input image sequence.

[0076] Optionally, in step 250, the frame rate setting of the medical viewing device 100 may be dynamically adjusted. For example, the frame rate setting may be increased. Then, the amount of time that passes between consecutive images in the input image sequence or respectively in the primary input image sequence becomes shorter. This allows motion blur artifacts to be reduced. At the same time, the frame rate setting of the primary camera 130 or both cameras may be increased, that is, the frame rate setting of the secondary camera 150 may also be increased.

[0077] In step 260, intensity values ​​of output pixels to be generated for the output image may be calculated. The intensity values ​​may be calculated based on an input image sequence or a subset of an input image sequence, a primary input image sequence or a subset thereof, and / or a secondary input image sequence or a subset thereof. Intensity values ​​of output pixels located in the determined static image region may be calculated, wherein the intensity values ​​are calculated based on a combination of intensity values ​​of a subset of a predetermined input image sequence. The predetermined input image sequence may be indicated by an operational setting of the medical viewing device 100.

[0078] In the case where the medical observation device operates with only one camera, the predetermined input image sequence cannot be indicated by the operation setting, but is the same as the input image sequence. In the case where the medical observation device 100 operates with two cameras, the predetermined input image sequence may be a secondary input image sequence, and thus the intensity value of the output pixel in the determined static image area is calculated based on a combination of intensity values ​​of a subset of the secondary input image sequence.

[0079] In any case, in the combination of intensity values, input pixels of different input images in the subset that match the positions of output pixels with respect to the static image area are combined. For example, primary input pixels of different primary input images in the subset of the predetermined input image sequence that match the positions of output pixels with respect to the static image area are combined. According to another example, in the combination, secondary input pixels of different secondary input images in the subset of the secondary input image sequence that match the positions of output pixels with respect to the static image area are combined.

[0080] The position of the output pixel is related to the static image area by one or more indices or addresses associated with the output image.In addition, the output pixels of the output image located outside the static image area can be calculated from the input image sequence or respectively from the sub-input image sequence.

[0081] The intensity value of an input pixel or respectively a sub-input pixel may be used as the intensity of the output pixel, wherein the position of the input pixel matches the position of the output pixel in the static image area. Alternatively, the intensity value of the output pixel may be calculated based on a combination of input pixels that match the output pixel in the sub-area. For example, the intensity value of the output pixel is the average or median of the neighboring input pixels of the input pixel that match the output pixel with respect to the position, and the number of neighboring input pixels in the same input image is defined by a configurable parameter. For example, a configurable parameter for the size of neighboring pixels that should be considered when calculating the output pixel may be dynamically adjusted in response to an adjustment of the frame rate setting. Thus, if the frame rate setting used introduces too much noise component in the recorded image, it may be compensated for.

[0082] In step 270, the output image 170 may be output. For example, the output image may be output to the output device 160. The output device 160 may receive the output image 170 from the data processing device 140. Likewise, two separate output images may be provided to the output device 160, one for each camera simultaneously viewing the object scene 120.

[0083] Figure 3 A schematic flow chart of a method for evaluating an input image sequence to determine motion in an object scene is shown. In the following, the method is described from the perspective of input pixels. However, the same method is also applicable to primary input pixels or secondary input pixels without any limitation.

[0084] Since the input image (e.g., the primary input image or the secondary input image) is a digital image processed by or can be processed by a computer, the position of any input pixel in the input image can be represented by a pair of numbers or indices. Such a pair of numbers is also referred to as pixel coordinates by those skilled in the art. The pair of numbers represents the position of the input pixel in a two-dimensional area.

[0085] The method 300 for estimating motion in an object scene can be performed by parallel calculation or sequential calculation. When performed by sequential calculation, the first input pixel for starting calculation can be set in step 310.

[0086] In step 320, the variance or standard deviation of a given or set input pixel is calculated. Since the variance is defined as the square of the standard deviation, it is preferred that only the absolute value of the standard deviation can be calculated. In order to calculate the above statistical measure, the intensity values ​​of the input pixels in the input image sequence are used, wherein each input pixel is located at the same position in different input images in the input image sequence. Since the position of the given or set input pixel is converted to a position in the object scene, the input image sequence including the input image provides a time series of intensity values ​​at the given or set position of the input pixel. In step 330, the calculated measure (e.g., the absolute value of the standard deviation, variance, or standard deviation) can be compared with a predetermined threshold. The statistical measure can be compared to determine whether it exceeds the predetermined threshold. If the calculated statistical measure is low, it may indicate that there is no motion at the given or set input pixel.

[0087] In step 340, any input pixel in a given input image (e.g., the latest input image in the sequence of input images being evaluated) may be marked or labeled. For example, the input pixel may be marked as a moving pixel. Alternatively, instead of marking the moving pixels, any pixels that do not exceed a predetermined threshold may be marked. In this case, they may be marked as static pixels.

[0088] In step 350, in case of sequential processing, the above steps 310 to 340 may be repeated for all input pixels.

[0089] In step 360, the sum of the labeled input pixels may be determined.

[0090] In step 370, the sum may be compared to another predetermined threshold t2 indicating motion of objects in the scene.

[0091] If the sum exceeds another predetermined threshold value t2, it can be determined that there is motion in the object scene. In other words, fluctuations in intensity values ​​in subsequent input images indicate that there is motion in the object scene.

[0092] Optionally, in step 380, a static image region may be determined based on the result of marking the input pixels. For example, when the input pixels are marked as moving pixels, a complementary image mask of the marked input pixels may be determined. The mask represents a static image region because it indicates the locations of those input pixels where there is no motion in the object scene. Alternatively, when the static pixels are marked in step 340, this result may be used directly as a static image region.

[0093] Figure 4a and 4b A flow chart for determining a static image region is shown. Figure 4a As shown, in step 410a, a statistical metric of each input pixel may be calculated pixel by pixel. For example, the statistical metric may be the absolute value of the variance and the deviation or the standard deviation. The statistical metric M may be calculated along the same input pixel in the input image sequence.

[0094] In step 420a, the calculated statistical metric M of each input pixel may be compared with a given threshold value t3. If it is determined that the statistical metric M exceeds the predetermined threshold value t3, the input pixel may be marked as a moving pixel or a dynamic object region in step 425a. If it is determined that the statistical metric M is less than the predetermined threshold value t3, the input pixel may be marked as a static pixel or a static object region in step 426a.

[0095] In step 430a, a complementary mask of the marked pixels may be determined. The complementary mask represents input pixels that do not indicate moving pixels, or in other words, static input pixels. The mask may be used as a static image area.

[0096] exist Figure 4b The statistical measure M is calculated in step 410b in the same manner as in Figure 4a The calculation method in step 410a is the same as that in step 420b. Similarly, in step 420b, the calculated statistical measure M is compared with a predetermined threshold value t3. Figure 4aIn contrast to the method in , in step 426b, now only those input pixels for which the calculated statistic M is below a predetermined threshold t3 are marked. This allows the direct generation of a mask representing a static image region in step 430b.

[0097] Figure 5 A schematic flow chart of a method for determining a static image area is shown. Figure 5 , an exemplary input image sequence 510 is shown. In step 520, the input image sequence of the time-ordered input images 502 to 506 may be processed by a motion detection scheme, a motion estimation scheme, a segmentation analysis scheme, etc. Input images 503 and 504 are not shown in the figure.

[0098] Step 520 may be part of the previously described methods 300 and 400a and 400b. As a result of step 520, a static image region may be obtained. For example, it may be obtained from the result of a motion detection scheme, a motion estimation scheme, or a segmentation analysis scheme. For example, any or all of the above schemes may generate a mask that segments an input image in the input image sequence into at least two parts, wherein one part or segment represents a static image region and the other part represents a dynamic image region.

[0099] Dynamic image areas can represent moving pixels. Figure 5 At the bottom of FIG. 5 , an example mask 530 is shown including a dynamic image region 540 (shaded) and a static image region 550. In the given example, the static image region 550 is discontinuous, where the center input pixel is annotated by the moving pixel of the dynamic image region 540.

[0100] Figure 6 Another schematic flow chart for determining a static image region starting from another image sequence is shown. In the input image sequence, the chronologically arranged input images 602 to 606 represent an object scene observed by a camera of the medical observation device 100 .

[0101] In the given example, the structure 601 in the center of each image of the object scene moves from one direction to another direction through the center of the image. In step 620, static image regions may be determined from analyzing the input image sequence 610. In particular, as a result of the analysis or evaluation of the input image sequence 610, an image mask 630 may be generated, wherein the mask 630 also includes pixels, and depending on whether the pixels indicate motion in the object scene, the intensity values ​​of the pixels are set to values ​​indicating motion or not indicating motion.

[0102] In the illustrated image mask 630 , the static image region 650 is continuous and at the center it completely surrounds the dynamic image region 640 .

Claims

1. A data processing device (100) for motion-dependent image processing in a medical observation device, such as a microscope or an endoscope, the data processing device (100) being configured to: - retrieving an input set (102a) of input images from a plurality of input images (102), the plurality of input images being part of one or more temporal sequences of input images, in, The input images in the input set (102a) are all from the same input image time series, and wherein each input image comprises input pixels, and each input pixel comprises an intensity value; - determining static input image segments (104) in input images in the input set (102a) by applying a motion detection scheme to the input set (102a), the static input image segments (104) representing static image content in the input set (102a); - determining a static output image segment (108) in an output image (106), the static output image segment (108) having a position in the output image (106) that is the same as the position of the static input image segment (104) in one of the input images in the input set (102a); and - determining an intensity value of an output pixel of the output image (106) in the static output image segment (108) based on a combination of intensity values ​​of corresponding input pixels of different input images from a subset of input images combined with each other, wherein the corresponding pixels are located within the static input image segment (104).

2. The data processing device (100) according to claim 1, wherein: The data processing device (100) is further configured to: - determining intensity values ​​of output pixels of the output image (106) outside (110) the static output image segment (108) based on one or more corresponding input pixels of a subset of the input image outside the static input image segment; as well as The number of input pixels used to determine intensity values ​​of output pixels outside (110) the static output image segment (108) is less than the number of input pixels used to determine intensity values ​​of output pixels in the static output image segment (108).

3. The data processing device according to claim 2, wherein: The data processing device (100) is further configured to: - determining the intensity values ​​of output pixels outside (110) said static output image segment (108) as the intensity values ​​of corresponding input pixels of the most recent input image in said subset of input images.

4. The data processing device (100) according to claim 2 or 3, wherein: The number of input pixels used to determine intensity values ​​of output pixels in the static output image segment (108) is in the range of 3 to 9.

5. The data processing device (100) according to any one of claims 1 to 4, wherein: The data processing device (100) is configured to: - retrieving a second set of second input images from the plurality of input images (102b), wherein the second set (102b) of second input images are all from the same time series, the time series of the second set (102b) of input images is different from the time series of the input set (102a), and wherein each second input image comprises second input pixels, and each second input pixel comprises an intensity value; - determining a second static output image segment in each of the second input images in the second set (102b), the second static output image segment being located in the second output image and having the same position in the second output image as the static input image segment (104) in one of the input images in the input set (102a); - determining an intensity value of an output pixel in the second static output image segment of the second output image based on a combination of intensity values ​​of corresponding input pixels of different input images from a subset of the second set (102b), wherein the corresponding pixels are located within the static input image segment (104).

6. The data processing device (100) according to any one of the preceding claims, the data processing device (100) being further configured to: - registering an input image in said input set with a second input image in said second set.

7. The data processing device (100) according to claim 6, wherein: The input image and the second input image are recorded simultaneously and / or synchronously.

8. The data processing device (100) according to any one of the preceding claims, wherein: The data processing device (100) is further configured to: - determining the size of said static input image segment; and - when the determined size of the static input image segment exceeds a combination threshold, determining intensity values ​​of output pixels of the output image (106) in the static output image segment (108) according to the combination.

9. The data processing device (100) according to any one of the preceding claims, the data processing device (100) being further configured to: - determining a standard deviation of said output pixel in said static output image segment (108) from a corresponding pixel of said different input image located within said static input image segment (104).

10. The data processing device (100) according to any one of the preceding claims, wherein the data processing device (100) is configured to: - calculating the output image (106), the static input image segment (104) or the static output image segment (108); - providing said output image (106) to a display device (112).

11. A medical observation device, such as a microscope or an endoscope, comprising: - A data processing device (100) according to any one of claims 1 to 10; - an input camera (114) for recording a time sequence in one or more time sequences.

12. The medical observation device according to claim 11, wherein: The medical observation device also includes: - a second camera (116) for recording a second time series of the one or more time series.

13. The medical observation device according to claim 12, -in, The input camera (114) is a white light camera; and / or - wherein the second camera (116) is a fluorescence camera configured to record the emission spectrum of at least one fluorophore.

14. A computer-implemented method for motion-dependent image processing of medical observation input images, the computer-implemented method comprising: - retrieving an input set of input images (102a) from a plurality of input images, the plurality of input images being part of one or more temporal sequences of input images, The input images in the input set (102a) are all from the same input image time series, and wherein each input image comprises input pixels, and each input pixel comprises an intensity value; - determining static input image segments (104) in input images in the input set (102a) by applying a motion detection scheme to the input set (102a), the static input image segments (104) representing static image content in the input set (102a); - determining a static output image segment (108) in an output image (106), the static output image segment (108) having the same position in the output image as the static input image segment (104) in one of the input images in the input set (102a); - determining an intensity value of an output pixel in the static output image segment (108) of the output image based on a combination of intensity values ​​of corresponding input pixels from different input images of a subset of input images combined with each other, wherein the corresponding pixels are located within the static input image segment (104).

15. A computer readable medium or a computer program product comprising instructions which, when said program is executed by a computer, cause said computer to perform the computer-implemented method according to claim 14.

Images