Image processing system
Patent Information
- Application Number
- JP2022199974
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-12-16
- Filing Date
- 2022-12-15
- Publication Date
- 2025-12-22
AI Technical Summary
Conventional image processing systems in microscopy are limited by the need for oversampling to perform effective deconvolution, which is not guaranteed during real-time imaging with varying sampling densities, leading to ineffective or detrimental deconvolution and inability to handle undersampling.
A combined deconvolution and denoising approach that adapts to varying sampling densities in real-time, blending deconvolution and denoising results based on sampling density to enhance image quality independently of oversampling conditions.
Enables reliable real-time image enhancement in microscopy by avoiding abrupt changes in image quality, effectively improving spatial resolution and noise reduction across a wide range of sampling densities, including undersampling conditions.
Smart Images
Figure 00000000_0000_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an image processing system, a microscope having an optical imaging system and an image processing system, an image processing method, and a computer program having program code for implementing such a method.
Background Art
[0002] Deconvolution and noise removal are well-known image optimization algorithms used in various image modalities, including, for example, fluorescence microscopy and confocal beam scanning microscopy. Conventionally, deconvolution and noise removal are performed as part of image post-processing, i.e., only after microscope visualization processing.
[0003] Deconvolution is used to restore the spatial structure of the original object of imaging, but these structures are blurred due to the limited spatial resolution of the optical imaging system of the microscope. In conventional deconvolution approaches, the restoration of the original object from the blurred image is achieved by virtually reversing the visualization process. This reversal process is based on empirical knowledge of the image projection operation of the optical imaging system. This knowledge of the image projection operation is represented by a so-called point spread function (PSF). The PSF represents a blurred image of an ideal point-shaped object imaged onto the image plane through the optical system. The spatial spread of the PSF is a measure of the spatial resolution of the optical imaging system. With an ideal deconvolution algorithm, a point as the original object is restored from the input image represented by the PSF.
[0004] There are also deconvolution algorithms that do not rely on a prior PSF. These so-called blind deconvolution algorithms are rather used to reconstruct an unknown PSF from the image itself. Two types of algorithms, namely conventional deconvolution based on prior knowledge of the PSF and blind deconvolution without such knowledge, are useful for reversing image blur and spatial information loss caused by imperfect image projection through optical imaging systems.
[0005] As a prerequisite for deconvolution, the image to be deconvolved must be oversampled. Oversampling means that the sampling density of the image pixel data generated by the optical imaging system exceeds a threshold expressed by the Nyquist criterion. In particular, the pixel size of the image detector must be smaller than the spatial resolution of the optical imaging system. Preferably, the pixel size of the image detector should be smaller than half the spatial resolution of the optical imaging system (Nyquist criterion). In other words, in the case of oversampling, the PSF is spatially decomposed by multiple pixels. If the prerequisite for oversampling is not met, meaningful deconvolution is impossible. In fact, performing deconvolution on an undersampled image will result in artifacts that render the deconvolution useless or even disadvantageous.
[0006] Unlike deconvolution, denoising algorithms do not increase the spatial resolution of an image. Rather, denoising algorithms extract the estimated real intensity of each pixel in an image that is noisy, i.e., an image with limited signal and / or acquired by an imperfect detector. Furthermore, a major difference from deconvolution is that denoising does not require oversampling. For example, denoising is typically applied to images that are not oversampled, such as those from a mobile phone camera.
[0007] As described above, conventional microscope systems provide deconvolution and denoising algorithms in the post-processing stage to improve the quality of already recorded microscope images. While some deconvolutions exist that utilize embedded denoising to reduce the risk of image artifacts, given the deviation from the preconditions, deconvolution and denoising are typically applied alternately. Examples are disclosed in the publication "Joint denoising-deconvolution approach for fluorescence microscopy" by SK Maji et al., 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI), 2016, pp. 128-131, doi: 10.1109 / ISBI.2016.7493227 and "Simultaneous Denoising, Deconvolution, and Demixing of Calcium Imaging Data" by Pnevmatikakis et al., Neuron. 89. 10.1016 / j.neuron.2015.11.037. However, in these examples, denoising becomes an embedding step in deconvolution, and deconvolution is still limited to oversampled images. Therefore, these approaches cannot be applied to sampling densities that fall within the undersampling region. [Overview of the project] [Problems that the invention aims to solve]
[0008] Therefore, the objective is to provide an image processing system, a microscope, an image processing method, and a computer program capable of improving images over a wide range of sampling densities. [Means for solving the problem]
[0009] The above-mentioned problems are achieved by the matters described in the independent claims. Advantageous embodiments are defined by the dependent claims and the following description.
[0010] The image processing system includes a processor. This processor is configured to acquire image pixel data generated by the optical imaging optics of a microscope. This processor is further configured to perform deconvolution on the acquired image pixel data to generate deconvolved image pixel data. This processor is configured to perform denoising on the acquired image pixel data to generate denoised image pixel data. The processor is configured to acquire the sampling density generated by the optical imaging system for the image pixel data. When the sampling density exceeds the oversampling limit, the processor is configured to mix the deconvolved image pixel data and the denoised image pixel data to change (particularly increase) the ratio of the deconvolved image pixel data to the denoised image pixel data, thereby generating mixed image pixel data with a weighting according to the sampling density.
[0011] The proposed solution is configured for an approach that combines deconvolution and denoising, which can be automatically applied to improve image quality regardless of the sampling density at which the microscope image is acquired. In particular, according to preferred applications, this image processing system can be used to apply a combination of deconvolution and denoising to microscope images during real-time recording with visualization parameters that are previously unknown and may change. Thus, deconvolution and denoising filtering are performed in real time and during the live setup of the microscope.
[0012] Therefore, the typical problems of conventional deconvolution approaches can be avoided. Hence, as explained above, the necessary condition for conventional deconvolution is that oversampling is what makes the increase in spatial resolution by deconvolution possible in the first place. However, the condition of oversampling is not guaranteed during real-time setup. Rather, if the number of pixels in the image detector is constant, the sampling density may change during real-time visualization if the user changes visualization parameters, such as zoom setup, scan field, etc. When visualization parameters change, the sampling density may change from oversampling to undersampling, and deconvolution may no longer be effective or even detrimental to the reliability of the intended image improvement. As a result, real-time deconvolution during imaging where the sampling density changes is impossible in conventional systems.
[0013] This problem is overcome by an image processing system that enables real-time image improvement, aided by deconvolution and denoising. The insight underlying the proposed solution is that when image parameters change from an oversampling region to a non-oversampling region due to changes in the relevant visualization parameters of the microscope during microscope setup or, for example, during image acquisition, deconvolution is technically hindered. In such a situation, the image data no longer benefits from virtually possible deconvolution, because the pixel resolution is no longer sufficient to represent structures smaller than the PSF size, in any case. However, denoising remains possible and desirable. Another insight is that spontaneous changes from one image improvement algorithm to another (which could be considered a trivial solution to this problem) should be avoided because such changes can result in jarring changes in the live image stream. Thus, a simple switch from one image improvement technique to another is easy but unsatisfactory.
[0014] The proposed solution is a combined approach of deconvolution and denoising, preferably performed in real time during the acquisition of a live visualization stream of an optical imaging system, followed by a weighted blending of the deconvolution and denoising results. On the one hand, this weighted blending is preferably performed such that deconvolution predominates in the overall result when the sampling density exceeds the oversampling limit induced by the current imaging conditions in the live visualization stream. On the other hand, when the imaging conditions fall below the oversampling limit, substantially pure denoising can be weighted against the output image. In the parameter region close to the oversampling limit, the weighting of one image improvement approach against the other can be changed to monotonic and continuous in order to avoid spontaneous changes in the image data stream. As a result, the proposed image processing system makes it possible to visualize dark, noisy, and spatially indistinct structures during live imaging in optical microscopes, particularly fluorescence microscopes.
[0015] As explained above, the image processing system described above is advantageous for image enhancement in real-time visualization. However, this is not limited to real-time visualization applications. Rather, the approach combining deconvolution and denoising can be very useful for image post-processing (offline) as well.
[0016] Furthermore, it should be noted that the weighting of one image improvement approach relative to the other can also be influenced by image parameters, namely the sampling density, as well as by the other. For example, the weighting may be adapted according to image intensity and / or signal-to-noise ratio. Therefore, in noisy images, deconvolution may be more problematic, and it may be advantageous to prioritize the denoising result and weight the overall mixture of deconvolution and denoising.
[0017] In a preferred embodiment, the oversampling limit is defined by a pixel size smaller than the spatial resolution of the optical imaging system. In a more preferred embodiment, the oversampling limit is defined by a pixel size that is less than or equal to half the spatial resolution of the optical imaging system. In particular, the oversampling limit can be defined by the Nyquist criterion.
[0018] The processor can be configured to determine weightings based on a property representing deconvolution weighting coefficients that increase monotonically and linearly or nonlinearly as the sampling density increases, within at least one range including the oversampling limit. Thus, this property may be considered a function of parameters affecting the sampling density.
[0019] The processor can be configured to determine weightings based on a characteristic representing deconvolution weighting coefficients that are zero when the sampling density is below the oversampling limit, and that increase monotonically and nonlinearly as the sampling density increases when the sampling density is above the oversampling limit. Therefore, deconvolution, which is meaningless or harmful due to artifacts, can be avoided in the undersampling region. Thus, deconvolution can be omitted entirely under undersampling conditions. Conversely, applying deconvolution in the oversampling region can increase spatial resolution.
[0020] Preferably, the monotonic and linear or nonlinear increase in the characteristic is mainly limited to the region around or at the oversampling limit. This advantageous embodiment is based on the insight that as soon as the oversampling limit is reached, the resolution-enhancing effect of deconvolution increases significantly.
[0021] In any case, the above characteristics represent a weighting function that may be predetermined and static, but are not limited to this. Rather, the weighting function can be adapted to the current imaging situation, and thus the weighting function may depend on, for example, the signal-to-noise ratio, the intensity of the detected light, the scan speed, etc.
[0022] The above range may have a width that is, namely, (preferably) 7 times, 5 times, 3 times, 2 times, or 1 times the spatial resolution of the optical imaging system.
[0023] The processor can be configured to generate mixed image pixel data while the weighting changes in response to variations in sampling density. The weighting changes can be determined by considering the characteristics of specific algorithms for deconvolution and denoising.
[0024] The processor can be configured to acquire a sampling density based on at least one visualization parameter and to determine weighting according to the sampling density. The image parameter may include, for example, a parameter that affects the zoom setup for finding a suitable region of interest (ROI), such as the scan field in a confocal microscope.
[0025] In a preferred embodiment, the processor can be configured to perform deconvolution based on a point spread function (PSF) that characterizes the spatial resolution of the optical imaging system. The PSF can be known in advance by considering the optical properties of the imaging system. Alternatively, the PSF may not be known a priori and can be derived from the image according to a blind deconvolution scheme.
[0026] The deconvolution process and the noise removal process can be executed independently or simultaneously, at least one of them. In particular, deconvolution and noise removal are preferably executed in the background as separate image enhancement processes. Subsequently, the results of these processes are combined into the overall result according to the weighting according to the sampling density. Image data mixing can be executed for each pixel, whereby the mixed image pixel data is generated based on the deconvolved image pixel data and the noise-removed image data at a mixing ratio according to the sampling density.
[0027] The deconvolution process may include at least one of the following, namely, Wiener deconvolution, Richardson-Lucy deconvolution, FFT-based deconvolution, Agard-Sedat deconvolution, and Meinel deconvolution.
[0028] The noise removal process may have at least one of the following, namely, smoothing, blur removal, averaging of multiple images, and background difference.
[0029] The processor may have one of a CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), ASIC (Application-Specific Integrated Circuit), and DSP (Digital Signal Processor).
[0030] The image processing system can be configured to be used in wide-field microscopy, laser scanning microscopy, fluorescence microscopy, transmission or reflection optical microscopy, multiphoton microscopy, super-resolution microscopy, interference contrast or polarization microscopy.
[0031] According to another aspect, a microscope having the optical imaging system and the image processing system described above is provided.
[0032] In another embodiment, an image processing method is provided, which includes the following steps: acquiring image pixel data generated by a microscope's optical imaging system; performing a deconvolution operation on the acquired image pixel data to generate deconvolved image pixel data; performing a denoising operation on the acquired image pixel data to generate denoised image pixel data; acquiring the sampling density generated by the optical imaging system for the image pixel data; and, if the sampling density exceeds the oversampling limit, mixing the deconvolved image pixel data and the denoised image pixel data to change (particularly increase) the ratio of the deconvolved image pixel data to the denoised image pixel data to generate mixed image pixel data having a weighting according to the sampling density.
[0033] In another embodiment, a computer program is provided that includes program code that performs the method described above when the computer program runs on a processor.
[0034] The following describes a specific embodiment with reference to the drawings. [Brief explanation of the drawing]
[0035] [Figure 1] Block diagram of a microscope according to one embodiment. [Figure 2] This figure shows the characteristics of the weighting coefficient according to the sampling density, according to one embodiment. [Figure 3] This figure shows the characteristics of the weighting coefficients according to the sampling density, according to another embodiment. [Figure 4] This is a flowchart illustrating an image processing method according to one embodiment. [Figure 5] This is a schematic diagram of the system for implementing the method shown in Figure 4. [Modes for carrying out the invention]
[0036] Figure 1 is a block diagram of a microscope 100 according to one embodiment. The microscope 100 may, but is not limited to, a confocal laser scanning microscope. Figure 1 shows only the components of the microscope 100 that are useful for understanding the mechanism of operation. Needless to say, the microscope 100 may include additional components not explicitly shown in the block diagram of Figure 1, such as a light source for emitting an illumination beam, a scanner for scanning the light beam across the sample to illuminate the sample point by point, and pinholes for removing out-of-focus light in the image during formation.
[0037] The microscope 100 has an optical imaging system 102 that collects detection light from an illuminated point of the sample and directs this detection light to a photodetector 104. The photodetector 104 may be a photomultiplier tube (PMT) or an avalanche photodiode that converts the detection light into an electrical image signal corresponding to one pixel. As a result, the optical imaging system 102 interacts with the photodetector 104 according to a confocal imaging scheme to generate point-by-point image pixel data representing an optical image of the sample.
[0038] The microscope 100 further includes an image processing system 106 with a processor 108, which may have, but is not limited to, a CPU, GPU, FPGA, ASIC, and / or DSP. The image processing system 106 may further include a data acquisition module 110 connected to a detector 104, from which it receives image pixel data. The processor 108 is configured to acquire image pixel data from the data acquisition module 110 and process the data as described below.
[0039] According to the embodiment shown in Figure 1, the microscope 100 has a controller 112 connected to a processor 108 and configured to control the optical imaging system 102 according to instructions received from the processor 108. The processor 108 can instruct the controller 112 to set specific visualization parameters in response to user input. For example, the user can change parameters such as the scan field during imaging to identify a sample ROI by zooming in or out, and to focus the optical imaging system on the ROI. Other changeable visualization parameters include, for example, exposure time, scan speed, specific detector settings, pinhole settings, etc.
[0040] The processor 108 is configured to perform both deconvolution and denoising on image pixel data acquired from the optical imaging system 102. Therefore, the image pixel data generated by the optical imaging system 102 can be considered as raw image data, and the processor 108 applies appropriate deconvolution and denoising algorithms to this raw image data to derive deconvolved image pixel data and denoised image pixel data, respectively. For example, the processor 108 can apply algorithms such as Wiener deconvolution, Richardson-Lucy deconvolution, blind deconvolution (i.e., deconvolution without prior knowledge of the PSF), FFT-based deconvolution, Agard-Sedat deconvolution, and Meinel deconvolution to perform deconvolution on the raw image data. Regarding denoising, the processor 108 can apply algorithms such as smoothing, deblurring, averaging of multiple images, and background subtraction. Needless to say, appropriate algorithms are not limited to the examples given above.
[0041] As explained above, a prerequisite for effective deconvolution is oversampling. Therefore, the sampling density generated by the optical imaging system 102 for image pixel data representing the raw image data for deconvolution should exceed the oversampling limit. If the pixel size of the detector 104 is smaller than the spatial resolution of the optical imaging system 102, the sampling density will be greater than the oversampling limit. The oversampling limit can be defined by the Nyquist criterion, which requires a sampling interval equal to twice the highest spatial frequency of the sample in order to accurately maintain the spatial resolution of the resulting digital image.
[0042] Considering the above preconditions, the processor 108 is adapted to determine weightings based on the sampling density. Based on these weightings, the results of deconvolution and denoising are combined into an overall result representing a mixture of deconvolutionated image pixel data and denoising image pixel data. Preferably, the processor determines the weightings such that the ratio of deconvolutionated image pixel data to denoising image pixel data increases when the sampling density exceeds the oversampling limit. Thus, if the sampling density is too low to effectively perform deconvolution, denoising will prevail in the resulting image pixel data. On the other hand, if the sampling density is sufficiently high to effectively perform deconvolution, deconvolution will prevail in the resulting image pixel data. Two examples of such a mixture of deconvolution and denoising weightings in the resulting image pixel data are shown in Figures 2 and 3.
[0043] Figure 2 is a diagram illustrating the characteristics that determine the weighting coefficient W_DECON in response to the sampling density, which may change during real-time visualization. The weighting coefficient W_DECON determines the ratio of deconvolutional image pixel data to denoised image pixel data in mixed image pixel data. The weighting coefficient W_DECON is complementary to the weighting coefficient W_DENOISE, which determines the ratio of denoised image pixel data. Therefore, the sum of the weighting coefficients W_DECON and W_DENOISE is 1.
[0044] In Figure 2, the horizontal axis represents the sampling density, and the vertical axis represents the weighting coefficient W_DECON (=1-W_DENOISE). The sampling density range is divided into an undersampling region A and an oversampling region B. Regions A and B are separated by the oversampling limit NL, which is determined by the Nyquist criterion. Roughly speaking, deconvolution is not substantially effective in region A, while it is substantially effective in region B. According to the properties in Figure 2, the deconvolution weighting coefficient W_DECON increases nonlinearly with increasing sampling density. In the intermediate range between regions A and B, around the oversampling limit N, the deconvolution weighting coefficient W_DECON increases steeply from a value close to 0 to a value close to 1, according to its nonlinear properties. Therefore, the most significant change in the weighting coefficient W_DECON occurs at the oversampling limit N, where, as the sampling density decreases, deconvolution remains possible but becomes increasingly ineffective. Furthermore, the characteristics shown in Figure 2 change continuously, thereby avoiding spontaneous changes in image perception caused by abrupt switching from denoising to deconvolution and vice versa. This is particularly advantageous when the sampling density changes during real-time visualization by changing one or more parameters, for example, by performing zoom-in or zoom-out operations to find the appropriate ROI of the sample.
[0045] Figure 3 shows another example of the characteristics by which the processor 108 determines the deconvolution weighting coefficient W_DECON. According to the characteristics in Figure 3, the deconvolution weighting coefficient W_DECON is zero when the sampling density is less than the oversampling limit N. Starting from the oversampling limit N, the weighting coefficient W_DECON increases steeply to a value close to 1, according to the nonlinear characteristics. Therefore, deconvolution that is meaningless or rather detrimental in terms of artifacts generated by noise amplification can be avoided in the undersampling region A. Thus, deconvolution can be omitted entirely in the undersampling region A. In contrast, resolution improvement is guaranteed in the oversampling region B, where deconvolution is effective and accurate.
[0046] To determine appropriate weighting for mixing deconvolutional image pixel data and denoising image pixel data, processor 108 is configured to obtain the currently set sampling density when the image pixel data is generated. For this purpose, processor 108 obtains one or more currently set visualization parameters that affect the sampling density, such as the spatial resolution of the optical imaging system 102, pixel size, scan field, etc., and can determine the corresponding sampling density based on these parameters using the characteristics illustrated in Figures 2 and 3. These characteristics can be predetermined. Alternatively, they can be adopted in real time by processor 108, taking into account the current visualization conditions. In other words, these characteristics are updated as a function of parameters such as signal-to-noise ratio, intensity, scan rate, etc.
[0047] Figure 4 shows a flowchart illustrating a method implemented by the image processing system 106 of a microscope 100 to generate a weighted mixture of deconvolved image pixel data and denoised image pixel data according to one embodiment. According to this embodiment, the method is performed during real-time visualization, but is not limited thereto.
[0048] The method shown in Figure 4 begins in step S1, where an optical image of the sample is acquired by the optical imaging system 102, and this optical image is converted into an electrical image signal by the detector 104. This image signal is stored in the data acquisition module 110 in the form of digital image pixel data. Furthermore, the processor 108 acquires one or more visualization parameters and determines the current sampling density based on these parameters. Accordingly, in step S2, the image pixel data and sampling density become available for further processing.
[0049] In the embodiment shown in Figure 4, it is assumed that the characteristics of the embodiment shown in Figure 3 are applied to combine deconvolution and denoising. As described above, in this embodiment, deconvolution can be omitted when the sampling density is in the undersampling region A. Therefore, in step S3, the processor 108 determines whether the sampling density is greater than or equal to the Nyquist limit NL. If yes (Y), the sampling density is in the oversampling region B, and in step S4, the processor 108 performs deconvolution on the image pixel data. If no (N), step S4 is skipped and no deconvolution is performed on the image pixel data. Therefore, in step S5, deconvolved image pixel data is available if deconvolution has been performed.
[0050] In step S6, the processor 108 performs denoising on the image pixel data. Therefore, in step S7, the denoised image pixel data is available.
[0051] In step S8, the processor 108 determines the deconvolution weight W_DECON as a function of sampling density based on the characteristics shown in Figure 3. Once W_DECON is determined, the complementary denoising weight W_DENOISE is also determined, since the sum of W_DECON and W_DENOISE is 1.
[0052] In step S9, the processor multiplies the deconvolved image pixel data, which became available in step S5, by the deconvolved weight W_DECON (if the sampling density is greater than or equal to the Nyquist limit NL). Thus, in step S10, the weighted deconvolved image pixel data is available.
[0053] In step S11, the processor 108 multiplies the denoised image pixel data, which became available in step S7, by the denoising weight W_DENOISE. Thus, in step S12, the weighted denoised image pixel data is available.
[0054] In step S13, the processor 108 adds the weighted deconvolution image pixel data and the weighted denoising image pixels that became available in steps S10 and S12, respectively. Thus, in step S14, the mixed image pixel data is available, and this data contains the deconvolution result and denoising result according to the weighting determined in step S8. In particular, when the sampling density is in the undersampling region A, the mixed image pixel data contains only the denoising image pixel data, in which case the weight W_DENOISE is equal to 1. On the other hand, when the sampling density is in the oversampling region, the mixed image pixel data contains both the denoising image pixel data and the deconvolution image pixel data, in which case the sum of the weights W_DECON and W_DENOISE is 1.
[0055] As used herein, the term "and / or" includes all possible combinations of one or more of the related items and may be abbreviated as " / ".
[0056] While several embodiments have been described in the context of the apparatus, it is clear that these embodiments also represent descriptions of the corresponding methods, where blocks or apparatus correspond to steps or features of steps. Similarly, embodiments described in the context of steps also represent descriptions of the corresponding blocks, items, or features of the corresponding apparatus.
[0057] Some embodiments relate to a microscope that includes a system such as those described in relation to one or more of Figures 1 to 4. Alternatively, the microscope may be part of a system such as those described in relation to one or more of Figures 1 to 4, or may be connected to a system such as those described in relation to one or more of Figures 1 to 4. Figure 5 shows a schematic diagram of a system 500 configured to carry out the methods described herein. The system 500 includes a microscope 510 and a computer system 520. The microscope 510 is configured to take images and is connected to the computer system 520. The computer system 520 is configured to carry out at least some of the methods described herein. The computer system 520 may be configured to run machine learning algorithms. The computer system 520 and the microscope 510 may be separate entities or may be integrated within a common housing. The computer system 520 may be part of the central processing system of the microscope 510, and / or the computer system 520 may be part of a dependent component of the microscope 510, such as a sensor, actor, camera, or illumination unit of the microscope 510.
[0058] The computer system 520 may be a local computer device (e.g., a personal computer, laptop, tablet computer, or mobile phone) comprising one or more processors and one or more storage devices, or it may be a distributed computer system (e.g., a cloud computing system comprising one or more processors and one or more storage devices distributed to various locations such as local clients and / or one or more remote server farms and / or data centers). The computer system 520 may include any circuit or combination of circuits. In one embodiment, the computer system 520 may include one or more processors, which can be of any kind. As used herein, the processor may be intended to be any kind of computing circuit, such as a microprocessor for a microscope or microscopic component (e.g., a camera), a microcontroller, a composite instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multicore processor, a field-programmable gate array (FPGA), or any other kind of processor or processing circuit. Other types of circuits that may be included in the computer system 520 may be custom circuits, application-specific integrated circuits (ASICs), etc., such as one or more circuits (communication circuits, etc.) used in wireless devices such as mobile phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. The computer system 520 may also include one or more storage devices that may include one or more memory elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard drives and / or one or more drives that handle removable media such as compact discs (CDs), flash memory cards, digital video discs (DVDs), etc.The computer system 520 may also include a display device, one or more speakers and a controller which may include a keyboard and / or mouse, trackball, touchscreen, voice recognition device, or any other device which enables a user of the system to input information into and receive information from the computer system 520.
[0059] Some or all of the steps may be performed by a hardware device (or by using a hardware device), such as a processor, microprocessor, programmable computer, or electronic circuit. In some embodiments, one or more of the most critical steps may be performed by such a device.
[0060] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. This implementation is feasible using a non-transient recording medium, which is a digital recording medium, etc., that stores electronically readable control signals and cooperates (or can cooperate) with a programmable computer system to carry out each method. Examples include floppy disks, DVDs, Blu-rays, CDs, ROMs, PROMs and EPROMs, EEPROMs, or FLASH memory. Thus, the digital recording medium may be computer-readable.
[0061] Some embodiments of the present invention include a data carrier having electronically readable control signals that can cooperate with a programmable computer system so as to carry out any of the methods described herein.
[0062] Generally, embodiments of the present invention can be implemented as a computer program product comprising program code, which operates to perform one of the methods when the computer program product is executed on a computer. This program code may be stored, for example, on a machine-readable carrier.
[0063] Another embodiment includes a computer program stored in a machine-readable carrier for carrying out any of the methods described herein.
[0064] Therefore, in other words, embodiments of the present invention are computer programs having program code for carrying out any of the methods described herein when the computer program is executed on a computer.
[0065] Accordingly, another embodiment of the present invention is a recording medium (or data carrier or computer-readable medium) containing a stored computer program for carrying out any of the methods described herein when executed by a processor. The data carrier, digital recording medium, or recording medium is typically tangible and / or non-transient. Another embodiment of the present invention is an apparatus, such as those described herein, comprising a processor and a recording medium.
[0066] Therefore, another embodiment of the present invention is a data stream or signal sequence representing a computer program for carrying out any of the methods described herein. The data stream or signal sequence may be configured to be transmitted, for example, over a data communication connection, such as the Internet.
[0067] Another embodiment includes processing means, for example, a computer or programmable logic device configured or adapted to carry out any of the methods described herein.
[0068] Another embodiment includes a computer having an installed computer program for carrying out any of the methods described herein.
[0069] Another embodiment of the present invention includes an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for carrying out any of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.
[0070] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, the field-programmable gate array may cooperate with a microprocessor to carry out any of the methods described herein. Generally, the methods are advantageously carried out by any hardware device. [Explanation of Symbols]
[0071] 100 Microscopes 102 Optical imaging system 104 detectors 106 Image Processing Systems 108 processors 110 Data Acquisition Module 112 Controllers A Undersampling Region B Oversampling region NL Nikest Limit
Claims
1. An image processing system (106), comprising: The image processing system (106) includes a processor (108); The processor (108) acquiring image pixel data produced by an optical imaging system of the microscope (100); performing a deconvolution process on the acquired image pixel data to generate deconvolved image pixel data; performing a noise reduction process on the acquired image pixel data to generate noise-reduced image pixel data; obtaining a sampling density at which the image pixel data is produced by the optical imaging system; and when the sampling density exceeds an oversampling limit (NL), the deconvolved image pixel data and the denoised image pixel data are mixed to generate mixed image pixel data having a weighting according to the sampling density, in order to change (in particular increase) a ratio of the deconvolved image pixel data to the denoised image pixel data. An image processing system (106).
2. the oversampling limit (NL) is determined by a pixel size smaller than the spatial resolution of the optical imaging system or by the Nyquist criterion; The image processing system (106) of claim 1.
3. the processor (108) is configured to determine weights based on a characteristic representing deconvolution weighting coefficients that increase monotonically and linearly or nonlinearly as sampling density increases in at least one range that includes the oversampling limit (NL). The image processing system (106) of claim 1.
4. the processor (108) is configured to determine weights based on a characteristic representing deconvolution weighting coefficients that are zero when the sampling density is less than the oversampling limit (NL) and that increase monotonically and non-linearly with increasing sampling density when the sampling density is equal to or greater than the oversampling limit (NL). The image processing system (106) of claim 1.
5. the monotonic and nonlinear increase of the characteristic is primarily limited to a region around or to the oversampling limit (NL); The image processing system (106) of claim 3.
6. said range having a width that is one of the following: 7 times, 5 times, 3 times, 2 times or 1 times the (preferably lateral) spatial resolution of said optical imaging system; The image processing system (106) of claim 3.
7. the processor (108) is configured to generate the blended image pixel data while the weightings vary as the sampling density varies. The image processing system (106) of claim 1.
8. the processor (108) is configured to obtain the sampling density based on at least one visualization parameter and to determine the weighting according to the sampling density. The image processing system (106) of claim 1.
9. the processor (108) is configured to perform the deconvolution process based on a point spread function that characterizes a spatial resolution of the optical imaging system. The image processing system (106) of claim 1.
10. the processor (108) is configured to generate the blended image pixel data while real-time visualization is performed by the optical imaging system. The image processing system (106) of claim 1.
11. The deconvolution process and the noise removal process are performed at least one of independently and simultaneously. The image processing system (106) of claim 1.
12. The deconvolution process comprises at least one of the following: Wiener deconvolution, Richardson-Lucy deconvolution, blind deconvolution, FFT-based deconvolution, Agard-Sedat deconvolution, and Meinel deconvolution. The image processing system (106) of claim 1.
13. The denoising process comprises at least one of the following: smoothing, deblurring, multi-image averaging and background subtraction. The image processing system (106) of claim 1.
14. the image processing system (106) is configured for use in wide-field microscopy, laser scanning microscopy, fluorescence microscopy, transmitted or reflected light microscopy, multiphoton microscopy, super-resolution microscopy, interference contrast or polarized light microscopy, The image processing system (106) of claim 1.
15. A microscope (100) having an optical imaging system and an image processing system (106) according to any one of claims 1 to 14.
16. An image processing method, the image processing method comprising: acquiring image pixel data produced by an optical imaging system of the microscope (100); performing a deconvolution process on the obtained image pixel data to generate deconvolved image pixel data; performing a noise reduction process on the acquired image pixel data to generate noise-reduced image pixel data; acquiring a sampling density at which the image pixel data is produced by the optical imaging system; blending the deconvolved image pixel data with the denoised image pixel data to generate blended image pixel data having weighting according to the sampling density, in order to change (in particular increase) a ratio of the deconvolved image pixel data to the denoised image pixel data when the sampling density exceeds an oversampling limit (NL); An image processing method comprising:
17. 17. A computer program having a program code for performing the method of claim 16, when the computer program runs on a processor (108).