Image processing method, program, and recording medium
The image processing method addresses the challenge of distinguishing between sensor and light-induced noise by generating a noise filter based on fluctuation and sensor variances, enhancing image processing accuracy.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-09-17
- Publication Date
- 2026-03-30
AI Technical Summary
Existing noise reduction methods fail to accurately distinguish between noise caused by image sensors and noise caused by fluctuations in the refractive index distribution of the medium, leading to inaccurate noise reduction.
An image processing method that filters noise by obtaining variances based on the power spectrum of optical fluctuations and image sensor characteristics, generating a noise filter to account for both types of noise sources.
Enables accurate noise reduction in moving images by accounting for noise variations due to both sensor and light fluctuations, improving image processing accuracy.
Smart Images
Figure 2026054599000001_ABST
Abstract
Description
[Technical Field]
[0001] This disclosure relates to an image processing method for reducing noise in moving images. [Background technology]
[0002] To address the degradation of image quality in video footage acquired by imaging devices due to noise, technologies for reducing noise are known. In addition to noise caused by sensor characteristics, captured video also generates noise caused by fluctuations in the refractive index distribution of the medium, such as the atmosphere (light fluctuations).
[0003] Patent Document 1 discloses a method for reducing noise in an image using adaptive filtering. [Prior art documents] [Patent Documents]
[0004] [Patent Document 1] Reprint of Bulletin No. 2008-093835 [Overview of the Initiative] [Problems that the invention aims to solve]
[0005] However, the method described in Patent Document 1 reduces noise using an index value that uniquely determines the noise in the image, and therefore does not clearly distinguish between noise caused by the sensor and noise caused by fluctuations. As a result, if either noise component increases, it may not be possible to reduce the noise accurately. [Means for solving the problem]
[0006] An image processing method according to an embodiment of the present invention is an image processing method for reducing noise by filtering from a frame of a moving image obtained by imaging using an image sensor, the method including: obtaining a first variance based on a power spectrum of fluctuation optical characteristics; obtaining a second variance based on characteristics of the image sensor; and generating a noise filter based on the first variance and the second variance.
Advantages of the Invention
[0007] According to the above-described image processing method, an image processing method capable of processing an image with high accuracy can be provided.
Brief Description of the Drawings
[0008] [Figure 1] This is a flowchart in the present embodiment. [Figure 2] This is a diagram showing an example of a point image distribution forming a speckle shape due to the influence of fluctuations. [Figure 3] This is a diagram showing an image processing apparatus in the present embodiment. [Figure 4] This is a diagram showing a system in the present embodiment.
Embodiments for Carrying Out the Invention
[0009] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings. Note that each drawing may be drawn at a scale different from the actual scale for convenience. Also, as shown in each drawing, the same members are given the same reference numerals, and redundant descriptions are omitted. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined.
[0010] Consider the process where light emitted from each point on the subject is affected by fluctuations before entering the imaging system and forming an incoherent image on the imaging surface. In this embodiment, fluctuations are variations in the refractive index distribution of a medium such as the atmosphere in the subject space. The subject space refers to the space between the subject and the imaging system. Due to fluctuations, the image formation results in different positional shifts and blurs over time, i.e., from frame to frame. Therefore, when these are concatenated in a time series, a video is obtained in which the image of the subject is degraded by noise caused by light fluctuations, etc. The pixel value y(r,t) at pixel position r at frame number (time) t of the video y is given by the following equation 1, where psf (Point Spread Function) is determined by the imaging system and fluctuations, o is the reflection distribution of the subject, and n is white noise. Note that * represents convolution. In the derivation of each of the following equations, it is assumed that the subject is stationary.
[0011]
number
[0012] When fluctuations are strong, or when the spatial characteristics of the imaging system change significantly, the PSF shape varies from pixel to pixel. In this case, the entire frame can be divided into patch regions smaller than the isoplanatic size, and the PSF shape can be determined for each pixel within those regions.
[0013] Next, we perform a Fourier transform on Equation 1. In this case, the Fourier transform Y of y at spatial frequency ν is given by Equation 2, where the Optical Transfer Function (OTF) due to the imaging system and fluctuations is O, the Fourier transform of the subject is O, and the Fourier transform of the noise is N.
[0014]
number
[0015] From now on, arguments for each term will be omitted unless necessary. Also, the frame average or statistical average of quantity A will be omitted. In this case, equation 2 can be expressed as equation 3. Furthermore, δOTF is expressed in equation 4.
[0016]
number
[0017]
number
[0018] OTF is an optical property that includes the effect of optical fluctuations, and its inverse transform, psf, becomes speckled as shown in Figure 2(a). δOTF is obtained by subtracting the average value <OTF> from OTF, and is shown as shown in Figure 2(b). Furthermore, the inverse Fourier transform of δOTF forms speckles as shown in Figure 2(c). In other words, according to Equation 3, an average image is first formed in the frame by <OTF>O. Noise N from sensors, etc., is then added. Furthermore, noise due to optical fluctuations other than N is generated by the fluctuation δOTF, which convolves the subject O in a speckled manner. Next, we show Equation 5 by absolutely squaring both sides of Equation 3. Here, the * above and to the right of the variable represents the complex conjugate, and Re(A) represents the real part of the complex number A.
[0019]
number
[0020] Next, we calculate the frame average of Equation 5. There is no correlation between the fluctuations of light due to fluctuations in the medium such as the atmosphere and the noise N from sensors, etc. Also, we can approximate that <δOTF> = <N> = 0. Therefore, the average of each of the last three terms in Equation 5 is 0, and we obtain Equation 6.
[0021]
number
[0022] In Equation 6, the power spectrum of the frame <|Y| 2 The first term in equation > represents the power spectrum of the image of the subject formed by the propagation of average light without fluctuations. The second term represents the power spectrum due to noise from sensors, etc. And the third term represents the power spectrum of noise due to light fluctuations. Therefore, in equation 6, the second and third terms are combined to represent the power spectrum P of the total noise. noise By doing so, it can be summarized as shown in equation 7 below. Note that P noise This is expressed by Equation 8.
[0023]
number
[0024]
number
[0025] In the following, let r be the central pixel position of the patch region, and Δr be the relative pixel position within the region with respect to r. That is, the real-space coordinate corresponding to the frequency ν above is Δr. Therefore, by performing an inverse Fourier transform on both sides of equation 8, we obtain equation 9.
[0026]
number
[0027] Here, the σ on the right side n 2 is the variance of noise n, which is uniform regardless of the pixel position r. Also, δ is the Kronecker delta. Furthermore, σ turb 2 Equation 10 is obtained by performing an inverse Fourier transform on the IFT based on Equation 7. Below, in order to clarify that these are values for a patch region centered on pixel position r, we will again express them by adding r as an argument, such as IFT[*](r,*), O(r,*), δOTF(r,*), etc.
[0028]
number
[0029] Here, in Equations (9) and (10), when the central pixel Δr = 0, by omitting the arguments, the variance at position r is represented by Equation (11).
[0030]
Equation
[0031] The left - hand - side σ in Equation (11) noise 2 is hereinafter referred to as the total noise variance, and the right - hand - side σ turb 2 is referred to as the variance of light fluctuation. Also, σ turb 2 can be expressed by Equation (12) based on Equation (10). Equation (12) is in the form of an integral in the frequency space, and the integration range of the frequency can be from the origin to the Nyquist frequency determined by the imaging system.
[0032]
Equation
[0033] From Equations (11) and (12), when there is an influence of degradation caused by light fluctuation, it can be seen that the evaluation of image noise is insufficient with only the uniform noise σ n 2 The noise in an image depends on the fluctuation of the medium, the imaging system, and the subject. In particular, the noise σ turb 2 caused by light fluctuation is not necessarily uniform for all pixel positions r in the image. Therefore, by generating a noise filter using the noise σ turb 2 caused by light fluctuation, denoising can be performed accurately. Furthermore, for the noise filter based on the variance σ n 2 of the existing noise, with respect to the noise variance σ turb 2 By adding this, a noise filter based on noise including light fluctuations can be generated. In other words, to improve the accuracy of denoising, it is preferable to generate a noise filter based on the dispersion based on the product of the image power spectrum (power spectrum of the subject) based on pixel values and the power spectrum of the optical properties of the fluctuations, and the dispersion based on the characteristics of the image sensor.
[0034] The pixel values should be the power spectra of one or more pixel values in the image (or frame). Furthermore, the dispersion based on the characteristics of the image sensor only needs to be based on the characteristics of the image sensor; if necessary, the dispersion may be determined by considering the settings of the imaging device used to capture the image, the imaging optical system, the light intensity of the subject, etc. This configuration can improve the accuracy of denoising.
[0035] Furthermore, if necessary, the image may be divided into multiple regions by processing such as cropping, and the power spectrum and variance may be calculated for each region. With this configuration, noise due to light fluctuations that differ at each pixel position can be estimated with greater accuracy. It is also preferable to calculate the variance by frequency integration of the product of the power spectrum of the image and the power spectrum of the optical properties of the fluctuations.
[0036] Here, the power spectrum of the subject appearing in equation 12 is |O|. 2 This is the estimated power spectrum of the image determined by the frame values in this embodiment. Here, as the power spectrum of the image, equation 13 is obtained by taking the absolute square of the Fourier transform Y(r,ν) of the patch at position r in frame y(r,t).
[0037]
number
[0038] Furthermore, the power spectrum described above can be obtained using Equation 14 by using the sum of the squares of the pixel values within the patch centered on r at the pixel position r value y(r,t) of the selected frame number t.
[0039]
number
[0040] Furthermore, the Fourier transform of the image obtained by applying arbitrary image corrections such as Wiener filtering to the power spectrum of the other image is O2. est Equation 15 is obtained by using the absolute square of (r).
[0041]
number
[0042] Next, the power spectrum of the optical properties of the fluctuations in Equation 12 <|δOTF| 2 Let's explain the following. The average value of the psf, <psf>, can be considered as a point-symmetric distribution with respect to its origin (the central pixel position r of the patch region), as shown in Figure 2(b). In that case, the Fourier transform, <OTF>, can be considered as a real number. Furthermore, when <δOTF> = <N> = 0, from the absolute square of both sides of equation 4, we get <|δOTF| 2 > is represented by the following equation 16.
[0043]
number
[0044] The right-hand side of Equation 16 can be obtained from the parameters of the imaging system and the interference length using analytical formulas of optical theory as follows. Here, assuming that the fluctuations of light follow Kolmogorov statistics, each term on the right-hand side of Equation 16 can be expressed by Equations 17 and 18. Note that the OTF of the imaging system is OTF lens Let D be the aperture diameter of the imaging system, and r0 be the interference size (Fried parameter) at the entrance pupil of the imaging system. In this case, it is independent of the position r of each pixel.
[0045]
number
[0046]
number
[0047] Furthermore, u is the diffraction-limited frequency, and is expressed in Equation 19. Note that f is the focal length of the imaging system and λ is the wavelength of interest.
[0048]
number
[0049] The interference length r0 can be obtained from the captured video using known methods. Alternatively, it may be determined by measurement using a laser measurement system, for example.
[0050] Also, <|δOTF| 2 The ≥|δOTF| can also be obtained by defining the optical properties power spectrum of the fluctuations based on Equations 4 and 16 and performing simulations. For example, by specifying the interference length r0 to define the power spectrum of the von Kármán model and repeatedly performing optical propagation simulations using the Monte Carlo method, the ensemble of PSFs can be obtained. Furthermore, by performing Fourier transforms and averaging operations corresponding to each term on the right-hand side of Equation 16 on these results, the ≥|δOTF| corresponding to r0 can be obtained. 2 > can be calculated.
[0051] In the embodiment described later, the calculated results are saved as electronic data in an arbitrary location, and the interference length r0 is specified during the denoising process to obtain the corresponding <|δOTF| 2 You may also refer to >. Alternatively, you can construct your own power spectrum model and use analytical formulas or simulations similar to those described above to determine <|δOTF| 2 You may also find 〉. In this case, 〈|δOTF| 2 > may depend on the pixel position r, but in that case, <|δOTF| for each pixel position in the frame. 2 It is preferable to obtain the value of > and make it available for reference during denoising.
[0052] Alternatively, only at locations where the value changes significantly, such as the boundaries of the subject, <|δOTF| 2 The '>' elements can be densely arranged, while those with gradual changes can be sparsely arranged, and the values at each pixel can be interpolated. Furthermore, the above simulation method can also be used to reflect detailed characteristics such as the complex structure and aberrations of the imaging system in the OTF. For example, if the image spectrum is frequency-independent as in Equation 14, the dispersion σ of the light fluctuations turb 2 (r) as, <|δOTF| 2 You can use the product of the frequency integral of only >.
[0053] Here, we show the specific denoising formula. In this embodiment, the noise dispersion σ n 2 In a denoising process using the parameter (hereinafter referred to as the noise dispersion parameter), the dispersion of light fluctuations σ turb 2 The total noise variance σ added noise 2 Replace with . For example, denoising is performed using the following equation 22 as an adaptive Wiener filter that changes the weight for each position.
[0054]
number
[0055] This is the pixel value after denoising.
[0056]
number
[0057] Here, σ x 2 σ is expressed by the following equation 22, y 2 (r) is the variance of the pixel values within the patch at position r in y.
[0058]
number
[0059] Furthermore, when denoising is performed using a bilateral filter, the pixel values after denoising are expressed by Equation 23.
[0060]
number
[0061] σ in Equation 23 r 2 This is an adjustment parameter that represents the area of the pixel range centered on the pixel position r where pixel values are mixed during denoising. For example, it can be a constant value approximately the square of the patch size. Note that this embodiment is not limited to the above, and any denoising process that has noise variance as a parameter can achieve a similar effect by replacing it with total variance. [Examples]
[0062] Next, with reference to Figures 3 and 4, the image processing system 1 according to Example 1 will be described.
[0063] The image processing system 1 includes an image processing device 100, an input device 111, and a display device 112, and performs image processing to reduce noise by filtering.
[0064] The image processing device 100 includes a storage unit 101, a processing unit 102, and a connection unit 103.
[0065] The storage unit 101 is a component for storing electronic data, such as a semiconductor memory, hard disk, or server on a network, and stores a program capable of executing a process to correct image degradation caused by filtering. It may also store execution values used to correct image degradation caused by filtering. Furthermore, the storage unit 101 may be configured to store results obtained from each process.
[0066] The processing unit 102 is composed of one or more processors such as CPUs and can perform processing to correct image degradation caused by filtering.
[0067] The connection unit 103 connects the image processing device 100 with the input device 111 and the display device 112, and transmits and receives various commands and data.
[0068] In the image processing system 1, images during and after processing can be viewed via the display device 112, and instructions for executing processing can be given via the input device 111. Here, the input device 111 is, for example, a keyboard or mouse. The display device 112 is, for example, a liquid crystal display or a projector.
[0069] The image processing system 1 according to this embodiment is not limited to this. For example, a program capable of performing a process to correct image degradation due to filtering may be stored in the storage medium 113 in Figure 3. In this case, the processing unit 102 is configured to read the program from the storage medium 113 and execute it. The storage medium 113 may be, for example, a semiconductor memory, a hard disk, or a server on a network.
[0070] In the image processing method for reducing noise by filtering in Example 1, the following information is used as needed: a moving image (a collection of frames), the frame to be processed, information about the imaging system (focal length, aperture diameter, wavelength of interest, sensor pixel spacing), and interference length r0. The size of the image patch and the dispersion of sensor noise may also be used.
[0071] The moving image may be a single frame or multiple still image files. There may be multiple interference lengths r0, and for each value of r0, the power spectrum of the fluctuation's optical characteristics may be stored as electronic data. The wavelength of interest may be the value that corresponds to the center or peak of the sensitivity characteristic distribution of the sensor mounted on the imaging system, or any value from the wavelength range handled by the imaging system may be used.
[0072] The following describes each step of the image processing flow in detail. In this embodiment, an example is shown in which a specific value is selected in advance from the above execution values, but it is not limited to this. For example, regarding interference length, the flow execution value may be set using separately measured values. Alternatively, it may be estimated from the video to be processed.
[0073] Here, referring to Figure 1, we will describe each step of the image processing method that reduces noise by filtering.
[0074] In step S101, one or more undenoised pixels are selected from the selected frame, and the image data is cropped to a patch size centered on those pixels.
[0075] In step S102, the power spectrum of the image is calculated from the image data cropped in S101. At this time, the Fast Fourier Transform can be used as the Fourier transform.
[0076] In step S103, noise dispersion (first dispersion) is obtained based on the power spectrum obtained in S102 and the power spectrum of the optical properties of the fluctuation, and noise dispersion (second dispersion) is obtained based on the characteristics of the image sensor.
[0077] In step S104, the total noise variance is calculated based on the first variance and the second variance. In this embodiment, the total noise variance is calculated by adding the first variance and the second variance, but the method for calculating the total noise variance is not limited to this.
[0078] In step S105, the total noise variance is set as the noise parameter, and a noise filter is generated for one or more pixels selected in S101.
[0079] Next, in step S106, denoising is performed using a noise filter, and the process is terminated. If there are pixels (or regions) in the selected frame that have not been denoised, one or more pixels are selected from them, and the process returns to step S101. If there are no pixels that have not been denoised, the process terminates.
[0080] Note that the flow in this embodiment is just one example and is not limited to it. For example, step S105 may be repeated to generate a noise filter for the frame, and then denoising may be performed on the frame. In that case, a termination check may be performed in S105 to determine whether the generation of the noise filter has finished.
[0081] Although preferred embodiments and examples of the present invention have been described above, the present invention is not limited to these embodiments and examples, and various combinations, modifications, and changes are possible within the scope of its gist.
[0082] Embodiments of the present invention include the following methods and configurations.
[0083] [Method 1] An image processing method for reducing noise from frames of moving images obtained by imaging using an image sensor, by filtering, A step of obtaining a first dispersion based on the power spectrum of the optical properties of the fluctuation, A step of obtaining a second dispersion based on the characteristics of the image sensor, An image processing method characterized by comprising the step of generating a noise filter based on the first variance and the second variance.
[0084] [Method 2] The image processing method according to Method 1, characterized in that, in the step of obtaining the first variance, the first variance is obtained based on the product of the power spectrum of the optical properties of the fluctuation and the power spectrum of the image based on the pixel values of the frame.
[0085] [Method 3] The image processing method according to Method 2, characterized in that the first variance is obtained by frequency integration of the product.
[0086] [Method 4] The image processing method according to any one of methods 1 to 3, characterized in that the noise filter is generated by adding the first variance and the second variance.
[0087] [Method 5] The image processing method according to any one of methods 1 to 4, characterized in that the noise filter is an adaptive filter.
[0088] [Method 6] The image processing method according to method 5, characterized in that the adaptive filter is an adaptive Wiener filter.
[0089] [Method 7] The image processing method according to method 5, characterized in that the adaptive filter is a bilateral filter.
[0090] [Method 8] The image processing method according to any one of methods 1 to 7, characterized in that the fluctuation is a variation in the refractive index distribution of the medium in the subject space.
[0091] [Method 9] The image processing method according to any one of methods 1 to 8, characterized in that the optical characteristics of the fluctuations are determined based on the interference length obtained from the captured video.
[0092] [Method 10] The image processing method according to any one of methods 1 to 9, characterized in that the optical properties of the fluctuations are obtained by an analytical formula of optical theory or by simulation.
[0093] [Method 11] An image processing method according to any one of methods 1 to 10, further comprising the step of reducing noise in the frame using the noise filter.
[0094] [Program 1] A program characterized by causing a computer to execute the image processing method described in any one of methods 1 to 11.
[0095] [Configuration 1] A storage medium characterized by storing the program described in Program 1. [Explanation of Symbols]
[0096] S103 Step to obtain noise dispersion S105 Step to generate a noise filter
Claims
1. An image processing method for reducing noise from frames of moving images obtained by imaging using an image sensor, by filtering, A step of obtaining a first dispersion based on the power spectrum of the optical properties of the fluctuation, A step of obtaining a second dispersion based on the characteristics of the image sensor, An image processing method characterized by comprising the step of generating a noise filter based on the first variance and the second variance.
2. The image processing method according to claim 1, characterized in that, in the step of obtaining the first variance, the first variance is obtained based on the product of the power spectrum of the optical properties of the fluctuation and the power spectrum of the image based on the pixel values of the frame.
3. The image processing method according to claim 2, characterized in that the first variance is obtained by frequency integration of the product.
4. The image processing method according to claim 1, characterized in that the noise filter is generated by adding the first variance and the second variance.
5. The image processing method according to claim 1, characterized in that the noise filter is an adaptive filter.
6. The image processing method according to claim 5, characterized in that the adaptive filter is an adaptive Wiener filter.
7. The image processing method according to claim 5, characterized in that the adaptive filter is a bilateral filter.
8. The image processing method according to claim 1, characterized in that the fluctuation is a variation in the refractive index distribution of the medium in the subject space.
9. The image processing method according to claim 1, characterized in that the optical characteristics of the fluctuations are determined based on the interference length obtained from the captured video.
10. The image processing method according to claim 1, characterized in that the optical properties of the fluctuations are obtained by an analytical formula of optical theory or by simulation.
11. The image processing method according to any one of claims 1 to 10, further comprising the step of reducing noise in the frame using the noise filter.
12. A program characterized by causing a computer to execute the image processing method described in any one of claims 1 to 10.
13. A storage medium characterized by storing the program described in claim 12.