Image processing device

Abstract

To improve the noise reduction effect. [Solution] The image processing device generates a difference signal between a first image signal, which is an image signal converted to a different number of bits than the input image signal, and a second image signal, which is the image signal before the delay with respect to the first image signal. Based on the input image signal and a signal obtained by multiplying the difference signal, which is expressed with a different number of bits than the input image signal, by a coefficient representing the strength of noise reduction, the device generates an output image signal.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus.

Background Art

[0002] Conventionally, a recursive noise reduction (Three-Dimension Noise Redu ction; 3DNR) method has been proposed (see, for example, Patent Document 1).

Prior Art Literature

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problem to be Solved by the Invention

[0004] However, with such a conventional 3DNR method, there have been cases where a sufficient noise reduction effect cannot be obtained when attempting to obtain a noise reduction effect without reducing resolution, which poses a problem.

[0005] The present invention has been made in view of such circumstances, and an object thereof is to provide an image processing apparatus capable of improving the noise reduction effect.

Means for Solving the Problem

[0006] One aspect of the present invention is an image processing apparatus comprising: a noise extraction unit that generates a difference signal between a first image signal, which is an image signal converted to a different number of bits than the input image signal, and a second image signal, which is an image signal before the delay with respect to the first image signal; and an output image generation unit that generates an output image signal based on the input image signal and a signal obtained by multiplying the difference signal, which is expressed with a different number of bits than the input image signal, by a coefficient representing the strength of noise reduction.

[0007] Furthermore, in an image processing apparatus according to one aspect of the present invention, the first image signal and the second image signal are image signals converted to a second number of bits smaller than the number of bits of the input image signal, and the output image generation unit multiplies the difference signal expressed in the second number of bits by a cyclic coefficient representing the strength of noise reduction, subtracts this from the input image signal, and generates an output image signal. [Effects of the Invention]

[0008] According to the present invention, the noise reduction effect can be improved. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of the functional configuration of the image processing apparatus of this embodiment. [Figure 2] This figure shows an example of the pixel arrangement of the image sensor in this embodiment. [Figure 3] This figure shows an example of the functional configuration of the noise reduction unit in this embodiment. [Figure 4] This figure shows an example of the conversion of the grayscale bit width by the limiter section of this embodiment. [Figure 5] This figure shows another example of the conversion of the grayscale bit width by the limiter section of this embodiment. [Figure 6] This figure shows an example of the operation flow of the noise reduction unit in this embodiment. [Figure 7] This figure shows an example of the functional configuration of the noise reduction unit of the second embodiment. [Figure 8]It is a diagram illustrating an example of a luminance calculation result obtained by the luminance signal generator of the present embodiment. [Figure 9] It is a diagram illustrating an example of a motion vector calculated by the motion detector of the present embodiment. [Figure 10] An example of the amplitude conversion result in the case of formula (5) is shown. [Figure 11] An example of the amplitude conversion result in the case of formula (6) is shown. [Figure 12] It is a diagram illustrating an example of the configuration of a conventional image processing apparatus. MODE FOR CARRYING OUT THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. The embodiments described below are merely examples, and the embodiments to which the present invention is applied are not limited to the following embodiments. [Embodiment 1] FIG. 1 is a diagram illustrating an example of a functional configuration of the image processing apparatus 1 according to the present embodiment. The image processing apparatus 1 includes a camera signal processing unit 10, a RAW data generation unit 20, and a video signal processing unit 30. The camera signal processing unit 10, the RAW data generation unit 20, and the video signal processing unit 30 are implemented as, for example, a camera provided in a drive recorder or a computer apparatus.

[0011] The RAW data generation unit 20 processes an analog signal photoelectrically converted by the image sensor of the camera , performs digital gradation conversion using a predetermined number of bits, and outputs the result as a digital image signal.

[0012] FIG. 2 is a diagram illustrating an example of a pixel array of the image sensor according to the present embodiment. The RAW data generation unit 20 of the present embodiment has RGB pixels arranged in a so-called Bayer pattern . The RAW data generation unit 20 performs compression and gradation value The data is output as RAW data to the camera signal processing unit 10 without performing image processing such as modification.

[0013] Returning to FIG. 1, the camera signal processing unit 10 processes the digital image generated by the RAW data generating unit 20 acquires an image signal (that is, RAW data).

[0014] The camera signal processing unit 10 includes an HDR combining unit 110, a noise reduction unit 120, and a correction unit 13 0 and a luminance-chrominance conversion unit 140 as its functional units. Further, the camera signal processing unit 1 0 includes a frame memory 300 that temporarily stores image signals in a semiconductor memory device or the like. .

[0015] The HDR combining unit 110 includes an HDR (High Dynamic Range) circuit, and processes the RAW data performs HDR combining on the RAW data generated by the data generating unit 20.

[0016] The image signal after this HDR combining is an image signal before γ (gamma) correction, and has a gradation bit width is relatively large. The image signal after HDR combining has, for each pixel, for example, a 24-bit gradation bit width. Specifically, for the image signal after HDR combining, when the image sensor has the aforementioned Bayer array in this case, the image signal after HDR combining includes a red (R) pixel, a first green (G1) pixel, and a second green (G2) pixel and a blue (B) pixel each has a 24-bit gradation bit width . In the following description, the gradation bit of the first number of bits generated by the HDR combining unit 110 a digital image signal having a bit width is referred to as an image signal of the first bit width or an input image signal . The HDR combining unit 110 outputs the image signal after HDR combining to the noise reduction unit 120.

[0017] Furthermore, the image processing device 1 does not necessarily have to include an HDR synthesis unit 110. In this case, the noise reduction unit 120 is supplied with RAW data generated by the RAW data generation unit 20. It may be configured to do so. In this case, the RAW data generation unit 20 generates RA The W data becomes the first bit width image signal, or input image signal.

[0018] If the camera signal processing unit 10 is equipped with an HDR synthesis unit 110, then the noise reduction unit 12 0 acquires the input image signal from the HDR synthesis unit 110. Also, the camera signal processing unit 10 If the HDR synthesis unit 110 is not provided, the noise reduction unit 120 generates RAW data. The input image signal (in this case, RAW data) is acquired from unit 20. In either case, the noise reduction unit 120 receives an input that represents the grayscale value of the pixel with a first number of bits. It functions as an image signal acquisition unit that acquires image signals. In the following description, the camera signal processing unit 10 is equipped with an HDR synthesis unit 110, H The image signal output by the DR synthesis unit 110 will be described as being the input image signal.

[0019] The noise reduction unit 120 processes the input image signal (in this example, the image signal after HDR synthesis). It reduces the noise components contained in the noise (number). A specific example of the functional configuration of this noise reduction unit 120 is as follows: Next, I will explain using Figure 3.

[0020] Figure 3 shows an example of the functional configuration of the noise reduction unit 120 of this embodiment. The reduction unit 120 includes a limiter unit 210, a noise extraction unit 220, and a cyclic coefficient calculation unit 230. It includes an output image generation unit 240 as its functional unit.

[0021] The limiter unit 210 converts the input image signal to a second number of bits that is smaller than the first number of bits. In one example of this embodiment, the grayscale bit width of the input image signal (i.e., the first number of bits) It is 24 bits. In another example of this embodiment, the second number of bits is 16 bits. Yes. In this example, the limiter unit 210 has an input with a 24-bit grayscale bit width. The image signal is converted into an image signal with a 16-bit grayscale bit width. In the following explanation, after the grayscale bit width has been converted by the limiter unit 210, The image signal is also called the converted image signal. Also, the limiter unit 210 converts the input image signal. Converting to an image signal is also called generating a converted image signal.

[0022] The limiter unit 210 limits the pixel values ​​of the input image signal to a range that can be represented by the second number of bits. Based on whether or not this occurs, the converted image signal is generated.

[0023] Figure 4 shows an example of the conversion of the grayscale bit width by the limiter unit 210 of this embodiment. Each square represents one bit, with the top bit representing the most significant bit and the bottom bit representing the most significant bit. The bit represents the least significant bit. In one example of this embodiment, the first number of bits (i.e., the input image) The image signal's grayscale bit width is 24 bits, and the second bit count (i.e., the converted image signal) The grayscale bit width is 16 bits. In this case, the pixel value of the input image signal is 0x000. The grayscale is represented within the range of 000 to 0xffffff (0x represents the hexadecimal representation). Furthermore, the pixel values ​​of the converted image are represented in a grayscale range of 0x0000 to 0xffff. For example, if the pixel value of the input image signal exceeds 0xffff (i.e., 0x0100) If the value is 00 or greater, it cannot be represented by the second bit number. In other words, the image of the input image signal If the prime value is greater than the maximum value that can be represented by the second number of bits, it cannot be represented by the second number of bits. It's not possible.

[0024] The limiter section 210 limits the pixel value of the input image signal to the maximum value that can be represented by the second number of bits. If the value is large, the maximum value that can be represented by the second number of bits is output as the converted image signal. In the example described above, the limiter unit 210 is limited when the pixel value of the input image signal is 0x010000~ If the range is 0xffffff, the pixel values ​​of the input image signal are represented by the second bit number. If it is determined to be greater than the maximum possible value, the pixel values ​​of the converted image signal are represented by a second number of bits. The value is set to the maximum possible value, 0xffff, and the converted image signal is generated.

[0025] Furthermore, the limiter unit 210 limits the input image signal to the maximum value that can be represented by the second number of bits. In this case, counting from the least significant bit of the input image signal, the second number of bits will be used to convert the image signal. It outputs as follows. In the example above, the limiter unit 210 outputs when the pixel value of the input image signal is 0. If the pixel values ​​of the input image signal are in the range of x000000 to 0x00ffff, then the second It is determined that the number of bits is less than or equal to the maximum value that can be represented, and the pixel values ​​of the converted image signal are set to the second The converted image signal is generated by converting the value to one that can be represented by the number of bits.

[0026] In the example described above, the limiter unit 210 sets the pixel value of the input image signal to 0x0000. When the range is 00 to 0x00ffff, the information in the upper 8 bits of the input image signal Excluding the lower 16 bits of the input image signal, the explanation states that the converted image signal is obtained by converting the lower 16 bits of the input image signal. As revealed, this is not the only example.

[0027] Figure 5 shows another example of the conversion of the grayscale bit width by the limiter unit 210 of this embodiment. For example, the limiter unit 210 limits the pixel value of the input image signal from 0x00000f to 0x When the range is 0fffff, the upper 4 bits and lower 4 bits of the input image signal are within that range. Excluding that information, the lower 5 bits to 20 bits of the input image signal are converted to the image signal. You can convert it.

[0028] In other words, the limiter unit 210 starts from the least significant bit of the input image signal and sets a predetermined number of bits (up In the example described above, the signal excluding 4 bits is used as the image signal to be converted. In other words, the limiter unit 210 does not necessarily start from the least significant bit of the input image signal to the second least significant bit. This is not limited to outputting the least significant bit of the input image signal after conversion, but also includes the least significant bit of the input image signal. The second set of bits counting from the first bit should be output as the converted image signal.

[0029] Returning to Figure 3, the limiter unit 210 processes the converted image signal through the noise extraction unit 220 and the cyclic coefficient. The output is sent to the calculation unit 230 and the frame memory 300, respectively.

[0030] The frame memory 300 (memory section) records the image signal after conversion by the limiter section 210. It delays the image signal. In the following explanation, among the image signals output from the limiter unit 210, the frame memo The delayed image signal stored in the RI300 is also called the first signal (or delayed image signal). Also, among the image signals output from the limiter unit 210, the frame memory 300 is used. The image signal before the delay, which is not recalled, is also called the second signal.

[0031] The limiter section 210 comprises a first limiter section 211 and a second limiter section 212. This configuration may also be used. In this case, the first limiter unit 211 has a frame memory 300 (me The second limiter section 212 outputs an image signal to be stored in the Mori section. The converted image signal (second image signal) is output to the cyclic coefficient calculation unit 230, which contains 0.

[0032] Furthermore, the limiter section 210 is a soft limiter to reduce the influence near the image boundary. It may also be structured as "ta".

[0033] The noise extraction unit 220 uses the image signal stored in the frame memory 300 (memory unit) The first image signal and the second image signal, which is the image signal before the delay output by the limiter unit 210. It generates a difference signal between , and . The circulating coefficient calculation unit 230 calculates the circulating coefficient based on the output of the limiter unit 210. The noise coefficient is calculated by multiplying it by the noise component and subtracting it from the input image signal to reduce the noise component. This reduces the noise and represents the strength of the noise reduction. In the above configuration, the cyclic coefficient calculation unit 230 calculates the cyclic coefficient based at least on the output of the second limiter section 212. The output image generation unit 240 multiplies the difference signal generated by the noise extraction unit 220 by a cyclic coefficient. The output image signal is generated by subtracting from the input image signal. The output image generation unit 240 generates... The output image signal is then output to the correction unit 130.

[0034] Returning to Figure 1, the correction unit 130 processes the output image signal output from the noise reduction unit 120. Then, various corrections such as white balance and gamma correction are performed. The correction unit 130 outputs the corrected image. The image signal is output to the luminance color difference conversion unit 140.

[0035] The luminance color difference conversion unit 140 applies the luminance ( The luminance (Y)-color difference conversion unit 140 performs a luminance (Y)-color difference conversion. The signal is output to the video signal processing unit 30.

[0036] The video signal processing unit 30 is equipped with a video output unit 310, and the camera signal processing unit 10 outputs The output image signal (that is, the output after luminance (Y)-color difference conversion by the luminance-color difference conversion unit 140) The image signal is output to a liquid crystal display or image storage device.

[0037] In other words, the image processing device 1 takes an input image signal in which the grayscale value of a pixel is represented by a first number of bits. By excluding the information of the higher bits that indicate a large gradation value, the number of bits is smaller than the first number of bits. It is converted into an image signal with a second bit depth. The image processing device 1 calculates the difference between the converted image signal and an image signal prior to that image signal. Generate a signal. The image processing device 1 calculates the cyclic coefficient based at least on the converted image signal, and the difference signal By applying a coefficient and a cyclic coefficient to the input image signal, the noise in the input image signal is reduced. do.

[0038] [Operation of Image Processing Device 1] Figure 6 shows an example of the operation flow of the noise reduction unit 120 in this embodiment. (Step S100) The noise reduction unit 120 acquires the input image signal. The input image signal has a grayscale bit width of a first number of bits (e.g., 24 bits). . (Step S110) The limiter unit 210 checks when the pixel value of the input image signal is the second bit number ( For example, it determines whether or not it falls within the range that can be represented by 16 bits. The limiter unit 210, If it is determined that the pixel values ​​of the input image signal are not within the range that can be represented by the second number of bits (S) Step S110 (NO) proceeds to step S120. The limiter section 210 is If it is determined that the pixel value of the image signal is within the range that can be represented by the second number of bits (step If S110 (YES), proceed to step S130. (Step S120) The pixel values ​​of the input image signal are not within the range that can be represented by the second number of bits. In this case, the limiter unit 210 sets the pixel value of the input image signal to the maximum value that can be represented by the second bit number. Convert it to (for example, 0xffff). (Step S130) The pixel values ​​of the input image signal are within the range that can be represented by the second number of bits. In this case, the limiter unit 210 limits the pixel values ​​of the input image signal to within a second number of bits (for example, Convert using values ​​from 0x0000 to 0xffff. (Step S140) The noise extraction unit 220 extracts the image information stored in the frame memory 300. The first image signal is the number, and the second image is the image signal before the delay output by the limiter unit 210. Generates a difference signal between the signal and the result. (Step S150) The circulating coefficient calculation unit 230 calculates the circulating coefficient based on the output of the limiter unit 210. Calculate the coefficient. (Step S160) The output image generation unit 240 receives the difference signal generated by the noise extraction unit 220. The output image signal is generated by multiplying the result by a cyclic coefficient and subtracting it from the input image signal. Output image generation Unit 240 outputs the generated output image signal to the correction unit 130. [Differentiation] Furthermore, the noise reduction unit 120 may also include a level detection unit 250 as its functional unit. stomach.

[0039] The level detection unit 250 detects pixels in the HDR-composited image signal output by the HDR synthesis unit 110. The magnitude of the value is detected. Here, the magnitude of the pixel value is the brightness of the image signal after HDR synthesis. It indicates (or brightness). For example, the pixel values ​​of the image signal after HDR synthesis are generally small. If this is the case, it indicates that the image signal was captured in a dark place. Also, the image signal after HDR synthesis. A generally high pixel value indicates that the image signal was captured in bright light. (Level detection) The unit 250 outputs the magnitude of the detected pixel value to the cyclic coefficient calculation unit 230.

[0040] The cyclic coefficient calculation unit 230 calculates the cyclic coefficient based on the magnitude of the pixel value detected by the level detection unit 250. The cyclic coefficient is calculated. The output image generated by the output image generation unit 240 described above is noise-reduced. The magnitude of the cyclic effect depends on the magnitude of the cyclic coefficient. When the cyclic coefficient is large, noise reduction When the duction effect is greater and the cyclic coefficient is smaller, the noise reduction effect is better. It gets smaller.

[0041] The cyclic coefficient calculation unit 230 calculates the value when the pixel values ​​detected by the level detection unit 250 are generally small. (In other words, if the image is dark) the cyclic coefficient is increased during calculation. The unit 230 is used when the pixel values ​​detected by the level detection unit 250 are generally large (i.e., the image If the light is bright, the calculation is performed by reducing the circulating coefficient.

[0042] According to the noise reduction unit 120 configured in this way, noise reduction is performed depending on the brightness of the image. The strength of the action can be controlled, for example, for images taken in dark places. This allows for a greater noise reduction effect. Furthermore, with the noise reduction unit 120 configured in this way, when the cyclic coefficient is reduced... Therefore, the effect of noise reduction becomes smaller, and the target pixels for noise reduction and This makes the boundary between pixels that are not subject to noise reduction less noticeable.

[0043] The cyclic coefficient calculation unit 230 uses a predetermined number of stages (for example, 8 stages) It may be configured to select from among the cyclic coefficients. Furthermore, if the level of the image signal detected by the level detection unit 250 exceeds the bit width of the second signal In cases where (for example, the pixel values ​​of the input image signal are in the range of 0x010000 to 0xffffff) There are cases where this occurs. In this case, the cyclic coefficient calculation unit 230 calculates the cyclic coefficient for the pixel in question. It can also be calculated as 0 (zero). If the cyclic coefficient is 0 (zero), then the corresponding pixel Therefore, there is no effect of noise reduction; in other words, the pixel in question is not subjected to noise reduction. stomach.

[0044] Furthermore, the level detection unit 250 detects the red (R) pixel, the first green (G1) pixel, and the second green (G2 You can also detect the pixel value (signal level) for each of the blue (B) pixels. Alternatively, the signal level may be detected based on the brightness value obtained from these pixel values. The rotation coefficient calculation unit 230 calculates the red (R) pixel, the first green (G1) pixel, the second green (G2) pixel, Based on the pixel value for each blue (B) pixel, the process for each pixel is performed. The coefficient of rotation may be calculated.

[0045] [Comparison with the configuration of conventional image processing devices] Here, we compare the configuration of the image processing apparatus 1 of this embodiment with the configuration of a conventional image processing apparatus. Next, the effects of the image processing apparatus 1 of this embodiment will be explained. Figure 12 shows an example of the configuration of a conventional image processing device 9. The image processing device 9 is a It comprises a camera signal processing unit 91 and a video signal processing unit 93. The camera signal processing unit 91 is a real In this embodiment, the video signal processing unit 93 processes the video signal in the camera signal processing unit 10. Each corresponds to a control unit 30, but the noise reduction unit 920 is not the camera signal processing unit 91. This embodiment differs from the image processing apparatus 1 in that it includes a video signal processing unit 93.

[0046] In other words, the camera signal processing unit 91 includes an HDR synthesis unit 910 and a noise reduction unit 920, The system includes a correction unit 930. The video signal processing unit 93 includes a noise reduction unit 920 and a video output unit It is equipped with 310. In this conventional image processing device 9, RAW data from the image sensor (or after HRD synthesis) The correction unit 930 corrects the image signal from the RAW data, and the conversion unit 940 converts the gradation bit width. After being converted to a second number of bits (for example, 16 bits), the noise reduction unit 920 removes the noise Reduction processing is performed. The correction unit 930 performs automatic processing such as white balance and scratch correction. It performs various correction processes such as positive correction and Y / chrominance signal conversion. During this process, the correction unit 930 performs Gamma correction is sometimes applied to low-light image signals, but this gamma correction removes noise components. The gain also increases. The noise reduction unit 920, as a result of the gain increase due to the correction by the correction unit 930, Noise reduction is performed on image signals that contain noise components. According to the previous image processing device 9, it is difficult to obtain sufficient noise reduction performance.

[0047] Furthermore, in order to obtain sufficient noise reduction performance in the conventional image processing device 9, supplement It is also possible to perform noise reduction on the RAW data before correction by the positive part 930. However, in this configuration, the correction unit 930 determines the first number of bits (for example). This involves processing an image signal (24 bits), and a second number of bits (for example, 16 bits) Compared to processing image signals, the circuit size, power consumption, and frame memory are significantly smaller. This creates the problem of the size becoming larger.

[0048] Furthermore, in the conventional image processing device 9, while suppressing the memory size of the frame memory, To obtain noise reduction performance, the image is reduced in size and noise reduction is performed, and the original image is restored. One possible method is to enlarge the image to the desired size. However, if configured in this way, the resolution... This creates a problem where it becomes difficult to obtain a noise reduction effect without reducing the signal strength. Furthermore, the addition of shrinking and expanding circuits can lead to problems such as increased circuit size and power consumption.

[0049] On the other hand, according to the image processing apparatus 1 of this embodiment, the limiter unit 210 reduces the grayscale bit width. By calculating noise extraction and cyclic coefficients based on the generated image signal, the circuit size and power consumption can be determined. This can reduce power consumption and the memory size of the frame memory. Furthermore, according to the image processing apparatus 1 of this embodiment, the noise extraction result and the calculation result of the cyclic coefficient are Using this, the output image generation unit 240 generates RAW data (or RAW data after HRD synthesis). By applying noise reduction processing to the image signal, the grayscale bit width and pixel depth of the RAW data are reduced. Noise reduction can be achieved while maintaining the image size (i.e., maintaining high resolution). Furthermore, the image processing apparatus 1 of this embodiment has noise reduction performed by the noise reduction unit 120. The correction unit 130 performs gamma correction on the image signal after it has been processed. According to the processing device 1, the amplification effect of noise components due to γ ​​correction can be reduced, and low light conditions The noise reduction effect can also be improved in this domain. In other words, according to the image processing device 1 of this embodiment, the circuit size, power consumption and frame memory While suppressing the memory size of the image signal and maintaining the resolution of the image signal, the image signal noise is reduced. The removal effect can be improved.

[0050] [Second Embodiment] Figure 7 shows an example of the functional configuration of the noise reduction unit 120a of the second embodiment. In addition to the functions of the noise reduction unit 120 described above, the noise reduction unit 120a also generates a luminance signal. It comprises a unit 260, a delayed brightness signal generation unit 270, a motion detection unit 280, and a motion correction unit 290. They differ in certain aspects. Note that components identical to those in the first embodiment are denoted by the same reference numerals, and their descriptions are omitted. .

[0051] Here, the signal is the difference between the first signal (delayed image signal) and the second signal (image signal before delay). In other words, the frame difference signal contains motion components between frames along with noise components. The frame difference signal, which includes motion components between frames, is input to the noise extraction unit 220a. When this happens, the noise extraction unit 220a can extract motion components between frames as noise. Yes, in this case, afterimages or motion blur may occur. Therefore, the noise reduction unit 120a of this embodiment reduces the motion component included in the frame difference signal. By detecting and correcting motion based on the detection results, the noise extraction unit 220a removes noise It improves extraction performance and reduces afterimages and motion blur. The specific functional configuration of the noise reduction unit 120a in this embodiment will be described below.

[0052] The luminance signal generation unit 260 converts the pixel values ​​of the input image signal output from the HDR synthesis unit 110. Convert to luminance values. The input image signal, which is RAW data, is converted to a Bayer array as shown in Figure 2. Pixel values ​​of the red (R) pixel, first green (G1) pixel, second green (G2) pixel, and blue (B) pixel. This includes the red (R) pixels, the first green (G1) pixels, and the second From the pixel values ​​of the green (G2) pixels and blue (B) pixels, based on equation (1) or equation (2) Then, calculate the brightness value.

[0053]

number

[0054]

number

[0055] Figure 8 shows an example of the luminance calculation result by the luminance signal generation unit 260 of this embodiment. . Note that the coefficients for each of RGB are not limited to these values; for example, the coefficient for B can be set to 0. You could also do that, or increase the coefficient of G. Furthermore, the luminance signal generation unit 260 calculates the luminance value for a pixel, and the surrounding pixels... Alternatively, the pixel values ​​may be interpolated using the values ​​before calculating the brightness values ​​as described above.

[0056] For example, in the green (G12) pixel, Red(R12) = (Red(R22) + Red(R02)) / 2 ·Blue(B12)=(Blue(B13)+Blue(B11)) / 2 We will seek it as follows. In the blue (B11) pixels, Red(R11) = (Red(R00) + Red(R02) + Red(R20) + Red(R22)) / 4 Green(G11) = (Green(G01) + Green(G21) + Green(G10) + Green(G12)) / 4 This is how it is calculated. In this case, the coefficients of each color component may be weighted instead of being 1.

[0057] The delayed brightness signal generation unit 270 generates the first stroke stored in the frame memory 300 (memory unit) The brightness value is calculated for the image signal in the same manner as the brightness signal generation unit 260.

[0058] The motion detection unit 280 detects the first image signal stored in the frame memory 300 (memory unit) Based on first luminance information indicating brightness and second luminance information indicating brightness of the input image signal before delay, Then motion detection of the input image signal is performed. Specifically, the motion detection unit 280 uses the first brightness information and By calculating the inter-frame difference based on the second luminance information, the motion vector is calculated. ru.

[0059] Figure 9 shows an example of a motion vector calculated by the motion detection unit 280 of this embodiment. . The motion detection unit 280, when Y12 shown in Figure 8 is the pixel of interest, • Horizontal gradient: dxi = luminance(Y13) - luminance(Y11) Vertical gradient: dyi = luminance(Y22) - luminance(Y02) • Frame difference: dfi = 2 * (Brightness (Y12) - Brightness (Y12_1f)) (However, luminance (Y12_1f) is a 1-frame delayed signal of luminance (Y12)) The motion vector is calculated. In the example shown in Figure 9, the motion detection unit 280 calculates the motion vector. Calculate the value of (-6,2). The motion vector calculated by the motion detection unit 280 is also referred to as the motion detection result.

[0060] Returning to Figure 7, based on the motion detection result of the motion detection unit 280, the frame memory 300 (Me The motion correction unit 290 corrects the motion of the first image signal stored in the Mori unit. The resulting first image signal is also called the motion correction result. The noise extraction unit 220a further analyzes the difference based on the motion correction result from the motion correction unit 290. Generates a minute signal.

[0061] Here, without generating a brightness signal, the RAW data is used as is, for example, horizontal Equation (3) describes the motion in the x-axis direction, and equation (4) describes the motion in the vertical direction (y-axis direction). It is also possible to detect the motion vector (a,b) using gradient methods as shown in the respective sections. It is possible.

[0062]

number

[0063]

number

[0064] The motion detection unit, for example, if the green (G12) shown in Figure 2 is the pixel of interest, • Horizontal gradient: dxi = Blue (B13) - Blue (B11) Vertical slope: dyi = red(R22) - red(R02) • Frame difference: dfi = 2 * (green(G12) - green(G12_1f)) (However, green (G12_1f) is a 1-frame delayed signal of green (G12)) Detect motion vectors (a,b) for each pixel block (e.g., 16 pixels × 8 pixels). ru.

[0065] However, the three types of gradients and differences mentioned above are different in terms of signal type (e.g., red, green, blue). Therefore, compared to calculating the motion vector using the same signal type (e.g., luminance signal), it is more accurate. In some cases, it may not be possible to determine the motion vector.

[0066] As described above, the noise reduction unit 120a of this embodiment uses the RAW data of the input signal, A luminance signal is generated from the RAW data with the output signal delayed by one frame, and these two luminance signals Motion detection is performed from the number. According to the noise reduction unit 120a configured in this way, the output from the pixels of the Bayer array Compared to using the pixel values ​​of the raw data directly for motion detection, this method offers higher accuracy. Motion detection can be performed.

[0067] [Differentiation] The noise reduction unit 120a may perform noise reduction in the following manner. The noise extraction unit 220a outputs a signal after noise extraction, and amplitude conversion is performed on that signal.

[0068] For example, each pixel shown in Figure 2 is red (R00), green (G01), green (G10), blue (B11 For each of the above, the output signals of the noise extraction unit are r0n, g0n, g1n, and b1n. . • Average values ​​of g0n and g1n: ave(g0n, g1n) = (g0n + g1n) / 2 The difference between r0n and ave(g0n, g1n): dif_rg=r0n-ave(g0n, g1n) The difference between b1n and ave(g0n, g1n): dif_bg = b1n-ave(g0n, g1n) • Absolute values ​​of each: abs(dif_rg), abs(dif_bg) • The thresholds for red (R00) and blue (B11) are r0n_th and b1n_th, respectively. Therefore, the amplitude conversion outputs r0n_lim and b for red (R00) and blue (B11) respectively are 1n_lim is given by equations (5) and (6).

[0069]

number

[0070]

number

[0071] Figure 10 shows an example of the amplitude transformation result in case (B) of equation (5). Figure 11 shows an example of the amplitude transformation result in case (C) of equation (5). The noise reduction unit 120a processes the amplitude-converted signal as described above into the output image generation unit 24. By assigning 0, noise reduction of the image signal may be performed.

[0072] Although embodiments of the present invention have been described above, the present invention is not limited to the above embodiments. Rather, various modifications can be made without departing from the spirit of the present invention. Furthermore, the embodiments described above may be combined as appropriate.

[0073] Furthermore, the entire function or part thereof of each part of the image processing apparatus 1 in the above-described embodiment is The program to implement these functions is written on a computer-readable recording medium. The program recorded on this recording medium is then loaded into a computer system and executed. It may also be achieved by taking action. Note that the term "computer system" here refers to: This includes hardware such as the operating system and peripheral devices. Furthermore, "computer-readable recording media" refers to flexible disks and magneto-optical media. Portable media such as disks, ROMs, and CD-ROMs, and hardware built into computer systems. This refers to storage devices such as disks. Furthermore, it refers to "computer-readable recording media." This involves transmitting programs via networks such as the internet or communication lines such as telephone lines. Like a communication line in such a case, something that dynamically holds a program for a short period of time, in that case Like volatile memory inside a computer system that acts as a server or client, at a certain time It may also include programs that hold intermediate programs. Furthermore, the above programs have the functions described above. It may also be used to implement a part of the aforementioned functions in a computer system It may also be possible to achieve this by combining it with programs already recorded in the program. [Explanation of Symbols]

[0074] 1...Image processing unit, 10...Camera signal processing unit, 20...RAW data generation unit, 110...H DR synthesis section, 120...noise reduction section, 130...correction section, 140...luminance color difference conversion section, 210 ...limiter section, 211...first limiter section, 212...second limiter section, 220...noise extraction Section 230...Circulation coefficient calculation section, 240...Output image generation section, 250...Level detection section, 260 ...Brightness signal generation unit, 270...Delayed brightness signal generation unit, 280...Motion detection unit, 290...Motion compensation Main unit, 300...frame memory, 310...video output unit

Claims

1. A noise extraction unit generates a difference signal between a first image signal, which is an image signal converted to a different number of bits than the input image signal, and a second image signal, which is the image signal before the delay relative to the first image signal. The system includes an output image generation unit that generates an output image signal based on the input image signal and a signal obtained by multiplying the difference signal, which is expressed with a different number of bits than the input image signal, by a coefficient representing the strength of noise reduction. Image processing device.

2. The first image signal and the second image signal are image signals converted to a second number of bits smaller than the number of bits of the input image signal. The output image generation unit multiplies the difference signal, which is represented by the second number of bits, by a cyclic coefficient representing the strength of noise reduction, and subtracts this from the input image signal to generate an output image signal. The image processing apparatus according to claim 1.

Images