Image processing device, image processing method and program

Abstract

To provide an image processing device, image processing method, and image processing program, which enable good noise reduction processing while minimizing the circuit scale.SOLUTION: An image processing device is provided, comprising a noise reduction intensity data generation unit and noise reduction processing unit. The noise reduction intensity data generation unit generates first noise reduction intensity data for an image, and then generates second noise reduction intensity data for the image in response to an input of processed data that is generated based on the first noise reduction intensity data. The noise reduction processing unit applies noise reduction processing to the image according to the second noise reduction intensity data.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] Books invention Image processing device and image processing method and Regarding the program. [Background technology]

[0002] For example, Patent Document 1 describes an image processing device that performs tone correction and noise reduction processing on an image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-127388 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional image processing devices including the one disclosed in Patent Document 1 are not sufficient in terms of reducing the circuit scale, and in the case where desired noise reduction processing cannot be performed, degradation of image quality occurs.

[0005] In view of the above-mentioned circumstances, an object of one aspect of the present invention is to provide an image processing device and an image processing method that can realize good noise reduction processing while suppressing the circuit scale. and The aim is to provide a program. [Means for solving the problem]

[0006] An image processing device according to a first aspect of the present invention includes a noise reduction intensity data generating unit that outputs first noise reduction intensity data generated based on predetermined image data to a filter processing unit, and receives processed data generated by the filter processing unit performing a predetermined filter processing on the first noise reduction intensity data, thereby generating second noise reduction intensity data that takes into account the processed data, and a noise reduction processing unit that performs noise reduction processing on the image data by applying the second noise reduction intensity data. the filtering unit performs the filtering process using a low-pass filter, an epsilon filter, a BiLateral filter, or a NonLocalMeans filter. It is characterized by: Also, an image processing device according to a second aspect of the present invention includes a gradation correction unit that generates gradation corrected image data by performing gradation correction on predetermined RGB image data, a YUV conversion unit that generates YUV image data by YUV conversion of the gradation corrected image data, a noise reduction intensity data generation unit that outputs first noise reduction intensity data to a filter processing unit, the noise reduction intensity data being generated based on gain map data corresponding to an amount of gradation correction for the RGB image data when the gradation correction unit generates the gradation corrected image data, luminance data in the YUV image data, or likelihood data indicating a likelihood between a color indicated by the YUV image data and a predetermined color, and receives processed data generated by the filter processing unit performing a predetermined filter processing on the first noise reduction intensity data, thereby generating second noise reduction intensity data taking the processed data into consideration, and a noise reduction processing unit that performs noise reduction processing on the YUV image data by applying the second noise reduction intensity data. the filtering unit performs the filtering process using a low-pass filter, an epsilon filter, a BiLateral filter, or a NonLocalMeans filter. It is characterized by: Further, an image processing method according to a first aspect of the present invention is an image processing method executed by an image processing device, comprising: a noise reduction intensity data generating step of outputting first noise reduction intensity data generated based on predetermined image data to a filter processing unit, and receiving processed data generated by the filter processing unit performing a predetermined filter processing on the first noise reduction intensity data, thereby generating second noise reduction intensity data taking into account the processed data; and a noise reduction processing step of applying the second noise reduction intensity data to perform noise reduction processing on the image data. the filtering unit performs the filtering process using a low-pass filter, an epsilon filter, a BiLateral filter, or a NonLocalMeans filter. It is characterized by: Further, an image processing method according to a second aspect of the present invention is an image processing method executed by an image processing device, the image processing method including: a gradation correction step of generating gradation corrected image data by performing gradation correction on predetermined RGB image data; a YUV conversion step of generating YUV image data by YUV converting the gradation corrected image data; a noise reduction intensity data generating step of outputting first noise reduction intensity data generated based on gain map data corresponding to an amount of gradation correction for the RGB image data when generating the gradation corrected image data in the gradation correction step, luminance data in the YUV image data, or likelihood data indicating a likelihood between a color indicated by the YUV image data and a predetermined color, to a filter processing unit, and receiving processed data generated by the filter processing unit performing a predetermined filter processing on the first noise reduction intensity data, thereby generating second noise reduction intensity data taking into account the processed data; and a noise reduction processing step of performing noise reduction processing on the YUV image data by applying the second noise reduction intensity data. the filtering unit performs the filtering process using a low-pass filter, an epsilon filter, a BiLateral filter, or a NonLocalMeans filter. It is characterized by: Further, a program according to a first aspect of the present invention causes a computer to function as: noise reduction intensity data generating means for outputting first noise reduction intensity data generated based on predetermined image data to a filter processing section; and, upon receiving processed data generated by the filter processing section performing a predetermined filter processing on the first noise reduction intensity data, generating second noise reduction intensity data taking into account the processed data; and noise reduction processing means for performing noise reduction processing on the image data by applying the second noise reduction intensity data. The filtering unit performs the filtering process using a low-pass filter, an epsilon filter, a BiLateral filter, or a NonLocalMeans filter. It is characterized by: A program according to a second aspect of the present invention causes a computer to function as: gradation correction means for generating gradation corrected image data by performing gradation correction on predetermined RGB image data; YUV conversion means for generating YUV image data by YUV converting the gradation corrected image data; noise reduction intensity data generation means for outputting to a filter processing unit first noise reduction intensity data generated based on gain map data corresponding to an amount of gradation correction for the RGB image data when the gradation correction means generates the gradation corrected image data, luminance data in the YUV image data, or likelihood data indicating a likelihood between a color indicated by the YUV image data and a predetermined color, and for generating second noise reduction intensity data taking into account the processing data generated by the filter processing unit performing a predetermined filter processing on the first noise reduction intensity data; and noise reduction processing means for applying the second noise reduction intensity data to perform noise reduction processing on the YUV image data. The filtering unit performs the filtering process using a low-pass filter, an epsilon filter, a BiLateral filter, or a NonLocalMeans filter. It is characterized by: Effect of the Invention

[0007] The present invention According to the present invention, an image processing device and an image processing method are capable of implementing good noise reduction processing while suppressing the circuit scale. and Programs can be provided. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is an external view showing an example of a digital camera 1X. [Diagram 2] 1 is a first block diagram showing a configuration of an image processing device 1. FIG. [Diagram 3] 2 is a second block diagram showing the configuration of the image processing device 1. FIG. [Figure 4] FIG. 13 is a conceptual diagram showing an example of noise reduction intensity data D9. [Diagram 5] FIG. 2 is a diagram showing a detailed configuration of a noise reduction intensity data generating unit 50. [Figure 6] FIG. 4 is a conceptual diagram showing the flow of noise reduction processing using first and second noise reduction intensity data. [Figure 7] 13 is a diagram showing an example of noise reduction intensity data before and after map processing (filter processing) is performed by the filter processing unit 80. FIG. [Figure 8] 4 is a flowchart showing an example of an image processing method performed by the image processing device 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The image processing device 1 of this embodiment will be described in detail with reference to Fig. 1 to Fig. 8. The image processing method and image processing program of this embodiment are realized by causing a computer (circuit components) of the image processing device 1 to execute various processing steps. The image processing device 1 may be called, for example, a Color Process Pipeline. The image processing device 1 may be implemented as a large scale integrated circuit (LSI: Large Scale Integration) or as a part thereof.

[0010] The image processing device 1 may be mounted on a fixed photographing device such as a security camera or a surveillance camera, or may be mounted on a portable photographing device such as a digital camera or various terminals. The image processing device 1 may be specialized for performing image processing on images transmitted by wire or wirelessly from a photographing device or other device, or images stored in various storage media. In other words, the image processing device 1 has a degree of freedom in its configuration, and various design changes are possible.

[0011] FIG. 1 is an external view showing an example of a digital camera (photographing device) 1X. In the digital camera 1X, light from a subject is guided into a camera body 11X via a lens 10X, reflected by a mirror 12X, and a real image of the subject is formed on a focusing glass 13X. This real image of the subject is guided to a viewfinder 15X by a pentaprism 14X so that it can be observed. A display device 16X is disposed between the focusing glass 13X and the pentaprism 14X, and displays various information superimposed on the subject image reflected on the focusing glass 13X. The broken lines in FIG. 1 indicate the optical path of the light from the subject that is guided to the viewfinder 15X. In the digital camera 1X, the mirror 12X is raised, the shutter 17X is opened, and the subject light is guided to an image sensor 18X to perform photography, so that the subject image does not reach the display device 16X in the mirror-up state.

[0012] The digital camera 1X may incorporate an image processing device 1 according to an embodiment of the present invention that performs image processing on an image captured by the image sensor 18X, or may have a communication unit (such as a connection terminal or an antenna) that transmits an image captured by the image sensor 18X to the external image processing device 1 according to an embodiment of the present invention by wire or wirelessly. That is, the present invention is not limited to the digital camera 1X having a photographing function, but can also be applied to any device that can obtain an image captured by an external device and apply noise reduction processing to it. In other words, the present invention can also be applied to a system composed of multiple devices.

[0013] 2 and 3 are first and second block diagrams showing the configuration of the image processing device 1. The image processing device 1 has a demosaic processing unit 10, a gradation correction unit 20, a YUV conversion unit 30, a color analysis unit 40, a noise reduction intensity data generation unit (NR intensity data generation unit) 50, a noise reduction processing unit (NR processing unit) 60, and a post-processing unit 70. In addition, as shown in Fig. 2, a filter processing unit 80 as an external circuit and a DRAM (Dynamic Random Access Memory) 90 as an external memory are connected to the image processing device 1.

[0014] 2, the demosaic processing unit 10, the gradation correction unit 20, the YUV conversion unit 30, the color analysis unit 40, and the noise reduction intensity data generation unit 50 are enclosed by a solid line as a common circuit unit 100 for generating noise reduction intensity data (first and second noise reduction intensity data) described later (the demosaic processing unit 10, the gradation correction unit 20, the YUV conversion unit 30, the color analysis unit 40, and the noise reduction intensity data generation unit 50 constitute the common circuit unit 100). Furthermore, in FIG. 2, the filter processing unit 80 and the DRAM 90 are enclosed by a dashed line as the device outside (outside the common circuit) 200 that contributes to the generation of the noise reduction intensity data (first and second noise reduction intensity data) (the filter processing unit 80 and the DRAM 90 are included in the device outside (outside the common circuit) 200).

[0015] The demosaic processor 10 performs demosaic processing on an input image (RAW image, Bayer image) D1, which is an image before gradation correction, and converts it into an RGB image D2. Although not shown here, a block that performs isolated point removal processing and chromatic aberration correction on the input image D1 may be provided before the demosaic processor 10.

[0016] The gradation correction unit 20 performs gradation correction on the RGB image D2 before gradation correction from the demosaic processing unit 10. The gradation correction is adaptively performed on a pixel-by-pixel basis based on the image analysis results, for example, when shooting a subject with a large difference in brightness such as backlight, or when it is desired to brighten a specific area such as a face. In addition, in processes such as backlight correction and multiple exposure synthesis (HDR: High Dynamic Range), an adaptive gain-up amount is calculated for each pixel or pixel block, and gradation correction is performed on the entire image. Furthermore, in gamma correction, etc., a different gain-up amount may be applied to each pixel or pixel block. This gain-up amount may be uniform for RGB. There is a degree of freedom in the manner of gradation correction by the gradation correction unit 20, and various design changes are possible.

[0017] The gradation correction unit 20 performs gradation correction and outputs a gradation corrected image (RGB image after gradation correction) D3 and gain map data (Gain Map) D4. The gain map data D4 is data based on the amount of gradation correction of the gradation corrected image D3, and corresponds to the logarithm of the amount of gradation correction (gain) of the gradation corrected image D3. The gain map data D4 divides the gradation corrected image D3 into regions for each pixel or pixel block, and has feature values ​​for each divided region. The gradation corrected image D3 is input to a YUV conversion unit 30 and a color analysis unit 40, and the gain map data D4 is input to a noise reduction intensity data generation unit 50.

[0018] The YUV conversion unit 30 obtains luminance data (luminance data of the gradation corrected image) D5 by YUV converting the gradation corrected image (RGB image after gradation correction) D3 from the gradation correction unit 20. The luminance data D5 divides the gradation corrected image D3 into regions for each pixel or pixel block, and has feature values ​​for each divided region. The luminance data D5 is input to a noise reduction intensity data generation unit 50. The YUV conversion unit 30 also outputs the YUV converted gradation corrected image D6 to the noise reduction processing unit 60.

[0019] The color analysis unit 40 performs color analysis (RGB analysis) on the gradation corrected image (RGB image after gradation correction) D3 from the gradation correction unit 20 to obtain color likelihood data (color likelihood data of the gradation corrected image) D7. The color likelihood data D7 indicates the likelihood of a specific color, such as the color of human skin or flesh color. The color likelihood data D7 divides the gradation corrected image D3 into regions for each pixel or pixel block, and has a feature value for each divided region. The color likelihood data D7 is input to the noise reduction strength data generation unit 50.

[0020] The noise reduction intensity data generating unit 50 receives the gain map data D4 from the tone correcting unit 20, the luminance data D5 from the YUV converting unit 30, and the color likelihood data D7 from the color analyzing unit .

[0021] In addition, an Additional Map and / or an Extra Map are input to the noise reduction intensity data generating unit 50 as external map data D8. The Additional Map is used to filter the noise reduction intensity data (NR intensity data) D9 as the once generated NR level output D10 as "first noise reduction intensity data (first NR intensity data) D12" in the filter processing unit 80, and to loop (input) the result temporarily stored in the DRAM 90 as "processed data D12X" to the noise reduction intensity data generating unit 50. In other words, the processed data D12X generated based on the first noise reduction intensity data D12 is included in the external map data D8 (Additional Map) and looped (input) to the noise reduction intensity data generating unit 50. In addition, the "processed data D12X generated based on the first noise reduction intensity data (first NR intensity data) D12" looped (input) to the noise reduction intensity data generating unit 50 is used to improve the accuracy (reliability) of the "second noise reduction intensity data (second NR intensity data) D11" to be applied to the actual noise reduction processing (NR processing). This point will be described in detail later. The Extra Map can be the gain amount of the HDR processing for the image before the tone correction, for example, the logarithm of the gain amount during the HDR processing. Note that in FIG. 2, the Additional Map and the Extra Map as the external map data D8 are drawn together as a single Extra Map, and in FIG. 3, the Additional Map and the Extra Map as the external map data D8 are drawn separately.

[0022] The noise reduction intensity data generating unit 50 generates noise reduction intensity data (NR intensity data) D9 based on any combination data of a plurality of data generated according to an image after gradation correction. More specifically, the noise reduction intensity data generating unit 50 generates noise reduction intensity data D9 based on any combination data of the luminance data D5 from the YUV conversion unit 30, the color likelihood data D7 from the color analysis unit 40, the gain map data D4 from the gradation correction unit 20, and the external map data D8. This makes it possible to provide a good quality image in which noise reduction processing has been performed satisfactorily for captured images captured under various conditions. The noise reduction intensity data generating unit 50 generates noise reduction intensity data D9 based on any one of the luminance data D5, the color likelihood data D7, the gain map data D4, and the external map data D8, and any combination data of these data. In particular, the noise reduction intensity data generating unit 50 generates second noise reduction intensity data D11 to be applied to actual noise reduction processing based on at least the external map data D8 among the above-mentioned data, i.e., processed data D12X obtained by processing the first noise reduction intensity data D12 looped (input) from an input circuit for external map data (the filter processing unit 80 and DRAM 90 outside the device (outside the common circuit) 200).

[0023] In the image processing device 1, the second noise reduction intensity data D11 to be applied to the actual noise reduction processing and the processed data D12X obtained by processing the first noise reduction intensity data D12 used to improve the accuracy (reliability) of the second noise reduction intensity data are generated while sharing (suppressing the circuit scale) the circuit (common circuit portion 100) including the demosaic processing unit 10, the gradation correction unit 20, the YUV conversion unit 30, the color analysis unit 40, and the noise reduction intensity data generation unit 50. That is, the noise reduction intensity data generation unit 50 generates the first noise reduction intensity data D12 for the image by the common circuit (common circuit portion 100), and generates the second noise reduction intensity data D11 for the image in response to the input of the processed data D12X generated based on the first noise reduction intensity data D12.

[0024] In the initial operation (first round operation) of the image processing device 1, the noise reduction intensity data D9 generated by the noise reduction intensity data generating unit 50 is output as first noise reduction intensity data D12 as a noise reduction level output (NR level output) D10 to the outside of the device (outside of the common circuit) 200. The first noise reduction intensity data D12 output to the outside of the device 200 is filtered by a filter processing unit (external circuit) 80, temporarily stored in a DRAM (external memory) 90, and then included in external map data D8 as "processed data D12X of the first noise reduction intensity data D12" and looped (input) to the noise reduction intensity data generating unit 50.

[0025] The filter processing unit 80 executes various types of filter processing, such as a low-pass filter, an epsilon filter, a BiLateral filter, a NonLocalMeans filter, etc. The filter processing unit 80 may be realized by a circuit equipped with a general-purpose image processing group, and may use, for example, a CPU (Central Processing Unit) or a DRP (Dynamic Reconfigurable Processor).

[0026] In the above description, the case has been exemplified where the noise reduction intensity data D9 that has been subjected to the filter process is stored, and then looped (input) as processed data D12X of the first noise reduction intensity data D12 to the noise reduction intensity data generation unit 50. However, the noise reduction intensity data D9 may be stored as is, and then filtered after being read out, and looped (input) as processed data D12X of the first noise reduction intensity data D12 to the noise reduction intensity data generation unit 50.

[0027] Even in the second and subsequent operations of the image processing device 1, until the actual noise reduction processing is performed using the "second noise reduction strength data D11", the loop (input) of "the first noise reduction strength data D12 and its processed data D12X" via the noise reduction strength data generation unit 50, the filter processing unit 80, and the DRAM 90 is repeated. That is, the noise reduction strength data D9 generated by the noise reduction strength data generation unit 50 becomes "the first noise reduction strength data D12 and its processed data D12X" when looped (input) to the noise reduction strength data generation unit 50, and becomes "the second noise reduction strength data D11" when input to the noise reduction processing unit 60.

[0028] When the predetermined condition is satisfied, the loop (input) of the first noise reduction strength data D12 and its processed data D12X is terminated, and the noise reduction strength data D9 generated by the noise reduction strength data generation unit 50 is input as "second noise reduction strength data D11" to the noise reduction processing unit 60. There is a degree of freedom in how the predetermined condition is set, and for example, the predetermined condition can be set such that the loop (input) of the first noise reduction strength data D12 and its processed data D12X is repeated a predetermined number of times, a predetermined time has elapsed, or the quality of the first noise reduction strength data D12 and its processed data D12X exceeds a predetermined level.

[0029] In this way, by keeping the circuit usage portion (common circuit portion 100) of the image processing device 1 to a necessary minimum while efficiently using calculations outside the device (outside the common circuit) 200, it is possible to improve the accuracy (reliability) of the first noise reduction intensity data D12 and ultimately the second noise reduction intensity data D11 while suppressing the circuit scale, thereby achieving excellent image quality based on flexible and appropriate noise reduction processing.

[0030] In addition, in this embodiment, a mechanism for memory output of NR intensity is provided before NR processing in the image processing pipeline. Also, a mechanism for input from memory is provided for NR intensity calculation before NR processing in the image processing pipeline. Also, the external map (including the first noise reduction intensity data), luminance, color likelihood, and tone correction gain input from the memory are recalculated to generate NR intensity (second noise reduction intensity data). Also, by performing filter processing (epsilon filter, bilateral filter, etc.) on the NR intensity input / output to the memory to mitigate the influence of noise on the NR intensity map, it becomes possible to set an appropriate NR intensity with more reduced noise influence. In this way, noise reduction is performed on values ​​such as luminance, color likelihood, and tone correction gain before noise reduction and treated as NR intensity.

[0031] The noise reduction intensity data D9 (first noise reduction intensity data D12, second noise reduction intensity data D11) is for applying an optimized noise reduction process to the gradation corrected image D6, taking into consideration the extent to which gradation correction has been performed on which portion of the gradation corrected image (level distribution of gradation correction gain). In other words, the noise reduction intensity data D9 (first noise reduction intensity data D12, second noise reduction intensity data D11) can be in a map format in which the gradation corrected image D3 is divided into regions for each pixel or pixel block, and the level of noise reduction process to be applied to each divided region is associated with each divided region.

[0032] 4A and 4B are conceptual diagrams showing an example of noise reduction intensity data D9 (first noise reduction intensity data D12, second noise reduction intensity data D11). In FIG. 4A and FIG. 4B, for convenience of explanation, the tone correction image is simplified and drawn into a divided area of ​​25 cells in total, 5 cells vertically and 5 cells horizontally. In FIG. 4A, the noise reduction intensity data of each divided area is drawn in three stages, H (High), M (Middle), and L (Low). In FIG. 4B, the noise reduction intensity data of each divided area is drawn in five stages, 1, 2, 3, 4, and 5 (the larger the number, the higher the intensity, and the smaller the number, the lower the intensity). In FIG. 4A and FIG. 4B, the larger the degree of tone correction (level of tone correction gain) of the divided area, the higher the intensity of the noise reduction intensity data is set, and the smaller the degree of tone correction (level of tone correction gain) of the divided area, the lower the intensity of the noise reduction intensity data is set. It should be noted that there is a degree of freedom as to how each divided region of the gradation corrected image is set and how many stages the intensity of the noise reduction intensity data is divided into, and various design modifications are possible.

[0033] How to generate the above-mentioned noise reduction intensity data D9 (first noise reduction intensity data D12, second noise reduction intensity data D11) will be described in detail later.

[0034] The noise reduction processing unit 60 performs noise reduction processing (NR processing) on ​​the gradation corrected image D6 that has been gradation corrected by the gradation correction unit 20 and YUV converted by the YUV conversion unit 30, based on the noise reduction intensity data D9 (second noise reduction intensity data D11) (see, for example, FIGS. 4A and 4B) generated by the noise reduction intensity data generation unit 50. By performing noise reduction processing using the noise reduction intensity data D9 (second noise reduction intensity data D11) optimized in consideration of the extent of gradation correction performed on which part of the gradation corrected image D3 (level distribution of gradation correction gain), it is possible to achieve a good balance between gradation correction and noise reduction processing, and to realize excellent image quality based on flexible and appropriate noise reduction processing.

[0035] The post-processing unit 70 performs post-processing such as gamma correction, edge enhancement, and super-resolution processing on the noise-reduction-processed YUV image (NR YUV) D13 to generate an output image (YUV) D14. Note that the post-processing performed by the post-processing unit 70 has a degree of freedom, and various design changes are possible.

[0036] 5 is a diagram showing a detailed configuration of the noise reduction intensity data generating unit 50. As described above, the noise reduction intensity data generating unit 50 receives the luminance data D5, the color likelihood data D7, the gain map data D4, and the external map data D8. The external map data D8 includes processed data D12X of the first noise reduction intensity data D12 that is looped (input) to the noise reduction intensity data generating unit 50.

[0037] The noise reduction intensity data generating unit 50 has a selection input unit 51 that selectively passes some or all of the luminance data D5, the color likelihood data D7, the gain map data D4, and the external map data D8. In the following explanation, it is assumed that all of the luminance data D5, the color likelihood data D7, the gain map data D4, and the external map data D8 have passed through the selection input unit 51. In particular, it is assumed that the external map data D8 including the first noise reduction intensity data D12 has passed through the selection input unit 51.

[0038] It is also possible to omit the selection input unit 51 from the noise reduction intensity data generation unit 50. In this case, only the data to be used among the luminance data D5, the color likelihood data D7, the gain map data D4, and the external map data D8 may be input to the noise reduction intensity data generation unit 50, and the data not to be used may not be input to the noise reduction intensity data generation unit 50.

[0039] The noise reduction intensity data generating unit 50 has a multiplication unit 52 that multiplies the gain map data D4 passed through the selection input unit 51 by the external map data D8. The noise reduction intensity data generating unit 50 has a selection input unit 53 that selectively passes either the gain map data D4 or the multiplication result of the gain map data D4 and the external map data D8 by the multiplication unit 52. In the following description, the gain map data D4 or the multiplication result of the gain map data D4 and the external map data D8 by the multiplication unit 52 will be collectively referred to as "gain map data D4" (the output of the selection input unit 53 is "gain map data D4").

[0040] In this embodiment, when two pieces of data are multiplied, if there is no input of one of the pieces of data (for example, if it does not pass through the selection input unit 51), one of the pieces of data may be set to 1, and the other piece of data may be output as the multiplication result as is.

[0041] The noise reduction intensity data generating unit 50 has a front-stage lookup table 54 for application to each (at least any one of) the luminance data D5, the color likelihood data D7, the gain map data D4, and the external map data D8. More specifically, the front-stage lookup table 54 has a luminance data lookup table (LUT(Y)) 54A applied to the luminance data D5, a color likelihood data lookup table (LUT(color)) 54B applied to the color likelihood data D7, a gain map data lookup table (LUT(Gain)) 54C applied to the gain map data D4, and an external map data lookup table (LUT(EXT)) 54D applied to the external map data D8.

[0042] The noise reduction intensity data generating unit 50 has a multiplication unit 55A that multiplies the luminance data D5 passed through the luminance data lookup table 54A by the color likelihood data D7 passed through the color likelihood data lookup table 54B, a multiplication unit 55B that multiplies the multiplication result by the multiplication unit 55A by the gain map data D4 passed through the gain map data lookup table 54C, and a multiplication unit 55C that multiplies the multiplication result by the multiplication unit 55B by the external map data D8 passed through the external map data lookup table 54D. The multiplication result by the multiplication unit 55C is output to a selection input unit (Sel) 57.

[0043] The noise reduction intensity data generating unit 50 has an addition value calculation unit (Sum) 56A that calculates the sum of luminance data D5 passed through a luminance data lookup table 54A, color likelihood data D7 passed through a color likelihood data lookup table 54B, gain map data D4 passed through a gain map data lookup table 54C, and external map data D8 passed through an external map data lookup table 54D. The addition result by the addition value calculation unit 56A is output to a selection input unit (Sel) 57.

[0044] The noise reduction intensity data generating unit 50 has a maximum value selection calculation unit (Max) 56B that selectively passes the maximum value of the luminance data D5 passed through the luminance data lookup table 54A, the color likelihood data D7 passed through the color likelihood data lookup table 54B, the gain map data D4 passed through the gain map data lookup table 54C, and the external map data D8 passed through the external map data lookup table 54D. The maximum value selection calculation result by the maximum value selection calculation unit 56B is output to a selection input unit (Sel) 57.

[0045] The noise reduction intensity data generating unit 50 has a minimum value selection calculation unit (Min) 56C that selectively passes the minimum value among the luminance data D5 passed through the luminance data lookup table 54A, the color likelihood data D7 passed through the color likelihood data lookup table 54B, the gain map data D4 passed through the gain map data lookup table 54C, and the external map data D8 passed through the external map data lookup table 54D. The minimum value selection calculation result by the minimum value selection calculation unit 56C is output to a selection input unit (Sel) 57.

[0046] The noise reduction intensity data generating unit 50 has a selection input unit (Sel) 57. The selection input unit 57 selectively passes one of the multiplication result by the multiplication unit 55C, the addition calculation result by the addition value calculation unit 56A, the maximum value selection calculation result by the maximum value selection calculation unit 56B, and the minimum value selection calculation result by the minimum value selection calculation unit 56C. The selection input unit 57 may select, for example, the multiplication result by the multiplication unit 55C, the addition calculation result by the addition value calculation unit 56A, the maximum value selection calculation result by the maximum value selection calculation unit 56B, or the minimum value selection calculation result by the minimum value selection calculation unit 56C at random, or the calculation unit to be used may be set in advance. Alternatively, the selection input unit 57 may determine whether to select the multiplication result by the multiplication unit 55C, the addition calculation result by the addition value calculation unit 56A, the maximum value selection calculation result by the maximum value selection calculation unit 56B, or the minimum value selection calculation result by the minimum value selection calculation unit 56C, based on some criterion for generating more appropriate noise reduction intensity data.

[0047] Here, the multiplication unit 55A, the multiplication unit 55B, the multiplication unit 55C, the addition value calculation unit 56A, the maximum value selection calculation unit 56B, the minimum value selection calculation unit 56C, and the selection input unit 57 function as a "calculation unit" that applies at least one of addition processing, multiplication processing, maximum value selection processing, minimum value selection processing, and random selection processing to each output of the previous-stage lookup table 54 (54A, 54B, 54C, 54D) to calculate a single output. Note that the configuration of the "calculation unit" described with reference to FIG. 5 is merely an example, and it is sufficient if it can calculate a single output from each output of the previous-stage lookup table 54 (54A, 54B, 54C, 54D).

[0048] The noise reduction intensity data generating unit 50 has a subsequent lookup table 58 for application to the single output calculated by the above-mentioned "calculation unit". The subsequent lookup table 58 is composed of a single lookup table (LUT) 58A corresponding to the single output calculated by the above-mentioned "calculation unit". The output that has passed through the subsequent lookup table 58 (single lookup table 58A) becomes noise reduction intensity data optimized in consideration of the extent to which tone correction has been performed on which part of the tone correction image (level distribution of tone correction gain) (for example, FIG. 4A, FIG. 4B).

[0049] In this way, the front-stage lookup table 54 (54A, 54B, 54C, 54D) for application to each (at least one of) the luminance data D5, the color likelihood data D7, the gain map data D4, and the external map data D8, and the rear-stage lookup table 58 (58A) for application to a single output calculated from each output of the front-stage lookup table 54 (54A, 54B, 54C, 54D) are arranged in the front and rear stages sandwiching the "calculation unit". This enables flexible and appropriate characteristic conversion. For example, the front-stage lookup table applies logarithmic processing to handle a wide range, and the rear-stage lookup table applied to the single output calculated from each output of the front-stage lookup table enables characteristic conversion such as returning the logarithm to an antilogarithm. Alternatively, conversely, the antilogarithm may be calculated in the front-stage lookup table and the logarithm may be calculated in the rear-stage lookup table.

[0050] In this manner, in the image processing device, image processing method, and image processing program of the present embodiment, the input image is subjected to the following in order: tone correction, noise reduction intensity data generation, and noise reduction processing. The tone correction block outputs tone correction gain (gain map) along with the tone correction image. The noise reduction intensity data generation block receives the luminance and gain map from the tone correction image, and further performs color analysis to generate color likelihood. Furthermore, a map for each pixel is input from the outside, and the gain amount, luminance, color likelihood, and external map (including processed data of the first noise reduction intensity data) are each applied with a lookup table (LUT) in the previous stage, and each of the addition, multiplication, maximum value, minimum value, etc. is selected, and the result of applying the lookup table (LUT) in the subsequent stage is used as noise reduction intensity data. In addition, the external map can also input the gain amount when the gain is multiplied by HDR processing or the like before the tone correction input, and in that case, it can also be added directly to the gain map. By treating the HDR gain and tone correction gain in logarithms, simple addition is possible. The noise reduction strength data is realized by contributing it as a coefficient to the correlation between the subject pixel and the surrounding pixels. For example, in the case of a BiLateral filter, it is contributed as a coefficient to the difference between the subject pixel and the surrounding pixels, and in the case of a NonLocalMeans filter, it is contributed as a coefficient to the patch correlation, so that the strength of the noise reduction effect can be controlled according to the noise reduction strength.

[0051] In processes such as backlight correction and multiple exposure synthesis (HDR), an adaptive gain-up amount is calculated for each pixel or pixel block, and tone correction of the entire image may be performed. Also, in gamma correction and the like, a different gain-up amount is applied to each pixel or pixel block. If the intensity of noise reduction can be changed according to each of these states, tone correction and noise reduction can be achieved in a well-balanced manner. Therefore, by enabling processing that takes into account not only the gain amount and brightness but also color likelihood and an external map (including processed data of the first noise reduction intensity data), it is possible to control the smoothness of skin-colored parts and the smoothness of a specific area that is freely set. Also, by enabling free map input from the outside, the intensity of noise reduction can be changed according to elements other than gain, brightness, and color likelihood, and it can be applied to applications such as blurring specific areas such as backgrounds, faces, and vehicle numbers.

[0052] Fig. 6 is a conceptual diagram showing the flow of noise reduction processing using the first and second noise reduction intensity data. Fig. 6 illustrates a case where the processed data D12X of the first noise reduction intensity data D12 is looped (input) only once to the noise reduction intensity data generating unit 50. That is, the noise reduction intensity data in the first round becomes "the first noise reduction intensity data D12 and its processed data D12X", and the noise reduction intensity data in the second round becomes "the second noise reduction intensity data D11".

[0053] In step ST100, the common circuit part 100 (demosaic processing part 10, gradation correction part 20, YUV conversion part 30, color analysis part 40 and noise reduction intensity data generation part 50) of the image processing device 1 executes the first round processing by the input image (RAW image, Bayer image) and outputs the first round noise reduction intensity data D9 (first noise reduction intensity data D12) (NR Level output D10). In step ST200, the first round noise reduction intensity data D9 (first noise reduction intensity data D12) is subjected to map processing (filter processing) by the filter processing part 80 and is stored in the memory in the DRAM 90. Then, the noise reduction intensity data read from the DRAM 90 is looped (input) to the noise reduction intensity data generation part 50 as "processed data D12X of the first noise reduction intensity data D12" included in the external map data D8. As is clear from the images before and after filter processing, the processed data D12X of the first noise reduction strength data D12 is of higher quality in which the adverse effects of noise are mitigated compared to the first noise reduction strength data D12 before processing.

[0054] In step ST300, the common circuit portion 100 (the demosaic processing portion 10, the gradation correction portion 20, the YUV conversion portion 30, the color analysis portion 40, and the noise reduction intensity data generation portion 50) of the image processing device 1 executes the second round processing on the input image (RAW image, Bayer image) and outputs the second round noise reduction intensity data D9 (NR Level output D10). This second round noise reduction intensity data is the "second noise reduction intensity data D11" based on the "processed data D12X of the first noise reduction intensity data D12" included in the external map data D8 input in the first round processing.

[0055] Then, a noise reduction processing section 60 separate from the common circuit of the image processing device 1 executes noise reduction processing using the second noise reduction strength data D11, and a post-processing section 70 executes post-processing to produce a developed output (final image) after NR processing. Note that in the second round of processing, the noise reduction strength data (NR Level) is processed internally and is not output to memory. What is output to memory is the YUV image after noise reduction processing (NR YUV).

[0056] 7A and 7B are diagrams showing an example of noise reduction strength data before and after map processing (filter processing) by the filter processing unit 80. As shown in Fig. 7A, in the noise reduction strength data before filter processing, the wall and the doll's face are rough (rough grain and resolution) and the influence of noise is large, whereas as shown in Fig. 7B, in the noise reduction strength data after filter processing, the wall and the doll's face are smooth (fine grain and resolution) and the influence of noise is suppressed. Furthermore, in Fig. 7B, the contour parts with large light and dark differences are emphasized more than in Fig. 7A, and the influence of noise is suppressed.

[0057] Fig. 8 is a flowchart showing an example of an image processing method by the image processing device 1. Fig. 8 illustrates a case where the processed data D12X of the first noise reduction strength data D12 is looped (input) only once to the noise reduction strength data generating unit 50. That is, the noise reduction strength data in the first round becomes "first noise reduction strength data D12 and its processed data D12X," and the noise reduction strength data in the second round becomes "second noise reduction strength data D11."

[0058] In the first round of processing, first, a RAW image is input to the image processing device 1 (step ST1). Next, the demosaic processing unit 10 performs demosaic processing on the RAW image to convert it into an RGB image (step ST2). Next, the gradation correction unit 20 performs gradation correction processing on the RGB image to output a gradation corrected RGB image and gain map data D4 (step ST3). Next, the YUV conversion unit 30 YUV converts the gradation corrected RGB image to output luminance data D5 (step ST4). Next, the color analysis unit 40 performs color analysis on the gradation corrected RGB image to output color likelihood data (e.g., skin map) D7 (step ST5). Next, the noise reduction intensity data generation unit 50 generates noise reduction intensity data (NR level) D9 based on the input luminance data D5, color likelihood data D7, and gain map data D4 (step ST6). The generated noise reduction intensity data (NR level) D9 is output to the outside (outside the common circuit) 200 of the image processing device 1 (step ST7). In the first round of processing, since the "first noise reduction intensity data D12" does not exist, the "processed data D12X of the first noise reduction intensity data D12" is not looped (input) to the noise reduction intensity data generation unit 50 as the external map data D8 (see dashed line).

[0059] The processing in steps ST1 to ST7 in FIG. 8 corresponds to the processing in step ST100 in FIG.

[0060] The noise reduction intensity data (NR level) D9 obtained in the first round of processing is input to the outside of the device (outside of the common circuit) 200 (step ST8). Then, in the outside of the device (outside of the common circuit) 200, the noise reduction intensity data (NR level) D9 is filtered by the filter processing unit (external circuit) 80 and stored in the DRAM (external memory) 90 (step ST9). After that, the stored noise reduction intensity data (NR level) is read out and output as an image (step ST10). This noise reduction intensity data (NR level) is looped (input) to the image processing device 1 (noise reduction intensity data generating unit 50) as "processed data D12X of the first noise reduction intensity data D12" as the processed NR level.

[0061] The processing in steps ST8 to ST10 in FIG. 8 corresponds to the processing in step ST200 in FIG.

[0062] In the second round of processing, the processed data D12X (processed NR level) of the first noise reduction intensity data D12 is output to the noise reduction intensity data generating unit 50. Also, a RAW image is input to the image processing device 1 (step ST11). Next, the demosaic processing unit 10 performs demosaic processing on the RAW image to convert it into an RGB image (step ST12). Next, the gradation correction unit 20 performs gradation correction processing on the RGB image to output a gradation corrected RGB image (step ST13). Next, the YUV conversion unit 30 outputs a gradation corrected YUV image obtained by YUV converting the gradation corrected RGB image (step ST14). Note that in the second round of processing, since the luminance data D5, the color likelihood data D7, and the gain map data D4 have already been acquired in the first round of processing, the luminance data D5, the color likelihood data D7, and the gain map data D4 are not reacquired or reused (see dashed lines). However, if necessary, all or part of the luminance data D5, color likelihood data D7, and gain map data D4 may be reacquired or reused.

[0063] Next, the noise reduction intensity data generating unit 50 generates "second noise reduction intensity data D11" as a more accurate (highly reliable) NR level based on "processed data D12X (processed NR level) of the first noise reduction intensity data D12" looped (input) from the first round processing (step ST15). Next, the noise reduction processing unit 60 performs noise reduction processing (NR processing) on ​​the tone-corrected YUV image from the YUV conversion unit 30 based on the "second noise reduction intensity data (NR level) D11" (step ST16). Next, the post-processing unit 70 performs post-processing (YUV processing) such as gamma correction, edge enhancement processing, and super-resolution processing on the YUV image after the noise reduction processing (step ST17). Finally, the YUV image after the post-processing is output as a final image (step ST18).

[0064] The processing in steps ST11 to ST18 in FIG. 8 corresponds to the processing in step ST300 in FIG.

[0065] In addition, the present invention is not limited to the above-mentioned embodiment, and various modifications can be made in the implementation stage without departing from the gist of the invention. In addition, the functions executed in the above-mentioned embodiment may be implemented in appropriate combinations as much as possible. The above-mentioned embodiment includes various steps, and various inventions can be extracted by appropriate combinations of the multiple components disclosed. For example, if an effect can be obtained even if some components are deleted from all the components shown in the embodiment, the configuration from which the components are deleted can be extracted as an invention.

[0066] The invention as described in the claims of the original application is set forth below. [Appendix 1] a noise reduction intensity data generating unit that generates first noise reduction intensity data for an image, and generates second noise reduction intensity data for the image in response to input of processing data generated based on the first noise reduction intensity data; a noise reduction processing unit that performs a noise reduction process on the image based on the second noise reduction intensity data; 13. An image processing device comprising: [Appendix 2] After the first noise reduction intensity data is stored in an external memory, the processed data is input to the noise reduction intensity data generating unit. 2. The image processing device of claim 1. [Appendix 3] the first noise reduction intensity data is filtered by an external circuit, and then the processed data is input to the noise reduction intensity data generating unit; 3. The image processing device according to claim 1 or 2. [Appendix 4] A tone correction unit that performs tone correction on the image, the noise reduction intensity data generating unit generates the first noise reduction intensity data based on any combination data of a plurality of data generated according to the image. 4. The image processing device according to claim 1, [Appendix 5] the noise reduction intensity data generating unit generates the first noise reduction intensity data based on any combination of luminance data of the tone corrected image, color likelihood data of the tone corrected image, gain map data based on a tone correction amount of the tone corrected image, and external map data. 5. The image processing device according to claim 1, [Appendix 6] The processing data is input from an input circuit for the external map data. 6. The image processing device of claim 5. [Appendix 7] A method for causing a computer to execute each process of the image processing device according to any one of Supplementary Note 1 to Supplementary Note 6. [Appendix 8] A program for causing a computer to execute each process of the image processing device according to any one of appendix 1 to appendix 6. [Explanation of symbols]

[0067] 1. Image processing device 1X Digital camera (photography device) 10 Demosaic processing unit 20. Tone correction section 30...YUV conversion section 40...Color analysis section 50 Noise reduction intensity data generator (NR intensity data generator) 51 Selection input section 52 Multiplication unit 53 Selection input section 54...Pre-stage lookup table 54A···Look-up table for luminance data (LUT(Y)) 54B Lookup table for color likelihood data (LUT(color)) 54C···Lookup table for gain map data (LUT(Gain)) 54D···Lookup table for external map data (LUT(EXT)) 55A···Multiplication section (calculation section) 55B Multiplication unit (arithmetic unit) 55C···Multiplication section (arithmetic section) 56A···Additional value calculation unit (Sum) (calculation unit) 56B Maximum value selection calculation section (Max) (calculation section) 56C Minimum value selection calculation unit (Min) (calculation unit) 57 Selection input section (Sel) (calculation section) 58... Rear lookup table 58A···Single Look-Up Table (LUT) 60 Noise reduction processing section (NR processing section) 70 Post-processing section 80···Filter processing section (external circuit) 90···DRAM (Dynamic Random Access Memory) (External memory) 100...Common circuit part 200...External to the device (external to the common circuit)

Claims

1. a noise reduction intensity data generating unit that outputs first noise reduction intensity data generated based on predetermined image data to a filter processing unit, and receives processed data generated by the filter processing unit performing a predetermined filter process on the first noise reduction intensity data, and generates second noise reduction intensity data taking into account the processed data; a noise reduction processing unit that performs noise reduction processing on the image data by applying the second noise reduction intensity data; having The filtering process is performed by a low-pass filter, an epsilon filter, a Bilateral filter, or a NonLocalMeans filter.

13. An image processing device comprising:

2. a gradation correction unit that performs gradation correction on predetermined RGB image data to generate gradation-corrected image data; a YUV conversion unit that generates YUV image data by YUV converting the gradation corrected image data; a noise reduction intensity data generating unit which outputs to a filter processing unit first noise reduction intensity data generated based on gain map data corresponding to an amount of gradation correction for the RGB image data when the gradation correction unit generates the gradation corrected image data, luminance data in the YUV image data, or likelihood data indicating a likelihood between a color indicated by the YUV image data and a predetermined color, and which receives processed data generated by the filter processing unit performing a predetermined filter process on the first noise reduction intensity data, thereby generating second noise reduction intensity data taking into account the processed data; a noise reduction processing unit that performs noise reduction processing on the YUV image data by applying the second noise reduction intensity data; having The filtering process is performed by a low-pass filter, an epsilon filter, a Bilateral filter, or a NonLocalMeans filter.

13. An image processing device comprising:

3. An image processing method executed by an image processing device, comprising: a noise reduction intensity data generating step of outputting first noise reduction intensity data generated based on predetermined image data to a filter processing unit, and receiving processed data generated by the filter processing unit performing a predetermined filter process on the first noise reduction intensity data, thereby generating second noise reduction intensity data taking into account the processed data; a noise reduction processing step of performing noise reduction processing on the image data by applying the second noise reduction intensity data; having The filtering process is performed by a low-pass filter, an epsilon filter, a Bilateral filter, or a NonLocalMeans filter.

13. An image processing method comprising:

4. An image processing method executed by an image processing device, comprising: a gradation correction step for generating gradation-corrected image data by performing gradation correction on predetermined RGB image data; a YUV conversion step of generating YUV image data by YUV converting the gradation corrected image data; a noise reduction intensity data generating step of outputting to a filter processing unit first noise reduction intensity data generated based on gain map data corresponding to an amount of gradation correction for the RGB image data when generating the gradation corrected image data in the gradation correction step, luminance data in the YUV image data, or likelihood data indicating a likelihood between a color indicated by the YUV image data and a predetermined color, and generating second noise reduction intensity data taking into account processing data generated by the filter processing unit performing a predetermined filter processing on the first noise reduction intensity data; a noise reduction processing step of performing noise reduction processing on the YUV image data by applying the second noise reduction intensity data; having The filtering process is performed by a low-pass filter, an epsilon filter, a Bilateral filter, or a NonLocalMeans filter.

13. An image processing method comprising:

5. A computer comprising: a noise reduction intensity data generating means for outputting first noise reduction intensity data generated based on predetermined image data to a filter processing unit, and for receiving processed data generated by the filter processing unit performing a predetermined filter process on the first noise reduction intensity data, thereby generating second noise reduction intensity data taking into account the processed data; a noise reduction processing means for performing noise reduction processing on the image data by applying the second noise reduction intensity data; Function as a The filtering process is performed by a low-pass filter, an epsilon filter, a Bilateral filter, or a NonLocalMeans filter. A program characterized by:

6. Computer, A tone correction means for generating tone-corrected image data by performing tone correction on predetermined RGB image data; YUV conversion means for generating YUV image data by YUV conversion of the gradation corrected image data; a noise reduction intensity data generating means for outputting to a filter processing unit first noise reduction intensity data generated based on gain map data corresponding to an amount of gradation correction for the RGB image data when the gradation correction means generates the gradation corrected image data, luminance data in the YUV image data, or likelihood data indicating a likelihood between a color indicated by the YUV image data and a predetermined color, and for receiving processed data generated by the filter processing unit performing a predetermined filter process on the first noise reduction intensity data, thereby generating second noise reduction intensity data taking into account the processed data; a noise reduction processing means for performing noise reduction processing on the YUV image data by applying the second noise reduction intensity data; Function as a The filtering process is performed by a low-pass filter, an epsilon filter, a Bilateral filter, or a NonLocalMeans filter. A program characterized by:

Images