Image processing device, image processing method and program

Abstract

To provide an image processing device, image processing method, and image processing program, which enable good gradation correction and noise reduction processing.SOLUTION: An image processing device is provided, comprising a gradation correction unit, noise reduction intensity data generation unit, and noise reduction processing unit. The gradation correction unit applies gradation correction to an image. The noise reduction intensity data generation unit generates noise reduction intensity data after the gradation reduction on the basis of data consisting of an arbitrary combination of a plurality of sets of data that is generated according to the image. The noise reduction processing unit applies noise reduction processing to a gradation-corrected image obtained by the gradation correction according to the noise reduction intensity data.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] Books invention Image processing device and image processing method and Regarding the program. [Background technology]

[0002] For example, Patent Document 1 describes an image processing device that performs tone correction and noise reduction processing on an image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-127388 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in conventional image processing devices including the one disclosed in Patent Document 1, it is difficult to achieve both tone correction and noise reduction processing, and image quality deteriorates when desired noise reduction processing cannot be performed.

[0005] In view of the above circumstances, an object of one aspect of the present invention is to provide an image processing device and an image processing method capable of satisfactorily implementing gradation correction and noise reduction processing. and The aim is to provide a program. [Means for solving the problem]

[0006] The image processing device according to the present invention comprises: an RGB image data generating unit that generates RGB image data by performing demosaic processing on the Bayer image data generated by the multiple exposure synthesis processing; By performing gradation correction on RGB image data, 、a gradation correction unit that generates gradation corrected image data; a YUV conversion unit that generates YUV image data by YUV converting the gradation corrected image data; and a plurality of input data including at least gain map data corresponding to an amount of gradation correction for the RGB image data when the gradation correction unit generates the gradation corrected image data, luminance data in the YUV image data, and likelihood data indicating a likelihood between a color indicated by the YUV image data and a predetermined color. A gain amount for each pixel or pixel block during the multiple exposure synthesis process; and a noise reduction intensity data generating unit configured to generate noise reduction intensity data by a predetermined combination of the plurality of input data, and a noise reduction processing unit configured to perform noise reduction processing on the YUV image data by applying the noise reduction intensity data. Further, an image processing method according to the present invention comprises: An image processing method executed by an image processing device, comprising: an RGB image data generating step of generating RGB image data by performing demosaic processing on Bayer image data generated by multiple exposure synthesis processing; a gradation correction step of generating gradation corrected image data by performing gradation correction on RGB image data; a YUV conversion step of generating YUV image data by YUV converting the gradation corrected image data; and a plurality of input data including at least gain map data corresponding to an amount of gradation correction for the RGB image data when generating the gradation corrected image data in the gradation correction step, luminance data in the YUV image data, and likelihood data indicating a likelihood between a color indicated by the YUV image data and a predetermined color. A gain amount for each pixel or pixel block during the multiple exposure synthesis process; and a noise reduction processing step for performing noise reduction processing on the YUV image data by applying the noise reduction intensity data. The program according to the present invention also includes: an RGB image data generating means for generating RGB image data by performing demosaic processing on the Bayer image data generated by the multiple exposure synthesis processing;a gradation correction means for performing gradation correction on RGB image data to generate gradation corrected image data, a YUV conversion means for performing YUV conversion on the gradation corrected image data to generate YUV image data, and a plurality of input data including at least gain map data corresponding to an amount of gradation correction for the RGB image data when the gradation correction means generates the gradation corrected image data, luminance data in the YUV image data, and likelihood data indicating a likelihood between a color indicated by the YUV image data and a predetermined color; A gain amount for each pixel or pixel block during the multiple exposure synthesis process; and a noise reduction intensity data generating means for generating noise reduction intensity data by a predetermined combination of the plurality of input data, and a noise reduction processing means for performing noise reduction processing on the YUV image data by applying the noise reduction intensity data. Effect of the Invention

[0007] The present invention According to the present invention, an image processing device and an image processing method capable of satisfactorily implementing gradation correction and noise reduction processing are provided. and Programs can be provided. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is an external view showing an example of a digital camera 1X. [Diagram 2] 1 is a first block diagram showing a configuration of an image processing device 1. FIG. [Diagram 3] 2 is a second block diagram showing the configuration of the image processing device 1. FIG. [Figure 4] FIG. 13 is a conceptual diagram showing an example of noise reduction intensity data D9. [Diagram 5] FIG. 2 is a diagram showing a detailed configuration of a noise reduction intensity data generating unit 50. [Figure 6] FIG. 13 is a diagram showing a first example of a generation pattern of noise reduction intensity data D9. [Figure 7]FIG. 13 is a diagram showing a second example of a generation pattern of noise reduction intensity data D9. [Figure 8] FIG. 13 is a diagram showing a third example of a generation pattern of noise reduction intensity data D9. [Figure 9] FIG. 13 is a diagram showing a fourth example of a generation pattern of noise reduction intensity data D9. [Figure 10] FIG. 13 is a diagram showing a fifth example of a generation pattern of noise reduction intensity data D9. [Figure 11] FIG. 13 is a diagram showing a sixth example of a generation pattern of noise reduction intensity data D9. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The image processing device 1 of this embodiment will be described in detail with reference to Fig. 1 to Fig. 11. The image processing method and image processing program of this embodiment are realized by causing a computer (circuit components) of the image processing device 1 to execute various processing steps. The image processing device 1 may be called, for example, a Color Process Pipeline. The image processing device 1 may be implemented as a large scale integrated circuit (LSI) or as a part thereof.

[0010] The image processing device 1 may be mounted on a fixed photographing device such as a security camera or a surveillance camera, or may be mounted on a portable photographing device such as a digital camera or various terminals. The image processing device 1 may be specialized for performing image processing on images transmitted by wire or wirelessly from a photographing device or other device, or images stored in various storage media. In other words, the image processing device 1 has a degree of freedom in its configuration, and various design changes are possible.

[0011] FIG. 1 is an external view showing an example of a digital camera (photographing device) 1X. In the digital camera 1X, light from a subject is guided into a camera body 11X via a lens 10X, reflected by a mirror 12X, and a real image of the subject is formed on a focusing glass 13X. This real image of the subject is guided to a viewfinder 15X by a pentaprism 14X so that it can be observed. A display device 16X is disposed between the focusing glass 13X and the pentaprism 14X, and displays various information superimposed on the subject image reflected on the focusing glass 13X. The broken lines in FIG. 1 indicate the optical path of the light from the subject that is guided to the viewfinder 15X. In the digital camera 1X, the mirror 12X is raised, the shutter 17X is opened, and the subject light is guided to an image sensor 18X to perform photography, so that the subject image does not reach the display device 16X in the mirror-up state.

[0012] The digital camera 1X may incorporate an image processing device 1 according to an embodiment of the present invention that performs image processing on an image captured by the image sensor 18X, or may have a communication unit (such as a connection terminal or an antenna) that transmits an image captured by the image sensor 18X to the external image processing device 1 according to an embodiment of the present invention by wire or wirelessly. That is, the present invention is not limited to the digital camera 1X having a photographing function, but can also be applied to any device that can obtain an image captured by an external device and apply noise reduction processing to it. In other words, the present invention can also be applied to a system composed of multiple devices.

[0013] 2 and 3 are first and second block diagrams showing the configuration of the image processing device 1. The image processing device 1 has a demosaic processing unit 10, a gradation correction unit 20, a YUV conversion unit 30, a color analysis unit 40, a noise reduction intensity data generation unit (NR intensity data generation unit) 50, a noise reduction processing unit (NR processing unit) 60, and a post-processing unit 70.

[0014] The demosaic processor 10 performs demosaic processing on an input image (RAW image, Bayer image) D1, which is an image before gradation correction, and converts it into an RGB image D2. Although not shown here, a block that performs isolated point removal processing and chromatic aberration correction on the input image D1 may be provided before the demosaic processor 10.

[0015] The gradation correction unit 20 performs gradation correction on the RGB image D2 before gradation correction from the demosaic processing unit 10. The gradation correction is adaptively performed on a pixel-by-pixel basis based on the image analysis results, for example, when shooting a subject with a large difference in brightness such as backlight, or when it is desired to brighten a specific area such as a face. In addition, in processes such as backlight correction and multiple exposure synthesis (HDR: High Dynamic Range), an adaptive gain-up amount is calculated for each pixel or pixel block, and gradation correction is performed on the entire image. Furthermore, in gamma correction, etc., a different gain-up amount may be applied to each pixel or pixel block. This gain-up amount may be uniform for RGB. There is a degree of freedom in the manner of gradation correction by the gradation correction unit 20, and various design changes are possible.

[0016] The tone correction unit 20 performs tone correction and outputs a tone corrected image (RGB image after tone correction) D3 and gain map data (Gain Map) D4. The gain map data D4 is data based on the tone correction amount of the tone corrected image D3 and corresponds to the logarithm of the tone correction amount (gain) of the tone corrected image D3. The gain map data D4 divides the tone corrected image D3 into regions for each pixel or pixel block, and has feature values ​​for each divided region. The tone corrected image D3 is input to a YUV conversion unit 30 and a color analysis unit 40, and the gain map data D4 is input to a noise reduction intensity data generation unit 50.

[0017] The YUV conversion unit 30 obtains luminance data (luminance data of the gradation corrected image) D5 by YUV converting the gradation corrected image (RGB image after gradation correction) D3 from the gradation correction unit 20. The luminance data D5 divides the gradation corrected image D3 into regions for each pixel or pixel block, and has feature values ​​for each divided region. The luminance data D5 is input to a noise reduction intensity data generation unit 50. The YUV conversion unit 30 also outputs the YUV converted gradation corrected image D6 to the noise reduction processing unit 60.

[0018] The color analysis unit 40 performs color analysis (RGB analysis) on the gradation corrected image (RGB image after gradation correction) D3 from the gradation correction unit 20 to obtain color likelihood data (color likelihood data of the gradation corrected image) D7. The color likelihood data D7 indicates the likelihood of a specific color, such as the color of human skin or flesh color. The color likelihood data D7 divides the gradation corrected image D3 into regions for each pixel or pixel block, and has a feature value for each divided region. The color likelihood data D7 is input to the noise reduction strength data generation unit 50.

[0019] The noise reduction intensity data generating unit 50 receives the gain map data D4 from the tone correcting unit 20, the luminance data D5 from the YUV converting unit 30, and the color likelihood data D7 from the color analyzing unit .

[0020] In addition, the noise reduction intensity data generating unit 50 receives an Additional Map and / or an Extra Map as external map data D8. The Additional Map can be generated by filtering the generated noise reduction intensity data (NR intensity data), for example, by externally low-pass filtering (LPF) the map data (NR level output) input from a DRAM (Dynamic Random Access Memory). The Extra Map can be a gain amount of HDR processing for an image before gradation correction, for example, a logarithmic gain amount during HDR processing. In FIG. 2, the Additional Map and the Extra Map as external map data D8 are collectively drawn as a single Extra Map, and in FIG. 3, the Additional Map and the Extra Map as external map data D8 are separately drawn. In FIG. 3, only one of the Additional Map and the Extra Map may be input to the noise reduction intensity data generating unit 50.

[0021] The noise reduction intensity data generating unit 50 generates noise reduction intensity data (NR intensity data) D9 based on any combination data of a plurality of data generated according to an image after gradation correction. More specifically, the noise reduction intensity data generating unit 50 generates noise reduction intensity data D9 based on any combination data of the luminance data D5 from the YUV conversion unit 30, the color likelihood data D7 from the color analysis unit 40, the gain map data D4 from the gradation correction unit 20, and the external map data D8. This makes it possible to provide a good quality image in which noise reduction processing has been performed satisfactorily for captured images captured under various conditions. The noise reduction intensity data D9 generated by the noise reduction intensity data generating unit 50 may be stored in a DRAM (Dynamic Random Access Memory) 80 as a noise reduction level output (NR level output) D10. The DRAM 80 is a component separate from the noise reduction intensity data generating unit 50 and, ultimately, the image processing device 1.

[0022] The noise reduction intensity data D9 is for applying an optimized noise reduction process to the gradation corrected image D6, taking into consideration the extent to which gradation correction has been performed on which portion of the gradation corrected image D3 (level distribution of gradation correction gain). In other words, the noise reduction intensity data D9 can be in the form of a map in which the gradation corrected image D3 is divided into regions for each pixel or pixel block, and the level of noise reduction process to be applied to each divided region is associated with the divided region.

[0023] 4A and 4B are conceptual diagrams showing an example of noise reduction intensity data D9. For convenience of explanation, in FIG. 4A and FIG. 4B, the tone correction image is simplified and drawn into a divided area of ​​25 cells in total, 5 cells vertically and 5 cells horizontally. In FIG. 4A, the noise reduction intensity data of each divided area is drawn in three stages, H (High), M (Middle), and L (Low). In FIG. 4B, the noise reduction intensity data of each divided area is drawn in five stages, 1, 2, 3, 4, and 5 (the larger the number, the higher the intensity, and the smaller the number, the lower the intensity). In FIG. 4A and FIG. 4B, the larger the degree of tone correction (level of tone correction gain) of the divided area, the higher the intensity of the noise reduction intensity data is set, and the smaller the degree of tone correction (level of tone correction gain) of the divided area, the lower the intensity of the noise reduction intensity data is set. It should be noted that there is a degree of freedom as to how each divided region of the gradation corrected image is set and how many stages the intensity of the noise reduction intensity data is divided into, and various design modifications are possible.

[0024] How the noise reduction intensity data D9 is generated will be described in detail later.

[0025] The noise reduction processing unit 60 performs noise reduction processing (NR processing) on ​​the gradation corrected image D6 that has been gradation corrected by the gradation correction unit 20 and YUV converted by the YUV conversion unit 30, based on the noise reduction intensity data D9 (see, for example, FIGS. 4A and 4B) generated by the noise reduction intensity data generation unit 50. By performing noise reduction processing using noise reduction intensity data D9 optimized in consideration of the extent of gradation correction performed on which part of the gradation corrected image D3 (level distribution of gradation correction gain), it is possible to achieve a good balance between gradation correction and noise reduction processing, and to realize excellent image quality based on flexible and appropriate noise reduction processing.

[0026] The post-processing unit 70 performs post-processing such as gamma correction, edge enhancement, and super-resolution processing on the noise-reduction-processed YUV image (NR YUV) D11 to generate an output image (YUV) D12. Note that the post-processing performed by the post-processing unit 70 has a degree of freedom, and various design changes are possible.

[0027] 5 is a diagram showing a detailed configuration of the noise reduction intensity data generation unit 50. As described above, the noise reduction intensity data generation unit 50 receives the luminance data D5, the color likelihood data D7, the gain map data D4, and the external map data D8.

[0028] The noise reduction intensity data generating unit 50 has a selection input unit 51 that selectively passes some or all of the luminance data D5, the color likelihood data D7, the gain map data D4, and the external map data D8. In the following explanation, it is assumed that all of the luminance data D5, the color likelihood data D7, the gain map data D4, and the external map data D8 pass through the selection input unit 51.

[0029] It is also possible to omit the selection input unit 51 from the noise reduction intensity data generation unit 50. In this case, only the data to be used among the luminance data D5, the color likelihood data D7, the gain map data D4, and the external map data D8 may be input to the noise reduction intensity data generation unit 50, and the data not to be used may not be input to the noise reduction intensity data generation unit 50.

[0030] The noise reduction intensity data generating unit 50 has a multiplication unit 52 that multiplies the gain map data D4 passed through the selection input unit 51 by the external map data D8. The noise reduction intensity data generating unit 50 has a selection input unit 53 that selectively passes either the gain map data D4 or the multiplication result of the gain map data D4 and the external map data D8 by the multiplication unit 52. In the following description, the gain map data D4 or the multiplication result of the gain map data D4 and the external map data D8 by the multiplication unit 52 will be collectively referred to as "gain map data D4" (the output of the selection input unit 53 is "gain map data D4").

[0031] In this embodiment, when two pieces of data are multiplied, if there is no input of one of the pieces of data (for example, if it does not pass through the selection input unit 51), one of the pieces of data may be set to 1, and the other piece of data may be output as the multiplication result as is.

[0032] The noise reduction intensity data generating unit 50 has a front-stage lookup table 54 for application to each (at least any one of) the luminance data D5, the color likelihood data D7, the gain map data D4, and the external map data D8. More specifically, the front-stage lookup table 54 has a luminance data lookup table (LUT(Y)) 54A applied to the luminance data D5, a color likelihood data lookup table (LUT(color)) 54B applied to the color likelihood data D7, a gain map data lookup table (LUT(Gain)) 54C applied to the gain map data D4, and an external map data lookup table (LUT(EXT)) 54D applied to the external map data D8.

[0033] The noise reduction intensity data generating unit 50 has a multiplication unit 55A that multiplies the luminance data D5 passed through the luminance data lookup table 54A by the color likelihood data D7 passed through the color likelihood data lookup table 54B, a multiplication unit 55B that multiplies the multiplication result by the multiplication unit 55A by the gain map data D4 passed through the gain map data lookup table 54C, and a multiplication unit 55C that multiplies the multiplication result by the multiplication unit 55B by the external map data D8 passed through the external map data lookup table 54D. The multiplication result by the multiplication unit 55C is output to a selection input unit (Sel) 57.

[0034] The noise reduction intensity data generating unit 50 has an addition value calculation unit (Sum) 56A that calculates the sum of luminance data D5 passed through a luminance data lookup table 54A, color likelihood data D7 passed through a color likelihood data lookup table 54B, gain map data D4 passed through a gain map data lookup table 54C, and external map data D8 passed through an external map data lookup table 54D. The addition result by the addition value calculation unit 56A is output to a selection input unit (Sel) 57.

[0035] The noise reduction intensity data generating unit 50 has a maximum value selection calculation unit (Max) 56B that selectively passes the maximum value of the luminance data D5 passed through the luminance data lookup table 54A, the color likelihood data D7 passed through the color likelihood data lookup table 54B, the gain map data D4 passed through the gain map data lookup table 54C, and the external map data D8 passed through the external map data lookup table 54D. The maximum value selection calculation result by the maximum value selection calculation unit 56B is output to a selection input unit (Sel) 57.

[0036] The noise reduction intensity data generating unit 50 has a minimum value selection calculation unit (Min) 56C that selectively passes the minimum value among the luminance data D5 passed through the luminance data lookup table 54A, the color likelihood data D7 passed through the color likelihood data lookup table 54B, the gain map data D4 passed through the gain map data lookup table 54C, and the external map data D8 passed through the external map data lookup table 54D. The minimum value selection calculation result by the minimum value selection calculation unit 56C is output to a selection input unit (Sel) 57.

[0037] The noise reduction intensity data generating unit 50 has a selection input unit (Sel) 57. The selection input unit 57 selectively passes one of the multiplication result by the multiplication unit 55C, the addition calculation result by the addition value calculation unit 56A, the maximum value selection calculation result by the maximum value selection calculation unit 56B, and the minimum value selection calculation result by the minimum value selection calculation unit 56C. The selection input unit 57 may select, for example, the multiplication result by the multiplication unit 55C, the addition calculation result by the addition value calculation unit 56A, the maximum value selection calculation result by the maximum value selection calculation unit 56B, or the minimum value selection calculation result by the minimum value selection calculation unit 56C at random, or the calculation unit to be used may be set in advance. Alternatively, the selection input unit 57 may determine whether to select the multiplication result by the multiplication unit 55C, the addition calculation result by the addition value calculation unit 56A, the maximum value selection calculation result by the maximum value selection calculation unit 56B, or the minimum value selection calculation result by the minimum value selection calculation unit 56C, based on some criterion for generating more appropriate noise reduction intensity data.

[0038] Here, the multiplication unit 55A, the multiplication unit 55B, the multiplication unit 55C, the addition value calculation unit 56A, the maximum value selection calculation unit 56B, the minimum value selection calculation unit 56C, and the selection input unit 57 function as a "calculation unit" that applies at least one of addition processing, multiplication processing, maximum value selection processing, minimum value selection processing, and random selection processing to each output of the previous-stage lookup table 54 (54A, 54B, 54C, 54D) to calculate a single output. Note that the configuration of the "calculation unit" described with reference to FIG. 5 is merely an example, and it is sufficient if it can calculate a single output from each output of the previous-stage lookup table 54 (54A, 54B, 54C, 54D).

[0039] The noise reduction intensity data generating unit 50 has a subsequent lookup table 58 for application to the single output calculated by the above-mentioned "calculation unit". The subsequent lookup table 58 is composed of a single lookup table (LUT) 58A corresponding to the single output calculated by the above-mentioned "calculation unit". The output that has passed through the subsequent lookup table 58 (single lookup table 58A) becomes noise reduction intensity data optimized in consideration of the extent to which tone correction has been performed on which part of the tone correction image (level distribution of tone correction gain) (for example, FIG. 4A, FIG. 4B).

[0040] In this way, the front-stage lookup table 54 (54A, 54B, 54C, 54D) for application to each (at least one of) the luminance data D5, the color likelihood data D7, the gain map data D4, and the external map data D8, and the rear-stage lookup table 58 (58A) for application to a single output calculated from each output of the front-stage lookup table 54 (54A, 54B, 54C, 54D) are arranged in the front and rear stages sandwiching the "calculation unit". This enables flexible and appropriate characteristic conversion. For example, the front-stage lookup table applies logarithmic processing to handle a wide range, and the rear-stage lookup table applied to the single output calculated from each output of the front-stage lookup table enables characteristic conversion such as returning the logarithm to an antilogarithm. Alternatively, conversely, the front-stage lookup table may calculate an antilogarithm, and the rear-stage lookup table may calculate a logarithm.

[0041] In this manner, in the image processing device, image processing method, and image processing program of the present embodiment, the input image is subjected to the following in order: tone correction, noise reduction intensity data generation, and noise reduction processing. The tone correction block outputs tone correction gain (gain map) along with the tone correction image. The noise reduction intensity data generation block receives the luminance and gain map from the tone correction image, and further performs color analysis to generate color likelihood. Furthermore, a map for each pixel is input from the outside, and the front-stage lookup table (LUT) is applied to each of the gain amount, luminance, color likelihood, and external map, and the respective addition, multiplication, maximum value, minimum value, etc. are selected, and the result of applying the rear-stage lookup table (LUT) is used as noise reduction intensity data. In addition, the external map can also input the gain amount when the gain is multiplied by HDR processing or the like before the tone correction input, and in that case, it can also be added directly to the gain map. By treating the HDR gain and tone correction gain in logarithms, simple addition is possible. The noise reduction strength data is realized by contributing it as a coefficient to the correlation between the subject pixel and the surrounding pixels. For example, in the case of a BiLateral filter, it is contributed as a coefficient to the difference between the subject pixel and the surrounding pixels, and in the case of a NonLocalMeans filter, it is contributed as a coefficient to the patch correlation, so that the strength of the noise reduction effect can be controlled according to the noise reduction strength.

[0042] In processes such as backlight correction and multiple exposure synthesis (HDR), an adaptive gain-up amount is calculated for each pixel or pixel block, and tone correction of the entire image may be performed. Also, in gamma correction and the like, a different gain-up amount is applied to each pixel or pixel block. If the intensity of noise reduction can be changed according to each of these states, tone correction and noise reduction can be achieved in a well-balanced manner. Therefore, by enabling processing that takes into account not only the gain amount and brightness but also color likelihood and an external map, it is possible to control the smoothness of skin-colored parts and the smoothness of a specific area that is freely set. Also, by enabling free map input from the outside, the intensity of noise reduction can be changed according to elements other than gain, brightness, and color likelihood, and it can be applied to applications such as blurring specific areas such as backgrounds, faces, and vehicle numbers.

[0043] Fig. 6 is a diagram showing a first example of a generation pattern of the noise reduction intensity data D9. The first example in Fig. 6 is specialized for enhancing skin color noise reduction (enhancing skin color NR).

[0044] Color likelihood data D7 is obtained by performing color likelihood analysis (skin mapping) on ​​the gradation corrected image (RGB image) D3. The color likelihood data D7 is map data for highlighting skin-like characteristics, for example, for making pixels determined to be skin-like brighter than the gradation corrected image (RGB image). A color likelihood data lookup table 54B is applied to the color likelihood data D7 as a skin map multiplication table. The color likelihood data lookup table 54B has a characteristic that the noise reduction strength increases as the skin likelihood increases, in order to increase the noise reduction strength of the skin-colored parts. The color likelihood data lookup table 54B has a characteristic that the noise reduction strength increases at a likelihood of 0, and at a likelihood of 255, and is 1 / 4 of that at this point. Color likelihood data D7 that has passed through color likelihood data lookup table 54B has skin likelihood converted into a contribution rate of noise reduction strength, and as a result, the pixels in the skin-colored areas are brighter than before passing through color likelihood data lookup table 54B.

[0045] On the other hand, a gain map data lookup table 54C serving as a logarithm to real number table is applied to gain map data (Gain Map) D4, which is a logarithm-to-real number table, in which the tone correction gain of the tone correction image is logarithmically expressed. The gain map data lookup table 54C converts the gain map data (Gain Map) D4 from logarithm to real number in order to perform multiplication between real numbers in a subsequent multiplier. Therefore, the gain map data (Gain Map) D4 that has passed through the gain map data lookup table 54C has been converted from a logarithmic expression to a real number expression.

[0046] The color likelihood data D7 passed through the color likelihood data lookup table 54B is multiplied by the gain map data (Gain Map) D4 passed through the gain map data lookup table 54C. The result of this multiplication is the product of the skin-likeness and the gain amount, so that even if the skin-likeness is high, the area with low gain becomes dark, and even if the gain is high, the area with low skin-likeness becomes dark. The single lookup table 58A as a level adjustment table is applied to the multiplication result of the color likelihood data D7 passed through the color likelihood data lookup table 54B and the gain map data (Gain Map) D4 passed through the gain map data lookup table 54C. The single lookup table 58A returns the data, which has been scaled to 1 / 4 by the multiplication of the color likelihood, to the original scale by multiplying it by 4. In this way, the noise reduction intensity data D9 optimized based on the result of taking into account both the skin color and the gain amount is output.

[0047] Fig. 7 is a diagram showing a second example of a generation pattern of the noise reduction intensity data D9. The second example in Fig. 7 is specialized for pre-addition of gamma.

[0048] By performing tone correction on the uncorrected RGB image D2, a tone-corrected image (RGB image) D3 and gain map data (Gain Map) D4 that logarithmically represents the tone correction gain of the tone-corrected image are obtained.

[0049] Luminance data D5 is obtained by performing luminance analysis (luminance conversion) on the gradation corrected image (RGB image) D3. A luminance data lookup table 54A serving as a gamma gain multiplication table is applied to the luminance data D5. The luminance data lookup table 54A is a table that specifies the amount of gamma correction to be applied after noise reduction processing. The luminance data lookup table 54A has a characteristic of being 4.0 times at low luminance and 1.0 times at saturation, and at this point, the output is 1 / 4 of that. The luminance data D5 that has passed through the luminance data lookup table 54A is optimized according to the amount of gamma correction to be applied after noise reduction processing.

[0050] On the other hand, a gain map data lookup table 54C serving as a logarithm to real number table is applied to gain map data (Gain Map) D4, which is a logarithm-to-real number table, in which the tone correction gain of the tone correction image is logarithmically expressed. The gain map data lookup table 54C converts the gain map data (Gain Map) D4 from logarithm to real number in order to perform multiplication between real numbers in a subsequent multiplier. Therefore, the gain map data (Gain Map) D4 that has passed through the gain map data lookup table 54C has been converted from a logarithmic expression to a real number expression.

[0051] The luminance data D5 passed through the luminance data lookup table 54A is multiplied by the gain map data (Gain Map) D4 passed through the gain map data lookup table 54C. The result of this multiplication reflects both the gamma correction and tone correction components (gamma correction gain and tone correction gain). The single lookup table 58A as a level correction table is applied to the multiplication result of the luminance data D5 passed through the luminance data lookup table 54A and the gain map data (Gain Map) D4 passed through the gain map data lookup table 54C. The single lookup table 58A returns the data, which has been multiplied by the gamma gain to 1 / 4 scale, to the original scale by multiplying it by 4. In this way, the noise reduction intensity data optimized based on the result of taking into account both the gamma correction and tone correction components (gamma correction gain and tone correction gain) is output.

[0052] Fig. 8 is a diagram showing a third example of a generation pattern of the noise reduction intensity data D9. The third example in Fig. 8 is specialized for HDR gradation correction.

[0053] Before being input to the demosaic processing unit 10 in Fig. 1 and Fig. 2, an HDR correction process is performed on the Under RAW image and the Over RAW image. As a result, an HDR RAW image obtained by combining the Under RAW image and the Over RAW image and an HDR gain obtained by logarithmically expressing the HDR correction gain are output. For example, the HDR gain can be set to a high gain for moving parts in order to perform motion-guaranteed HDR combining. The HDR RAW image is converted into RGB and tone corrected to output an HDR RGB image, and tone correction gain (logarithmic expression) of the HDR RGB image is also output.

[0054] The HDR gain, which is a logarithmic representation of the HDR correction gain, and the tone correction gain (logarithmic representation) of the HDR RGB image are added in logarithmic space. The lookup table 54C for gain map data, which is a logarithm to real number table, is applied to the result of this addition. The lookup table 54C for gain map data converts the result of the addition of the HDR gain and the tone correction gain from logarithm to real number. The result of the addition of the HDR gain and the tone correction gain that has passed through the lookup table 54C for gain map data becomes an HDR-compatible gain map. The single lookup table 58A, which is a linear table, is applied to this HDR-compatible gain map. By passing through the linear table when no conversion is performed in the final stage, noise reduction intensity data optimized based on the result taking into account HDR tone correction is output.

[0055] Fig. 9 is a diagram showing a fourth example of a generation pattern of the noise reduction intensity data D9. The fourth example in Fig. 9 is a combination of the first to third examples in Figs.

[0056] Before being input to the demosaic processing unit 10 in Fig. 1 and Fig. 2, an HDR correction process is performed on the Under RAW image and the Over RAW image. As a result, an HDR RAW image obtained by combining the Under RAW image and the Over RAW image and an HDR gain obtained by logarithmically expressing the HDR correction gain are output. For example, the HDR gain can be set to a high gain for moving parts in order to perform motion-guaranteed HDR combining. The HDR RAW image is converted into RGB and tone corrected to output an HDR RGB image, and tone correction gain (logarithmic expression) of the HDR RGB image is also output.

[0057] The HDR gain, which is a logarithmic representation of the HDR correction gain, and the tone correction gain (logarithmic representation) of the HDR RGB image are added in logarithmic space. The gain map data lookup table 54C, which is a logarithm to real number table, is applied to the result of this addition. The gain map data lookup table 54C converts the addition result of the HDR gain and the tone correction gain from a logarithm to a real number. The addition result of the HDR gain and the tone correction gain that has passed through the gain map data lookup table 54C becomes an HDR-compatible gain map.

[0058] Color likelihood data D7 is obtained by performing color likelihood analysis (skin mapping) on ​​the gradation corrected image (HDR RGB image). A color likelihood data lookup table 54B serving as a skin map multiplication table is applied to the color likelihood data D7. The color likelihood data lookup table 54B has a characteristic of multiplying the data by 1.0 at a likelihood of 0 and by 4.0 at a likelihood of 255, and at this point, the output is 1 / 4 of that. After passing through the color likelihood data lookup table 54B, the skin likelihood of the color likelihood data D7 is converted into a contribution rate of noise reduction strength, and as a result, the pixels in the skin-colored areas are brighter than before passing through the color likelihood data lookup table 54B.

[0059] Luminance data D5 is obtained by performing luminance analysis (luminance conversion) on the tone corrected image (HDR RGB image). A luminance data lookup table 54A serving as a gamma gain multiplication table is applied to the luminance data D5. The luminance data lookup table 54A has a characteristic of multiplying the data by 4.0 at low luminance and 1.0 at saturation, and the output at this point is 1 / 4 of that. The luminance data D5 that has passed through the luminance data lookup table 54A is optimized according to the amount of gamma correction to be applied after noise reduction processing.

[0060] In this way, the luminance data D5 passed through the luminance data lookup table 54A, the color likelihood data D7 passed through the color likelihood data lookup table 54B, and the HDR-compatible gain map passed through the gain map data lookup table 54C are converted into real number space, and are multiplied by a multiplier. This multiplication result is optimized in consideration of skin color noise reduction enhancement (skin color NR enhancement), both components of gamma correction and gradation correction (gamma correction gain and gradation correction gain), and HDR gradation correction. A single lookup table 58A as a level adjustment table is applied to the multiplication result of the luminance data D5 passed through the luminance data lookup table 54A, the color likelihood data D7 passed through the color likelihood data lookup table 54B, and the HDR-compatible gain map passed through the gain map data lookup table 54C. The single lookup table 58A returns data that has been scaled to 1 / 16 by the multiplication of the color likelihood and gamma to the original scale by multiplying it by 16. In this way, noise reduction intensity data optimized taking into account enhanced skin tone noise reduction (skin tone NR enhancement), both gamma correction and tone correction components (gamma correction gain and tone correction gain), and HDR tone correction is output.

[0061] Fig. 10 is a diagram showing a fifth example of a generation pattern of noise reduction intensity data D9 (NR level output D10). The fifth example in Fig. 10 is specialized for background blurring using a subject distance map.

[0062] For example, the distance between the camera (imaging device) and the subject is measured using a TOF (Time Of Flight) sensor or image analysis, and this is generated as a subject distance map. External map data lookup table 54D, which serves as a noise reduction level (NR Level) table, is applied to the generated subject distance map. External map data lookup table 54D increases the output as the input subject distance increases, and can, for example, output at 1x magnification when subject distance is 0 (close state), and output at up to 4.0x magnification as the subject distance increases.

[0063] The subject distance map that has passed through external map data lookup table 54D becomes a blur amount map that converts subject distance into a blur amount. A single lookup table 58A as a linear table is applied to this blur amount map. By passing the data through the linear table when no conversion is performed in the final stage, noise reduction intensity data optimized based on the result that takes into account the blur amount caused by the subject distance is output (NR level output).

[0064] Fig. 10 shows an example of an image based on normal RGB output without background blurring, and an image based on RGB output with background blurring using the object distance map. The farther the object distance is, the stronger the noise reduction strength is, resulting in an RGB image with a blurred background.

[0065] Fig. 11 is a diagram showing a sixth example of a generation pattern of noise reduction intensity data D9. The sixth example in Fig. 11 is a combination of the first to third and fifth examples in Figs. 6 to 8 and Fig. 10.

[0066] Before input to the demosaic processing unit 10 in FIG. 1 and FIG. 2, the Under RAW image and the Over RAW image are subjected to HDR correction processing, and as a result, an HDR RAW image obtained by combining the Under RAW image and the Over RAW image is output. The HDR RAW image is converted into RGB and subjected to gradation correction, and an HDR RGB image is output, and a gradation correction gain (logarithmic expression) of the HDR RGB image is output. The gradation correction gain is added to an HDR gain that is a logarithmic expression of the HDR correction gain, and becomes gain map data D4. A gain map data lookup table 54C as a logarithm to real number table is applied to the gain map data D4. The gain map data lookup table 54C converts the gain map data D4 from logarithm to real number. The gain map data D4 that has passed through the gain map data lookup table 54C becomes an HDR-compatible gain map.

[0067] Color likelihood data D7 is obtained by performing color likelihood analysis (skin mapping) on ​​the gradation corrected image (HDR RGB image). A color likelihood data lookup table 54B serving as a skin map multiplication table is applied to the color likelihood data D7. The color likelihood data lookup table 54B has a characteristic of multiplying the data by 1.0 at a likelihood of 0 and by 4.0 at a likelihood of 255, and at this point, the output is 1 / 4 of that. After passing through the color likelihood data lookup table 54B, the skin likelihood of the color likelihood data D7 is converted into a contribution rate of noise reduction strength, and as a result, the pixels in the skin-colored areas are brighter than before passing through the color likelihood data lookup table 54B.

[0068] Luminance data D5 is obtained by performing luminance analysis (luminance conversion) on the tone corrected image (HDR RGB image). A luminance data lookup table 54A serving as a gamma gain multiplication table is applied to the luminance data D5. The luminance data lookup table 54A has a characteristic of multiplying the data by 4.0 at low luminance and 1.0 at saturation, and the output at this point is 1 / 4 of that. The luminance data D5 that has passed through the luminance data lookup table 54A is optimized according to the amount of gamma correction to be applied after noise reduction processing.

[0069] For example, the distance between the camera (imaging device) and the subject is measured using a TOF sensor or image analysis, and this is generated as a subject distance map. External map data lookup table 54D, which serves as a noise reduction level (NR Level) table, is applied to the generated subject distance map. External map data lookup table 54D increases the output as the input subject distance increases, and can output, for example, at 1x output when subject distance is 0 (close state), and up to 4.0x output as the subject distance increases. The subject distance map that passes through external map data lookup table 54D becomes a blur amount map that converts subject distance into a blur amount.

[0070] In this way, the luminance data D5 passed through the luminance data lookup table 54A, the color likelihood data D7 passed through the color likelihood data lookup table 54B, the HDR-compatible gain map data D4 passed through the gain map data lookup table 54C, and the blur amount map (external map data D8) passed through the external map data lookup table 54D are converted into real number space and multiplied by a multiplier. The multiplication result is optimized in consideration of skin color noise reduction enhancement (skin color NR enhancement), both components of gamma correction and gradation correction (gamma correction gain and gradation correction gain), HDR gradation correction, and subject distance. A single lookup table 58A is applied as a level adjustment table to the multiplication result of the luminance data D5 passed through the luminance data lookup table 54A, the color likelihood data D7 passed through the color likelihood data lookup table 54B, the HDR-compatible gain map data D4 passed through the gain map data lookup table 54C, and the blur amount map (external map data D8) passed through the external map data lookup table 54D. The single lookup table 58A returns the data, which has been multiplied by the color likelihood and gamma to 1 / 16 scale, to the original scale by multiplying it by 16. In this way, noise reduction intensity data optimized in consideration of skin tone noise reduction enhancement (skin tone NR enhancement), both components of gamma correction and gradation correction (gamma correction gain and gradation correction gain), HDR gradation correction, and subject distance is output.

[0071] The advantages of the image processing device, image processing method, and image processing program of this embodiment include the ability to freely associate multiple elements (brightness, color likelihood, gain map (tone correction gain), external map (external element), etc.) (such as a two-stage lookup table with front and rear stages or relational equation selection (addition, multiplication, maximum value, minimum value)), in addition to excellent tone correction, to realize the following functions: (1) The strength of noise reduction in areas such as skin tones and blue skies can be freely changed (for example, allowing for flexible and appropriate control, such as making skin tones smoother). (2) It is possible to allow the components of each process (such as gamma correction) performed after noise reduction processing to be contributed in advance. (3) The HDR gain and tone compensation gain used during HDR processing can be multiplied together. HDR gain and tone compensation gain can be added together in a special first-stage (previous stage) lookup table. Since they are both gains, they are multiplied together, but in the case of logarithmic gains, they are added together. (4) You can control the degree of background blur by entering the subject distance in the Extra Map. (5) Anything else can be input to the external map (Extra Map), allowing flexible and appropriate control of noise reduction intensity according to the input of the external map.

[0072] In addition, the present invention is not limited to the above-mentioned embodiment, and various modifications can be made in the implementation stage without departing from the gist of the invention. In addition, the functions executed in the above-mentioned embodiment may be implemented in appropriate combinations as much as possible. The above-mentioned embodiment includes various steps, and various inventions can be extracted by appropriate combinations of the multiple components disclosed. For example, if an effect can be obtained even if some components are deleted from all the components shown in the embodiment, the configuration from which the components are deleted can be extracted as an invention.

[0073] The invention as described in the claims of the original application is set forth below. [Appendix 1] a gradation correction unit that performs gradation correction on an image; a noise reduction intensity data generating unit that generates noise reduction intensity data based on any combination data of a plurality of data generated according to the image after the gradation correction; a noise reduction processing unit that performs a noise reduction process on the tone corrected image based on the noise reduction intensity data; 13. An image processing device comprising: [Appendix 2] the noise reduction intensity data generating unit generates the noise reduction intensity data based on any combination of luminance data of the gradation corrected image, color likelihood data of the gradation corrected image, gain map data based on a gradation correction amount of the gradation corrected image, and external map data. 2. The image processing device of claim 1. [Appendix 3] the noise reduction intensity data generating unit has a front-stage lookup table to be applied to at least one of the luminance data, the color likelihood data, the gain map data, and the external map data; 3. The image processing device of claim 2. [Appendix 4] the noise reduction intensity data generating unit has a calculation unit that applies at least one of an addition process, a multiplication process, a maximum value selection process, a minimum value selection process, and a random selection process to each output of the previous-stage lookup table to calculate a single output; 4. The image processing device of claim 3. [Appendix 5] The noise reduction intensity data generating unit has a post-stage lookup table for application to the single output calculated by the calculation unit. 5. The image processing device of claim 4. [Appendix 6] The external map data includes at least one of a gain amount of HDR processing for an image before gradation correction and a data obtained by filtering the generated noise reduction intensity data. 6. The image processing device according to claim 2, [Appendix 7] A method for causing a computer to execute each process of the image processing device according to any one of Supplementary Note 1 to Supplementary Note 6. [Appendix 8] A program for causing a computer to execute each process of the image processing device according to any one of appendix 1 to appendix 6. [Explanation of symbols]

[0074] 1. Image processing device 1X Digital camera (photography device) 10 Demosaic processing unit 20. Tone correction section 30...YUV conversion section 40...Color analysis section 50 Noise reduction intensity data generator (NR intensity data generator) 51 Selection input section 52 Multiplication unit 53 Selection input section 54...Pre-stage lookup table 54A···Look-up table for luminance data (LUT(Y)) 54B Lookup table for color likelihood data (LUT(color)) 54C···Lookup table for gain map data (LUT(Gain)) 54D···Lookup table for external map data (LUT(EXT)) 55A···Multiplication section (calculation section) 55B Multiplication unit (arithmetic unit) 55C···Multiplication section (arithmetic section) 56A···Additional value calculation unit (Sum) (calculation unit) 56B Maximum value selection calculation section (Max) (calculation section) 56C Minimum value selection calculation unit (Min) (calculation unit) 57 Selection input section (Sel) (calculation section) 58... Rear lookup table 58A···Single Look-Up Table (LUT) 60 Noise reduction processing section (NR processing section) 70 Post-processing section 80...DRAM(Dynamic Random Access Memory)

Claims

1. An RGB image data generation unit that generates RGB image data by performing demosaic processing on Bayer image data generated by multiple exposure synthesis processing; a gradation correction unit that performs gradation correction on the RGB image data to generate gradation-corrected image data; a YUV conversion unit that generates YUV image data by YUV converting the gradation corrected image data; a noise reduction intensity data generating unit configured to be able to receive as input, as a plurality of pieces of input data, at least gain map data corresponding to a gradation correction amount for the RGB image data when the gradation correction unit generates the gradation corrected image data, luminance data in the YUV image data, likelihood data indicating a likelihood between a color indicated by the YUV image data and a predetermined color, and a gain amount for each pixel or pixel block during the multiple exposure synthesis process, and generating noise reduction intensity data by a predetermined combination of the plurality of input data; a noise reduction processing unit that performs noise reduction processing on the YUV image data by applying the noise reduction intensity data; 13. An image processing device comprising:

2. The multiple exposure compositing process is a motion guaranteed multiple exposure compositing process, The gain amount for each pixel or pixel block during the multiple exposure synthesis process is a gain amount based on the presence or absence of motion.

2. The image processing device according to claim 1,

3. An image processing method executed by an image processing device, comprising: an RGB image data generating step of generating RGB image data by performing demosaic processing on the Bayer image data generated by the multiple exposure synthesis processing; a gradation correction step of generating gradation-corrected image data by performing gradation correction on the RGB image data; a YUV conversion step of generating YUV image data by YUV converting the gradation corrected image data; a noise reduction intensity data generating step in which at least gain map data corresponding to a gradation correction amount for the RGB image data when the gradation corrected image data is generated in the gradation correction step, luminance data in the YUV image data, likelihood data indicating a likelihood between a color indicated by the YUV image data and a predetermined color, and a gain amount for each pixel or pixel block during the multiple exposure synthesis process can be input as a plurality of input data, and noise reduction intensity data is generated by a predetermined combination of the plurality of input data; a noise reduction processing step of performing noise reduction processing on the YUV image data by applying the noise reduction intensity data; 13. An image processing method comprising:

4. A computer comprising: RGB image data generating means for generating RGB image data by performing demosaic processing on the Bayer image data generated by the multiple exposure synthesis processing; a tone correction means for generating tone-corrected image data by performing tone correction on the RGB image data; YUV conversion means for generating YUV image data by YUV conversion of the gradation corrected image data; a noise reduction intensity data generating means configured to be able to receive as input, as a plurality of pieces of input data, at least gain map data corresponding to a gradation correction amount for the RGB image data when the gradation correction means generates the gradation corrected image data, luminance data in the YUV image data, likelihood data indicating a likelihood between a color indicated by the YUV image data and a predetermined color, and a gain amount for each pixel or pixel block in the multiple exposure synthesis process, and generating noise reduction intensity data by a predetermined combination of the plurality of input data; a noise reduction processing means for performing noise reduction processing on the YUV image data by applying the noise reduction intensity data; A program characterized by causing the program to function as a

Images