Imaging device, imaging method, and program

Abstract

The present disclosure relates to an imaging device, an imaging method, and a program that make it possible to appropriately achieve gradation compression by non-linear conversion related to development processing. Since the correction curve applied to the non-linear conversion and a non-linear inverse conversion can be switched according to an imaging mode, appropriate non-linear conversion and non-linear inverse conversion can be performed according to the imaging mode, making it possible to achieve appropriate gradation compression. The present disclosure can be applied to imaging devices.

Description

Imaging device, imaging method, and program

[0001] The present disclosure relates to an imaging device, an imaging method, and a program, and more particularly to an imaging device, an imaging method, and a program that are capable of appropriately achieving gradation compression by nonlinear conversion related to development processing.

[0002] When compressing image data, there is a process of applying nonlinear compression (tone quantization) accompanied by gamma conversion etc. immediately before encoding using a main compression method.

[0003] This process is known to be particularly effective for display devices with non-linear characteristics, or when gamma conversion is applied in the development process of image sensor output.

[0004] As this processing, for example, a technique has been proposed in which compression efficiency is improved by performing gradation compression using nonlinear conversion, focusing on gradation ranges to which humans have low visual sensitivity (see Patent Document 1).

[0005] Japanese Patent Application Laid-Open No. 2005-191939

[0006] However, in the technology of Patent Document 1, gradation compression is performed by uniform nonlinear conversion based on a correction curve centered on the gradation range where humans have low visual sensitivity, so nonlinear conversion matching the correction curve corresponding to the subsequent development process cannot be performed, and there is a risk that gradation compression cannot be properly achieved. In particular, it is difficult to take into account optical black, which is specific to image processing of sensor signals, and to handle signal processing such as HDR compositing.

[0007] The present disclosure has been made in view of such circumstances, and in particular, makes it possible to appropriately realize gradation compression by nonlinear conversion related to image development processing.

[0008] An imaging device and a program according to one aspect of the present disclosure include a pixel array unit in which pixels are arranged in an array, the pixels performing photoelectric conversion on incident light in accordance with the amount of light and generating pixel signals in accordance with the amount of light; a nonlinear conversion unit that applies a nonlinear transformation to the pixel signals; an encoding unit that encodes the nonlinearly converted pixel signals; a decoding unit that decodes the pixel signals encoded by the encoding unit; a nonlinear inverse conversion unit that applies a nonlinear inverse conversion to the pixel signals decoded by the decoding unit; and a development processing unit that generates a developed image by development processing using the pixel signals that have been nonlinearly inversely converted by the nonlinear inverse conversion unit, wherein the nonlinear conversion unit applies a nonlinear transformation having a gradient smaller than the maximum gradient of the nonlinear transformation applied to pixel signals with values ​​equal to or greater than optical black to some or all of the pixel signals with values ​​equal to or less than optical black, and the nonlinear inverse conversion unit applies a nonlinear inverse transformation having a gradient smaller than the maximum gradient of the nonlinear transformation applied to pixel signals with values ​​equal to or greater than optical black to some or all of the pixel signals with values ​​equal to or less than optical black.

[0009] An imaging method according to one aspect of the present disclosure is an imaging method for an imaging device having a pixel array unit in which pixels are arranged in an array, each pixel performing photoelectric conversion on incident light in accordance with an amount of light and generating a pixel signal in accordance with the amount of light, the imaging method including: a nonlinear transformation process that applies a nonlinear transformation to the pixel signals; an encoding process that encodes the nonlinearly transformed pixel signals; a decoding process that decodes the pixel signals encoded by the encoding process; a nonlinear inverse transformation process that applies a nonlinear inverse transformation to the pixel signals decoded by the decoding process; and a development process that generates a developed image using the pixel signals that have been nonlinearly inversely transformed by the nonlinear inverse transformation process, wherein the nonlinear transformation process applies a nonlinear transformation having a smaller gradient to some or all of the pixel signals having values ​​equal to or less than optical black than the maximum gradient of the nonlinear transformation applied to pixel signals having values ​​equal to or greater than optical black, and the nonlinear inverse transformation process applies a nonlinear inverse transformation having a smaller gradient to some or all of the pixel signals having values ​​equal to or less than optical black than the maximum gradient of the nonlinear transformation applied to pixel signals having values ​​equal to or greater than optical black.

[0010] In one aspect of the present disclosure, pixels arranged in an array in a pixel array photoelectrically convert incident light in accordance with the amount of light to generate pixel signals in accordance with the amount of light, the pixel signals are subjected to a nonlinear transformation, the nonlinearly converted pixel signals are encoded, the encoded pixel signals are decoded, the decoded pixel signals are subjected to an inverse nonlinear transformation, and a developed image is generated by a development process using the inverse nonlinearly converted pixel signals, where a nonlinear transformation having a gradient smaller than the maximum gradient of the nonlinear transformation applied to pixel signals with values ​​equal to or greater than optical black is applied to some or all of the pixel signals with values ​​equal to or less than optical black, and a nonlinear inverse transformation having a gradient smaller than the maximum gradient of the nonlinear transformation applied to pixel signals with values ​​equal to or greater than optical black is applied to some or all of the pixel signals with values ​​equal to or less than optical black.

[0011] 10 is a diagram illustrating an example of the configuration of an imaging device for explaining an overview of the present disclosure. FIG. 11 is a diagram illustrating correction curves in nonlinear transformation and inverse nonlinear transformation. FIG. 2 is a diagram illustrating an LUT corresponding to the correction curve. FIG. 2 is a diagram illustrating a correction curve assumed in nonlinear transformation in the imaging device of FIG. 1 and a correction curve at the time of development. FIG. 12 is a diagram illustrating variations of the correction curve. FIG. 13 is a diagram illustrating HDR compositing. FIG. 14 is a diagram illustrating histograms of long-exposure and short-exposure images used in HDR compositing. FIG. 15 is a diagram illustrating an example of the configuration of a first embodiment of an imaging device of the present disclosure. FIG. 16 is a diagram illustrating a priority table and corresponding correction curves. FIG. 17 is a diagram illustrating an example of a priority table based on the correction curve of FIG. 9. FIG. 18 is a diagram illustrating consideration of Optical Black in the priority table of FIG. 10. FIG. 19 is a diagram illustrating consideration of reproduction of saturated pixel values ​​in the priority table of FIG. 10. FIG. 19 is a diagram illustrating that calculations using bit shifts can be performed by using the priority table of FIG. 10. FIG. 20 is a diagram illustrating an example of bit shifts for each priority. FIG. 21 is a diagram illustrating a procedure for generating a priority table. FIG. 22 is a diagram illustrating a procedure for generating a priority table. FIG. 23 is a diagram illustrating a procedure for generating a priority table. FIG. 24 is a flowchart illustrating a priority table generation process. FIG. 25 is a flowchart illustrating an imaging process by the imaging device of FIG. 8. 25 is a diagram illustrating a plurality of correction curves in a first modified example of the present disclosure. 26 is a diagram illustrating a priority table corresponding to the plurality of correction curves in FIG. 20. 27 is a diagram illustrating an LUT corresponding to the correction curve in a second modified example of the present disclosure. 28 is a diagram illustrating an example configuration of an imaging device when the LUT in FIG. 22 is used. 29 is a diagram illustrating an encoding process or a decoding process using already-encoded or already-decoded pixels around a pixel to be processed in an encoding process and a decoding process that are application examples of the present disclosure. 30 is a diagram illustrating an example configuration of an imaging device that performs an encoding process or a decoding process using already-encoded or already-decoded pixels around a pixel to be processed in an encoding process and a decoding process. 31 is a diagram illustrating pixel signals before being stored in memory and pixel signals restored from memory when pixel signals are stored in memory as reference pixels in the imaging device of FIG. 25.27 is a diagram illustrating an example configuration of an imaging device according to the present disclosure, in which encoding and decoding processes are performed using pixels that have already been encoded or decoded around a pixel to be processed. FIG. 28 is a diagram illustrating pixel signals before being stored in memory and pixel signals restored from memory when pixel signals are stored in memory as reference pixels in the imaging device of FIG. 27. FIG. 29 is a diagram illustrating an overview of a second embodiment of an imaging device according to the present disclosure. FIG. 30 is a flowchart illustrating imaging processing by the imaging device of FIG. 30. FIG. 31 is a diagram illustrating an example configuration of a general-purpose computer.

[0012] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0013] Hereinafter, embodiments of the present technology will be described in the following order.

[0014] 1. Overview of the present disclosure 2. First embodiment 3. First modified example 4. Second modified example 5. Application example 6. Second embodiment 7. Example of execution by software

[0015] <<1. Overview of the Present Disclosure>> The present disclosure is directed to, in particular, appropriately achieving tone compression through nonlinear conversion related to development processing. Therefore, first, an overview of the present disclosure will be described.

[0016] FIG. 1 shows an example of the configuration of an imaging device for explaining encoding by nonlinear transformation and decoding by nonlinear inverse transformation.

[0017] The imaging device 1 includes an image sensor 11 and a processor unit 12 .

[0018] The image sensor 11 captures an image made up of pixel signals corresponding to the amount of incident light, encodes and compresses the image, and outputs the encoded image to the processor unit 12. More specifically, the image sensor 11 includes a pixel array 31, a nonlinear conversion unit 32, a quantization unit 33, and an encoding unit 34.

[0019] The pixel array 31 is configured with pixels made of CMOS (Complementary Metal Oxide Semiconductor) or the like arranged in an array, and each pixel generates a pixel signal corresponding to the amount of incident light, and outputs the pixel signal consisting of two-dimensional RAW data to the nonlinear conversion unit 32.

[0020] The nonlinear conversion unit 32 performs nonlinear conversion on the pixel signals supplied from the pixel array 31 using a correction curve such as gamma conversion applied to a display device (not shown) or in the development process in the processor unit 12, thereby achieving gradation compression processing and outputting the result to the quantization unit 33.

[0021] The quantization unit 33 removes a predetermined number of lower bits from the nonlinearly converted n-bit pixel signal, quantizes it to m (<n)-bit data, and outputs it to the encoding unit 34 .

[0022] The encoding unit 34 compresses the pixel signal consisting of m-bit quantized data using a predetermined encoding method (compression method), and outputs the compressed data to the processor unit 12. The predetermined encoding method (compression method) may be any of a variety of methods, such as standard encoding methods defined in the MIPI (Mobile Industry Processor Interface) standard, such as COMP6, MPC (Multi Pixel Coding), and JPEG XS (Joint Photographic Experts Group XS), or a proprietary compression method.

[0023] The processor unit 12 decodes the encoded and compressed image supplied from the image sensor 11 and generates a developed image by performing a predetermined development process on the pixel signals that constitute the image. More specifically, the processor unit 12 includes a decoding unit 51, an inverse quantization unit 52, a nonlinear inverse transformation unit 53, and a signal processing unit 54.

[0024] The decoding unit 51 has a configuration corresponding to the encoding unit 34, and decodes a pixel signal consisting of encoded m-bit quantized data compressed using a predetermined encoding method (compression method), and outputs it to the inverse quantization unit 52.

[0025] The inverse quantization unit 52 has a configuration corresponding to the quantization unit 33 , and inversely quantizes a pixel signal consisting of m-bit quantized data to restore it to an n-bit pixel signal, and outputs it to the nonlinear inverse transformation unit 53 .

[0026] The nonlinear inverse conversion unit 53 has a configuration corresponding to the nonlinear conversion unit 32, and restores the gradation by performing a nonlinear inverse conversion on the n-bit pixel signal that has been subjected to gradation compression processing by nonlinear conversion, and outputs the pixel signal with the restored gradation to the signal processing unit 54.

[0027] The signal processing unit 54 performs a predetermined development process based on the pixel signals with restored gradation, and generates and outputs a developed image.

[0028] More specifically, the signal processing unit 54 includes an OPB subtraction unit 71 , a gamma correction unit 72 , a development processing unit 73 , a saturation detection unit 74 , and a noise amount detection unit 75 .

[0029] The OPB (Optical Black) subtraction unit 71 regards the pixel values ​​of light-shielded pixels formed in a part of the pixel array 31 called Optical Black as noise and treats them as an offset, subtracts them from pixel values ​​other than the light-shielded pixels to remove the noise, and outputs the result to the gamma correction unit 72.

[0030] Generally, pixel signals do not output 0 even when there is no incident light, but have a certain offset added to them. This certain offset can be considered as noise generated in the image sensor that makes up the pixel.

[0031] Therefore, the OPB subtraction unit 71 removes noise from the pixel signals by regarding the pixel signal values ​​of the light-shielded pixels as noise and subtracting the pixel signal values ​​from the pixel signals of pixels other than the light-shielded pixels to perform offsetting.

[0032] The gamma correction unit 72 performs gamma correction related to image development on the pixel signals from which noise has been removed, and outputs the result to the development processing unit 73 .

[0033] The development processing unit 73 executes predetermined processes related to image development, such as demosaic and white balance correction, based on the pixel signals that have been subjected to gamma correction, and generates and outputs a developed image.

[0034] The saturation detection unit 74 detects areas where saturated pixels exist in the pixel signals that have been subjected to nonlinear inverse transformation, and outputs the detected areas as saturated area information. This saturated area information can be used to determine whether the exposure was appropriate, for example, based on whether the saturated pixel area is higher than a predetermined ratio. Therefore, it is possible to perform processing such as correcting the exposure depending on the result of determining whether the exposure is appropriate based on the saturated area information.

[0035] The noise amount detector 75 detects and outputs noise intensity based on the pixel signals of the light-shielded pixels, which makes it possible to adjust the sensitivity of the pixels in the pixel array 31, for example, based on the noise intensity.

[0036] <Nonlinear Transformation Unit and Nonlinear Inverse Transformation Unit> Next, the nonlinear transformation unit 32 and the nonlinear inverse transformation unit 53 will be described with reference to FIGS.

[0037] The nonlinear conversion unit 32 performs nonlinear conversion based on a correction curve corresponding to gamma correction or the like as shown in FIG. 2 as pre-processing on the linear pixel signal before the pixel signal is encoded by the encoding unit 34.

[0038] In FIG. 2, the solid curved line represents the correction curve, and the dotted straight line represents the RAW data that constitutes the pixel signal.

[0039] The correction curve in FIG. 2 can be expressed, for example, by the following equation (1).

[0040] y=Pmax×(x / Pmax) １／γ ...(1)

[0041] Here, x is the RAW data of the pixel signal, y is the correction value, Pmax is the maximum value of the RAW data of the pixel signal within the frame, and γ is the γ value in the gamma correction.

[0042] In FIG. 2, a gamma 2.2 curve is shown as an example of a correction curve, but the RAW data indicated by the dotted line that constitutes the pixel signals captured by the pixel array 31 is converted into correction curve values ​​that correspond to the gamma value.

[0043] Nonlinear conversion is possible by multiplying the pixel signal of each pixel by the above-mentioned formula (1). However, in order to reduce the processing load involved in the calculation, this is often done by providing an LUT and converting the RAW data into a correction value.

[0044] That is, the nonlinear conversion unit 32 may be provided with an LUT (Look Up Table) as shown in FIG. 3, and may refer to the LUT for the pixel signal of each pixel, convert the pixel signal into a corresponding pixel signal, and output the converted signal.

[0045] In FIG. 3, the left column shows the values ​​of the raw data (RAW), and the right column shows the corresponding correction values ​​(Gamma).

[0046] That is, for example, when the pixel signal supplied from the pixel array 31 is 34, the nonlinear conversion unit 32 converts it to 217 and outputs it.

[0047] The nonlinear inverse transform unit 53 basically performs processing to restore pixel signals that have been nonlinearly transformed by a transform that is the inverse of the nonlinear transform performed by the nonlinear transform unit 32 .

[0048] Generally, pixel signals made up of RAW data are subjected to nonlinear transformation, followed by quantization to reduce the bit depth of the input data. When the post-processing is a predetermined signal processing, sufficient image quality can be obtained by gradation compression using such nonlinear transformation.

[0049] As described above, the pixel values ​​of light-shielding pixels called Optical Black (hereinafter also referred to as OPB) pixels are added as an offset to the RAW data, which is the pixel signal of the pixel array 31. This is because the pixel value is not 0 but has a value even when there is no incident light, and the pixel value in this state of no incident light becomes the offset.

[0050] When an image is developed, the pixel signals of the OPB pixels are offset by subtraction, and gamma correction and development processing are performed.

[0051] Here, if the pixel signal of the OPB pixel is ob, the dynamic range of the pixel signal after ob subtraction is Pmax-ob.

[0052] Taking the offset operation due to the OPB into consideration, the processes of the OPB subtraction unit 71 and the gamma correction unit 72 can be formulated together as shown in the following equation (2).

[0053] y=Pmax×(max(0,x-ob) / (Pmax-ob)) １／γ ... (2)

[0054] Here, max(0, x-ob) is the maximum value of either the offset RAW data (x-ob) or 0. In other words, max(0, x-ob) essentially represents clipping the RAW data (x-ob) to 0 or more.

[0055] As a result, the correction curve applied in the nonlinear conversion unit 32 becomes the one shown in the left part of Figure 4, and the correction performed by the OPB subtraction unit 71 and gamma correction unit 72 in the signal processing unit 54 becomes the one shown in the right part of Figure 4, resulting in a mutual discrepancy.

[0056] Furthermore, as shown in the left part of Figure 4, the correction curve assumed on the compression side does not take into account the subtraction of the offset due to the OPB pixel, and a correction curve with a large slope is applied to the pixel signal subtracted by the OPB.

[0057] Correction using a correction curve with a steep slope means that the pixel signals are protected in terms of image quality, and the RAW data of OPB pixels that do not contribute after development processing is largely protected.

[0058] As a result, there is a risk that pixel signals may not be restored appropriately by nonlinear transformation and nonlinear inverse transformation based on a correction curve that does not take OPB pixels into consideration.

[0059] It is also possible to apply simple subtraction to the compression side, but in this case, it may become difficult to detect noise intensity using the RAW data of the OPB pixels.

[0060] (Quantization Unit and Inverse Quantization Unit) Next, the quantization unit 33 and the inverse quantization unit 52 will be described.

[0061] The quantization unit 33 quantizes the pixel signal by, for example, the calculation expressed by the following equation (3).

[0062] Qv=int(In / 2 ｑ ) ... (3)

[0063] Here, Qv is a quantized value, In is a pixel signal obtained by nonlinear conversion of each pixel, int(A) is the value of the integer part of A, and q is a quantization bit.

[0064] Furthermore, the inverse quantization unit 52 corresponding to the quantization unit 33 inversely quantizes the quantized value Qv by the calculation expressed by the following equation (4), for example, to restore the pixel signal.

[0065] Out = Qv x 2 ｑ +2 ｑ－１ ...(4)

[0066] Here, Out is the restored pixel signal, Qv is the quantization value, and q is the quantization bit.

[0067] The calculations of the above-described equations (3) and (4) in the quantization unit 33 and the inverse quantization unit 52 are merely examples. For example, consider a case where the pixel signal In is a saturated value of 1023 when the quantization bit number q is 10, and 2 bits are reduced by quantization.

[0068] At this time, the quantization unit 33 quantizes the saturated value 1023 to output a quantized value Qv=255.

[0069] However, even if the inverse quantization unit 52 performs inverse quantization based on this quantization value Qv=255, the restored pixel signal Out becomes 1022.

[0070] In other words, when the RAW data constituting the pixel signal is a saturated value, if a nonlinear transformation and a nonlinear inverse transformation using a uniform correction curve are applied, the pixel signal may not be properly restored in processing by the quantization unit 33 and the inverse quantization unit 52.

[0071] <Variations of Correction Curve> Next, variations of the correction curve will be described with reference to FIG.

[0072] In the above, an example has been described in which a gamma curve is used as a correction curve (tone curve), but correction curves that are set during development processing exist other than gamma curves, and there are various variations, such as the high-key curve shown on the left side of Fig. 5, the low-key curve shown in the center of Fig. 5, and the curve that provides a hard tone expression shown on the right side of Fig. 5. Although not shown, correction curves also exist that provide a soft tone expression.

[0073] However, when nonlinear transformation and nonlinear inverse transformation are performed assuming only a gamma curve as the correction curve described above, applying a correction curve other than a gamma curve may result in inappropriate transformation in the nonlinear transformation and nonlinear inverse transformation.

[0074] <Complexity of Development Processing> Next, the complexity of development processing will be described with reference to FIG.

[0075] Current imaging devices do not only generate a developed image from a single frame of captured image, but also generate a developed image by combining multiple frames of captured image.

[0076] One technique for generating a developed image by synthesizing multiple frames of captured images is HDR (High Dynamic Range). As shown in Figure 6, HDR synthesis generates an illuminance image from a multiple-exposure image consisting of multiple images with different exposure times by linearizing each image, and generates a weight map by calculating the weights for each image. The multiple illuminance images are then integrated and synthesized by taking a weighted average based on the weight map to generate a single image.

[0077] For example, in the case of normal imaging, since imaging is performed using a single exposure, the exposure time is generally set so that a wide range of images can be captured, from dark areas to saturated areas. For example, as shown in the left part of Figure 7, the exposure time is set so that the number of saturated pixels in the histogram is constant. As a result, for an image made up of normal RAW data, the histogram will have a peak in the area with low pixel values.

[0078] In contrast, HDR blending captures two images with different exposure times: a long-exposure image and a short-exposure image. The long-exposure image is set to capture dark areas clearly and without noise, and is set to allow a larger number of saturated pixels than a general short-exposure image. Therefore, the distribution of pixel values ​​for the long-exposure image, as shown in the center of Figure 7, is shifted toward the high-luminance side compared to the distribution for normal imaging, as shown in the left part of Figure 7.

[0079] Similarly, in a short-exposure image, the exposure time is set to reduce saturated pixels and saturated pixels in high-brightness areas, and therefore the distribution of pixel values ​​in the short-exposure image, as shown in the right part of Fig. 7, is shifted toward the low-brightness side relative to the distribution in the normal image, as shown in the left part of Fig. 7.

[0080] In this way, the pixel value distribution differs between a long-exposure image and a short-exposure image captured in HDR compositing.

[0081] For this reason, there is a risk that pixel signals may not be restored appropriately by nonlinear transformation and inverse nonlinear transformation based on a uniform correction curve.

[0082] As described above, the imaging device 1 of Figure 1 may not be able to perform nonlinear transformation and inverse nonlinear transformation corresponding to the subsequent development processing, making it impossible to properly restore pixel signals, and as a result, there is a risk that proper development processing may not be achieved.

[0083] Therefore, in the present disclosure, nonlinear transformation and inverse nonlinear transformation are performed in the subsequent development process, thereby allowing pixel signals to be appropriately restored and enabling appropriate development processing to be achieved.

[0084] <<2. First Embodiment>> Next, with reference to FIG. 8, a configuration example of a first embodiment of an imaging device according to the present disclosure will be described.

[0085] The imaging device 101 in FIG. 8 is composed of an image sensor 111, a processor unit 112, and an operation unit 113.

[0086] The image sensor 111 and the processor section 112 basically have the same functions as the image sensor 11 and the processor section 12 in FIG.

[0087] The image sensor 111 includes a pixel array 131, a nonlinear conversion unit 132, a quantization unit 133, and an encoding unit 134. The pixel array 131, the quantization unit 133, and the encoding unit 134 have the same functions as the pixel array 31, the quantization unit 33, and the encoding unit 34 in FIG. 1, respectively.

[0088] However, the nonlinear conversion section 132 has a configuration with functions different from those of the nonlinear conversion section 32 .

[0089] That is, the nonlinear conversion unit 132 performs nonlinear conversion, such as gamma conversion applied to a display device (not shown) or in the development process in the processor unit 12, on the pixel signals supplied from the pixel array 31 based on the priority table supplied from the priority table setting unit 170 of the signal processing unit 154 in the processor unit 112, thereby achieving gradation compression processing and outputting the result to the quantization unit 133. The priority table corresponds to a correction curve, and details will be described later.

[0090] The processor unit 112 includes a decoding unit 151, an inverse quantization unit 152, a nonlinear inverse transformation unit 153, and a signal processing unit 154. The decoding unit 151, the inverse quantization unit 152, and the signal processing unit 154 have the same functions as the decoding unit 51, the inverse quantization unit 52, and the signal processing unit 54 in FIG.

[0091] However, the nonlinear inverse transform unit 153 has a different function from that of the nonlinear inverse transform unit 53 .

[0092] That is, the nonlinear inverse transform unit 153 has a configuration corresponding to the nonlinear transform unit 132, and restores the nonlinearly transformed pixel signals based on the priority table supplied from the priority table setting unit 170 of the signal processing unit 154 in the processor unit 112, and outputs the restored pixel signals to the signal processing unit 154. As mentioned above, the priority table will be described in detail later.

[0093] In addition to the functions of the signal processing unit 54 in Figure 1, the signal processing unit 154 accepts input of an imaging mode from the operation unit 113 and outputs a priority table corresponding to the accepted operation mode to the nonlinear conversion unit 132 and the nonlinear inverse conversion unit 153.

[0094] More specifically, the signal processing unit 154 includes a priority table setting unit 170 , an OPB subtraction unit 171 , a gamma correction unit 172 , a development processing unit 173 , a saturation detection unit 174 , and a noise amount detection unit 175 .

[0095] Of these, the OPB subtraction unit 171, gamma correction unit 172, saturation detection unit 174, and noise amount detection unit 175 have the same functions as the OPB subtraction unit 71, gamma correction unit 72, saturation detection unit 74, and noise amount detection unit 75 in Figure 1, respectively.

[0096] The priority table setting unit 170 stores correction curves corresponding to operation modes, and generates and stores a priority table based on the correction curves for each imaging mode in advance. The priority table setting unit 170 accepts an imaging mode input by operating the operation unit 113, which is composed of a keyboard, operation buttons, a touch panel, or the like, reads out a priority table corresponding to the accepted imaging mode, and supplies it to the nonlinear conversion unit 132 and the inverse nonlinear conversion unit 153 of the image sensor 111.

[0097] The development processing unit 173 uses the gamma-corrected pixel signals to perform development processing according to the operation mode, and generates and outputs a developed image.

[0098] With this configuration, it is possible to essentially switch the correction curve depending on the imaging mode, making it possible to realize nonlinear transformation and nonlinear inverse transformation depending on the imaging mode, and as a result, it becomes possible to realize appropriate development processing depending on the imaging mode.

[0099] <Priority Table> Next, the priority table will be described.

[0100] The priority table is a table that divides a nonlinear transformation based on a correction curve into multiple sections and enables linear transformation in each section, thereby converting the nonlinear transformation into a pseudo-linear transformation for each section.

[0101] For example, consider a case where a correction curve represented by a dotted line as shown in Fig. 9 is divided into sections #0 to #5 and expressed so that each section can be linearly transformed. Here, the function of each section from #0 to #5 is expressed by the following equation (5).

[0102] ft(n,x)=(xx-x ｎ－１，ｍａｘ ) x K Ｐｎ +ft(n-1,x ｎ－１，ｍａｘ ) ... (5)

[0103] Here, ft(n, x) is a function that expresses a straight line in the section n, x is a pixel signal consisting of RAW data, and x ｎ－１，ｍａｘ is the maximum value of the pixel signal in the section (n-1) immediately before the section n, and ft(n-1, x ｎ－１，ｍａｘ ) is the linear transformation result for the maximum value of the pixel signal in the section (n-1) (nonlinear transformation is performed for all sections from section #0 to #5, but linear transformation is performed for each section #n). Ｐｎis the slope (gradient) of the function ft(n, x), K is 2 or 4, and Pn is the priority of section n and is an integer. Note that K may be a value greater than 4 as long as it is a power of 2. Although K is described as a power of 2 and Pn as an integer, these are constraints imposed when the computational cost during implementation is taken into consideration. When computational cost is not taken into consideration or when implementation is performed using an LUT, the above constraints can be ignored. However, for convenience of explanation, this embodiment will be described as having a value of K=2 or 4.

[0104] The priority is an index indicating the degree of protection of the pixel signal x made up of RAW data, which is an input value. That is, in equation (5), when the priority Pn is 0, the slope K Ｐｎ is 1, the linear transformation result ft is equivalent to the pixel signal x.

[0105] Furthermore, when the priority Pn is greater than 1, the linear transformation result ft changes more significantly than the change in the pixel signal x, and is therefore treated as a pixel signal whose image quality is protected within the image.

[0106] Furthermore, when the priority Pn is smaller than 1, the linear transformation result ft changes less than the change in the pixel signal x, and therefore is treated as a pixel signal that is not protected (unprotected) in terms of image quality within the image.

[0107] To summarize, when the priority Pn is greater than 1, the larger the priority Pn, the greater the change in pixel value of the pixel signal x made of RAW data will be converted so that the pixel signal x will be protected within the image. On the other hand, when the priority Pn is less than 1, the smaller the priority Pn, the smaller the change in pixel value of the pixel signal x made of RAW data will be converted so that the pixel signal x will not be protected within the image.

[0108] On the other hand, by performing a linear transformation for each section based on the function of each section from section #0 to #5 expressed in equation (5), the overall nonlinear transformation result is restored by an inverse transformation expressed in the following equation (6).

[0109] bt(n,x)=(x-ft(n-1,xｎ－１，ｍａｘ )) x K （－Ｐｎ） +x ｎ－１，ｍａｘ +offset ｎ ...(6)

[0110] Here, bt(n, x) is the nonlinear inverse transform result for the nonlinear transform result x, and offset ｎ is an offset that prevents saturated values ​​from becoming unsaturated values ​​in the nonlinear inverse transformation. ｎ is the slope K Ｐｎ If decreases monotonically with respect to each interval, it will be non-zero only in the final interval (interval #5 if the intervals are from interval #0 to interval #5), and will be 0 in the other intervals.

[0111] In this way, the correction curve is divided into a plurality of sections #0 to #5, and in each section, the functions expressed by equations (5) and (6) are used to express the nonlinear transformation and inverse nonlinear transformation as a combination of linear transformations set for each of the plurality of divided sections. This is the priority table shown in FIG.

[0112] In FIG. 10, the range (minimum value x) of the pixel signal x is shown for each section n from the left. ｎ，ｍｉｎ and the maximum value x ｎ，ｍａｘ ), priority Pn, and offset offset ｎ is stated.

[0113] In FIG. 10, in section #0, the minimum value x ｎ，ｍｉｎ is set to 0, and the maximum value x ｎ，ｍａｘ is set to 16, the priority Pn is set to 0, and the offset offset ｎ is set to 0.

[0114] In addition, in section #1, the minimum value x ｎ，ｍｉｎ is set to 17, and the maximum value x ｎ，ｍａｘ is set to 128, the priority Pn is set to +2, and the offset ｎ is set to 0.

[0115] Similarly, in section #2, the minimum value xｎ，ｍｉｎ is set to 129, and the maximum value x ｎ，ｍａｘ is set to 256, the priority Pn is set to +1, and the offset ｎ is set to 0.

[0116] In section #3, the minimum value x ｎ，ｍｉｎ is set to 257, and the maximum value x ｎ，ｍａｘ is set to 384, the priority Pn is set to 0, and the offset offset ｎ is set to 0.

[0117] In section #4, the minimum value x ｎ，ｍｉｎ is set to 385, and the maximum value x ｎ，ｍａｘ is set to 962, the priority Pn is set to -2, and the offset offset ｎ is set to 0.

[0118] In section #5, the minimum value x ｎ，ｍｉｎ is set to 963, and the maximum value x ｎ，ｍａｘ is set to 1023, the priority Pn is set to -1, and the offset ｎ is set to 1.

[0119] In the sections #0 and #1, in a typical gamma correction curve, the slope (the slope of the tangent to the correction curve) is assumed to monotonically decrease with respect to the change in the pixel signal x, as described above.

[0120] However, in the priority table of the present disclosure, as shown in FIG. 11, in the section #0 and #1, the priority Pn is set to 0 for the section #0, which is the range up to x=16 of the pixel signal that is the OPB pixel, and the gradient K Ｐｎ is set to 1. That is, the slope K Ｐ０ is the slope K of section #1 Ｐ１ and is not monotonically decreasing. That is, the slope K Ｐ１ is the maximum gradient (gradient), and the gradient K in section #0 that becomes the pixel signal of the OPB pixel Ｐ０ is the slope K of section #1 Ｐ１ As a result, when the gradient of the correction curve monotonically decreases, the gradient K in section #0 becomes maximum. Ｐ０However, in FIG. 11, the slope K Ｐ１ will be smaller than

[0121] This is to prevent the pixel signal of the OPB from being overly protected by nonlinear conversion and output as is, which enables the noise amount detection unit 175 to appropriately measure noise intensity by using a signal that changes linearly with respect to the pixel signal x made up of RAW data restored by nonlinear inverse conversion.

[0122] 11 , a priority table is shown on the left side of the figure, and an enlarged view of range Z1 in FIG. 9 is shown on the right side of the figure. The information in the bold rectangular range in the priority table on the left side of FIG. 11 corresponds to the straight lines in each of ranges #0 and #1 on the right side of FIG. 11 . Furthermore, in this example, the priority Pn is set to 0 for all pixels in the range of pixel signals x up to 16 that are considered to be OPB pixels (the range of pixel signals x equal to or less than 16). However, the priority Pn may be set to 0 only for pixels in a portion of the range of pixel signals x up to 16 that are considered to be OPB pixels that are close to pixel signal x=0, for example, pixels with pixel signals x=14 or less. In other words, since it is sufficient to be able to measure the amount of noise to a certain extent, a portion of pixel signals close to pixel signal x=0 among pixel signals x=0 to 16 that are considered to be OPB pixels may be used.

[0123] In addition, in sections #4 and #5, in a typical gamma correction curve, the slope (the slope of the tangent to the correction curve) is assumed to monotonically decrease with respect to the change in pixel signal x, as described above.

[0124] However, in the priority table of the present disclosure, as shown in FIG. 12, the gradient K in section #4 is set to 1023 so that the saturated value 1023 is appropriately restored in sections #4 and #5. Ｐ０ is the slope K of section #5 Ｐ１ is smaller than and does not decrease monotonically.

[0125] This is to ensure that the saturation value of the pixel signal can be appropriately restored as a saturation value by inverse nonlinear transformation, even if the saturation value is nonlinearly transformed. As a result, the saturation detection unit 174 can appropriately measure the saturation region by using the pixel signal that is appropriately restored as a saturation value.

[0126] In FIG. 12, the priority table is shown on the left side of the drawing, and an enlarged view of the range Z2 in FIG. 9 is shown on the right side of the drawing.

[0127] Furthermore, the nonlinear transformation unit 132 and the inverse nonlinear transformation unit 153 need to perform calculations related to linear transformation and inverse linear transformation using the pixel signal x in each section based on the priority table, but since the calculations can be reused, the calculations can be performed only once for each pixel signal x, which reduces the load related to the calculations.

[0128] <Bit Shift of Pixel Signal Based on Priority Pn> Next, bit shift of pixel signal based on priority Pn will be described, taking the calculation method of (x-16)×4+16 as an example.

[0129] As described above, the function f(n, x) related to the nonlinear transformation has a slope K Ｐｎ where K is 2 or 4. Furthermore, a priority Pn is set for each section.

[0130] Since the priority Pn is an integer, when K=2, the slope K Ｐｎ is expressed as a power of two.

[0131] For example, as shown in the upper part of Figure 13, if the pixel signal x that is the boundary between section #0 and section #1 is x = 16, the nonlinear transformation result f(0, 16) becomes 16 through calculation based on the priority table for nonlinear transformation.

[0132] Furthermore, since the priority Pn in section #0 is 0, the gradient K Ｐ０ is 1, but the priority Pn in section #1 is +2, so the slope K Ｐ１ is 4 (=2 ２ )

[0133] Furthermore, when x=25, 16, x, x-16, and (x-16)×4 are expressed in bit notation, they are expressed as shown in order from left to right in the bottom of Fig. 13. Note that in the bit notation in the bottom row of Fig. 13, the bottom is the LSB (Least Significant Bit) and the top is the MSB (Most Significant Bit).

[0134] In the lower part of FIG. 13, 16 is expressed as 0000010000, x (=25) is expressed as 0000011001, x-16 (=9) is expressed as 0000001001, and (x-16) x 4 (=36) is expressed as 0000100100.

[0135] For the data in section #1 in the upper part of FIG. 13, the value of (x-16), which is the main component of the pixel value, is multiplied by four, and 0 is inserted into the two least significant bits.

[0136] In this case where K=2, nonlinear transformation and inverse nonlinear transformation can be performed simply by shifting (bit shifting) the bit-represented pixel signal x so that a number of 0s corresponding to the absolute value of the priority Pn are inserted into the LSB or MSB depending on whether the value of the priority Pn is positive or negative. Furthermore, by adding 0000100100, which is (x-16) x 4 (=36), to 0000010000, which is 16, (x-16) x 4 + 16 is obtained as 0000110100. In this way, nonlinear transformation and inverse nonlinear transformation of pixel signals can be achieved by bit shifting and addition / subtraction.

[0137] The bit shift in this example can be summarized, for example, as shown in Fig. 14. Note that Fig. 14 shows the bit shift when pixel signals with priority levels Pn = +2, +1, 0, -1, and -2 are expressed in 10-bit notation from the left in the figure. Here, the values ​​of each bit from the LSB to the MSB for priority level Pn = 0 are expressed as x0 to x9.

[0138] That is, when the priority Pn=+1, one bit of 0 is inserted on the LSB side, so the most significant bit x9 is deleted. Also, when the priority Pn=+2, two bits of 0 are inserted on the LSB side, so the most significant bits x8 and x9 are deleted.

[0139] Furthermore, when the priority Pn=-1, one bit of 0 is inserted on the MSB side, so the least significant bit value x0 is deleted. Furthermore, when the priority Pn=-2, two bits of 0 are inserted on the MSB side, so the most significant bit values ​​x0 and x1 are deleted.

[0140] In this way, in quantization after nonlinear transformation, the number of bits deleted as a result of compression changes according to the priority Pn.

[0141] In other words, when 3 bits are deleted by the quantization unit 33, if the priority Pn = +2, 0 is inserted into the 0th and 1st bits, so that essentially only the value x0 of the 2nd bit, i.e., essentially the least significant bit of the original pixel signal x, is deleted.

[0142] On the other hand, when 3 bits are deleted by the quantization unit 33, if the priority Pn = -2, 2 bits of 0 are inserted on the MSB side of the original pixel signal x, thereby deleting the lower 2 bits of values ​​x0 and x1, and further 3 bits of values ​​x2, x3, and x4 are deleted by quantization.

[0143] That is, when 3 bits are deleted by the quantization unit 33, if the priority Pn=-2, the lower 5 bits of the original pixel signal x are deleted by quantization.

[0144] The above-mentioned formula (5) can be modified as shown in the following formula (5').

[0145] ft(n,x)=(xx-x ｎ－１，ｍａｘ ) x K Ｐｎ +ft(n-1,x ｎ－１，ｍａｘ ) = x × K Ｐｎ +ft(n-1,x ｎ－１，ｍａｘ) -x ｎ－１，ｍａｘ ×K Ｐｎ ...(5')

[0146] In equation (5'), the first term (x × K Ｐｎ ) and subsequent terms (ft(n-1, x ｎ－１，ｍａｘ ) -x ｎ－１，ｍａｘ ×K Ｐｎ ) are fixed values ​​and can be calculated in advance from the priority table, which can further reduce the load of calculations due to bit shifts and additions and subtractions.

[0147] Similarly, the above-mentioned formula (6) can be transformed into the following formula (6').

[0148] bt(n,x)=(x-ft(n-1,x ｎ－１，ｍａｘ )) x K （－Ｐｎ） +x ｎ－１，ｍａｘ +offset ｎ = x x K （－Ｐｎ） +x ｎ－１，ｍａｘ +offset ｎ -ft(n-1, x ｎ－１，ｍａｘ ) x K （－Ｐｎ） ...(6')

[0149] In equation (6'), the first term (x × K （－Ｐｎ） ) and subsequent terms (+x ｎ－１，ｍａｘ +offset ｎ -ft(n-1, x ｎ－１，ｍａｘ ) x K （－Ｐｎ） ) are fixed values ​​and can be calculated in advance from the priority table, which can further reduce the load of calculations due to bit shifts and additions and subtractions.

[0150] <Procedure for Generating Priority Table> Next, a procedure for generating a priority table will be described.

[0151] The priority table needs to be tailored to the imaging mode, and various imaging modes are envisaged, but here we will explain the procedure for generating a priority table in an imaging mode in which a gamma correction curve and a correction curve for adding hard contrast are applied sequentially in a development process that takes into account the pixel signal of the OPB.

[0152] Here, we will explain an example of a procedure for generating a priority table for an imaging mode corresponding to a development process in which a gamma correction curve and a correction curve for adding hard contrast are sequentially applied, but this is merely one example of a procedure for generating a priority table, and the present invention is not limited to this procedure.

[0153] For example, the gamma correction curve during development that takes into account the pixel signal of the OPB is expressed by the function y = g(x), and the correction curve when adding hard contrast is expressed by y = t(x), and each correction curve is assumed to be the waveform shown by the solid lines on the left and center of Figure 15. Note that each dotted line is the waveform when the pixel signal x is directly subjected to nonlinear conversion.

[0154] In the first step, the gamma correction curve during development that takes into account the pixel signal of the OPB and the correction curve when adding hard contrast are combined to generate a composite curve (y = t(g(x))) as shown on the right side of Figure 15.

[0155] In the second step, the synthesis curve is adjusted so as to convert the pixel signal x of the OPB to be used as is.

[0156] That is, the composite curve (y = t(g(x))) has a waveform as shown in the left part of Fig. 16, but as mentioned above, if the OPB pixel signal x is set to 0, there is a risk that the noise intensity will not be measured properly, so the composite curve is transformed into a compression protection curve consisting of a waveform that outputs a value that changes linearly with the pixel signal x up to the OPB pixel signal th, as shown in the right part of Fig. 16. Note that the OPB pixel signal th does not have to be the actual pixel value output from the OPB pixel formed in the pixel array 131, and may be a value that varies slightly above or below the actual pixel value.

[0157] In the compression protection curve on the right side of Figure 16, when the pixel signal x is smaller than the pixel signal th of the OPB, the waveform is a function y = αx (α is a constant), and in other cases the waveform is a function y = (max - α × th) / max × t (g(x)) + α × th.

[0158] Note that α, which is the slope of the function, is in principle 1 or a value at least close to 1, but it may also be used to adjust the sensitivity when measuring noise intensity. That is, for example, if α is greater than 1, the noise level will be greater than the pixel signal x, so the sensitivity in measuring noise intensity will be high, and conversely, if α is greater than 1, the noise level will be smaller than the pixel signal x, so the sensitivity in measuring noise intensity can be lowered.

[0159] In the third step, the compression protection curve is set with a slope K Ｐｎ The interval is set by approximating the line according to the

[0160] For example, in the case of the compression protection curve shown in the left part of FIG. 17, the gradient K set by the priority Pn is as shown in the center part of FIG. Ｐｎ are approximated by a straight line according to the above equation, and sections #1 to #4 are set.

[0161] Then, in the fourth step, the minimum value x for each of the set sections #1 to #4 is calculated. ｎ，ｍｉｎ and the maximum value x ｎ，ｍａｘ , priority Pn, and offset offset ｎ A priority table is generated based on the above.

[0162] That is, as shown in the center of FIG. 17, the gradient K Ｐｎ When intervals #1 to #4 are set by approximating the intervals with straight lines according to the above, a priority table such as that shown in the right part of FIG. 17 is generated.

[0163] That is, in section #0, the minimum value x ｎ，ｍｉｎ is set to 0, and the maximum value x ｎ，ｍａｘ is set to th-1, the priority Pn is set to 0, and the offset offset ｎ is set to 0. In addition, in section #1, the minimum value xｎ，ｍｉｎ is set to th, and the maximum value x ｎ，ｍａｘ is set to a-1, the priority Pn is set to +1, and the offset offset ｎ is set to 0.

[0164] Similarly, in section #2, the minimum value x ｎ，ｍｉｎ is set to a, and the maximum value x ｎ，ｍａｘ is set to b-1, the priority Pn is set to +2, and the offset offset ｎ is set to 0. In section #3, the minimum value x ｎ，ｍｉｎ is set to b, and the maximum value x ｎ，ｍａｘ is set to c-1, the priority Pn is set to 0, and the offset offset ｎ is set to 0. In section #4, the minimum value x ｎ，ｍｉｎ is set to c, and the maximum value x ｎ，ｍａｘ is set to d-1, the priority Pn is set to -1, and the offset offset ｎ The ofst value is an offset value for restoring a saturated pixel value after nonlinear inverse transformation when the saturated pixel value is nonlinearly transformed.

[0165] Through the above-described procedure, a priority table is generated based on the correction curves set in association with the imaging modes.

[0166] The above describes the process of generating a necessary correction curve when a corresponding correction curve is not pre-stored when generating a priority table according to an imaging mode. However, when a correction curve corresponding to an imaging mode is pre-stored, the stored correction curve may be read and used as is without the need to generate a new correction curve. Furthermore, variations of the pre-stored correction curves include, for example, the high-key curve shown in the left part of Fig. 5, the low-key curve shown in the center of Fig. 5, the curve that provides a high-key representation shown in the right part of Fig. 5, and a curve that provides a soft-key representation (not shown).

[0167] <Priority Table Generation Process> Next, the priority table generation process will be described with reference to the flowchart in Fig. 18. Note that, here, a case will be described as an example in which there are a plurality of imaging modes and priority tables corresponding to all of the plurality of imaging modes are generated, but the process may also be a process in which one priority table for a specific imaging mode is generated.

[0168] In step S31, the priority table setting unit 170 sets an unprocessed imaging mode as a processing target imaging mode.

[0169] In step S32, the priority table setting unit 170 reads out a compression curve (correction curve) for the imaging mode to be processed. If no compression curve is available, the priority table setting unit 170 generates a compression curve corresponding to the imaging mode to be processed, for example, by synthesizing necessary compression curves, as described with reference to Fig. 15. Alternatively, if a compression curve specified by the user is available, the specified compression curve may be read out.

[0170] In step S33, the priority table setting unit 170 sets the compression curve of the imaging mode to be processed with a slope K Ｐｎ The distance is approximated by a straight line to set sections, and a priority Pn is set for each section.

[0171] In step S34, the priority table setting unit 170 calculates the offset of the final section ｎ More specifically, if a saturation value exists in the final section, and if the saturation value cannot be restored after nonlinear transformation, quantization, encoding, decoding, inverse quantization, and inverse nonlinear transformation, the priority table setting unit 170 sets an offset offset that can restore the saturation value. ｎ Set.

[0172] In step S35, the priority table setting unit 170 calculates the minimum value x ｎ，ｍｉｎ , maximum value x ｎ，ｍａｘ , priority Pn, and offset offset ｎ A priority table is generated based on the above.

[0173] In step S36, the priority table setting unit 170 stores the generated priority table in association with the imaging mode to be processed.

[0174] In step S37, the priority table setting unit 170 determines whether or not there are any unprocessed imaging modes, and if there are, the process returns to step S31, and the subsequent processes are repeated. That is, the same processes are repeated until priority tables are generated for all imaging modes.

[0175] Then, in step S37, if it is determined that priority tables have been generated for all the imaging modes and that there are no unprocessed imaging modes, the processing ends.

[0176] By the above processing, priority tables are generated for all imaging modes. After the above processing has been performed once, the generated priority table can be used, and no further processing is required. However, when a new imaging mode is set, it is necessary to generate a priority table corresponding to the newly set imaging mode.

[0177] <Image capture process by the image capture device of Fig. 8> Next, the image capture process by the image capture device 101 of Fig. 8 will be described with reference to the flowchart of Fig. 19. Note that the image capture process is performed on the premise that a priority table has been generated for each image capture mode by the priority table generation process described above.

[0178] In step S51, the priority table setting unit 170 accepts the setting of the operation mode input by the user operating the operation unit 113. At this time, the development processing unit 173 of the signal processing unit 154 also accepts the setting of the imaging mode. Note that once the operation mode has been set, the same setting may be used thereafter, and in this case, the processing of step S51 is not essential.

[0179] In step S52, the priority table setting unit 170 reads out the priority table stored in association with the set imaging mode, and supplies the read priority table to the nonlinear conversion unit 132 and the nonlinear inverse conversion unit 153 of the image sensor 111.

[0180] In step S53 , the pixel array 131 captures an image made up of pixel signals corresponding to the amount of incident light, and outputs the captured pixel signals made up of RAW data to the nonlinear conversion unit 132 .

[0181] In step S54 , the nonlinear conversion unit 132 performs nonlinear conversion on the pixel signal made up of RAW data based on the priority table corresponding to the imaging mode, and outputs the converted pixel signal to the quantization unit 133 .

[0182] In step S55 , the quantization unit 133 quantizes the nonlinearly converted pixel signal and outputs the quantized signal to the encoding unit 134 .

[0183] In step S56, the encoding unit 134 compresses the quantized pixel signal using a predetermined encoding method, and transmits the compressed signal to the processor unit 112 in step S57.

[0184] In step S58 , the decoding unit 151 decodes the pixel signal compressed using a predetermined encoding method, restores the quantized pixel signal, and outputs the restored pixel signal to the inverse quantization unit 152 .

[0185] In step S59 , the inverse quantization unit 152 inverse quantizes the quantized pixel signal to restore the nonlinearly transformed pixel signal, and outputs the restored pixel signal to the inverse nonlinear transformation unit 153 .

[0186] In step S60, the nonlinear inverse transform unit 153 performs a nonlinear inverse transform on the restored nonlinearly transformed pixel signal based on the priority table corresponding to the imaging mode to restore the pixel signal, and outputs the restored pixel signal to the signal processing unit 154.

[0187] In step S61, the signal processing unit 154 generates and outputs a developed image based on the pixel signal through development processing corresponding to the imaging mode.

[0188] More specifically, the OPB subtraction unit 171 of the signal processing unit 154 removes noise by subtracting the pixel signal value of the OPB pixel, which is a light-shielded pixel, from the pixel signal, and outputs the result to the gamma correction unit 172.

[0189] The gamma correction unit 172 performs gamma correction related to image development on the pixel signals from which noise has been removed, and outputs the result to the development processing unit 173 .

[0190] The development processing unit 173 uses the pixel signals that have been subjected to gamma correction to perform image development processing corresponding to the imaging mode, such as demosaic and white balance correction, to generate an image and output it as a developed image.

[0191] At this time, the saturation detection unit 174 detects areas where saturated pixels exist from the restored pixel signals by performing nonlinear inverse transformation, and outputs this as information on the saturated areas.

[0192] Furthermore, the noise amount detection unit 175 detects and outputs noise intensity based on the pixel signals of OPB pixels, which are light-shielded pixels, among the pixel signals restored by performing nonlinear inverse transformation.

[0193] By the above processing, nonlinear conversion is performed using a correction curve that is optimal for the imaging mode and corresponds to the priority table generated according to the imaging mode, and appropriate gradation compression processing is realized, making it possible to realize appropriate development processing for the imaging mode.

[0194] Furthermore, at this time, a priority table based on a correction curve that takes into account the subtraction of pixel signals of OPB pixels from pixel signals of pixels other than OPB pixels is supplied to both the nonlinear conversion unit 132 and the inverse nonlinear conversion unit 153, and the nonlinear conversion process and the inverse nonlinear conversion process are performed based on the same priority table, thereby preventing discrepancies between the two processes. This makes it possible to achieve appropriate gradation compression through appropriate nonlinear conversion process and inverse nonlinear conversion process, and as a result, it becomes possible to achieve appropriate image development processing.

[0195] Furthermore, the priority table is configured so that saturated pixel values ​​are properly restored by nonlinear inverse transformation after being nonlinearly transformed, making it possible to properly detect saturated areas using saturated pixel values.

[0196] Furthermore, since it is possible to set a priority table for each correction curve corresponding to the imaging mode, it is possible to realize nonlinear conversion and nonlinear inverse conversion corresponding to the various correction curves required in the development process. As a result, it is possible to realize gradation compression corresponding to each of the various imaging modes, and it is possible to realize appropriate image development processing.

[0197] Furthermore, since the calculations for the nonlinear transformation and the inverse nonlinear transformation can be realized only by bit shifts and additions and subtractions according to the priority Pn, it is possible to reduce the cost of the calculations. Although it is possible to implement an LUT to suppress the increase in the calculation cost, the priority table is sufficiently small compared to the LUT and the implementation cost is not high, so it is possible to suppress the increase in the calculation cost as well as the implementation cost.

[0198] <<3. First Modification>> In the above, an example has been described in which one priority table is generated in association with one imaging mode, but multiple priority tables may be set in accordance with one imaging mode.

[0199] For example, when HDR is set as the imaging mode, two types of images are captured: an image captured with a long exposure time and an image captured with a short exposure time, so two priority tables may be set according to the respective exposure times.

[0200] For example, consider the case where a 14-bit HDR composite image is generated from a 10-bit short-exposure image and a 10-bit long-exposure image.

[0201] For the sake of explanation, let us consider a 10-bit pixel signal of a short-exposure image as 16 (=2 4 ) to effectively create a 14-bit pixel signal, which is then combined with the 10-bit pixel signal of the long exposure by taking a weighted average on a pixel-by-pixel basis.

[0202] The pixel signals in this HDR synthesis are synthesized using the following equation (7).

[0203] HDR composite pixel value (14 bits) = w × short-exposure pixel signal (10 bits) × 16 + (1 − w) × long-exposure pixel signal (10 bits) (7)

[0204] Here, w is the weighting factor for the short-exposure pixel signal, and (1-w) is the weighting factor for the long-exposure pixel signal.

[0205] In this case, if an attempt is made to correct a 14-bit pixel signal of a short-exposure image and a 10-bit pixel signal of a long-exposure image using the same correction curve, a nonlinear transformation will be performed using the correction curve shown by the solid line in the upper part of Figure 20.

[0206] However, the correction curve for the 10-bit pixel signal of the long-exposure image is applied to the 14-bit pixel signal of the short-exposure image only in the left 1 / 16 range of the correction curve in the drawing, which is smaller than 1023, which is the saturated value of the 10-bit pixel signal, and appropriate correction cannot be performed.

[0207] Therefore, in such a case, as shown in the lower part of Figure 20, a correction curve for the 10-bit pixel signal of the long-exposure image is set separately, and a correction curve different from the correction curve for the 14-bit pixel signal of the short-exposure image is set.

[0208] In this case, a priority table is also generated for each correction curve. For example, a priority table for a 10-bit pixel signal of a long-exposure image is generated as shown in the left part of FIG. 21, and a priority table for a 10-bit pixel signal of a short-exposure image is generated as shown in the right part of FIG.

[0209] The correction curve shown in the lower part of FIG. 20 has a waveform with a smaller curvature than the correction curve shown in the upper part of FIG. 20, and therefore the priority table on the right side of FIG. 21 shows an example in which the section is divided into four sections, from section #0 to section #3.

[0210] That is, in the priority table on the left side of FIG. 21, in section #0, the minimum value x ｎ，ｍｉｎ is set to 0, and the maximum value x ｎ，ｍａｘ is set to 16, the priority Pn is set to 0, and the offset offset ｎ is set to 0. In addition, in section #1, the minimum value x ｎ，ｍｉｎ is set to 17, and the maximum value x ｎ，ｍａｘ is set to 144, the priority Pn is set to +1, and the offset ｎ is set to 0. Furthermore, in section #2, the minimum value x ｎ，ｍｉｎ is set to 145, and the maximum value x ｎ，ｍａｘ is set to 768, the priority Pn is set to 0, and the offset offset ｎ is set to 0. In addition, in section #3, the minimum value x ｎ，ｍｉｎ is set to 765, and the maximum value x ｎ，ｍａｘ is set to 1023, the priority Pn is set to -1, and the offset ｎ is set to 1.

[0211] Note that the example of the priority table on the right side of Fig. 21 is similar to the example of the priority table in Fig. 10, and therefore description thereof will be omitted. In this case, since the imaging mode is HDR blending, the development processing unit 173 generates and outputs a developed image by HDR blending the above-mentioned long exposure image and short exposure image according to the imaging mode, for example, as described with reference to Fig. 6.

[0212] As described above, by setting multiple priority tables according to the processing content of the subsequent development process corresponding to the imaging mode, it is possible to achieve appropriate nonlinear transformation and inverse nonlinear transformation for multiple pixel signals with different ranges. Furthermore, even when switching between multiple priority tables, nonlinear transformation and inverse nonlinear transformation can be achieved using only bit shifts and addition / subtraction, thereby reducing calculation costs. While using an LUT could be considered to reduce calculation costs, implementing multiple LUTs could increase equipment costs. In contrast, priority tables have a much smaller configuration than LUTs, and even when multiple priority tables are set, the increase in equipment costs associated with implementation can be suppressed.

[0213] <<4. Second Modification>> Although an example has been described above in which nonlinear transformation and inverse nonlinear transformation are realized by calculation based on a priority table, a look-up table (LUT) may also be used.

[0214] However, when generating an LUT, for example, a correction curve is set so that no correction is applied to the pixel signals of OPB pixels as shown in the left part of Figure 22, and so that the saturated values ​​remain saturated values ​​even after nonlinear inverse transformation, and an LUT corresponding to the set correction curve is generated.

[0215] Furthermore, an LUT corresponding to the correction curve set as shown in the left part of FIG. 22 is generated as shown in the right part of FIG. 22, for example.

[0216] Fig. 23 shows an example of the configuration of an imaging device when an LUT is used instead of a priority table. In the imaging device 101' of Fig. 23, components having the same functions as those of the imaging device 101 of Fig. 8 are given the same reference numerals, and their description will be omitted as appropriate. Furthermore, components having the same main functions but some different functions are given a "'" next to their reference numerals.

[0217] 23 differs from the imaging device 101 in FIG. 8 in that an image sensor 111′ and a processor unit 112′ are provided instead of the image sensor 111 and the processor unit 112.

[0218] Furthermore, in the image sensor 111′, in addition to the pixel array 131 to the encoding unit 134 in the image sensor 111, a selection unit 181 and a table storage unit 182 are newly provided.

[0219] Furthermore, in the processor section 112', a signal processing section 154' is provided instead of the signal processing section 154 in the processor section 112', and a selection section 191 and a table storage section 192 are also provided.

[0220] In the signal processing unit 154 ′, an imaging mode notification unit 201 is provided instead of the priority table setting unit 170 of the signal processing unit 154 .

[0221] The image capture mode notification unit 201 notifies the selection units 181 and 191 of information on the image capture mode input by operating the operation unit 113 .

[0222] The selection unit 181 reads out an LUT corresponding to the correction curve corresponding to the imaging mode, which is stored in the table storage unit 182, based on the imaging mode information supplied from the imaging mode notification unit 201, and supplies the LUT to the nonlinear conversion unit 132. The nonlinear conversion unit 132 performs nonlinear conversion based on the LUT supplied from the selection unit 181.

[0223] Furthermore, the selection unit 191 reads out an LUT corresponding to the correction curve corresponding to the imaging mode, which is stored in the table storage unit 192, based on the imaging mode information supplied from the imaging mode notification unit 201, and supplies the LUT to the nonlinear inverse transformation unit 153. The nonlinear inverse transformation unit 153 performs nonlinear inverse transformation based on the LUT supplied from the selection unit 191.

[0224] Note that the LUTs stored in the table storage units 182 and 192 are merely stored in place of the LUTs for the process of generating a priority table from the compression protection curve described with reference to FIG. 17, and therefore will not be described here.

[0225] Also, with regard to the imaging processing, the processing in step S52 is changed so that, when an imaging mode is notified from the imaging mode notifying unit 201, the selecting units 181 and 191 read out an LUT corresponding to the notified imaging mode from the table storage units 182 and 192 and supply it to the nonlinear conversion unit 132 and the nonlinear inverse conversion unit 153, instead of notifying a priority table, and therefore the processing in steps S54 and S60 is changed so that nonlinear conversion and nonlinear inverse conversion based on the LUT are performed instead of the priority table, and therefore a description thereof will be omitted.

[0226] Furthermore, in the second modified example, an example has been described in which LUTs are stored in the storage units 182, 192, and the selection units 181, 191 select the corresponding LUTs from the storage units 182, 192 according to the respective operation modes, and supply the LUTs to the nonlinear transformation unit 132 and the inverse nonlinear transformation unit 153. However, instead of the LUTs, the priority tables shown in the first embodiment may be stored in the table storage units 182, 192, and the selection units 181, 191 may select the priority table from the table storage units 182, 192 according to the respective imaging modes, and supply the priority table to the nonlinear transformation unit 132 and the inverse nonlinear transformation unit 153.

[0227] <<5. Application Examples>> In the processing of the encoding unit and the decoding unit, as shown in Fig. 24, the encoding or decoding process may be performed by using, as reference pixels, pixels that have already been coded or decoded and are present around the pixel to be coded or decoded, which are represented by gray hatching, as represented by diagonally shaded areas in the figure. This type of processing makes it possible to improve coding efficiency.

[0228] FIG. 25 shows an example of the configuration of an image pickup device 1' that uses reference pixels that have already been coded or decoded for coding or decoding a pixel to be processed.

[0229] In the imaging device 1' in Fig. 25, components having the same functions as those in the imaging device 1 in Fig. 1 are given the same reference numerals, and descriptions thereof will be omitted. Also, components that correspond to each other but have partially different functions are marked with "'" after the reference numeral.

[0230] That is, the imaging device 1' in Figure 25 differs from the imaging device 1 in Figure 1 in that instead of the encoding unit 34 and the decoding unit 51, an encoding unit 34' and a decoding unit 51' are provided, and further, a bit reduction unit 251, a memory 252, a bit restoration unit 253, a bit reduction unit 261, a memory 262, and a bit restoration unit 263 are provided.

[0231] The encoding unit 34 ′ supplies the encoded pixels that can be used as reference pixels to the bit reduction unit 251 .

[0232] The bit reduction unit 251 reduces a predetermined number of bits on the LSB side of pixel signals that can be used as a reference image, and stores the reduced image in the memory 252. More specifically, for example, the bit reduction unit 251 performs bit reduction by the calculation shown in the following equation (8), and stores the reduced image in the memory 252.

[0233] Cb=int(Nb / 2 ｍ ) ... (8)

[0234] Here, int(A) represents the value of the integer part of A, Nb represents the pixel signal before bit reduction, Cb represents the pixel signal after bit reduction, and m represents the number of bits that can be reduced.

[0235] The bit restoration unit 253 reads pixel signals that can be used as reference pixels from the memory 252, restores the reduced bits, and outputs the restored pixel signals to the encoding unit 34'. More specifically, for example, the bit restoration unit 253 restores the reduced bit information from the bit-reduced pixel signals read from the memory 252 by the calculation shown in the following equation (9), and outputs the restored bit information to the encoding unit 34'.

[0236] Nb = Cb × 2 ｍ +2 ｍ－１ ... (9)

[0237] Here, Nb is the pixel signal to be restored before bit reduction, and Cb is the pixel signal after bit reduction.

[0238] The encoding unit 34' encodes the pixel to be processed using the pixel signals of the encoded reference pixels read from the memory 252 via the bit restoration unit 253 as needed.

[0239] The bit reduction unit 261 , the memory 262 , and the bit restoration unit 263 have the same functions as the bit reduction unit 251 , the memory 252 , and the bit restoration unit 253 .

[0240] That is, the decoding unit 51' decodes the pixel to be processed using the pixel signals of the decoded reference pixels that are read out from the memory 262 via the bit restoration unit 263 as needed.

[0241] In this configuration, the cost of the memories 252 and 262 is high, and there are many cases where a reduction in memory capacity is required, but in recent years, a technology has been developed in DSC (Display Stream Compression) standardized by VESA that can reduce the bit depth per bit of the line buffer. By adopting this technology, the bit reduction unit 261, memory 262, and bit restoration unit 263, as well as the bit reduction unit 251, memory 252, and bit restoration unit 253, can limit the bit depth of the reference image to, for example, 8 bits by reducing about 2 bits for an input with a bit depth of 10 bits per pixel.

[0242] However, even if the bit size that can be reduced is 2 bits, in the imaging device 1′ of Figure 25, there are cases where the encoded or decoded pixel signals cannot be restored to appropriate pixel signals when they are stored in memories 252 and 262 using bit reduction units 251 and 261 and bit restoration units 253 and 263, and restored, resulting in a decrease in the accuracy of the encoding and decoding.

[0243] For example, consider the case where the pixel signal stored as the reference pixel is 127, and is represented as a 10-bit signal, and the lower four bits are reduced in the bit reduction units 251 and 261 and stored in the memories 252 and 262, and the upper two bits of the reduced four bits are restored by the bit restoration units 243 and 263.

[0244] In this case, as shown in the left part of FIG. 26, when the pixel signal is 127, the pixel signal of 127 is expressed as 0001111111 when expressed in 10-bit binary notation.

[0245] When this pixel signal is stored in the memories 252 and 262, the lower four bits are reduced by the bit reduction units 251 and 261, as shown in the center of Fig. 26, to 000111xxxx, calculated using the above-mentioned equation (8). Note that "x" indicates that a reduction has been made.

[0246] Furthermore, when the bit restoration units 253 and 263 read out and restore this pixel signal from the memories 252 and 262, the result is 0001111000 according to the calculation of the above-mentioned equation (9).

[0247] As a result, the pixel signal, which was 127 (0001111111) before being reduced by the bit reducing units 251 and 261, changes to 120 (=0001111000).

[0248] As a result, the reference pixels cannot be properly restored, which reduces the accuracy of encoding and decoding.

[0249] However, by applying this to the imaging device 101 of the present disclosure, it is possible to suppress a decrease in accuracy in restoring pixel signals of reference pixels.

[0250] Fig. 27 shows an example configuration of an imaging device when the present disclosure is applied with a function of using already-encoded or already-decoded reference pixels for encoding or decoding a pixel to be processed. In the imaging device 101'' in Fig. 27, components having the same functions as those in the imaging device 101 in Fig. 8 are given the same reference numerals, and descriptions thereof will be omitted as appropriate. Furthermore, components having the same main functions but some different functions are marked with "''" next to their reference numerals.

[0251] That is, the imaging device 101'' in FIG. 27 differs from the imaging device 101 in FIG. 8 in that an image sensor 111'' and a processor unit 112'' are provided instead of the image sensor 111 and the processor unit 112.

[0252] In addition, the image sensor 111'' is provided with an encoding unit 134'' instead of the encoding unit 134, and is newly provided with a bit reduction unit 301, a memory 302, and a bit restoration unit 303.

[0253] Furthermore, in the processor unit 112 ″, a decoding unit 151 ″ is provided in place of the decoding unit 151 , and a bit reduction unit 311 , a memory 312 , and a bit restoration unit 313 are newly provided.

[0254] Note that the encoding unit 134'', bit reduction unit 301, memory 302, bit restoration unit 303, decoding unit 151'', bit reduction unit 311, memory 312, and bit restoration unit 313 are identical to the encoding unit 34', bit reduction unit 251, memory 252, bit restoration unit 253, decoding unit 51', bit reduction unit 261, memory 262, and bit restoration unit 263 in FIG. 25, respectively.

[0255] That is, in the imaging device 101'' of Figure 27, the functions of the encoding unit 134'' and the decoding unit 151'' are basically the same as those of the encoding unit 34' and the decoding unit 51', but the functions of the nonlinear transformation unit 132 and the nonlinear inverse transformation unit 153 of the present disclosure can suppress a decrease in the accuracy of pixel signals during restoration even when reference pixels are restored after being stored in memories 302, 312.

[0256] That is, consider the case where the pixel signal of the reference pixel is 127 as described above.

[0257] In this case, as shown in the left part of Fig. 28, when the pixel signal of the reference pixel is 127, it is treated as a 10-bit signal of 0001111111, but based on the priority table of Fig. 10, it is treated as a pixel signal of section #1, and the priority Pn is set to +2. Therefore, as shown second from the left in Fig. 28, 0 is inserted into the two least significant bits, and the pixel signal becomes 0111111100.

[0258] Furthermore, when the bit reduction unit 301 reduces 4 bits from the pixel signal of the reference pixel and stores it in the memory 302, the pixel signal becomes 011111xxxx, as shown third from the left in FIG.

[0259] Furthermore, by restoring the 4 bits removed by the above-described equation (9), the pixel signal becomes 0111111000, as shown in the right part of FIG.

[0260] Since the pixel signal has a priority Pn=+2, it is shifted by two bits, and therefore, essentially, the second bit is simply inverted from "1" to "0".

[0261] As a result, the imaging device 101'' of the present disclosure shows that the decrease in accuracy is improved compared to the imaging device 1' in which the lower three bits are all inverted from "1" to "0", as described with reference to Figure 26.

[0262] That is, according to the imaging device 101'' disclosed herein, when encoding and decoding a pixel to be processed in an encoding process and a decoding process, even if the pixel signals of surrounding reference pixels are bit-reduced and stored in memory when processing the pixel to be processed, and then the reduced bits are restored and used, it is possible to suppress a decrease in accuracy related to the restoration.

[0263] <<6. Second embodiment>> In the above, an example has been described in which the priority table is switched based on the imaging mode input by the user operating the operation unit 113, and the nonlinear conversion unit 132 and nonlinear inverse conversion processing are performed.

[0264] However, it is also possible to set imaging parameters such as exposure time and white balance on a frame-by-frame basis based on pixel signals consisting of captured RAW data, and to select a priority table based on the imaging parameters.

[0265] Also, as shown in FIG. 29, imaging parameters may be set based on pixel signals consisting of captured RAW data, and a priority table may be set for each area within the image based on the set imaging parameters.

[0266] 29, for example, for a bright area within a rectangular range enclosed by a dotted line in the figure, imaging parameters for bright areas for capturing a bright image may be set, and a bright area priority table corresponding to the imaging parameters for bright areas may be set. In this case, for a dark area outside the rectangular range enclosed by a dotted line in the figure, imaging parameters for dark areas may be set, and a dark area priority table corresponding to the imaging parameters for dark areas may be set.

[0267] Note that Figure 29 is an example of an image captured at night, with a tent and bonfire in the lower left corner and the moon in the upper right corner. The rectangular area close to the bonfire is considered a bright area, and the rest of the area is considered a dark area.

[0268] FIG. 30 shows an example configuration of an imaging device in which imaging parameters are set based on pixel signals consisting of captured RAW data, and a priority table is set on a frame-by-frame or region-by-region basis based on the set imaging parameters.

[0269] In the imaging device 101''' in Fig. 30, components having the same functions as those in the imaging device 101 in Fig. 8 are given the same reference numerals, and descriptions thereof will be omitted as appropriate. Furthermore, components having the same main functions but some different functions are given a "'''" next to their reference numerals.

[0270] That is, the imaging device 101''' in FIG. 30 differs from the imaging device 101 in FIG. 8 in that an image sensor 111''' and a processor unit 112''' are provided instead of the image sensor 111 and the processor unit 112.

[0271] Furthermore, in the image sensor 111''', a pixel array 131''' is provided instead of the pixel array 131, and an imaging parameter determination unit 351 and a priority table setting unit 352 are newly provided.

[0272] Furthermore, in the processor section 112 ″, a development processing section 173 ′″ is provided in place of the development processing section 173 .

[0273] That is, the pixel array 131''' captures an image based on imaging parameters such as exposure time and sensitivity setting supplied from the imaging parameter determination unit 351, and outputs a pixel signal consisting of RAW data to the imaging parameter determination unit 351 and the nonlinear conversion unit 132.

[0274] The imaging parameter determination unit 351 determines imaging parameters on a frame-by-frame or region-by-region basis based on pixel signals consisting of RAW data supplied from the pixel array 131''', and outputs them to the pixel array 131''', the priority table setting unit 352, and the development processing unit 173'''.

[0275] The priority table setting unit 352 basically has the same functions as the priority table setting unit 170, but further determines an imaging mode based on imaging parameters, and outputs a priority table corresponding to the determined imaging mode to the nonlinear conversion unit 132 and the nonlinear inverse conversion unit 153.

[0276] The development processing unit 173''' basically has the same functions as the development processing unit 173, but it identifies an imaging mode corresponding to the imaging parameters and performs development processing in the identified imaging mode, thereby generating and outputting a developed image based on gamma-corrected pixel signals.

[0277] <Image capture process by the image capture device of Fig. 30> Next, the image capture process by the image capture device 101''' of Fig. 30 will be described with reference to the flowchart of Fig. 31. Note that the image capture process is performed on the premise that the priority table setting unit 352 has generated a priority table for each image capture mode by the priority table generation process described above.

[0278] In step S81, the pixel array 131 captures an image consisting of pixel signals corresponding to the amount of incident light based on the imaging parameters determined by the imaging parameter determination unit 351 on a frame-by-frame or region-by-region basis in the immediately preceding frame, and outputs the pixel signals consisting of the captured RAW data to the imaging parameter determination unit 351 and the nonlinear conversion unit 132.

[0279] In step S82, the imaging parameter determination unit 351 determines imaging parameters for the next frame based on the pixel signal consisting of RAW data supplied from the pixel array 131, and supplies them to the pixel array 131''', the priority table setting unit 352, and the development processing unit 173'''.

[0280] In step S83, the priority table setting unit 352 determines the imaging mode based on the imaging parameters supplied from the imaging parameter determination unit 351, and supplies a priority table corresponding to the determined imaging mode to the nonlinear conversion unit 132 and the nonlinear inverse conversion unit 153.

[0281] In step S84 , the nonlinear conversion unit 132 performs nonlinear conversion on the pixel signal made up of RAW data based on the priority table corresponding to the imaging mode, and outputs the converted pixel signal to the quantization unit 133 .

[0282] In step S85, the quantization unit 133 quantizes the nonlinearly converted pixel signal and outputs the quantized signal to the encoding unit 134.

[0283] In step S86, the encoding unit 134 compresses the quantized pixel signals using a predetermined encoding method, and in step S87 outputs the compressed signals to the processor unit 112'''.

[0284] In step S88, the decoding unit 151 decodes the pixel signal compressed using a predetermined encoding method, restores the quantized pixel signal, and outputs the restored pixel signal to the inverse quantization unit 152.

[0285] In step S89 , the inverse quantization unit 152 inverse quantizes the quantized pixel signal to restore the nonlinearly transformed pixel signal, and outputs the restored pixel signal to the inverse nonlinear transformation unit 153 .

[0286] In step S90, the nonlinear inverse transform unit 153 performs a nonlinear inverse transform on the restored nonlinearly transformed pixel signal based on the priority table corresponding to the imaging mode to restore the pixel signal, and outputs the restored pixel signal to the signal processing unit 154.

[0287] In step S91, the signal processing unit 154 identifies an imaging mode based on the imaging parameters supplied from the imaging parameter determination unit 351, and performs development processing on the pixel signal in the identified imaging mode to generate and output a developed image.

[0288] Through the above processing, imaging parameters are determined based on pixel signals consisting of RAW data, an imaging mode is determined based on the determined imaging parameters, nonlinear processing is performed by the nonlinear conversion unit 132 based on a correction curve that is optimal for the imaging mode based on a priority table generated in accordance with the determined imaging mode, and nonlinear inverse conversion is performed by the nonlinear inverse conversion unit 153.As a result, it is possible to achieve development processing appropriate for the imaging mode even if the imaging mode is not input by the user.

[0289] <<7. Example of Execution by Software>> The above-described series of processes can be executed by hardware, but can also be executed by software. When the series of processes is executed by software, the programs that make up the software are installed from a recording medium into a computer that is built into dedicated hardware, or into, for example, a general-purpose computer that can execute various functions by installing various programs.

[0290] 32 shows an example of the configuration of a general-purpose computer. This personal computer has a built-in CPU (Central Processing Unit) 1001. An input / output interface 1005 is connected to the CPU 1001 via a bus 1004. A ROM (Read Only Memory) 1002 and a RAM (Random Access Memory) 1003 are connected to the bus 1004.

[0291] The input / output interface 1005 is connected to an input unit 1006 including input devices such as a keyboard and a mouse through which a user inputs operation commands, an output unit 1007 that outputs a processing operation screen and images of processing results to a display device, a storage unit 1008 including a hard disk drive or the like that stores programs and various data, and a communication unit 1009 including a LAN (Local Area Network) adapter or the like that executes communication processing via a network typified by the Internet. Also connected is a drive 1010 that reads and writes data from / to a removable storage medium 1011 such as a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory) and a DVD (Digital Versatile Disc)), a magneto-optical disk (including an MD (Mini Disc)), or a semiconductor memory.

[0292] The CPU 1001 executes various processes in accordance with a program stored in a ROM 1002 or a program read from a removable storage medium 1011 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, installed in a storage unit 1008, and loaded from the storage unit 1008 into a RAM 1003. The RAM 1003 also stores data necessary for the CPU 1001 to execute various processes as appropriate.

[0293] In a computer configured as described above, the CPU 1001 performs the above-described series of processes by, for example, loading a program stored in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing it.

[0294] The program executed by the computer (CPU 1001) can be provided by being recorded on a removable storage medium 1011 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0295] In a computer, a program can be installed in the storage unit 1008 via the input / output interface 1005 by inserting a removable storage medium 1011 into the drive 1010. The program can also be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. Alternatively, the program can be installed in advance in the ROM 1002 or the storage unit 1008.

[0296] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0297] 32 realizes the functions of the nonlinear conversion unit 132 and the nonlinear inverse conversion unit 153 in FIGS. 8, 23, and 30, as well as the priority table setting unit 170 in FIG. 8, the imaging mode notification unit 201, the selection unit 181, the table storage unit 182, the selection unit 191, the table storage unit 192 in FIG. 23, and the imaging parameter determination unit 351 in FIG. 30.

[0298] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0299] The embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure.

[0300] For example, the present disclosure can be configured as a cloud computing system in which a single function is shared and processed collaboratively by multiple devices via a network.

[0301] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0302] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0303] The present disclosure may also have the following configuration: <1> An imaging device including: a pixel array unit in which pixels are arranged in an array, the pixels photoelectrically converting incident light in accordance with the amount of light and generating pixel signals in accordance with the amount of light; a nonlinear conversion unit that applies a nonlinear conversion to the pixel signals; a coding unit that encodes the nonlinearly converted pixel signals; a decoding unit that decodes the pixel signals encoded by the coding unit; a nonlinear inverse conversion unit that applies a nonlinear inverse conversion to the pixel signals decoded by the decoding unit; and a development processing unit that generates a developed image by development processing using the pixel signals that have been nonlinearly inversely converted by the nonlinear inverse conversion unit, wherein the nonlinear conversion unit applies a nonlinear conversion having a gradient smaller than the maximum gradient of the nonlinear conversion applied to pixel signals with values ​​equal to or greater than optical black to some or all of the pixel signals with values ​​equal to or less than optical black, and the nonlinear inverse conversion unit applies a nonlinear inverse conversion having a gradient smaller than the maximum gradient of the nonlinear conversion applied to pixel signals with values ​​equal to or greater than optical black to some or all of the pixel signals with values ​​equal to or less than optical black. <2> The imaging device described in <1>, wherein the nonlinear transformation unit performs nonlinear transformation by applying an offset to the pixel signal so that the pixel signal consisting of a saturated value is restored as the pixel signal consisting of the saturated value by being nonlinearly transformed, then encoded by the encoding unit, decoded by the decoding unit, and nonlinearly inversely transformed by the nonlinear inverse transformation unit. <3> The imaging device described in <1>, wherein the nonlinear transformation unit and the nonlinear inverse transformation unit each perform nonlinear transformation and nonlinear inverse transformation according to a nonlinear correction curve set for each mode of the development processing of the development processing unit. <4> The imaging device described in <3>, wherein the nonlinear transformation unit and the nonlinear inverse transformation unit each perform nonlinear transformation and nonlinear inverse transformation based on a table for expressing the nonlinear correction curve set for each mode of the development processing of the development processing unit as a combination of linear functions set for each of a plurality of divided sections.<5> The imaging device according to <4>, wherein the slope of the linear function set for each of the plurality of divided sections is set in units of powers of 2 expressed by an exponent consisting of integers with base 2 being the base that approximates the nonlinear correction curve, and the sections have the same slope of the linear function. <6> The imaging device according to <5>, wherein a minimum value and a maximum value of the pixel signal and a priority consisting of the exponent are set for each of the plurality of divided sections and registered in the table. <7> The imaging device according to <6>, wherein the nonlinear transformation unit and the nonlinear inverse transformation unit perform nonlinear transformation and nonlinear inverse transformation of the pixel signal by bit shifting and addition / subtraction based on the priority registered in the table, respectively. <8> The imaging device according to <6>, wherein, among the multiple divided sections, in end sections where the pixel signal is maximum, an offset is further set to nonlinearly transform the pixel signal of the pixel consisting of the pixel of the saturated value, after which the pixel signal is nonlinearly transformed, encoded by the encoding unit, decoded by the decoding unit, and nonlinearly inverse transformed by the nonlinear inverse transform unit, thereby restoring the pixel signal consisting of the saturated value, and the offset is further set and registered in the table. <9> The imaging device according to <4>, wherein, when the development processing mode of the development processing unit is HDR blending (High Dynamic Range), two tables are set, one for a long exposure image and one for a short exposure image, and the nonlinear transform unit and the nonlinear inverse transform unit switch the tables between the pixel signals of the long exposure image and the pixel signals of the short exposure image, respectively, to perform nonlinear transform and nonlinear inverse transform. <10> The imaging device described in <4>, wherein the table is set for each frame or for each region within the same frame, and the nonlinear transformation unit and the nonlinear inverse transformation unit switch the table for each frame or each region to perform nonlinear transformation and nonlinear inverse transformation, respectively.<11> The imaging device described in <3>, wherein the nonlinear conversion unit and the nonlinear inverse conversion unit each perform nonlinear conversion and nonlinear inverse conversion based on a look-up table (LUT) that indicates the correspondence between the input pixel signal and the nonlinear conversion result when using the nonlinear correction curve that is set for each mode of the development processing of the development processing unit. <12> The imaging device described in <9>, further including: a storage unit that stores the LUT in association with the mode; and a selection unit that selects the LUT from the storage unit based on the mode, wherein the nonlinear conversion unit and the nonlinear inverse conversion unit each perform nonlinear conversion and nonlinear inverse conversion using the LUT selected from the storage unit by the selection unit based on the mode of the development processing of the development processing unit. <13> The imaging device described in <3>, wherein the mode of the development processing of the development processing unit is set based on an operation input by a user. <14> The imaging device according to <3>, further including an imaging parameter setting unit that sets imaging parameters for the development processing of the development processing unit based on the pixel signals captured by the pixel array unit, wherein the nonlinear conversion unit and the nonlinear inverse conversion unit respectively perform nonlinear conversion and nonlinear inverse conversion according to the nonlinear correction curve that is set for the mode of the development processing of the development processing unit, which are specified based on the imaging parameters. <15> The imaging device according to <3>, wherein the nonlinear correction curve that is set for each mode of the development processing of the development processing unit includes a gamma curve, a high-key curve, a curve that provides hard contrast, and a curve that provides soft contrast.<16> An imaging method for an imaging device having a pixel array unit in which pixels are arranged in an array, each pixel performing photoelectric conversion on incident light in accordance with an amount of light and generating a pixel signal in accordance with the amount of light, the imaging method comprising: performing a nonlinear transformation process of applying a nonlinear transformation to the pixel signals; performing an encoding process of encoding the pixel signals that have been nonlinearly transformed; performing a decoding process of decoding the pixel signals that have been encoded by the encoding process; performing a nonlinear inverse transformation process of applying a nonlinear inverse transformation to the pixel signals that have been decoded by the decoding process; and performing a development process of generating a developed image using the pixel signals that have been nonlinearly inversely transformed by the nonlinear inverse transformation process, wherein the nonlinear transformation process applies a nonlinear transformation having a smaller gradient to some or all of the pixel signals having a value equal to or less than optical black than a maximum gradient of the nonlinear transformation applied to the pixel signals having a value equal to or greater than optical black, and the nonlinear inverse transformation process applies a nonlinear inverse transformation having a smaller gradient to some or all of the pixel signals having a value equal to or less than optical black than a maximum gradient of the nonlinear transformation applied to the pixel signals having a value equal to or greater than optical black. <17> A program that causes a computer to function as: a pixel array unit in which pixels are arranged in an array, the pixels performing photoelectric conversion on incident light in accordance with the amount of light and generating pixel signals in accordance with the amount of light; a nonlinear conversion unit that applies a nonlinear conversion to the pixel signals; a coding unit that encodes the pixel signals that have been nonlinearly converted; a decoding unit that decodes the pixel signals encoded by the coding unit; a nonlinear inverse conversion unit that applies a nonlinear inverse conversion to the pixel signals decoded by the decoding unit; and a development processing unit that generates a developed image by development processing using the pixel signals that have been nonlinearly inversely converted by the nonlinear inverse conversion unit, wherein the nonlinear conversion unit applies a nonlinear conversion having a gradient smaller than the maximum gradient of the nonlinear conversion applied to the pixel signals of values ​​equal to or greater than optical black to some or all of the pixel values ​​having a value equal to or less than optical black, and the nonlinear inverse conversion unit applies a nonlinear inverse conversion having a gradient smaller than the maximum gradient of the nonlinear conversion applied to the pixel signals of values ​​equal to or greater than optical black to some or all of the pixel signals having a value equal to or less than optical black.

[0304] DESCRIPTION OF SYMBOLS 101, 101', 101'', 101''' imaging device, 111, 111', 111'', 111''' image sensor, 112, 112', 112'', 112''' processor unit, 131, 131''' pixel array, 132 nonlinear conversion unit, 133 quantization unit, 134, 134'' encoding unit, 151, 151'' decoding unit, 152 inverse quantization unit, 153 nonlinear conversion unit, 154 signal processing unit, 170 imaging mode setting unit, 171 OPB subtraction unit, 172 gamma correction unit, 173, 173''' development processing unit, 174 saturation detection unit, 175 noise amount detection unit, 181 selection unit, 182 table storage unit, 191 selection unit 192 Table storage unit, 201 Imaging mode notification unit, 301 Bit reduction unit, 302 Memory, 303 Bit restoration unit, 311 Bit reduction unit, 312 Memory, 313 Bit restoration unit, 351 Imaging parameter determination unit, 352 Priority table determination unit

Claims

1. An imaging device comprising: a pixel array unit in which pixels are arranged in an array, the pixels performing photoelectric conversion on incident light in accordance with the amount of light and generating pixel signals in accordance with the amount of light; a nonlinear conversion unit that applies a nonlinear conversion to the pixel signals; a coding unit that encodes the pixel signals that have been nonlinearly converted; a decoding unit that decodes the pixel signals encoded by the coding unit; a nonlinear inverse conversion unit that applies a nonlinear inverse conversion to the pixel signals decoded by the decoding unit; and a development processing unit that generates a developed image by development processing using the pixel signals that have been nonlinearly inverse converted by the nonlinear inverse conversion unit, wherein the nonlinear conversion unit applies a nonlinear conversion having a smaller gradient to some or all of the pixel signals having a value equal to or less than optical black than the maximum gradient of the nonlinear conversion applied to pixel signals having a value equal to or greater than optical black, and the nonlinear inverse conversion unit applies a nonlinear inverse conversion having a smaller gradient to some or all of the pixel signals having a value equal to or less than optical black than the maximum gradient of the nonlinear conversion applied to pixel signals having a value equal to or greater than optical black.

2. The imaging device of claim 1, wherein the nonlinear transformation unit performs a nonlinear transformation by applying an offset to the pixel signal so that the pixel signal consisting of a saturated value is restored as the pixel signal consisting of the saturated value by being nonlinearly transformed, encoded by the encoding unit, decoded by the decoding unit, and nonlinearly inverse transformed by the nonlinear inverse transformation unit.

3. The imaging device according to claim 1, wherein the nonlinear conversion section and the nonlinear inverse conversion section respectively execute a nonlinear conversion and a nonlinear inverse conversion according to a nonlinear correction curve set for each mode of the development processing of the development processing section.

4. The imaging device described in claim 3, wherein the nonlinear conversion unit and the nonlinear inverse conversion unit each perform a nonlinear conversion and a nonlinear inverse conversion based on a table for expressing the nonlinear correction curve, which is set for each mode of the development processing of the development processing unit, as a combination of linear functions set for each of a plurality of divided sections.

5. The imaging device of claim 4, wherein the slope of the linear function set for each of the multiple divided sections is set in units of powers of 2 expressed as an integer exponent with base 2 that approximates the nonlinear correction curve, and the sections are sections in which the slope of the linear function is the same.

6. The imaging device according to claim 5, wherein a priority consisting of the minimum and maximum values ​​of the pixel signal and the index is set for each of the multiple divided sections and registered in the table.

7. The imaging device according to claim 6, wherein the nonlinear conversion unit and the nonlinear inverse conversion unit respectively perform nonlinear conversion and nonlinear inverse conversion of the pixel signal by bit shifting based on the priority registered in the table and addition and subtraction.

8. An imaging device as described in claim 6, wherein, among the multiple divided sections, in the end sections where the pixel signal is maximum, an offset is further set so that the pixel signal of the pixel consisting of the saturated value is nonlinearly transformed, then encoded by the encoding unit, decoded by the decoding unit, and nonlinearly inverse transformed by the nonlinear inverse transformation unit, thereby being nonlinearly transformed to be restored as the pixel signal consisting of the saturated value, and the offset is further set and registered in the table.

9. The imaging device of claim 4, wherein when the development processing mode of the development processing unit is HDR compositing (High Dynamic Range), two tables are set, one for a long exposure image and one for a short exposure image, and the nonlinear conversion unit and the nonlinear inverse conversion unit switch the tables for the pixel signals of the long exposure image and the pixel signals of the short exposure image, respectively, to perform nonlinear conversion and nonlinear inverse conversion.

10. The imaging device of claim 4, wherein the table is set on a frame-by-frame basis or on a region-by-region basis within the same frame, and the nonlinear transformation unit and the nonlinear inverse transformation unit switch between the tables on a frame-by-frame basis or on a region-by-region basis, respectively, to perform nonlinear transformation and nonlinear inverse transformation.

11. The imaging device described in claim 3, wherein the nonlinear conversion unit and the nonlinear inverse conversion unit each perform a nonlinear conversion and a nonlinear inverse conversion based on a LUT (Look Up Table) indicating the correspondence between the input pixel signal and the nonlinear conversion result when using the nonlinear correction curve set for each mode of the development processing of the development processing unit.

12. The imaging device of claim 9, further comprising: a storage unit that stores the LUT in correspondence with the mode; and a selection unit that selects the LUT from the storage unit based on the mode, wherein the nonlinear transformation unit and the nonlinear inverse transformation unit each perform a nonlinear transformation and a nonlinear inverse transformation using the LUT selected from the storage unit by the selection unit based on the mode of the development processing of the development processing unit.

13. The imaging device according to claim 3, wherein the development processing mode of the development processing section is set based on an operational input by a user.

14. The imaging device according to claim 3, further comprising an imaging parameter setting unit that sets imaging parameters for the development processing of the development processing unit based on the pixel signals captured by the pixel array unit, wherein the nonlinear transformation unit and the nonlinear inverse transformation unit respectively perform nonlinear transformation and nonlinear inverse transformation according to the nonlinear correction curves that are set in a mode of the development processing of the development processing unit, the nonlinear correction curves being specified based on the imaging parameters.

15. The imaging device according to claim 3, wherein the nonlinear correction curves set for each mode of the development processing of the development processing section include a gamma curve, a high key curve, a curve for hard contrast expression, and a curve for soft contrast expression.

16. An imaging method for an imaging device having a pixel array section in which pixels are arranged in an array, the pixels performing photoelectric conversion on incident light in accordance with the amount of light and generating pixel signals in accordance with the amount of light, the imaging method comprising: performing a nonlinear conversion process to apply a nonlinear conversion to the pixel signals; performing an encoding process to encode the nonlinearly converted pixel signals; performing a decoding process to decode the pixel signals encoded by the encoding process; performing a nonlinear inverse conversion process to apply a nonlinear inverse conversion to the pixel signals decoded by the decoding process; and performing a development process to generate a developed image using the pixel signals nonlinearly inversely converted by the nonlinear inverse conversion process, wherein the nonlinear conversion process applies a nonlinear conversion having a smaller gradient to some or all of the pixel signals having a value equal to or less than optical black than the maximum gradient of the nonlinear conversion applied to the pixel signals having a value equal to or greater than optical black, and the nonlinear inverse conversion process applies a nonlinear inverse conversion having a smaller gradient to some or all of the pixel signals having a value equal to or less than optical black than the maximum gradient of the nonlinear conversion applied to the pixel signals having a value equal to or greater than optical black.

17. A program that causes a computer to function as: a pixel array unit in which pixels are arranged in an array, which photoelectrically converts incident light in accordance with the amount of light and generates pixel signals in accordance with the amount of light; a nonlinear conversion unit that applies a nonlinear transformation to the pixel signals; a coding unit that encodes the pixel signals that have been nonlinearly converted; a decoding unit that decodes the pixel signals encoded by the coding unit; a nonlinear inverse conversion unit that applies a nonlinear inverse conversion to the pixel signals decoded by the decoding unit; and a development processing unit that generates a developed image by development processing using the pixel signals that have been nonlinearly inverse converted by the nonlinear inverse conversion unit, wherein the nonlinear conversion unit applies a nonlinear transformation having a smaller gradient to some or all of the pixel signals having a value below optical black than the maximum gradient of the nonlinear transformation applied to the pixel signals having a value above optical black, and the nonlinear inverse conversion unit applies a nonlinear inverse conversion having a smaller gradient to some or all of the pixel signals having a value below optical black than the maximum gradient of the nonlinear transformation applied to the pixel signals having a value above optical black.